Loup Ventures proves Siri is improving — and it’s getting better soon

You say tomatoes, I say tomatoes.

I know that’s not a difference in verbosity that comes across particularly well in words, but when I see the Loup Ventures test of voice assistants show some impressive improvements in Siri running on iOS 11.4, I’m not thinking about the here and now, I’m thinking about what’s coming next.

Hey Siri, when will you get smarter?

iOS 12 will introduce a bunch of new Siri features. Siri Shortcuts (and the built in AI recommending you new shortcuts Siri thinks you need to use) will add a whole new level to this, as it will eventually enable you to get third-party apps to do complex tasks for you by uttering just a few words.

Think about that.

A quick recap of the test results shows Siri to be the second place best voice assistant out there – and private too.

More importantly, these tests show that Siri has some specific strengths and weaknesses: Yes, Google is better at answering knowledge questions (“What is the capital of Mexico?”), but less good at getting your device to show you all your images of Mexico.

I won’t go into the Loup tests here – you can read them for yourself. I just want to draw your attention to a few thoughts.

Command and control

They asked 800 questions in five categories:

  • Local – Where is the nearest coffee shop?
  • Commerce – Can you order me more paper towels?
  • Navigation – How do I get to uptown on the bus?
  • Information – Who do the Twins play tonight?
  • Command – Remind me to call Steve at 2pm today.

Google Assistant answered 86% of questions correctly and understood all 800 questions. Siri got 79% percent right and misunderstood 11, while Alexa got 61 percent right (misunderstanding 13). Cortana? 52%/19.

However, when you break down the Siri v Google Assistant thing more you see that Google leads on shopping (commerce), local search (it owns a search engine), information (it owns a search engine) and navigation (Apple Maps is about to improve).

The importance of Maps

On the basis of this I’m going to make a prediction. Apple is engaged in a big improvement in Maps. When it introduces those improvements it hopes to make Maps an incredibly effective navigation tool (closing the gap there), and that effectiveness will likely also see it deliver more accurate listings for local things, such as businesses – which means while it may not get into international search, it may well have a plan to expand into local search.

The latter really wouldn’t surprise me. Apple needs to get into local search because it needs to improve Maps. Improving Maps is a foundation tech, on which the company will base a swathe of future tech and service improvements, from wearables to self-driveables.

With this in mind, I’ll be very interested to see how much Apple is going to close the Siri gap between itself and Google Assistant over the next 12 months – because its Maps improvements indicate we’ll see big leaps forward in local and navigation, while Siri Shortcuts will do a great deal to massively increase its advantage when it comes to use of the Voice Assistant to command.

Don’t neglect that over the last 15-months Google Assistant has improved by 11 per cent while Siri has gotten 13 points better – and while Google is working to enhance its core competencies, too, it’s pretty clear that it will need to go some to match the platform competencies Apple provides.

Still, when it comes to Siri, the battle between all these players is a great illustration as to how competition can at times benefit us all.

Jonny Evans

Watching Apple since 1999. I don't say what they should do. I say what they might do. They sometimes do.

1 Response

  1. Bern says:

    If (big if) Siri use increases significantly due to IOS 12 AND Apple collects randomized data of all that usage there is then the potential for seriously updating Siri’s smarts. Much hinges on deeply the AI goes and what it has access to. Of course Apple’s iterative approach means Siri’s showing isn’t over in the days or weeks post IOS 12 rollout but will ripen with age.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.