At I/O 2024, Google DeepMind unveiled Project Astra, which the company says is the future of AI assistants. Project Astra won’t be what the feature is called when it’s on your device, but it will be implemented into Gemini Live, the company’s upcoming conversational experience for mobile. Think of Project Astra as the concept or technology behind certain features, whereas Gemini Live is the finished product.

At Google I/O, in a small room, I looked at a Project Astra demo. In this demo, Project Astra was connected to a monitor and a camera, and I saw some of the upcoming functionalities that Gemini Live might be able to use in the future. However, Google didn’t allow me to record or photograph the demo.

The demo separated the features into four parts: Alliteration, Storytelling, Pictionary, and an open experience.

With its advanced reasoning capabilities, our prototype agent Project Astra was able to identify a number of famous faces in science – from just a few drawings. ✏️ pic.twitter.com/dK4O8vpqQx

— Google DeepMind (@GoogleDeepMind) May 15, 2024

I think Alliteration is the least useful feature out of the bunch. You can have Project Astra look at an object; the AI will see it and then speak only in alliterations regarding the object. For instance, they showed a plastic fruit in the demo, and Project Astra spoke about it in alliteration. Afterwards, the Google developer asked what would be good to eat with the fruit, and Project Astra continued speaking in alliteration, offering options on what to eat.

I didn’t see how this was useful. It could be fun for kids, but they would probably get bored after using the feature a few times.

Following Alliteration, we did a Pictionary demo, which I found pretty fun. In the demo, the Google employee drew stick figures and had Project Astra guess what show they were thinking about. One picture showed two stickmen with a conjoined thought bubble and an alien emoji inside the bubble. We then asked what show it was, and Astra guessed it was the X-Files. To make it more interesting, I asked what other show this could be, and it guessed Roswell.

Again, this probably isn’t the most useful feature, but I can see some use cases. Perhaps you couldn’t remember the name of a series or a song, but you could remember certain things about it, so you draw what you remember and ask Astra to identify the series or the song. It could be a visual expansion of the ‘hum song’ feature. While I couldn’t care less if the Alliteration feature makes it to Gemini, it’d be cool to see this functionality make it to the AI for everyone to use. It’s not the most practical feature, but playing around with your kids could be helpful and fun.

We’re sharing Project Astra: our new project focused on building a future AI assistant that can be truly helpful in everyday life. 🤝

Watch it in action, with two parts – each was captured in a single take, in real time. ↓ #GoogleIO pic.twitter.com/x40OOVODdv

— Google DeepMind (@GoogleDeepMind) May 14, 2024

During the demo, we didn’t dive too deep into Storytelling, but as the name says, Astra could make up stories about anything it sees. As someone trying to write a fantasy novel, this feature might be useful if I get stuck in a certain spot and need ideas. While many people are worried about AI replacing artistic jobs, I really believe that it could be a useful tool, and I tend to bounce ideas off ChatGPT whenever I’m lost. Speaking to Project Astra and bouncing ideas back and forth could be helpful when writing, although, to be clear, the application of Storytelling didn’t go that far, and it could only make up stories about what it saw on the screen.

In a separate Gemini demo, an engineer explained that Gemini Live could eventually do this, allowing users to bounce ideas.

Last but not least, we saw a demo of its open experience, where we tested out Project Astra’s memory. It identified certain plushy toys; they then gave each of the plushies a name, and Project Astra could remember the names of each. It was cool, and while not the most useful, it was a fun application of the feature we saw in the Project Astra video that Google showed off during its I/O press conference that allowed the user to ask if Astra knew where she placed her glasses. Ahead of my little hands-off demo, an Android developer had the Project Astra sandbox on his smartphone, showcasing how Project Astra could remember certain people. He showed the handset my face and told it my name, showed it his face and name and then asked Astra if it could remember who was who, which the AI did perfectly.

The coolest part of our hands-off demo was when someone asked Project Astra if it could identify the quote tattooed on his arm. In just a few seconds, Astra recognized that the quote was from Battlestar Galactica. This feels like an advanced form of Google Lens, the company’s image recognition technology, but also offering a natural language experience.

The Future of Google AI

Live audience request for GPT-4o vision capabilities pic.twitter.com/FPRXpZ2I9N

— OpenAI (@OpenAI) May 13, 2024

Project Astra is an interesting concept, and I like to see what Google is doing with the technology. If Astra ends up anything like the demo we saw at Google I/O, it’ll make Gemini a very capable AI. Remembering where your glasses are, quickly understanding coding, identifying objects in the real world, speedily learning about them, and recognizing where you are in the world are useful applications that can assist people in many different ways.

Of course, Google has a lot of competition in this field, such as OpenAI, which recently showed off a new ChatGPT-4o’s vision capabilities. I did like that Project Astra seemed to flow continuously throughout its demo, whereas ChatGPT-4o needed to be prompted a few times to activate.

The benefit of Gemini is that it’ll be on most Android devices, and users won’t have to download anything extra. ChatGPT-4o won’t have that benefit. Rumour has it that Apple is planning on putting ChatGPT on iOS 18. It’ll be interesting to see which AI people will find more useful, but considering Gemini has the benefit of expansions such as connecting between Google Maps and Gmail, as well as Google Keep and other apps in the future, Gemini might have a leg up, at least for now.

Image credit: Google