In my office, there are bookshelves where I keep my magic tricks. Next to them are panoramic windows looking out on Paris — except, not really. They’re mixed reality overlays that greet me when I wear Apple’s Vision Pro headset, mapped to my actual windows. A virtual clock hangs on the wall.
The bookshelves are real; the Paris windows and wall clock aren’t. But Siri, which I invoke as a glowing orb in front of me, sees them all. When I ask what’s in front of me, Siri’s text box describes it all.
My first little moments wearing the Vision Pro and testing an all-seeing Siri AI in the developer preview of VisionOS 27 are surprising but very familiar. I’ve had moments like this in Samsung’s Galaxy XR headset, which already has a Gemini Live mode that can recognize your room and the virtual apps and windows that are open. I’ve also worn many smart glasses where I’ve asked camera-aware AI to tell me about the world as I’ve walked around.
As I expected, Apple’s got a new supercharged Siri coming to a range of devices this fall, from Watches to iPhones to iPads, and is expanding its visual intelligence capabilities beyond the iPhone. The Vision Pro’s Siri is the most fascinating, though, and maybe the most ambitious. It can see everything in your field of view on command, like a sensory companion.
Not everything is perfect in this very early preview build, revealed just days ago at Apple’s WWDC, but it already works well enough to show what could come next. Yes, glasses that could do this feel entirely possible now. In the meantime, the Vision Pro’s assistive functions are going to get pretty interesting, too.
Siri’s glowing orb is ready to see from the get-go
Unlike the Visual Intelligence modes on iOS and iPadOS, which launch via the Camera app, the Vision Pro’s Siri can see things when you simply say, “Hey, Siri,” and ask a question.
Siri appears as a 3D glowing orb, manifesting in my room like a spirit. You can drag and drop it anywhere, but it’s neat to see the glowing marble cast light effects on my desk or the room using VisionOS’ spatial graphics engine.
I was able to say, “Hey, Siri, what’s in front of me?” and it triggered the Visual Intelligence, making a new-sounding chirp and then flashing a quick visual scan around the area my eyes were looking, using eye tracking. It casts a wide net: It could see my bookshelf full of books in front of me and all the little action figures and toys I put there. It read a few book titles back to me, like Uzumaki and Wonderbook.
The AI feedback comes after the camera snaps a still image of what’s in front of me, either real or virtual. It’s similar to the snap-response cadence of other camera-based AI on Meta’s glasses or Samsung’s Galaxy XR.
But it doesn’t have a live mode like those glasses and headsets do. It’s a one-snap-per-request affair.
Recognizing real and virtual
On my desk, Siri was able to spot a red virtual reality headset (Virtual Boy) and a Steam Deck, both real. After that, I tried looking at my virtual Paris windows and wall clock widget and my bookshelf full of magic, and it recognized them too.
In this early beta, Siri tended to stick to a particular view for a while and analyze that capture unless I closed Siri or moved the orb to try again. Or, I guess, I’m just getting used to it.
But I’m already wondering how I could use this visual awareness to quickly handle work. I asked Siri to summarize a Notes app about my recent playwriting, which I pulled up just by asking (it summarized the key points), and asked it about browser windows I had open on my MacBook via the virtual display. (It’s telling me right now, as I work on this story in Vision Pro via my Mac, that I have a Google Doc window open where I’m writing about Siri recognizing real and virtual objects.)
Turning my own photos into panoramic 3D backgrounds
I also tried the new panoramic photo conversion effect in VisionOS 27, which lets any panoramic photo in your library turn into a 3D wraparound background you can use while working.
The results aren’t as full-3D wraparound as Apple’s own immersive environments, and they don’t move or have ambient sound effects. They’re more like very big super-wraparound 3D windows, with some of my office still visible around the edges.
It looks lovely, even though not all my panoramic photos converted (it’s very early days; I’m sure it’ll get worked out). I pulled up a photo of my mom in her backyard when I visited her during the pandemic, and now it can greet me like I’m almost there again. I’d love it if the Vision Pro added fully immersive Gaussian splat captures from multiple photos, as I do on the Meta Quest.
I can’t help thinking about how Apple’s expected smart glasses will use a seeing Siri for assistive purposes. But I’m getting ahead of myself. Right now, you need a $3,499 Vision Pro to make this happen. But when Apple has AR glasses like Google and Xreal’s Project Aura are smaller and more affordable? And when other apps can hook into Siri’s framework? It’s going to be fascinating to see what happens next. I feel like I’m just peeking over the edge at things to come.
Read the full article here