Apple’s futuristic Vision Pro headset hit the one-year mark, but during that initial year, it still felt like it was missing some key features. Now, Apple is addressing a few of those features, at least in small steps, with VisionOS 2.4. Apple Intelligence is finally arriving on Vision Pro, along with some connected apps and guest mode features for iPhones and iPads.
The new OS update can be tested in a developer beta arriving today, but the official OS update isn’t coming until April. I’ve been waiting for these new features for a while.
Apple Intelligence might be the flashiest new feature, but it’s not likely to be the most meaningful yet — the AI extras launching in this wave don’t include any camera-enabled Visual Intelligence multimodal features like those Google Gemini will flex on Android XR. But it should at least be a foot in the door toward more AI getting added over time. I’m more excited that the Vision Pro is going to work better with iPhones and iPads. Not necessarily in all the ways I want, but a new app and a Guest Mode feature should give the headset remote-access functions that Meta’s Quest headset has had for years.
Apple Intelligence: Text and image tools, but no visual intelligence or new Siri yet
The Apple Intelligence features coming to Vision Pro are familiar; they mostly mirror what’s already on Macs, iPhones and iPads. Their arrival is overdue: I expected them last year. Writing tools, a mode integrated into several apps, is back again to summarize or generate text. There are message summaries, and ChatGPT can be invoked for extra assistance. There’s Apple’s generative AI GenMoji, which makes emoji on demand, and Image Playground, which makes 2D images. None of these will generate anything 3D on Vision Pro… not yet. And Apple’s AI-driven Memory Movies feature, which can generate photo and video galleries on demand, won’t show any 3D “spatial” movies or photos yet. That’s a bummer.
Still, the big missing thing is Visual Intelligence. Apple’s camera-enabled AI functions on iPhones, which are summoned using the Camera Button, scan the world and search or identify what’s in view. Visual Intelligence would make sense on Vision Pro, which is literally a giant wearable display with world-viewing cameras studded everywhere. Right now, though, there’s no Visual Intelligence feature in the mix. Or new Siri, for that matter. Apple’s revamped Siri should be a part of iOS 18.4, but it won’t be on the Vision Pro with this OS update.
Google, meanwhile, is already integrating a multimodal camera-assisted Gemini AI into Android XR, seemingly on course to be available on day one of that OS’ release.
Apple could still introduce Visual Intelligence later this year. It’s a likely candidate for VisionOS 3, which should be announced at Apple’s WWDC developer conference, usually held in June. But, at least, Apple Intelligence arriving on Vision Pro — as Apple already indicated it would — shows that the current hardware can do more than Apple has allowed.
A Vision Pro phone app and a remote-viewing guest mode
I was frustrated that the Vision Pro never had a good working relationship, or even a connection, with iPhones. Meta’s Quest headsets have had apps for phones for years that can browse and remote-download apps to the headset, sync phone notifications and remote-control Quest headsets to help people demo apps while you watch their experience on your phone screen.
Apple’s adding a lot of this with VisionOS 2.4 and iOS 18.4. An overdue Vision Pro iPhone app lets you remote-download apps and discover experiences coming to VisionOS. The app will also store details about the headset and prescription lens inserts. According to Apple, the app automatically appears on your iPhone with iOS 18.4 if you own a Vision Pro, but can also be downloaded from the App Store.
There’s also a new Guest Mode experience for sharing the headset. Apple’s current process is weird and clunky, and it doesn’t let you remotely observe what’s going on in-headset to help. The new mode kicks in when someone else puts on the headset, and your nearby iPhone or iPad has a button to start a connection. It has an app picker that’ll only make certain movies or apps appear on the headset, and it starts an AirPlay stream to watch whatever your guest is doing so you can guide them.
It’s a little strange that the new guest experience isn’t launched from the Vision Pro app, and right now, the app won’t let you remotely launch or pause apps; it’s a passive AirPlay stream. But again, it’s a start — and it sounds far better than what the Vision Pro had before.
A 3D video app called Spatial Gallery
Apple is also introducing a curious new app called Spatial Gallery, which is described as a curated showcase of 3D photos and videos shot using the iPhone-based “spatial” capture format. The app sounds like a way to find other 3D content to watch in the Vision Pro, which is odd since Apple already has bullishly but slowly showcased its 180-degree 3D Immersive Video format in the headset, too.
This is likely a tell that Immersive Video content production is facing a bottleneck. It’s an expensive format to shoot and edit, requiring very specific high-end cameras. Meanwhile, with more basic 3D videos shot on an iPhone or other cameras and edited with a few apps that support spatial video editing (Final Cut Pro, DaVinci Resolve), it seems like Apple is splitting the difference to offer more Vision Pro-ready experiences as quickly as possible.
In an ideal world, the future probably involves iPhone cameras developing better immersive 3D capture formats to make more impressive VR-ready content. The present, however, is split between phones having passable 3D capture and high-end professional cameras developing a different tier. I’d rather see Apple make further investments in useful immersive apps, but it looks like spatial video is going to be Apple’s most achievable content move in the short term.
Read the full article here
