Passthrough
I was reading a piece about Apple Vision Pro in the Weekend FT and a quote from CEO Tim Cook stayed with me.
Cook said that AVP is the first Apple product that its users look through, not at.
This sentiment, that screens are merging with our field of vision, reflects a changing notion of what screens are and do.
It connects to a relatively new area of computer vision research and experience design called passthrough technology.
Passthrough is a feature that allows users to see their real-world surroundings while wearing a virtual reality (VR) headset.
Passthrough uses cameras on the headset to capture a real-time video feed of the user's environment. This video feed is then displayed on the headset, allowing the user to interact with the real world while also seeing digital content.
Passthrough is a key component of mixed reality (MR) experiences, which allow users to place virtual 3D objects in their field of view. Passthrough helps blur the lines between the real and virtual worlds.
It takes a lot of work (research and design) to make a passthrough experience feel real to users.
In terms of research, a quick look on Google Scholar for "passthrough view synthesis" comes back with almost 20k articles, the vast majority written since 2020.
Many of these are academic, from computer science labs at universities. Some are from commercial players, like Facebook Reality Lab (FRL) which has been generating lots of valuable patents in this space.
When it comes to research, for now, it seems Meta via FRL is breaking the new paths. But when it comes to design, Apple lights the way. And in the FT article, various employees go on the record about it.
"We wanted to make sure that you felt like you were where you were,” said Richard Howarth, Apple’s vice-president of industrial design. “So we put an awful lot of effort into the clarity — of the glass, the optics, the whole system — to make sure you didn’t feel separated from it.”
I demo-ed the AVP recently at an Apple Store and can confirm the feeling of thereness.
But there are some parts that still need refinement. For example, if spatial computing is not going to be a fully isolating experience, shouldn't the passthrough metaphor be extended? Shouldn't it be two-way? You see people and people see you?
Interestingly, FRL explored this but at the moment it looks super strange to have full-def eyes displayed on the outside of a headset. See above. Lower definition mirroring like AVP’s seems to work a little better, I think.
Excited to see where this goes next.
You can read the entire FT article here:
https://www.ft.com/content/387f3293-01f1-4e95-b226-2db93c00b52d
For some interesting related research bites, look here:
