A couple of months ago when Apple introduced its newest product: Vision Pro, the whole world went wild. It was in every article, in every social media post, and even in every meme. It was like humanity stepped for the first time on Mars. Don’t get me wrong, it’s not like it is not exciting to see a product that looks so futuristic in its characteristics that it is actually real and people are about to purchase it, but it has to do with the fact that it is not the first Mixed Reality (MR) product. It’s just an Apple product, and we all know the popularity of the company and everything it produces. It is bound to stir up interest. People are going to talk, and consider it the first of its generation, and how it is so groundbreaking and innovative that it makes sense for Apple to be the first one to do it. But in fact, it isn’t.
Image 1: Apple's Vision Pro
MR products have been around for some time, since 1968 to be exact, with The Sword of Damocles which is considered to be one of the earliest forms of Mixed Reality.
Image 2: The Sword of Damocles
And companies like Microsoft have already created MR products in the past (HoloLens – 2016).
Image 3: Microsoft's HoloLens
Virtual Reality (VR) products have been around for a longer time even. Sensorama Simulator is one of the earliest VR devices, created in 1962. And every tech company possible has to this day released one or at least created one.
Image 4: Sensorama Simulator
But I have to admit that when the news came out, I was intrigued just like everyone else. I did some research on how it differentiates from other products, what makes it special, and how it works in general, and I was genuinely impressed by the amount of work, and detail that everyone had put through to create this incredible product. The crazy numbers advertised were like:
- high-resolution cameras transmit over 1 Billion pixels per second to the displays
- custom micro-OLED display system features 23 million pixels, more than a 4K TV for each eye
- an R1 Chip streams images to the display within 12 ms
And that impressed me alright. But at that moment I couldn’t help but wonder at the role this technology is playing in the transformation of the User Experience/Interaction Realm.
At the same time when the news broke, I was in the middle of my UI for UX Designers course, and the task I was working on was called Interactions & Gestures. I was learning all about gestures, transitions, animations, and feedback. But what I was learning was all about having a 2D display in front of you.
Image 5: 2D Display Devices
Then I was thinking about how all VR products have always a controller that powers all the human-to-device interactions. Ok, in the case of VR, the designer has to take into consideration that they are dealing with spatial design and they should design navigations, menus, and interactions that feel natural in the 3D space.
Image 6: VR Kit
But with Augmented Reality (AR) products and MR products, you are working with the display and the display alone. This is the time for designers to put their creativity to use and utilize everything available.
New, unique hand gestures need to be used for interactions, in order for the camera to understand and capture them.
Image 7: Microsoft's Hand Tracking Presentation for HoloLens
Eye-tracking methodologies can be used to select objects.
Image 8: Labvanced Eye Tracking Metrics
A virtual keyboard needs to be used for typing.
Video 1: Apple's Vision Pro Virtual Keyboard and Writing Presentation
Voice commands are also a good idea (although speech recognition for a large number of languages and dialects needs a lot of improvement).
Image 9: Voice Commands and Speech Recognition
Another key element that is to be taken seriously is the way these digital objects interact with the real environment. How colors change in different lights. How the luminosity and contrast need to be adjusted automatically to match the environment in which the objects are being overlaid on.
Image 10: Color Theory
The audio comes next. The fact that the user needs to be aware of the surroundings while listening to the digital audio, adds another element that needs to be adjusted accordingly.
Image 11: Apple's Vision Pro Spatial Audio Presentation
That is quite the change! No pressure on the designer, right? And this is not even half of it. But that’s the beauty of the times we are living in. Everything is moving so fast. Even new fields such as UX and UI, they are restless. They adapt and evolve in the blink of an eye. And for me (a UX Designer who is starting just now) it is exciting, challenging, and beautiful altogether. I am thrilled to start this journey now and I love that not only I will be able to experience what the future has in store for it, but also actively participate in it.
