The Myriad Postures of XR

If you were watching the live stream of Microsoft’s 2019 Surface event there was a moment when you’d be forgiven for thinking you’d strayed onto a random YouTube Chiropractic Channel. About an hour into the annual hardware extravaganza Panos Panay - Chief Product Office (and Head of Dramatic Pauses) - took back the stage to wax more lyrically and philosophically than ever before, in a verbal drum-roll for the reveal of…. [long dramatic pause]... the Surface Neo.

"This product is the product that’s made to adapt to you in both form and function, not just in two postures but in every posture, creating limitless flow, uninterrupted."

And there it was, a word that Panay would go on to use not twice, not thrice but a grand total of ten times before the end of the presentation - posture. “Different postures”, “flexible postures”, “adaptable postures”, “every posture”, “in all its postures”, “unlimited postures”. 

And yet, for all the yoga poses he went on to highlight for their upcoming foldables, it's striking that these hinged slabs of glass will adapt to barely more postures than their older, arthritic kin. But you know what computing form factor will actually deliver a myriad of flexible postures?…. [dramatic pause, even though the title gives it away]… aha - XR (aka Spatial Computing). Myriad but not infinite.

A week before Microsoft’s event Facebook’s Oculus Connect 6 took place in California. Despite the reveal of Horizon, Facebook’s upcoming VR massive-multiplayer world, the most important development was, for my money, the announcement of hand tracking for Oculus Quest. 

At a breakout session entitled ‘Hand Tracking: Designing a New Input Modality’ Facebook Product Design Prototyper Jonathan Ravasz put it thus:

“Hand input is a super approachable and low-friction input that requires no additional hardware. It’s a foundational technology that crosses all of spatial computing including augmented reality.”

And that’s great but we must not forget about the Gorilla in the room (yes, gorilla) - ‘The Minority Report interface is far from practical’ - “It's a great interface for a 15-minute presentation, but for an 8-hour stint in a cubicle it's a recipe for cramps and soreness.”

Indeed this problem was well recognized by Leap Motion, the original enabler of hand interaction for VR - 

“According to some reports [Tom Cruise] had to frequently rest his arms while shooting the Minority Report Pre-Crime scenes. This condition is often called gorilla arm – when your muscles feel sore and stiff after holding out your arm for extended periods of time.”

In fairness to Oculus they also seem keenly aware of this issue, as evidenced by the aforementioned Connect 6 session which addressed awkward and neutral arm positions (top diagram) when looking at the ergonomic properties of the body. They posit The Work Envelope of primary, secondary and tertiary zones, based on arm reach, as a guiding principle for understanding the ergonomic cost of certain distances from the body and developing mid-air interaction models that afford long repetitive uses while also minimizing muscle strain. Where primary, secondary and tertiary zone correspond respectively to frequent, infrequent and occasional reaches.

Mike Alger was an early thought leader in ideation of ergonomic VR zones. It’s nearly 5 years since he published his VR Interface Design Manifesto, leaning on postgraduate research focused on UI/UX design for ergonomic applications of multitasking and productivity in VR. Therein Alger explored the obvious dissolution of the flat rectangular boundaries that limit our interactions on screens today to be replaced in future by 360 degree x 3D digital environments. With the emphasis on ‘environments’ - “Some people who are web designers before will need to learn to be environment designers, really will have to find out a lot about what architects and interior designers have already learned”. 

So if the environment subsumes the form factor what does that imply for postures? That an abundance of postures relegates postures to irrelevance. And effectively equates to a posture-less environment?

In a follow up video Alger dives deeper into the affordances of immersive interfaces and his learnings on facilitated workflows -

• More screen space makes computer workers more productive
• Less time and cognition is spent navigating occluded things in the interface
• Stereoscopic images and head-tracking give users the illusion of depth thus allowing them to spatially organize tasks which in turn increases productivity

This leads on to Alger’s concept of Content Zones based on the realistic use of VR by seated office workers, analogous to Oculus’ Work Envelope framework. Sitting in a regular chair your Field-of-View is 94 degrees, roughly. Beyond that you’ve got a certain range of rotation of the head, as well as up and down movement that is comfortable. We can see strong stereoscopic 3D to about 10m, with some 3D to about 20m. And ergonomics dictates the vertical positioning of objects that we might spend a long time gazing at (like text). The intersection of these various constraints yields an optimal Touch Interface zone bounding the work area within arms reach. Seen below this zone may seem just as limiting as slabs of glass.

But returning to the Oculus 6 session on ‘Hand Tracking: Designing a New Input Modality’, we learn that Jonathan Ravasz has an answer -

“Another powerful technique to enhance the capability of hands is remapping them - hand mapping is a method for indirect selection that translates hand movements from the body space to the target space. Like moving a mouse on a table and driving a cursor on the screen, mapping enables more ergonomic and precise interactions that are independent of the distance between the body”.

And in XR neither the movement of your hand, nor the corresponding digital action are confined to 2D planes, but instead can be manifested across a multiplicity of volumes. And postures, or rather non-postures. 

In hindsight, when Alger dives into a comprehensive user narrative for real work and multitasking in VR, where he brings together a number of ideas together around zones, gestures, depth and context switching, what’s apparent now is that it’s not only flat screens and desks that have dissolved into the ether but also for the most part the actual notion of postures. Of course there are still restrictions based on physics, physiology and ergonomics so this doesn’t translate to an infinity of postures. But certainly a multiplicity with seamless transitions between them unrestricted by rotating hinges.

In his statement of belief Alger references the even earlier work of Alex Chu, an Interaction Designer at Samsung Research. During his 2014 presentation on ‘Transitioning from a 2D to 3D Design Paradigm’ Chu makes a brilliant observation, when talking about Environmental Design, that contrasts the exposition of content in traditional 2D apps versus the real world (and XR). In comparing a phone based gallery app to a real world gallery space and the relationship of the content to the container Chu notes that the app puts a value on the primacy of content...

“... and the space in-between isn’t really meant to be noticed at all. But when we look at the real gallery space the relationship is far different - the content seems very small relative to the environment. And the environment plays a very meaningful role in the experience of the content with intangible qualities of light, texture, psychology and comfort that comes from these in-between spaces.”

This observation is summed up perfectly in the phrase that Chu uses in preference to Environmental Design - ‘The Design of Everything Else’. For Panay the posture is the container for all content. For Chu the XR environment is the container for all postures and ‘everything else’. In other words XR is practically postures-less.

At last week’s Made by Google ‘19 event their hardware chief Rick Osterloh opened with a vision for ambient computing, aka another version of posture-less computing -

“It’s super useful to have a powerful computer everywhere you are. But it’s even more useful when computing is anywhere you need it. Helpful computing can be all around you, ambient computing. The technology just fades into the background when you don’t need it. So the devices aren't the centre of the system, you are".

At the same event Sabrina Ellis introduced Google’s new voice recorder app with on-device transcription, even when offline. And it’s brilliant. In fact, I used it to compose a small portion of this article while walking through the countryside, rather than hunched over a another device. Much better for my posture.