Optics the key challenge for AR, VR and mixed reality evolution

New types of optical and photonics technologies need to be implemented in next-generation virtual reality and augmented reality (VR/AR) systems in order to provide greater visual comfort for prolonged usage, and to achieve a better sense of display immersion for the user.

That is one of the main conclusions drawn by Bernard Kress, Partner Optical Architect at Microsoft HoloLens and chair of the inaugural VR, AR, MR (the hybrid "mixed reality") one-day industry conference and headset demonstrations event that took place in the Moscone Center during SPIE Photonics West. The new event featured no fewer than 18 invited talks from key systems developers, academics, and industrialists, as well as a panel covering the latest optical challenges and successes for VR, AR, and MR technologies.

Hot topics that this diverse community is currently addressing include some familiar and some not-so-familiar terminology: optical foveation, vergence, accommodation conflict mitigation, pixel occlusion, high dynamic range, and peripheral displays.

Foveation is a digital image processing technique in which image resolution is varied across the field of view according to so-called "fixation points" determined by the position of the user's fovea, at the center of the retina.

Vergence is the simultaneous movement of a user's pupils towards or away from one another during focusing. As AR,VR, MR have rapidly taken off, these issues have become real challenges for the optical engineers, who also have to consider size, weight, center of gravity, and thermal management of the equipment to provide true, wearable comfort.

Kress told Show Daily, "This conference at Photonics West is the first time we have in a single location all the major developers of VR, AR, and MR headsets. These include large corporations such as Microsoft, Google, Facebook, Amazon, Intel, and Huawei, but also exciting smaller start-ups such as DigiLens, Avegant, Meta, ODG, Lumus, and Leia, as well as renowned universities that have strong research programs in these fields such as University of Arizona, Berkeley University, and Stanford University."

AR/VR/MR demonstrations at SPIE Digital Optical Technologies 2017 in Munich.

This conference follows a demonstration at last year's Laser World of Photonics event in Munich, part of SPIE's Digital Optical Technologies conference, which Kress also chaired. That event featured demonstrations of headsets from Microsoft, Oculus, HTC, and Sony. Attendees had the opportunity to learn about and try on the headsets to experience a range of head-spinning entertainments like virtual rollercoaster rides, deep-sea diving, and car racing.

"With this new full-day conference, we are particularly glad to have hosted 18 exciting invited talks focussing on specific optical hardware from such companies and start-ups, 12 amazing headset demonstrations in the largest such session assembled to date, and a panel discussion of leading industry figures," says Kress.

These included world-renowned VR and AR pioneers such as Jaron Lanier (the so-called "inventor of VR" as well as an author, musician, and composer), Ronald Azuma (who defined the term "augmented reality"), Thad Starner (Professor at Georgia Tech and one of the first smart glasses pioneers) and Mark Bolas (Professor at USC, founder of Fakespace and mentor of Palmer Luckey, founder of Oculus).

Kress described his own personal highlights of the day: "I really enjoyed the various novel solutions described by the participants to solve main visual discomfits in AR systems, such as how to resolve Vergence Accommodation Conflict, and how to enable more comfortable arm's length display interaction in MR immersive display systems, now a key feature for enterprise-driven MR applications."

With at least 500 attendees expected to take part - many very actively - this was definitely an interactive experience. In April, Kress will continue the series of AR and VR events at SPIE Photonics Europe, with a conference entitled Digital Optics for Immersive Displays (DOID18). The Strasbourg, France, event will feature a student-focused optical design challenge dedicated solely to designing novel optics and optical architectures for imaging, optical combiners and sensing for AR and VR. "We have been pleased to receive around 70 submissions to be shortlisted," says Kress.

Emergence of "MR" hybrid
Virtual reality started to take off in the early 1990s, involving some of the pioneers that have been speaking at this week's panel sessions (notably Lanier and Bolas). AR and smart glasses also experienced an initial boom in the 1990s, thanks to the early work of Azuma and Starner.

Kress told Show Daily that he believes these rapidly evolving technologies will interact over the coming years. "Market analysts agree that the AR market will overtake the VR market as early as this year, 2018," he said. "The main revenues today and in previous years have come solely from VR systems, but revenues in the coming years will be
ever-increasing for AR.

"Now we have MR, which combines the concepts of VR and AR in an architecture that is more flexible, capable of delivering to the user both experiences, through high-performance imaging, and combiner optics capable of producing a quasi-lightfield over large fields of view. MR is supported by a battery of sensor technologies providing eye tracking, gesture sensing, depth map sensing, head tracking and iris authentication - all in a low-power and compact format."

One key issue to address in order to solve the ultimate MR hardware challenge is to better understand the limitations and specifics of the human visual system, which is very different to traditional imaging systems based on cameras and sensors. Combining efforts through collaboration between optical engineers and optometrists is key to achieving these goals. An example is what we call "high resolution:" for a camera just a simple, high-pixel-count image, but an entire sensory experience for the human visual system.

Kress commented, "This is why we involved renowned optometrists on our panel, such as Marty Banks from Berkeley University, who gives unique insights on the human visual system and how to take advantage of them in order to decrease the burden on optical design, while increasing the comfort for the user."

Equipment trends
What might be termed the "ultimate MR headsets" are now demanding new developments in all aspects of hardware. That ranges from battery technology, thermal management, material engineering, and custom silicon for display and sensor fusion, to custom mechanical design and novel optical elements and optical architectures, including micro- and nano-optics operating both in free space and wave-guide mode.



Microsoft's Hololens VR hardware assisting Ford's car designers. Photo: Microsoft/Ford.

Kress contends that the associated electronics will continue to follow Moore's law, providing smaller and less power-hungry circuits. Optical sensors will also be reduced in size, but classical optics will have a hard time adapting to the size, weight, and efficiency requirements. Unlike electronics, optics does not follow Moore's law, and is proving to be one of the hardest challenges to solve in AR/VR hardware.

"However, the optical engineer's toolbox is ever growing, with new and very exciting tools such as surface gratings,volume holograms, sub-wavelength relief structures, surface plasmons, photonic crystals, and metamaterials," Kress points out. "The task is not to replicate traditional optical functions with novel technologies but to use novel technologies to provide optical functionalities that [it has not been possible to] integrate with classical optics."

Application areas
Market analysts had predicted impressive growth for VR and AR in 2017, but the reality was a slower ramp than had been anticipated. However, Kress predicts that the scope of the VR/AR market by 2022 will cross the "$100 billion line" in terms of revenues. He added: "The pace of discovery in optics today is faster than any time in my own career in optics, even faster than through the glorified optical telecom boom of the late 1990s and early 2000s. Such technological revolutions - for example, optical metasurfaces - will feed the need for smaller, lighter, cheaper optics delivering functionality that was out of reach for traditional optics, including diffractive and holographic [elements]."

From VR to AR to smart glasses and MR, application fields vary significantly. VR tends to appeal to gaming and virtual presence and virtual experience purposes. Aside from games, examples include virtual home visits for the real estate sector, along with virtual travel, sports, art, and communications applications such as virtual conferences and meetings. There is also telepresence (the use of virtual reality technology, especially for remote control of machinery or for apparent participation in remote events), and Microsoft's own "Holoportation" application.

That Holoportation technology boasts of enabling users to "jump into someone else's reality." At its heart is a new 3D image capture system developed by Microsoft Research. A series of cameras are set up around a room, tracking shapes and movement for the observer and stitching together a 3D model in real time.



DigiLens is working with BMW on a near-to-eye technology for motorcycle helmets. Photo: DigiLens.

AR seems to have more appeal in professional and enterprise sectors, potentially boosting productivity and decreasing human errors in complex processes (such as in automotive or avionic assembly, on oil rig platforms, or for complex machinery support), but the technology is also finding markets in architecture, surgery, and industrial design. In gaming, fewer opportunities have emerged than with VR. AR also allows for arm's-length interaction of the real world, notably hands, and the display. Kress says this is only possible with light fields or holography (as provided with Avegant or Magic Leap hardware). Interaction between a user's hands and the virtual objects in holograms could become essential for designers, surgeons, and many other enterprise application examples.

The relatively established technology of smart glasses tends to appeal more to consumers. Such spectacles can provide a minimalistic AR approach, exemplified by the "Pokemon Go" game, including contextual display of features such as traffic conditions, food outlets.

Despite all of this apparent commercial and infotainment sector progress, Kress says that the full potential of mixed reality technology has not yet been realized: "One thing is for sure," he told Show Daily, "We have not yet found the killer app for VR, AR, MR, or smart glasses."

By way of explanation, he added, "The companies represented at this Photonics West conference are attempting to provide the ultimate MR user experience in terms of wearable comfort, visual comfort, and immersion, as well as the best development platform so that developers can figure out and develop the killer apps. "It's a chicken-and-egg situation: hardware functionality entices developers to develop new, exciting apps, and then the apps entice hardware developers to develop new
hardware functionality to enable them."

Right now, the key developmental challenges range from increasing the FOV without decreasing the so-called eye box (violating the etendue conservation principle), mitigating the vergence accommodation conflict (VAC) to allow arm's-length display interactions, including pixel occlusion to make holograms more realistic, and including optical foveation. Most of these developments will rely on accurate eye tracking.

For image generation, traditional technologies such as HTPS LCD, LCOS, micro-OLED, DLP, laser MEMS, or fiber scanners are used, while novel technologies capable of high-brightness, high-contrast emissive displays, are emerging. New challenges such as VAC mitigation call for additional optics such as tunable lenses either in reflection or transmission modes and/or liquid crystal or MEMS phase panels for holographic displays.

On the optics side, there are many sensor challenges. An MR device is only as good as its sensors, followed by the quality of its display - in that order. Options include head-tracking (requiring six degrees of freedom), but also depth mapping provided by time-of-flight sensors for 3D scanning, allowing for "world locking" of holograms.

Another rapidly emerging technology, artificial intelligence (AI), will also likely be used to recognize scanned objects such as tables, chairs, or people, and categorize them as such. To this end, Monday's conference featured Sense: a San Jose company specializing in optical sensors for AR and MR.

Other issues include the usual technological difficulties: improving batteries, thermal dissipation, and optimizing ergonomics. But Kress says that optics remains the main challenge. Considering the geography of the AR/VR/MR business, the key companies are mainly still based in the US. However, the key AR/VR market will be in Asia and the center of gravity could be starting to shift. Kress said, "Today, most of the investment in AR and VR startups is coming from China. Emerging startups in AR and VR are multiplying in China thanks to huge government investments in this very exciting field."

Of his own particular brand, Microsoft's Hololens project, Kress commented, "I cannot say much about our own internal development plans, of course, but we have a very aggressive and challenging MR hardware roadmap which reaches well in the 2020s. HoloLens V1 is only the tip of the iceberg."

In addition to MR hardware and an apps development platform, Microsoft is also actively pursuing various AI aspects dedicated to MR, especially in the image processing realm and optical sensors for MR - including continuous 3D world scanning for realistic hologram world locking and interaction.

Kress concluded, "We have one of the strongest optics teams on the planet, which continues to amaze me every day with its vision, inventiveness, excitement and productivity."

Matthew Peach is a contributing editor to optics.org and director of Original Content Ltd. A version of this article appeared in the Photonics West Show Daily in January.

Related SPIE content:

Virtual experience draws a real big crowd