THERE IS SOMETHING special happening in a generic office park in an uninspiring suburb near Fort Lauderdale, Florida. Inside, amid the low gray cubicles, clustered desks, and empty swivel chairs, an impossible 8-inch robot drone from an alien planet hovers chest-high in front of a row of potted plants. It is steampunk-cute, minutely detailed. I can walk around it and examine it from any angle. I can squat to look at its ornate underside. Bending closer, I bring my face to within inches of it to inspect its tiny pipes and protruding armatures. I can see polishing swirls where the metallic surface was “milled.” When I raise a hand, it approaches and extends a glowing appendage to touch my fingertip. I reach out and move it around. I step back across the room to view it from afar. All the while it hums and slowly rotates above a desk. It looks as real as the lamps and computer monitors around it. It’s not. I’m seeing all this through a synthetic-reality headset. Intellectually, I know this drone is an elaborate simulation, but as far as my eyes are concerned it’s really there, in that ordinary office. It is a virtual object, but there is no evidence of pixels or digital artifacts in its three-dimensional fullness. If I reposition my head just so, I can get the virtual drone to line up in front of a bright office lamp and perceive that it is faintly transparent, but that hint does not impede the strong sense of it being present. This, of course, is one of the great promises of artificial reality—either you get teleported to magical places or magical things get teleported to you. And in this prototype headset, created by the much speculated about, ultrasecretive company called Magic Leap, this alien drone certainly does seem to be transported to this office in Florida—and its reality is stronger than I thought possible.
I saw other things with these magical goggles. I saw human-sized robots walk through the actual walls of the room. I could shoot them with power blasts from a prop gun I really held in my hands. I watched miniature humans wrestle each other on a real tabletop, almost like a Star Wars holographic chess game. These tiny people were obviously not real, despite their photographic realism, but they were really present—in a way that didn’t seem to reside in my eyes alone; I almost felt their presence.
Virtual reality overlaid on the real world in this manner is called mixed reality, or MR. (The goggles are semitransparent, allowing you to see your actual surroundings.) It is more difficult to achieve than the classic fully immersive virtual reality, or VR, where all you see are synthetic images, and in many ways MR is the more powerful of the two technologies.
Magic Leap is not the only company creating mixed-reality technology, but right now the quality of its virtual visions exceeds all others. Because of this lead, money is pouring into this Florida office park. Google was one of the first to invest. Andreessen Horowitz, Kleiner Perkins, and others followed. In the past year, executives from most major media and tech companies have made the pilgrimage to Magic Leap’s office park to experience for themselves its futuristic synthetic reality. At the beginning of this year, the company completed what may be the largest C-round of financing in history: $793.5 million. To date, investors have funneled $1.4 billion into it.
That astounding sum is especially noteworthy because Magic Leap has not released a beta version of its product, not even to developers. Aside from potential investors and advisers, few people have been allowed to see the gear in action, and the combination of funding and mystery has fueled rampant curiosity. But to really understand what’s happening at Magic Leap, you need to also understand the tidal wave surging through the entire tech industry. All the major players—Facebook, Google, Apple, Amazon, Microsoft, Sony, Samsung—have whole groups dedicated to artificial reality, and they’re hiring more engineers daily. Facebook alone has over 400 people working on VR. Then there are some 230 other companies, such as Meta, the Void, Atheer, Lytro, and 8i, working furiously on hardware and content for this new platform. To fully appreciate Magic Leap’s gravitational pull, you really must see this emerging industry—every virtual-reality and mixed-reality headset, every VR camera technique, all the novel VR applications, beta-version VR games, every prototype VR social world.
Like I did—over the past five months.
Then you will understand just how fundamental virtual reality technology will be, and why businesses like Magic Leap have an opportunity to become some of the largest companies ever created.
Even if you’ve never tried virtual reality, you probably possess a vivid expectation of what it will be like. It’s the Matrix, a reality of such convincing verisimilitude that you can’t tell if it’s fake. It will be the Metaverse in Neal Stephenson’s rollicking 1992 novel, Snow Crash, an urban reality so enticing that some people never leave it. It will be the Oasis in the 2011 best-selling story Ready Player One, a vast planet-scale virtual reality that is the center of school and work. VR has been so fully imagined for so long, in fact, that it seems overdue.
The Untold Story of Magic Leap, the World’s Most Secretive Startup
I first put my head into virtual reality in 1989. Before even the web existed, I visited an office in Northern California whose walls were covered with neoprene surfing suits embroidered with wires, large gloves festooned with electronic components, and rows of modified swimming goggles. My host, Jaron Lanier, sporting shoulder-length blond dreadlocks, handed me a black glove and placed a set of homemade goggles secured by a web of straps onto my head. The next moment I was in an entirely different place. It was an airy, cartoony block world, not unlike the Minecraft universe. There was another avatar sharing this small world (the size of a large room) with me—Lanier.
We explored this magical artificial landscape together, which Lanier had created just hours before. Our gloved hands could pick up and move virtual objects. It was Lanier who named this new experience “virtual reality.” It felt unbelievably real. In that short visit I knew I had seen the future. The following year I organized the first public hands-on exhibit (called Cyberthon), which premiered two dozen experimental VR systems from the US military, universities, and Silicon Valley. For 24 hours in 1990, anyone who bought a ticket could try virtual reality. The quality of the VR experience at that time was primitive but still pretty good. All the key elements were there: head-mounted display, glove tracking, multiperson social immersion.
But the arrival of mass-market VR wasn’t imminent. The gear cost many scores of thousands of dollars. Over the following decades, inventors were able to improve the quality, but they were unable to lower the cost.
Twenty-five years later a most unlikely savior emerged—the smartphone! Its runaway global success drove the quality of tiny hi-res screens way up and their cost way down. Gyroscopes and motion sensors embedded in phones could be borrowed by VR displays to track head, hand, and body positions for pennies. And the processing power of a modern phone’s chip was equal to an old supercomputer, streaming movies on the tiny screen with ease. The cheap ubiquity of screens and chips allowed a teenage Palmer Luckey to gaffer-tape together his first VR headset prototypes, launching a Kickstarter campaign for the Oculus Rift in 2012. And the Rift was the starting signal that many entrepreneurs were waiting for. (Facebook bought the company for $2 billion in 2014.)
All of today’s head-mounted VR displays are built out of this cheap phone technology. Put on almost any synthetic-reality display and you enter a world born of billions of phones. Lanier, who has contributed to Microsoft’s HoloLens MR system, estimates it would have cost more than $1 million in 1990 to achieve the results that even simple phone-inserted headsets like the Samsung Gear or Google Cardboard do today.
The recurring discovery I made in each virtual world I entered was that although every one of these environments was fake, the experiences I had in them were genuine. VR does two important things: One, it generates an intense and convincing sense of what is generally called presence. Virtual landscapes, virtual objects, and virtual characters seem to be there—a perception that is not so much a visual illusion as a gut feeling. That’s magical. But the second thing it does is more important. The technology forces you to be present—in a way flatscreens do not—so that you gain authentic experiences, as authentic as in real life. People remember VR experiences not as a memory of something they saw but as something that happened to them.
Travel experiences—terror at the edge of an erupting volcano, wonder at a walking tour of the pyramids—will be accessible to anyone with a VR rig.
Experience is the new currency in VR and MR. Technologies like Magic Leap’s will enable us to generate, transmit, quantify, refine, personalize, magnify, discover, share, reshare, and overshare experiences. This shift from the creation, transmission, and consumption of information to the creation, transmission, and consumption of experience defines this new platform. As Magic Leap founder Rony Abovitz puts it, “Ours is a journey of inner space. We are building the internet of presence and experience.”
We haven’t yet fully absorbed the enormous benefit that the internet of information has brought to the world. And yet we are about to recapitulate this accomplishment with the advent of synthetic realities. With a VR platform we will create a Wikipedia of experiences, potentially available to anyone, anywhere, anytime. Travel experiences—terror at the edge of an erupting volcano, wonder at a walking tour of the pyramids—once the luxury of the rich (like books in the old days), will be accessible to anyone with a VR rig. Or experiences to be shared: marching with protesters in Iran; dancing with revelers in Malawi; how about switching genders? Experiences that no humans have had: exploring Mars; living as a lobster; experiencing a close-up of your own beating heart, live.
You’ve seen a lot of this in movies and on TV or read about it in books. But you haven’t experienced it, felt it below your intellect, had it lodge in your being in a way that you can call your own. Kent Bye, founder of the podcast Voices of VR, has conducted over 400 interviews with the people creating VR and has seen almost every possible prototype of VR there is. “VR talks to our subconscious mind like no other media,” he says.
The most intense and complete sense of subconscious presence that I experienced occurred with a system called the Void, which debuted at the 2016 TED conference. The Void isn’t as advanced as Magic Leap technologically, but it integrates the best off-the-shelf parts available with custom gear to create an unforgettable experience. For several hours I watched a line of people enter the Void. Almost every person squealed with delight, screamed, laughed, and staggered away asking for more. I felt the same; I’d be happy to pay for an hour’s visit.
The Void grew out of stage magic, a theme park, and a haunted house. Every year, Ken Bretschneider, one of the three cofounders, stages a gonzo haunted house in Utah that draws 10,000 people in two days. It occurred to him that he could amplify the interactions of his house with VR. Curtis Hickman , the second cofounder, is a professional illusionist, designed tricks for big-name magicians, and is also a visual-effects producer. The third, James Jensen, started out developing special effects for film and unique experiences for theme parks. He came up with the idea of layering VR over a physical playground. The common factor among the three was their realization that VR was a new way to trick the mind into believing something imaginary is real.
The Void takes place in a large room. You wear a 12-pound vest that carries batteries, a processor board, and 22 haptic patches that vibrate and shake you at the right moments. Your headset or goggles and earphones are connected to your vest, so you’re free to roam without a cord. Untethered, you’re released from worrying about tripping over a cable or tangling or straying too far. That relief heightens the effect of being present in the VR. Inside, you navigate an Indiana Jones-like adventure that seems to take place over a large territory. The illusion of unbounded space, or, as Hickman describes it, “a magical space bigger inside than it is outside,” is achieved by a trick called redirected walking.
As an example, whenever you turn 90 degrees in the room your VR will show you the room turning only 80 degrees. You don’t notice the difference, but the VR accumulates those small 10-degree cheats on each turn until it redirects your route away from a wall or even gets you to walk in a circle while making you think you’ve walked a mile in a straight line. Redirected touching does a similar trick. A room could contain one real block but display three virtual blocks on a shelf—blocks A, B, and C. You see your hand grab block B, but the VR system will direct your hand to touch the only real block in the room. You can replace block B and pick up block C, but in reality you’re picking up the same real block.
It’s astounding how those tiny misdirections fool your gut into believing that what you’re seeing is real. Stairs can be made to feel endless if they drop down as you walk upward. In fact, at one point in the Void a decaying floor collapses while you’re walking across it, and you see, hear, and feel—in all your body—a plunge down to the floor below. But in fact the real floor only sinks 6 inches. You can easily imagine a room 60 by 60 feet packed with a minimal set of elemental shapes, ramps, and seats, all recycled and redirected for a variety of multihour adventures.
Seeing, it turns out, is not believing. We use all our senses to gauge reality. Most of the high-end VR rigs on sale this year include dynamic binaural—that is, 3-D—audio. This is more than just stereo, which is fixed in space. To be persuasive, the apparent location of a sound needs to shift as you move your head. Deep presence includes the sensations of motion from your inner ear; if the two are out of sync with what you see, you get motion sickness. Good VR also includes touch. Jason Jerald, a professor at the Waterford Institute of Technology who wrote the book on VR (called The VR Book), claims that much of our sense of presence in VR comes from our hands. Gloves are still not consumer-ready, so hardware makers are using simple controllers with a few easily operated buttons. When you wave them, their positions are tracked, so you can manipulate virtual objects. As primitive as these stick-hands are, they double the sense of being present. Touch, vision, and sound form the essential trinity of VR.
HILE MAGIC LEAP has yet to achieve the immersion of the Void, it is still, by far, the most impressive on the visual front—the best at creating the illusion that virtual objects truly exist. The founder of Magic Leap, Rony Abovitz, is the perfect misfit to invent this superpower. As a kid growing up in South Florida, he was enthralled by science fiction and robots. He gravitated toward robots as a career and got a degree in biomedical engineering from the University of Miami. While still a grad student, he started a company that built robots for surgery. Before the company got off the ground, his only income was $30 a week drawing cartoons for his college newspaper. Most people find Abovitz’s cartoons more weird than funny. They are stream-of-consciousness doodles featuring alien creatures, annotated by tiny inscriptions that include secret messages to girlfriends. They do not appear to come from the mind of an engineer. As it happens, though, good virtuality takes both fantasy and physics.
Abovitz is heavyset, bespectacled, and usually smiling. He is warm and casual, at ease with himself. But he vibrates. He hums with ideas. Overflowing. One idea unleashes two more. He whips his large head around as he speaks, sweeping up more ideas. It’s hard for him to throttle their escape, to slow down how fast they issue from his brain. As in his cartoons, a discussion can leap almost anywhere. Most of his ideas seem to combine physics and biology. In his Twitter bio, Abovitz describes himself as a “friend of people, animals, and robots,” which is pretty accurate. In his conversation and his work he exhibits a rare sensitivity to both the logic of machines and the soul of biology. If you’re making robot arms that help human doctors carve into living flesh, you have to obey the laws of physics, the laws of biology, and the minds of humans. Abovitz has a knack for all three realms, and his surgery robots sold well. In 2008 his company, Mako, went public. It was sold in 2013 for $1.65 billion.
That success sparked a new idea. Could you make a virtual knee good enough to help repair a real knee? Could you augment a knee operation with an overlay of a virtual knee? Abovitz began thinking about the technology that could match virtual worlds with complex real-life surgery.
At the same time he began to create a graphic novel.
Abovitz has a deep love of science fiction, and he invented a whole world on another planet—flying whales, men in dragonfly gear, a young girl with a pet monkey-bat, and an invading army of robots. Flush with cash from his robotics company, he hired Weta Workshop, the New Zealand special-effects house co-owned by movie director Peter Jackson, to create a detailed realization of that world. The Weta team created all the props and practical effects for The Lord of the Rings, and they helped invent the culture of the Na’vi in Avatar. For Abovitz they designed his world, called Hour Blue, and filled in the details of flying whales and monkey-bats. It quickly mutated from graphic novel into virtual-reality precursor. Because what alien world would not be better experienced in immersive 3-D? Abovitz was already pioneering MR for doctors; this would be an extension of his ideas.
The company Abovitz set up to develop this immersive world was Magic Leap. Its logo would be his totem animal, the leaping whale. The hardware to create the MR would have to be invented. By this time, 2012, the Oculus Kickstarter campaign had launched, and other prototypes with similar phone-based technology were in the works. Here Abovitz deviated off the main path. Because of his work in biomedicine, he realized that VR is the most advanced technology in the world where humans are still an integral part of the hardware. To function properly, VR and MR must use biological circuits as well as silicon chips. The sense of presence you feel in these headsets is created not by the screen but by your neurology. Tricks like redirected walking operate in our brain as much as in the Nvidia processor. Abovitz saw artificial reality as a symbiont technology, part machine, part flesh. “I realized that if you give the mind and body what they want, they’ll give you back much more,” he says.
Artificial reality exploits peculiarities in our senses. It effectively hacks the human brain in dozens of ways to create what can be called a chain of persuasion. In a movie, our brains perceive real motion in a sequence of absolutely still images. In the same way, you can scan a blue whale from many angles and then render it as a 3-D volumetric image that can be displayed on a headset screen and viewed from any position. Even if we know the object isn’t real—say it’s Godzilla instead of a whale—we feel subconsciously that its presence is real.
But if even one small thing is misaligned, that discrepancy can break the gut-level illusion of presence. Something as simple as having to
A GAGGLE OF GOGGLES
As virtual (and mixed) technology improves—and as companies start smelling profits—everyone from phone manufacturers to tech giants is getting into the game. Here’s the hardware that VR’s and MR’s biggest players are cooking up. —Chelsea Leu
AVAILABILITY VR MR
OSVRDEV KIT AVAILABLE
MicrosoftDEV KIT AVAILABLE
MetaDEV KIT, FALL 2016
worry about tripping over a tethering cable can seed our unconsciousness with doubt. It might look like it’s there, but it won’t feel there.
Following his hunch to exploit human biology, Abovitz set off to make an artificial-reality display in a more symbiont way. The phonelike screens used in the majority of head-mounted displays created a nagging problem: They were placed right next to your eyeballs. If the device is generating the illusion of a blue whale 100 feet away, your eyes should be focused 100 feet away. But they’re not; they’re focused on the tiny screen an inch away. Likewise, when you look at a virtual jellyfish floating 6 inches from your face, your eyes are not crossed as they would be in real life but staring straight ahead. No one is conscious of this optical mismatch, but over long use the subconscious misalignment may contribute to frequently reported discomfort and weaken the chain of persuasion. Magic Leap’s solution is an optical system that creates the illusion of depth in such a way that your eyes focus far for far things, and near for near, and will converge or diverge at the correct distances.
In trying out Magic Leap’s prototype, I found that it worked amazingly well close up, within arm’s reach, which was not true of many of the other mixed- and virtual-reality systems I used. I also found that the transition back to the real world while removing the Magic Leap’s optics was effortless, as comfortable as slipping off sunglasses, which I also did not experience in other systems. It felt natural.
Magic Leap’s competition is formidable. Microsoft is now selling development versions of its mixed-reality visor called the HoloLens. The technology is unique (so far) in that the entire contraption—processor, optics, and battery—is contained in the visor; it is truly untethered. Meta, another startup, has released an MR device that began, like Oculus, with a Kickstarter campaign. The headset is tethered to a computer, and dev kits should hit the market this fall—likely well before Magic Leap.
Abovitz realized that VR is the most advanced technology in the world where humans are still an integral part of the hardware.
All three major MR headsets rely on images that are projected edgeways onto a semitransparent material—usually glass with a coating of nanoscale ridges. The user sees the outside world through the glass, while the virtual elements are projected from a light source at the edge of the glass and then reflected into the user’s eyes by the beam-splitting nano-ridges. Magic Leap claims that its device is unique in the way it beams light into the eye, though the company declines to explain it further at this time.
However Magic Leap works, its advantage is that pixels disappear. Most screen-based, head-mounted VR displays exhibit a faint “screen door” effect that comes from a visible grid of pixels. Magic Leap’s virtual images, by contrast, are smooth and incredibly realistic. But in truth, the quality of displays in all alternative-reality gear—VR and MR alike—is improving rapidly. Month by month the resolution of all visors increases, the frame rate jumps, the dynamic range deepens, and the color space widens. Within two decades, when you look into a state-of-the-art virtual-reality display, your eye will be fooled into thinking you’re looking through a real window into a real world. It’ll be as bright and crisp as what you see out your window.
Once this small display perfects realism, it becomes the one display to rule them all. If a near-eye screen offers sufficient resolution, brightness, breadth, and color richness, it can display any number of virtual screens, of any size, inside it. While I was wearing the photonic spectacles of Magic Leap, I watched an HD movie on a virtual movie screen. It looked as bright and crisp as my 55-inch TV at home. With Microsoft’s HoloLens on, I watched a live football game on a virtual screen hovering next to a web browser window, alongside a few other virtual screens. I could fill my office with as many screens as I wanted, as big (or small) as I desired. I could click for a screen overlaid anywhere in the real world.
One of Microsoft’s ambitions for the HoloLens is to replace all the various screens in a typical office with wearable devices. The company’s demos envision workers moving virtual screens around or clicking to be teleported to a 3-D conference room with a dozen coworkers who live in different cities. I found virtual screens and virtual media within a virtual reality surprisingly natural and practical. At Magic Leap, the development team will soon abandon desktop screens altogether in favor of virtual displays. Meron Gribetz, founder of Meta, says that its new Meta 2 mixed-reality glasses will replace monitors in his company of 100 employees within a year. It’s no great leap to imagine such glasses also replacing the small screens we all keep in our pockets. In other words, this is a technology that can simultaneously upend desktop PCs, laptops, and phones. No wonder Apple, Samsung, and everyone else is paying attention. This is what disruption on a vast scale looks like.
Exclusive Footage of What It’s Like to See Through Magic Leap