Storytellers have always been true magicians, transforming imagination into timeless tales that make incredible worlds. Roy Taylor and Sasa Marinkovic explain how VR can take storytelling to the next level.
Storytelling sorcerers have always been at the vanguard of the latest communication techniques and innovative technologies, weaving their spell with oral epics and poems in ancient times, gripping readers with printed books and novels, and enthralling audiences with today’s CGI-enhanced movies and cinematic and interactive video games.
Now we are witnessing the dawn of a new era of storytelling, this time using advanced virtual reality (VR) technology. VR promises to change forever how we experience electronic entertainment – along with computer simulations, gaming, education, social media, travel, medicine, real estate, ecommerce, and more. Virtual reality technology will also enable storytellers with an arsenal of new techniques to bring their imagination to life, and transform fantasies into believable tales.
Convincing Presence within the Virtual Environment
The challenge of creating a simulated, immersive and realistic computer-rendered world is in convincing participants of their presence within the virtual environment and making it seem utterly believeable. However, achieving this is a technological feat that necessitates advances in both computer hardware and software, and a »re-understanding« of how human senses truly work.
The complexity inherent in our perception of and interactions with the natural world is astounding. Visual and auditory stimuli blend to provide knowledge about the world around us – what objects are, where they are located, their color, how fast they are moving, and more. A simple everyday pleasure such as stopping to observe a rose and contemplate its grandeur is not easy to replicate from a sensory perspective.
Nevertheless, the experience of complete immersion in a hyper-realistic scenario is within reach today, thanks to recent advances in VR technology. By harnessing the graphics power of modern computing systems, sophisticated motion-sensing, and high-resolution head-mounted displays (HMDs), VR technology is now capable of generating simulated worlds where users can manipulate, explore and interact within a virtual environment.
VR Storytelling in Movies
Movies offer a great opportunity for virtual reality storytelling. Currently, investment banker-turned-film producer Jonah Hirsch is working on a project to tell the story and re-create the experience of the Wright Brothers’ first flight using VR. Hirsch and James Knight, one of Hollywood’s leading visual effects and motion capture experts, are collaborating with Smithsonian’s top Wright Brothers historians, Dr. Tom Crouch and Dr. Peter Jakab, to bring their dream to life.
Using virtual reality technology, people will be able to walk around, pause, rewind, and see the historic flight from different vantage points where it all began on the sand dunes of Kitty Hawk. Their goal is to recreate the most accurate two minutes of history ever viewed in a virtual experience – right down to minute details of the aircraft, including which side of the battery the ground cable is connected – with every detail thoroughly researched and reproduced.
The Wright Brothers: From Kitty Hawk to Virtual Reality
December 17th, 1903 was a day that changed the world forever. On that day, a pair of seemingly ordinary bicycle makers from Dayton, Ohio launched what looked like a propeller-driven box-kite along the Atlantic Ocean beach of Kitty Hawk, North Carolina – achieving history’s first powered flight.
Fast-forward 112 years. Most people today are familiar with the Wright Brothers – Wilbur and Orville Wright – but very little else is known about them. In fact, there is probably a better chance that somebody could tell you who your favorite celebrity is dating at the moment than the name of the Wright’s first powered airplane, now on permanent display at the Smithsonian’s National Air and Space Museum in Washington, DC. Significantly, it is the only aircraft in the entire Smithsonian museum that has a separate and dedicated exhibit hall all to itself.
Why would such a famous plane – an aircraft that achieved the world’s first powered flight, and occupies such exalted exhibition space at the most visited museum in the world – be a mere afterthought among today’s public?
»I think people just don’t really understand what an achievement it was that Wilbur and Orville did on that cold December day in 1903,« says Jonah Hirsch, who has spent the better part of the last five years attempting to secure funding for a feature film about the epic story of the Wright Brothers. »Studios want super-heroes, they want war, they want vampires – nobody seems to be interested in human achievement anymore except the indie financiers. To do this story justice, the film can’t be done on a typical indie budget so the options aren’t very good.« Not somebody to give up easily, Hirsch turned his sights on making an IMAX movie for the museum audiences.
Reaching the Museum Audiences
»I grew up around the Washington, DC area as a kid, and every year we would visit the Air and Space Museum on school field trips,« says Hirsch. »Without question the best part of the visit to the museum was buying the freeze-dried astronaut Neapolitan ice cream from the gift shop, and then settling in to watch the IMAX movie ›To Fly‹.«
Hirsch was visiting the museum on a trip to discuss the feature film with Dr. Tom Crouch and Dr. Peter Jakab. To his surprise, the same IMAX film he saw as a kid, was still playing at the Smithsonian. One of the original IMAX movies made for the museum’s giant screen, »To Fly« has been in continual exhibition at the museum for more than 30 years. Hirsch and his partners bought tickets and watched the iconic 40-minute movie. Afterward, Hirsch turned to his partners and said, »We’re making an IMAX movie«.
Hirsch had never made an IMAX movie – but he didn’t view it as a problem. Hirsch and his partners hired well-known Hollywood creative talent to develop an original script. They also hired acclaimed street artist Shepard Fairy to create original artwork for the project in an attempt to make the project »feel a bit hipper« for today’s audiences, and hopefully, more attractive to financiers.
The project started taking shape in October of 2014. James Knight, one of Hollywood’s leading visual effects and motion capture experts who was part of the pioneering never-seen-before motion capture techniques with James Cameron on the epic blockbuster »Avatar«, joined the effort and started scoping-out the project. Knight, the youngest member of the prestigious Academy of Motion Picture Arts and Science’s Sci-Tech Committee, was looking for something different – different than the typical 16-hour-day grind required the blockbuster hits he’s worked on, including »The Amazing Spiderman« and »Hulk«.
»When I first heard about the Wright Brothers project, I was immediately interested,« says Knight. »If I wasn’t working on films, I’d be teaching history at university – and this was a perfect opportunity to blend both.« A die-hard British soccer fan (football, more properly), Knight’s distinct Burberry’s uniform and High Street accent were a regular at weekend matches. There he met another die-hard soccer fan and ex-pat living in Los Angeles – Roy Taylor, who frequently scours the Los Angeles scene for talented virtual reality developers and unique ideas. He was immediately interested in the Wright Brother’s project, and suggested something even better than an IMAX film: a VR experience.
»IMAX films give the viewer an immersive entertainment experience, but VR enables a feeling of realistic presence, of actually being inside the experience,« says Knight. After a technical discussion while lunching at a local Hollywood hotspot, Taylor arranged to have an Oculus DK2 VR headset and a very high-powered notebook PC sent to Hirsch so he could experience first-hand what this virtual reality buzz was all about.
Knight set up the VR demo in the conference room, and Hirsch donned the VR headset. It only took about 30 seconds. Hirsch pulled off the headset and said, »OK – we’re doing virtual reality now.«
Working with Crouch and Jakab at the Smithsonian, Hirsch and his team have attempted to re-create the most accurate two minutes of history ever viewed in a virtual experience.
»Never before in the history of cinema has six months of production been devoted to creating a two-minute piece,« says Hirsch. »Even Spielberg takes small liberties on his historical films for the sake of a great story – but we could not, as we had one mission: To recreate history as it was.«
If only Wilbur and Orville Wright were alive to see this virtual reality re-creation of their first flight. They might conclude that the future has no need of an airplane, because virtual reality enables everyone to visit with each other in virtual space (though making that a reality is still years away). Hirsch and his team are busy working on their next historical piece, and plan to create a series of historical events that are best suited for the VR experience.
New Storytelling Techniques
VR technology is actively transforming the computer gaming industry. The game creators of Crytek were early to recognize the massive opportunities enabled by VR technology, and have been developing their industry-leading CRYENGINE to produce advanced visual gaming experiences and new storytelling techniques using VR technology.
»We see the potential of a new media and immediately start imagining stories that we can tell with it,« CRYENGINE Creative Director Frank Vitz explains. »That leads us to the development of software methods, animation tools and rendering modes that feed back into the storytelling, and, the next thing you know, we have what already feels like a full game. It’s ›just a tech demo‹ but it feels like a glimpse into a whole new world. That is what inspires us. When we see people’s reactions to the demo, their thirst for more, well, we know that we are on to something big.«
Hardware manufacturers such as AMD play a leading role in accelerating technological advances that enable the indispensable sense of »presence« sought by VR developers like Crytek. Developing advanced computing and graphics processors to render high fidelity imagery at high frame rates with low latency is vital to achieving presence.
Neuro VR Experience
We understand less about the human brain than any other organ of the body. However, this may soon change. General Electric scientists, working with top researchers and institutions around the world, are working to unlock new insights about the brain previously not possible. New tools and technologies that enable visualizing brain anatomy and function can give researchers new insights and a better understanding of treatments needed for brain illness or injury.
Among of the best visualization tools available today involves VR. Already beginning to revolutionize PC gaming, VR is poised to dramatically transform an array of applications and industries, including entertainment, business, education, communications, training, psychotherapy – and medical research.
»Visualizing the human brain in intricate detail in virtual reality is no easy feat,« notes Katrina Craigwell, Director of Global Content and Programming at GE. »It takes an incredible amount of computational horsepower to create the virtual world and ensure that you’re completely comfortable interacting within it. To power the experience, we’re using AMD’s next-generation GPUs which not only deliver the exceptional graphical detail we want, but also a number of optimizations via their LiquidVR technology that ensures that the virtual world feels every bit as responsive and natural as the real world.«
Combining the latest VR technologies with powerful AMD computing and graphics capabilities, GE scientists have created a virtual portal into the human brain, enabling anybody to enter, view, and explore the brain in ways never before possible. The Neuro VR Experience enables an interactive understanding of how GE scientists are breaking new ground in understanding how the brain works.
A Mind’s Representation
In GE’s Neuro VR Experience, viewers are virtually transported into Reuben Wu’s brain and are introduced to the inner workings of his mind – the interplay of billions of synaptic responses to sense and thought stimuli. This software-generated representation of Reuben’s mind simulates a physical presence within the universe of thoughts, desires, hopes and dreams contained within his brain.
Reuben is no stranger to the surreal. As an industrial designer, photographer, DJ, keyboardist and songwriter for the electro-synthpop band, Ladytron, the Liverpudlian artist captures and creates mind-bending auditory and visual experiences. His work not only stimulates the senses and elicits emotions, but also fuels new understanding of the grandeur of our natural environment and the imagining of what is possible. He has a beautiful mind, one that all will soon be able to explore through the latest in VR experiences.
As participants marvel at the intricate tapestry of intelligence, behavioral, sense and motion control centers that compose the masterpiece of the mind, they will also learn about the pivotal GE innovations that are propelling research into, and understanding of, the human brain. Enhanced visualization techniques, such as higher resolution and mobile magnetic resolution imaging (MRI) scanners, and integrated mechanisms to better analyze the volumes of data collected from brain research, are expediting the pursuit for cures to neurological diseases and disorders.
Offering a tantalizing virtual glimpse into the medical imaging and visualization advancements that will soon be reality, GE’s Neuro VR Experience immerses participants in a unique and visually stunning interactive journey through the wonder that is the human brain – the source of all qualities that define our humanity.
The capabilities of today’s VR technology are very good – but they still have not reached their full potential. Researchers working with computer hardware and software development, game design, human optics, and neuroscience are striving to further the VR experience. Their goal: to dissolve the last barriers to achieving true immersion, make a VR experience indistinguishable from reality, and create a new paradigm for the storytelling of tomorrow.
Roy Taylor, Sasa Marinkovic