On Pointe: First of its kind LiDAR-based VR documentary, The 100%, premieres at Tribeca Film Festival

On Monday, April 22, THE 100%—a “first experience of its kind” VR narrative—premiered at the Tribeca Film Festival in New York City. Winner of this year’s Tribeca X Award for Best VR Film, the film features the story of Maggie Kudirka, a young ballerina with the Joffrey Concert Group, whose career was derailed due to cancer. Using volumetric video and LiDAR to create a VR experience to tell Maggie’s story, Springbok Entertainment has broken new ground in immersive storytelling. Maggie Kudirka, a young ballerina with the Joffrey Concert Group, whose career was derailed due to cancer. Using volumetric video and LiDAR to create a VR experience to tell Maggie’s story, Springbok Entertainment has broken new ground in immersive storytelling.

We caught up with Andy Cochrane, Springbok Entertainment’s head of immersive content, about their use of the Leica BLK360 as an integral part of the project. 


What is the premise of THE 100%? 

THE 100% is an immersive VR narrative that takes place in the Orpheum Theater in Vancouver. This experience is a theatrical retelling of Maggie’s true story of having her promising career as a ballerina derailed by metastatic breast cancer at age 23. The piece is designed to benefit the nonprofit Stand Up To Cancer (SU2C). Ultimately, THE 100% is about triumph over adversity and perseverance in the face of dire obstacles, and aims to provide hope to those fighting cancer and their supporters.

Why VR for this project? 

Immersive documentaries are impactful due to a “transportation” effect, where the viewer feels like they are actually there. This effect has been likened to the difference between thinking about something you watched and remembering something you experienced – VR has the ability to create what feels like memories. This in turn leads to a far greater sense of empathy for the subject of an immersive documentary, something that we can attest to based on how many people that have been moved to tears after experiencing THE 100% in a VR headset. 

What is the biggest benefit of using Reality Capture technology to recreate this space? 

Some of the most realistic VR experiences have used photogrammetry and LiDAR as the basis of their environment creation. The sense of actually being in a real location is incredibly effective at helping a viewer feel like they are truly in a virtual reality, which in turn leads to much higher engagement with the overall experience.  

In the case of THE 100%, we added volumetric video captures to a LiDAR and HDRI captured environment for a whole new level of immersion. While this is a theatrical piece, we worked hard to make it feel like a real performance in a real space, to put the audience on stage in the middle of Maggie’s story. Without the ability to capture the real world in detail and quality we would have been left with an experience that fell flat and seemed more like a video game than a cinematic reality. 


Each yellow triangle represents a scan location in the Orpheum Theater captured with the BLK360

What's the part of this project that you are most proud of? 

The impact that this experience is already having on the world since its debut at Tribeca. In the entertainment business, we often strive to distract and amuse an audience, but in pieces like THE 100% we have a rare opportunity to create art that will have a measurable impact on those dealing with cancer by increasing funding for research into new ways to help them.  

We are all humbled to be working on this project, and we each have our own personal experiences with cancer, so we have our own individual reasons to want this piece to succeed. As a group, we all share a deep pride for what we have created, and a responsibility to carry it as far as possible to maximize the impact we can have. 

How do true-to-scale experiences help generate and establish empathy with an audience? And why is that important to storytelling and documentaries in particular? 

Since this is the first experience of its kind, many aspects of the production were theoretical or instinctual on our part. Yet we knew that the Microsoft volumetric capture technology was ready for prime-time production work, and we knew that we had the team, schedule, and budget to pull this off, but we were taking our best guess as to how the film would feel to viewers. We also knew that standing 1:1 in front of Maggie would be impactful. We’ve been pleasantly surprised at every stage of this process by how good it looks and how real everything feels.  

Some of our early test viewers did not know it was fully real-time rendered 3D, as they were used to seeing either pre-rendered 360º content or low-quality real-time content. Volumetric video is a whole new category of immersive technology, and it has delighted us with how personal and beautiful and engaging it can be.   

How was the Leica BLK360 used in the capture of the space?  

We used two BLK360s to scan every millimeter of the Orpheum Theater in Vancouver. We captured far more data than we needed, simply to have a massive data set to fine-tune our processing pipeline after the fact.  

We filled the BLK360s with over 60 high-quality scans, which provided an absolutely incredible amount of detail inside the auditorium and the lobby. We even discovered that we captured some of the street outside through the lobby windows!

 

Andy Cochrane and the Springbok Entertainment team scanning the Orpheum Theater with the BLK360

How was the process of using LiDAR? What would you do without LiDAR, and what are your perceived advantages of LiDAR? 

The BLK360 has revolutionized the way we capture environments. In the early days of VFX, we used survey equipment, blueprints, reference photography, and guesswork to create a 3D model matching a real location. Using photogrammetry, we were able to capture more details in less time, but the amount of effort it takes to capture thousands of well-exposed, sharp photographs in varying lighting has always been a challenge. The error inherent to photogrammetry has also been a bane of every 3D department, requiring lots of manual intervention.  

LiDAR was always considered to be a luxury that few could afford, so it wasn’t until the BLK360 came along that we could even consider building a reality capture pipeline around it. Now that we can capture billions of accurate 3D points on location without shutting everything down for hours at a time, LiDAR has become the cornerstone of our reality capture pipeline, with photogrammetry and HDRI contributing texture detail but not the underlying geometry. We’ve worked hard to develop a pipeline to bring these disparate technologies together into a single coherent flow that leverages the accuracy of LiDAR and the detail of photography, and THE 100% was just the first outing for this exciting process. 

What did you use in addition to the BLK360?  

We captured thousands of HDRI photos of the Orpheum Theater in concert with the BLK360 scans, and ended up using these high-quality panoramas to texture the 3D mesh that we got out of 3DReshaper. 

What do you hope that the viewers remember most? How do you hope this impacts them? 

Maggie’s positive message and incredible projection of hope is what has inspired us the most during this production. We want viewers to come away from this experience with a sense of empathy for Maggie, and urgency to help find a cure for so many affected by cancer. We hope more than anything that viewers come out of the headset resolved to do something, and to make that desire for action tangible to the benefit of SU2C.

Maggie Kudirka in film 100%