A new kind of story required rethinking the camera

Depthkit began with a practical problem: how do you film a person who will continue to occupy space?

The moving three-dimensional image is now familiar as “volumetric video.” When we began, we were still learning how to light, frame, direct, and edit it. The tools available to us came from research labs, industrial vision systems, and expensive capture facilities. Our production language developed on the projects themselves.

2011–2013: RGBDToolkit

RGBDToolkit was the first system we shared publicly. James George and I developed it through experiments combining a Microsoft Kinect depth sensor with a conventional video camera. The Kinect recorded scene geometry while a conventional camera contributed a richer color image. Calibration aligned the two views, letting us choose a camera position after recording.

Some of that development took place at Eyebeam during a residency formally held by James. Because the residency centered on our shared project, Eyebeam added me as an honorary resident.

The images were visibly computational, with broken edges and surfaces opening into point clouds. Their failures showed what each camera could see. We kept the artifacts and learned their cinematic possibilities.

RGBDToolkit was released as an open system. Its public life grew through workshops and demonstrations, where other filmmakers began testing three-dimensional cinematography with us.

2014 onward: rebuilding the tool as Depthkit

When virtual reality created a practical need for people inside spatial scenes, James George and I rebuilt the RGB-D workflow as Depthkit. We made the fragile experimental pipeline into a production tool that supported broken pointillist images and increasingly naturalistic performance.

Depthkit became part of Scatter, the artist-led immersive studio I co-founded with Yasmin Elayat and James George. We used our productions to set Depthkit's technical brief.

On set, a face might drift out of calibration under difficult lighting; longer performances strained storage and workflow. Installations forced us to rethink playback and the audience's path through the room. We changed the software while making the work.

I am interested in building the tools that not-yet-possible documentary stories require. I begin with the experience I want to create and work backward toward the system that could make it possible.

Productions as research

Giant used a compact capture system to hold an actor's presence inside a virtual scene. Blackout, which I directed, placed volumetric recordings of New York subway riders inside a generative documentary. Each visitor encountered a different group of people and inner monologues.

For Zero Days VR, Depthkit placed audiences inside the human and geopolitical story of the Stuxnet cyberattack. The project won the 2018 News & Documentary Emmy Award for Outstanding New Approaches: Documentary.

Captured bodies ranged from near-photographic figures to visible fragments. The viewer shared space with each recorded person. We treated that proximity as an ethical and cinematic choice.

From access to community

The camera needed people around it. New creators needed documentation and help carrying a first test into a finished work. Scatter and Depthkit worked directly with them through workshops, production support, and the Powered by Depthkit Creator Program.

Three of the five works in Tribeca's 2018 Storyscapes competition used Depthkit: Terminal 3 by Asad J. Malik, Queerskins: A Love Story by Illya Szilak and Cyril Tsiboulski, and Vestige by Aaron Bradbury. These were the Creator Program's three flagship projects. The program supplied a software license, creative and technical guidance, capture planning, production and postproduction help, and the shared work of translating very different artistic ideas into a still-emerging medium.

As I told Forbes at the time: “We created this system that allowed [us] to collaborate on capturing people basically using 3D sensing… If anyone else is interested in this crazy new art form, please come with us and experiment alongside us.”

People outside Scatter carried Depthkit into documentary, music, games, and performance. Their projects moved volumetric filmmaking beyond specialized facilities.

A continuing lineage

Depthkit was recognized as a Webby Awards Technical Achievement Honoree in 2019 and acquired by Evercoast in 2024.

I still build tools to test what an image can know and whose experience it can hold. The virtual-cinematography system built for Truth or Consequences and newer custom systems for Strata and Topography continue that method.