DEV Building an Effective nVP (Neural Virtual Production) Sound Stage
Scrivener Export - Reformatting Needed!
This article is an export of a Scrivener document. It will definitely need at least some reformatting to work in Obsidian and MkDocs. Delete this note once the article's formatting has been fixed to some extent.
Kartaverse Workflows | Building an Effective nVP (Neural Virtual Production) Sound Stage
Kartaverse | nVP On-set Solutions
First Draft
Created: 2022-12-07 Last Updated 2022-12-10 07:30 AM UTC -4
Overview¶
Overview
This guide is a companion resource for the OpenDisplayXR VDD (Virtual Device Driver) and Kartaverse "KartaVP" open-source immersive content production pipeline projects. This draft document explains how to approach driving large-scale nVP (Neural Virtual Production) workflows in an efficient and performant fashion.
Included with the guide is a high-level discussion of the back-end hardware systems integration considerations a new nVP sound stage operator needs to consider when switching away from traditional "machinima" style OpenGL/DirectX based LED panel rendering workflows.
OpenGL Rasterization Rendering¶
OpenGL Rasterization Rendering
OpenGL real-time rendering is still quite common, as of December 2022. OpenGL started out as IrisGL on SGI IRIX systems, and the descendant of that API is pretty much the number one option used in (legacy) rectilinear lens based rasterizer VP pipeline architectures.
The fact that OpenGL rasterization techniques have endured for decades is impressive. Many of the essential OpenGL API developer documentation resources pre-date the 1997 release of the Nintendo64 GoldenEye 007 game. The N64 hardware performed photo-realistic rendering via OpenGL running on SGI (Silicon Graphics) derived graphics chips. It's a stunning technical and engineering accomplishment to keep the OpenGL API going in such a consistent fashion for so long, really! 👍
Neural VP Rendering¶
Neural VP Rendering
KartaVP and the OpenDisplayXR VDD provided "NeuralFoam Engine" renderer are developed as cross-platform compatible, free open-source tools. This is an intentional choice to help move forward the objective of modernizing existing VP workflows with ML driven neural rendering techniques.
With NeRF approaches, content creators can embrace recent advances in the state of the art technology used in volumetric video capture/playback.
In 2023 the emergence of off-the-shelf nVP stage solutions will hopefully provide feature film-level DOPs (Director of Photography) with the chance to unlock exciting new potential for 100% real-time rendered digital environments. NeRF has already demonstrated its potential to deliver levels of photorealism (and rendering features) never witnessed in conventionalrasterized-based on-set virtual production systems deployed between 2015-2022.
Distributed NeRF Rendering for Live-Action Virtual Production Background Environments¶
Distributed NeRF Rendering for Live-Action Virtual Production Background Environments
If you review the OpenXR-VR Virtual Device Driver Google Docs file, you will see a "Demo Apps" section that has notes that cover a novel "distributed NeRF rendering" pipeline, and how that can be achieved in a modular fashion, (with relatively low design complexity), to power near-infinite LED video wall size, for an nVP stage setup that supports an arbitrary geometric screen surface shape.
This approach empowers the construction of highly-scalable virtual production workflows that use NeRF techniques on large-scale LED display panel based cylindrical VP sound stages.
The core aspect is the use of a conventional 10 Gb Ethernet networked client/server method. This drives a genlock synced, GPU-powered, rack-mounted NeRF-based distributed rendering appliance.
The key aspect is that the individual NeRF "worker nodes'' are used as multi-view tile generators. Of special importance in the system design is to avoid the need of OpenGL quad buffer rendering support in the worker node's GPUs.
One worker node is configured to direct-drive high-resolution 24Hz - 120 Hz output. The generated frame-buffer data is routed via an NewTek NDI IP video stream or a fibre-encoded SDI cable and either an ATEM switcher or SmartVideo Hub connection to either individual LED display panel's LED video processor unit.
AJA NDI to SDI Encoder BMD Smart Video Hub SDI Video Routing
Brompton Tessera S8 LED Processor 
Roe Visual LED panel
I feel this NeRF distributed rendering workflow best mirrors the concepts that Disguise, Unreal + nDisplay, Assimilate LiveFX, Notch, and TouchDesigner TouchEngine all support for OpenGL/DirectX rendered real-time "Machinima" content today.
All NVIDIA InstantNGP API based NeRF rendering implementations support a cropped 3D DoD (Domain of Definition) bounding box.To fully optimize the render pipeline, and efficiently extract all usable VRAM on the NVLink bridge connected GPUs pairs; one needs to bind the active GPU worker node's "nCam tracked" camera view frustum to the precise DoD region cropping control in NVIDIA's InstantNGP testbed program.
This DoD usage allows a VP production setup, that features an arbitrarily shaped LED panel based video wall, to handle voxel scene scales an order of magnitude larger than most people could ever conceivably imagine.