NVIDIA Omniverse NuRec is a neural reconstruction pipeline that builds high-fidelity 3D representations of real environments from multisensor inputs such as cameras and lidar. It is used to convert dynamic scenes captured by autonomous vehicle (AV) and robotics platforms into simulation-ready digital environments that can be rendered, replayed, and analyzed inside NVIDIA Omniverse and related simulation workflows.
These reconstructions are important for physical AI and autonomous systems development: engineers can capture a real driving or robotics scenario, reconstruct the environment, and then inspect or replay the scene to better understand model behavior, validate perception results, generate synthetic viewpoints, or create training data.
NuRec combines neural rendering techniques (for example Gaussian splatting) with GPU-accelerated rendering and simulation pipelines to produce realistic scene reconstructions. That realism comes at a high computational cost: reconstruction and rendering workflows involve large sensor datasets, complex PyTorch-based training loops, and specialized CUDA kernels that heavily load GPU resources.
Goals and starting point
Performance is critical for NuRec because reconstruction turnaround time directly affects engineering productivity. A common use case is reconstructing an interesting or problematic AV run quickly for debugging; if reconstruction takes hours, iteration and debugging slow down considerably. Initially, even relatively short captures could require hours to reconstruct depending on scene and configuration. The team’s long-term goal is ambitious: real-time reconstruction where a 30-second capture is reconstructed in roughly 30 seconds.
Performance matters beyond reconstruction as well: rendering-only workflows for reinforcement learning, synthetic data generation, and large-scale simulation can produce massive numbers of frames, so modest speedups can translate into significant GPU-time and infrastructure cost savings.
Profiling with Nsight Systems — where are the bottlenecks?
To address these challenges, NVIDIA used profiling and optimization tools, primarily NVIDIA Nsight Systems and NVIDIA Nsight Compute. The first step was to run an Nsight Systems profile to establish a baseline.
Using NVTX annotations available in PyTorch, the team zoomed into a single forward pass iteration. The initial assumption was that the rendering kernel would dominate runtime; instead, the CUDA hardware timeline showed long periods of low GPU utilization and many small kernels rather than a continuous heavy kernel.
Additional NVTX annotations revealed that the collect_gaussian_parameters function consumed a large portion of time before rendering began and was called multiple times per forward pass. Digging deeper identified the interpolate function as taking the plurality of time (~4.148 ms), invoking many tiny kernels and memory operations that throttled the GPU.
Fusing small kernels and removing synchronizations
Inspecting the code under interpolate, the team fused the many small kernels into a single larger kernel. This reduced the interpolate function runtime from 4.184 ms to 83.81 µs — nearly a 50× speedup.
Next, long cudaStreamSynchronize API calls were identified that prevented the CPU from enqueueing kernels while the GPU was active. Removing one synchronization often revealed the next bottleneck, so the team iteratively removed unnecessary synchronization points until the CPU could efficiently enqueue work while the GPU was busy. With these synchronizations reduced, the formerly tiny kernels were no longer limited by CPU launch time, and GPU utilization became more compact and continuous.
Identifying hot kernels and tuning renderBackward with Nsight Compute
Nsight Systems highlighted hot kernels, and Nsight Compute was used to profile individual kernels in depth. The renderBackward kernel, used for both camera and lidar processing, showed only ~15% occupancy and behaved differently depending on the input type.
Profiling multiple instances revealed that the three longest renderBackward kernels were from lidar data and three from camera data; all were statically allocating 167 registers per thread. Comparing a top lidar kernel to a camera kernel showed that although both performed most accesses in shared memory, the camera kernels issued roughly 75% fewer shared memory requests even though both statically allocated the same shared memory per block.
Given these behavioral differences, the team split the kernel into separate lidar and camera versions. For each, they experimented with register allocations using the launch_bounds qualifier and adjusted per-block shared memory. They also used the cudaFuncSetCacheConfig runtime API to prefer larger shared memory and smaller L1 cache.
After tuning, register requirements dropped from 167 to 64 for the lidar kernel and to 128 for the camera kernel, and both ran efficiently with about half of the formerly allocated shared memory. Occupancy improved from ~15% to roughly 30–50%, and overall runtime improved: the longest lidar kernel decreased from 31 ms to 18 ms.
Remaining work: long-tail and load imbalance
The authors note there is still room for improvement. One current issue is a long-tail effect in the kernel caused by workload imbalance: Nsight Compute PM Sampling shows the first half of the kernel averaging 32 active warps that taper off over time, leaving fewer than one active warp per cycle in the final milliseconds. Ideally all warps would remain active through the kernel.
Getting started with Nsight developer tools
Performance analysis and optimization is iterative: run a profile, identify problems, fix them, and repeat. Tools like Nsight Systems and Nsight Compute make this cycle easier when developing on NVIDIA GPUs. Both tools are free to download and try; the authors encourage users to experiment with their own workloads and to share questions or findings on the NVIDIA Developer Forums.
Acknowledgments
Special thanks to NVIDIA contributors Francois Trudel, Joey Lai, and Rodolfo Lima.
