Using cuDLA Python bindings for DLA inference on NVIDIA Jetson AGX Orin

NVIDIA’s cuDLA Python bindings (cuda.bindings.cudla) allow running Deep Learning Accelerator (DLA) inference on Jetson AGX Orin directly from Python, without writing C++ to manage the runtime. Offloading supported neural network operations to DLA can improve power efficiency and free GPU capacity for other tasks. The bindings appeared in cuda-bindings 13.3.0 and expose the cuDLA runtime (libcudla.so is provided by JetPack on Orin).

What this provides and why it matters

cuDLA is a fixed-function accelerator that efficiently executes commonly used neural network operators (convolutions, pooling, etc.). It supports fewer layer types than a GPU but covers many popular architectures. The cuda.bindings.cudla module is a thin Cython-generated Python layer over the cuDLA C++ API and supports hybrid-mode workflows where CUDA streams coordinate DLA execution with GPU work. This enables GPU preprocessing, DLA inference, and GPU postprocessing to be sequenced on the same CUDA stream.

Prerequisites and installation

• Install the Python bindings: pip install "cuda-bindings>=13.3.0".

• Minimum supported Python version is 3.10; the example uses Python 3.12 installation via the deadsnakes PPA if needed:

  sudo apt update && sudo apt install -y software-properties-common sudo add-apt-repository -y ppa:deadsnakes/ppa sudo apt update && sudo apt install -y python3.12 python3.12-venv python3.12-dev python3.12 -m venv ~/dla-env ~/dla-env/bin/pip install "cuda-bindings>=13.3.0"

• To run a loadable on-device you only need the JetPack-provided libcudla.so; TensorRT is required only to compile a trained model into a DLA loadable.

Compiling a DLA loadable (example)

A DLA loadable is a binary produced by TensorRT’s trtexec that encodes a trained network for DLA hardware. The example targets TensorRT 11.4.1 with CUDA 13.4 Update 1 on Jetson AGX Orin running JetPack 7.2.2 or later. For an FP16 CHW16 standalone loadable from an FP16 ONNX model with fixed input dimensions:

trtexec
--onnx=your_model_fp16.onnx
--useDLACore=0
--inputIOFormats=chw16
--outputIOFormats=chw16
--memPoolSize=dlaSRAM:1
--saveEngine=model.bin
--buildDLAStandalone

Key flags:

• --useDLACore=0: target the first DLA engine on Orin.
• --buildDLAStandalone: produce a raw DLA loadable that does not require TensorRT at inference time.

Note: TensorRT 11 derives precision from model tensor types (the --fp16 and --int8 flags are no longer supported). Not every network layer is DLA-compatible; preprocessing the ONNX graph to trim incompatible head/tail layers is often necessary. The Python runtime works with any .bin or .nvdla file.

Python runtime workflow (overview)

Required imports:

import numpy as np from cuda.bindings import cudla from cuda.bindings import driver, runtime

The driver and runtime modules are the existing cuda-python bindings used for stream and memory management; cudla is the new module exposing the cuDLA runtime.

The high-level steps:

1. Initialize CUDA and create a DLA device

   • Call driver.cuInit(0) and runtime.cudaSetDevice(0).
   • Create the device in hybrid mode: cudla.create_device(0, int(cudla.Mode.CUDA_DLA)). Hybrid mode enqueues DLA tasks through a CUDA stream so GPU and DLA work can be ordered on the same stream. Standalone mode is not supported by this release via these bindings.

2. Load the loadable into memory

   • Read the .bin file as bytes and call cudla.module_load_from_memory; no filesystem handle is required at the API level.

3. Introspect the loaded module

   • Query attributes with cudla.module_get_attributes to get number of input/output tensors and their descriptors (cudla.ModuleTensorDescriptor). Each descriptor provides buffer size (size_), shape (n,c,h,w), name, data format, and stride. Allocation sizes are driven from desc.size_.

4. Allocate GPU memory

   • Use runtime.cudaMalloc for each input/output descriptor size. The sample code checks return status in helper wrappers (omitted in short snippets) — proper error checking is necessary in production.

5. Register GPU buffers with cuDLA

   • Registering GPU buffers via cudla.mem_register returns a DLA-side handle required for tasks. The CUDA pointer is passed as an int() cast of the CUDA allocation object.

6. Prepare input data and copy to GPU

   • Preprocessing is model-specific: normalize values and pack data according to the descriptor’s stride, data_format and data_type. Copy input arrays to the GPU asynchronously with runtime.cudaMemcpyAsync and clear output buffers with runtime.cudaMemsetAsync on the same stream.

7. Build the task and submit

   • Create a cudla.Task, set module_handle, input_tensor and output_tensor fields (using the registered handles), then call cudla.submit_task to enqueue the DLA inference on the CUDA stream. The task.count fields are automatically kept in sync when assigning Python lists to the task fields.

8. Synchronize and read back

   • Synchronize the stream with runtime.cudaStreamSynchronize(stream). Allocate a numpy array sized to output_descs[0].size_ and copy back with runtime.cudaMemcpyAsync from device to host, then synchronize again before reading the data.

9. Cleanup

   • Call cudla.mem_unregister for registered buffers, cudla.module_unload, cudla.destroy_device, runtime.cudaFree for GPU allocations, and driver.cuStreamDestroy for the stream.

Error handling

cuDLA API calls raise cudla.CudlaError on failure, with both numeric status and readable name. Use try/finally for cleanup and branch on e.status compared to cudla.Status enum values to handle specific error modes (from ErrorInvalidParam up to hardware-level errors like ErrorDlaErrEngineTimeout).

Interpreting output

After cudaStreamSynchronize, the output numpy array (output_cpu) contains the raw DLA output bytes. Interpret and reshape these bytes using the output descriptor’s data_type, data_format, stride and shape fields and feed them into any downstream Python postprocessing (NumPy, PyTorch, OpenCV, etc.). The runtime hands you the bytes; model-specific postprocessing remains the same as for any inference pipeline.

Concrete example output (illustrative):

• cuDLA device created: 0x55a1b2c30000
• Module loaded: 0x55a5b6c70000
• numInputTensors = 1
• numOutputTensors = 1
• Input 0: size=..., shape=(1,64,56,56), name=resnetv17_pool0_fwd
• Output 0: size=..., shape=(1,2048,7,7), name=resnetv17_stage4_activation2
• SUBMIT IS DONE (task queued; completion pending)
• Output tensor written to host

These messages show the exact buffer sizes and tensor names the loadable expects, which your preprocessing and postprocessing code can read directly from the descriptors rather than hardcoding.

Cross-platform note

The same cuda.bindings.cudla API is available on NVIDIA RTX Spark (bringing DLA inference to Windows developer laptops), so the API surface and Python workflow in this article apply there as well.

Getting started

• Install: pip install "cuda-bindings>=13.3.0".
• Build a loadable with trtexec (TensorRT).
• Follow the nine-step Python lifecycle from create_device to cudaStreamSynchronize to get DLA inference results in Python on Jetson AGX Orin.

Unit tests for the cudla enums, POD types and API surface are included in the NVIDIA/cuda-python GitHub repository under cuda_bindings/tests/cudla/ and run without DLA hardware (libcudla.so not required): pytest cuda_bindings/tests/cudla/ -v

For full end-to-end examples, sample inputs, and model preparation scripts, consult the Deep Learning Accelerator samples and the NVIDIA/cuda-python GitHub repository.