NVIDIA outlines a workflow that combines agentic AI with the cuDF plugin for Apache Spark to convert user-defined functions (UDFs) into GPU-accelerated implementations in minutes. Using the provided "cuDF skills for Apache Spark," teams converted more than 30 production UDFs; one benchmarked example reduced operator time from 274.55 hours to 12.78 hours (over a 22x speedup) on a 600-billion-row workload. The modified UDF affected more than 3,000+ jobs.
Why UDFs are problematic in Spark
Apache Spark’s declarative SQL/DataFrame APIs make it possible for the cuDF plugin to map expressions to GPU implementations because the planner can see what the query should produce. UDFs, however, embed arbitrary user code that the system cannot generally reason about or parallelize. As a result, execution often falls back to the CPU and requires converting data from columnar (GPU-friendly) to row-wise format and back. Consequences include:
- overhead from data format conversion and CPU fallback;
- lost opportunity to exploit GPU compute speedups;
- additional memory and transfer costs.
These factors explain why UDFs can create significant slowdowns in otherwise GPU-accelerated pipelines.
Approaches to accelerate UDFs
NVIDIA describes three rewrite approaches, ordered by increasing complexity and control:
- SQL: low complexity, highest-level abstraction; best for simple relational transformations.
- cuDF: medium complexity, greater control using columnar GPU APIs (for example, implementing RapidsUDF and using ColumnVector operations); suited for more complex logic such as string parsing or nested types.
- CUDA: high complexity, kernel-level control via custom CUDA kernels (invoked through JNI) for the most complex or performance-critical logic.
A simple Scala example that converts Celsius to Fahrenheit can be replaced by a plain SQL expression so the planner and cuDF plugin handle it on-GPU, reimplemented as a RapidsUDF that implements evaluateColumnar and composes ColumnVector operations, or—when necessary—implemented with a custom CUDA kernel.
AI agents and the cuDF skills for Apache Spark
Although GPU rewrites yield large speedups, they can be labor-intensive: engineers must choose APIs, validate semantics, and measure performance. NVIDIA proposes that much of this work can be handled by coding agents, provided there are proper guardrails. The cuDF skills for Apache Spark decompose the rewrite into four focused feedback loops: scaffolding and design, conversion, benchmarking, and optimization. Each skill specifies the work and the verification evidence required before proceeding.
The skills package supplies a complete project template with cuDF- and Spark-specific scaffolding, test and benchmark harnesses, automated output verification routines, and build configurations. It also encodes GPU programming best practices (memory management, configuration) so agents can try different implementations and select the best based on correctness tests and performance measurements.
Performance in practice
A large consumer technology company used this workflow to convert 30+ production UDFs for their data pipelines. One highlighted benchmark shows operator time reduced from 274.55 hours to 12.78 hours—more than a 22x speedup—on a 600-billion-row job. The UDF in question impacted over 3,000+ jobs. NVIDIA notes that agent-assisted rewrites can make iteration cycles measured in hours rather than days or weeks while preserving the original UDF behavior.
How to get started
NVIDIA provides the cuDF skills for Apache Spark on GitHub and suggests installing via the npx skill installer:
npx skills add NVIDIA/cudf-spark --skill ‘*’
They also point to additional materials, including the Data + AI Summit session "Coding with AI Agents to Supercharge Your Spark UDFs with GPU Acceleration," the cuDF plugin for Apache Spark getting-started guide, and the cuDF documentation.
Summary
Rewriting Spark UDFs to use GPU-accelerated primitives can yield substantial speedups and TCO improvements. The cuDF plugin plus a structured, agent-driven workflow (SQL, cuDF, CUDA approaches) provides scaffolding, testing, and benchmarking to validate correctness and measure performance gains without manual infrastructure rebuilding.
