Modal recently closed a $355 million Series C and presents itself as building an "agent cloud." The company previously raised a $17 million Series A; founder and leadership conversations have frequently referenced CEO Erik Bernhardsson. In a podcast episode, Modal CTO Akshat Bubna spoke with swyx and Vibhu about how AI agents change infrastructure requirements.
Why traditional assumptions fall short
Legacy cloud and infra assumptions rely on human operators who read docs, reason about YAML and inspect dashboards. That worked because humans could fill missing context mentally. Agents lack that luxury: they must run code, inspect outputs, debug and re-run quickly, which demands tighter iteration and richer contextual tooling.
Sandboxes and programmatic environments for agents
Modal built sandbox support early: according to the company, a sandbox API was introduced in May 2023. Sandboxes let agents spin up isolated environments programmatically, which supports both research and production workflows.
From better runtime to AI cloud
Modal started from the idea of building a better runtime: serverless containers, decorative configuration colocated with code, and improved developer experience by avoiding YAML. Modal added GPUs to its product a year before ChatGPT’s release, anticipating the growth of compute‑heavy, bursty workloads.
From developer experience to agent experience
Modal shifted its SDK team to focus on agent experience (AX) rather than only developer experience (DX). The rationale is that co‑locating infra requirements with code (via decorators) speeds up agent iteration: changes can be provisioned and observed live. Bubna also stressed that although fewer people may inspect code directly, observability remains critical—dashboards, logs and CLI tools are needed for humans to interpret and decide.
Modal’s AI primitives today
According to Bubna, Modal has built primitives tailored to AI apps and supports inference, training, batch processing and sandbox workloads. Key platform elements include:
- serverless functions and decorator‑based configuration
- elastic inference for custom models (audio, video, robotics, computational biology)
- GPU snapshotting to reduce cold starts
- DeFlash: a block‑based speculative decoding implementation shared as open source
- Auto Endpoints: pre‑optimized endpoints that include Modal optimizations such as DeFlash
- sandboxes with persistent storage, networked containers and a private IPv6 overlay
- RDMA and serverless multi‑node training
- a capacity pool spanning 17 cloud providers
Technical details: burstiness, snapshotting and DeFlash
Modal emphasizes elasticity and autoscaling: customer workloads show highly variable inference traffic (diurnal patterns, regionally offset spikes, product launches). GPU snapshotting—capturing a GPU’s state—speeds up cold starts. DeFlash is a speculative decoding approach where a smaller draft model predicts blocks of tokens and the larger model verifies them; by increasing the accepted token length, this technique can multiply throughput without degrading quality.
Bursty RL rollouts and 100,000 sandboxes
Bubna noted that some applications, especially reinforcement learning rollouts, are extremely bursty. In such cases, rollouts can require very large numbers of isolated environments; he described scenarios where as many as 100,000 sandboxes are needed.
Networking, RDMA and distributed training
Modal supports networked sandboxes, sidecars, egress controls and an IPv6 private overlay (I6PN) so containers in the same workspace can address each other privately. For multi‑node training Modal uses RDMA to bypass TCP and accelerate transfers between nodes; this is important for distributed workloads and high bandwidth requirements.
Auto Endpoints and frontier inference performance
Auto Endpoints enable users to create endpoints (via UI or CLI) that include Modal’s inference optimizations like DeFlash. Modal open‑sources many improvements and works with projects such as SGLang; the company also provides a team that helps customers achieve frontier‑level performance. Customers receive the underlying code and can eject to a full Modal experience if they need deeper control.
Capacity strategy and the 17‑cloud supercloud
Modal does not run its own data centers. Instead, it aggregates capacity across 17 providers and runs a reliability layer on top to mask provider variability. This allows regional placement, better cost management and new product levers such as a lower‑cost batch tier for latency‑insensitive workloads.
Observability, guardrails and production agents
Bubna emphasized that production agents require hard guardrails (networking, storage, snapshot/restore controls) because relying solely on LLM‑mediated permissioning is risky at the sandbox level. Hard boundaries should be paired with softer, mediated controls where appropriate.
Customers, verticals and what’s next
Modal works with frontier startups and companies across multiple verticals (examples mentioned include Suno, Runway, Cognition and Ramp Inspect). Beyond LLM inference, Modal sees demand in audio‑video real‑time workloads, robotics and computational biology. The company continues to evolve primitives—regional routing, persistent file systems and additional agent stack features—that customers need for production agents.
SDKs, Modal Bench and auto research
Modal offers SDKs in Python, TypeScript and Go; the runtime is written in Rust but is language‑agnostic. The company runs Modal Bench to identify AX gaps and improve tooling so agents can better use observability and other platform features. Modal also prototypes "auto research" approaches: automated sweeps and resource tuning to accelerate inference and training optimizations.
Summary
Modal’s engineering and product direction aim to serve agent‑driven, bursty and specialized AI workloads by building infra primitives—sandboxes, elastic inference, GPU snapshotting, DeFlash speculative decoding, Auto Endpoints and a 17‑cloud capacity pool—that address production inference, training and agent lifecycles. Bubna highlighted the importance of observability, capacity planning, hard guardrails for production agents and close collaboration with customers at the frontier.
Key names and references
- Modal
- Akshat Bubna, CTO of Modal
- Erik Bernhardsson, CEO of Modal
- Customer examples referenced in the conversation: Suno, Runway, Cognition, Ramp Inspect
(The article summarizes claims and technical points made by Modal leadership in a public discussion and company announcements.)
