At VB Transform 2026, Ryan Phillips, head of enterprise product at Runway ML, explained how his team handled a persistent issue in their real‑time video model: AI‑generated avatars would drift away from the center of the frame. Instead of fixing the backend model immediately, Runway introduced a frontend feature that mitigated the problem and became a perceived benefit to users.
Runway and Runway Characters
Runway ML is an applied AI research company building general world models for generative tools. During his talk, Phillips showcased Runway Characters, a real‑time video model that enables zero‑latency, back‑and‑forth interactions with AI‑generated avatars. He noted that five years ago producing video often involved hundreds of hours of stitching individual frames together; today their models generate interactive video on the fly.
Demystifying evaluations (evals)
Phillips stressed that building a robust AI product starts with a high‑quality evaluation set, and creating that set is not purely an engineering exercise. It requires cross‑functional alignment across product, design, research, and sales to agree on what “quality” means. Runway runs internal workshops where team members review generated examples together to converge on shared definitions of success and failure down to small, picky details.
The evaluation set must include both common customer use cases and extreme edge cases. Runway, for example, used a non‑human character called “Tooth” (no nose and unusual teeth) to test behavior beyond standard human facial structures. The team grades subtle artifacts strictly; a video where the character’s face stayed intact but background elements like a net morphed was marked as a failure. To track tests they use a simple tool: Excel. Tests are logged daily and outputs are categorized as minor or major failures against a predetermined pass rate. “We set a bar before we get started on what percentage we need to pass, and when we hit that, we ship the model,” Phillips said.
For teams facing non‑deterministic quality drift in real‑time pipelines, manual evaluation at scale is a bottleneck. Phillips recommended leveraging large language models (LLMs) to automate visual grading. He said LLMs are becoming capable judges for many content issues, and can be fed behind‑the‑scenes context (like a sketch or an ad layout) to guide both generation and validation without increasing user cognitive load.
Model training, distillation and converting bugs into features
Delivering real‑time generative video requires an optimized technical stack. Runway begins with pre‑training a large foundation model, which is resource‑intensive and slow. To reach real‑time latency they use distillation: training a smaller, faster student model to mimic the large teacher model. Phillips estimated distillation cuts 80–90% of generation time.
After distillation they apply adversarial post‑training (APT) to force the distilled model to improve by exposing it to a system that seeks its weaknesses, regaining visual sharpness lost during distillation. However, these architecture changes introduced a persistent bug: characters would sway or drift from the frame center during real‑time generation.
The team spent weeks trying to fix the core model. They discovered that when the user’s input image was perfectly centered, the generated video remained stable. Rather than continue with a backend patch, Runway implemented a front‑end UX solution: a feature called “Optimize for Image Quality” that automatically re‑centers the user’s image before generation. Wrapping a backend limitation in a frontend tool turned what could be perceived as an engineering flaw into a user‑facing feature.
“Turn model limitations into product features so that you can actually expand how the model works,” Phillips advised. “It might feel like a limitation internally, but your customers will not see it that way if you’re kind of building this in as a product feature.”
Infrastructure details matter
Delivering video globally at 24 frames per second requires optimizing each layer of the infrastructure stack—from caching and parallel decoding to deep kernel changes in partnership with hardware vendors like Nvidia.
Shortly after launching Runway Characters, the team observed that 8% of API calls dropped to 16 fps, causing customer stutter. Root cause analysis required deep observability: an AI agent powered by Claude and monitoring tools such as Datadog and Sentry traced the issue to a single data center in the us‑east‑1 region. The fix wasn’t a config change; engineers physically replaced GPUs at that data center, which ultimately solved the problem.
For enterprise teams deploying real‑time applications, Phillips’ takeaway is clear: hardware and infrastructure anomalies directly affect model performance, requiring rigorous full‑stack debugging capabilities. “Don’t forget about all the small details, because there’s so many of them when you’re deploying these models,” he said.
Surviving “failure hell” and the future of world‑building
AI development is rarely linear. Phillips described a phase he calls “failure hell,” when teams can be stuck on a single problem for weeks. He argued that struggling through these periods is often necessary to reach breakthroughs; sustained iteration can lead to sudden, exponential improvements.
As underlying models overcome technical hurdles, the role of enterprise creatives is changing. Instead of designing single assets (an ad or an illustration), marketing and design teams are increasingly defining parameters, aesthetics and intellectual property—designing worlds that agents or real‑time models can later populate with generated content. “You might not be designing a single ad, but you might design a world that then the agent or a real‑time video model can generate ads from,” Phillips said.
Conclusion
Runway’s experience illustrates that real‑time generative video development requires disciplined evaluation practices, model engineering techniques like distillation and APT, deep infrastructure observability, and a product mindset that can convert model limitations into user‑facing features. A relatively simple frontend capability—automatic image re‑centering—proved sufficient to reframe a core model limitation as a practical product enhancement.
