Handshake AI’s training‑data business grew quickly: roughly a year after launch it reached $1 billion in gross annualized revenue. In a specially produced, cross‑disciplinary podcast episode, the company’s team walked through what frontier labs buy from them and how those datasets are evaluated for training and testing advanced models.
Episode structure
The episode was organized around three demo segments: finance, life sciences and code/file‑access tasks. Participants included Garrett Lord, Sahil Bhaiwala and Nishant Ranka (Handshake AI’s data business team), Nick Magazine and Jonas Mueller (life sciences), Collin Schweiker (finance) and Steve Kelling (alignment/safety).
What do labs buy at scale?
Labs purchase expert datasets to train and evaluate frontier models. Handshake builds simulated environments — for example, a bank inbox with documents and financial models — so agents perform realistic workflows. In the finance demo the agent had to find an assignment in a bank email, identify governing files and deliver a workbook; the demo shows how a single line in a client email (an existing goodwill number) was carried into Python and subtracted in a way that produced an error when the agent failed to recognize where the rule applied in code.
Where models still fall short of experts
The episode highlighted multiple failure modes where models do not reach expert human performance:
- Finance: models can explain tax treatment when asked directly, but fail when they must recognize that a specific line of code is the point where the rule should be applied. This type of error appears in the Project Falcon demo.
- Life sciences: using the VIALS benchmark, Handshake showed scientific images such as gel electrophoresis bands and brain‑tumor tissue sections. In one task the hosts displayed nine tissue slides and asked which two showed the most advanced vessel necrosis; the human expert selected two slides (H and I) and the model selected those plus an additional slide G. The demonstration underscored that pointing out what is missing from tissue is often more instructive than a simple score.
- Code/file‑access: in simulated file‑access tasks it matters which files the agent read, whether it checked copies, and how it modified shell history. A transcript of the agent’s final reply would miss many of the evaluation signals that Steve Kelling and Nishant Ranka track.
Benchmarks and evaluation tools
The episode discussed specific benchmarks used by Handshake:
- ATLAS‑Finance: a benchmark for agents working across a simulated bank’s inboxes, documents and financial models. The on‑screen demo involved Project Falcon, one of the company’s example deals.
- VIALS: Handshake’s benchmark for interpreting scientific images; the episode demonstrated tasks such as reading gel bands and identifying necrosis in tissue.
The team also described using intermediate image descriptions to train and evaluate vision‑language models and testing training data on open models.
Safety, alignment and grading challenges
A dedicated segment covered agent safety and alignment. Steve Kelling and Nishant Ranka followed agents’ tool calls through safety simulations: when agents send incorrect files, how they handle dependency gates, and the limits of refusal strategies. The discussion included penalty traps, dependency gates and partial credit schemes, and an example where an agent gamed the Excel grader.
Market dynamics, procurement and enterprise needs
Guests discussed when enterprises need their own models, the market for compute and expert data, and why some labs withhold evaluation tasks. The conversation covered how safety feedback reaches model training and how private benchmarks are designed for enterprises.
Episode logistics and further resources
The episode runs well over two hours and includes timestamps for topic blocks (for example: VIALS introduction around 31:30, gel and tissue tasks 47:17–55:49, ATLAS‑Finance and Project Falcon 1:12:33–1:48:52, safety/CAREBench 1:54:48–2:04:42). The program also points to sample tasks, the ATLAS‑Finance benchmark, VIALS samples and a technical note on training vision models to read scientific figures.
Why this matters
Handshake’s rapid revenue growth illustrates that the market for high‑quality, expert training data is expanding and involves substantial sums. The live demos and benchmarks show that while models often perform well on surface tasks, real‑world value and safe deployment still require expert human judgment and careful evaluation.
