Alibaba released Qwen 3.8‑Max this week and positioned its preview as second only to Claude Fable 5. Yet an independent harness produced nearly the opposite outcome: a VulcanBench run (apparently on the Preview) placed Qwen 3.8‑Max’s highest‑effort setting mid‑pack and its default setting last.
Both sets of results are valid. The primary driver of the discrepancy is token and wall‑clock time budgets — parameters that rarely feature in headlines but materially change who looks best.
How much do time and token budgets matter?
Alibaba’s published footnotes show generous limits: a five‑hour timeout for coding tasks and up to 12 hours per run on PaperBench. VulcanBench’s open harness, by contrast, allowed 45–60 minutes of wall‑clock time. An Alibaba time budget that is 5–16× larger explains much of the gap between the two outcomes.
Two practical changes should follow when choosing models:
- Use cost per successful task: count total spend (including failed attempts) and divide by the number of tasks that actually passed your acceptance criteria.
- Make time and token budgets explicit acceptance criteria rather than hidden configuration details.
Price per token no longer predicts the bill
Early comparisons for Qwen 3.8‑Max focused on per‑token price because that was the only public data. But reasoning models spend tokens on internal thinking, not only on final output. A model that uses most of its token allowance for reasoning can hit a token cap before emitting an answer, producing an empty or timeout result that costs a full run.
Artificial Analysis provides a clear illustration: running its Intelligence Index on DeepSeek‑V4‑Flash at maximum effort generated 210 million output tokens versus a class median of 100 million. Absolute monetary cost stayed low because tokens were cheap, but verbosity also costs time — and in many deployments time is a hard constraint.
You need a metric that counts everything you spent, including attempts that returned nothing, against the tasks that actually completed within the time and token budget you specified. That is what cost‑per‑success reveals.
Failure mode is partly a configuration choice
A run that produces a wrong answer and a run that exhausts its budget are different events with different remedies. Almost no harness currently separates these failure modes, and almost no leaderboard reports the split. The author encountered this building their own agent benchmark: the harness logged a failure but not its cause, so the distinction had to be added manually. When you separate timeouts from wrong answers, budget exhaustion tends to dominate.
Long‑Horizon‑Terminal‑Bench (published in July) ran 17 frontier models across 46 tasks with one 90‑minute attempt each. Timeouts accounted for 79% of unresolved runs, agents stopping on their own were 19%, and harness errors 3%. The timed‑out runs were not generally near completion (mean reward between 0.10 and 0.35), so more time would not necessarily have yielded success — but the benchmark was effectively measuring time efficiency whether that was stated or not.
VulcanBench and Claude Opus 5: extra reasoning can backfire
VulcanBench’s July 26 report shows Claude Opus 5 performing best at its lowest‑effort setting: 20 of 23 tasks solved at low effort versus 18 at high effort. High effort did reduce wrong answers (1 wrong vs. 3 wrong) but ran out of clock and scored zero for timeouts. Two of the three regressions were cutoffs on tasks the low‑effort setting solved; with unlimited time the high‑effort setting would only tie the cheap one, and at 3.1× the cost.
This has direct implications for routing ladders that escalate to stronger, more expensive reasoning after cheap attempts fail. For many model‑task combinations that escalation simply pays more and results in timeout or cap rather than a better answer.
Who is already measuring cost per success?
Several groups have converged on cost per successful task in recent months:
- VulcanBench has reported dollars per solved task as a headline column since its early reports.
- Long‑Horizon‑Terminal‑Bench publishes per‑task cost alongside accuracy; its instructive row shows GPT‑5.4 at roughly $26 per task with a lower pass rate than Grok 4.5 at about $11.
- TestEvo‑Bench runs agents under a cost cap: Claude Code’s test‑generation score falls from 71% to 44% under a tighter cap.
Commercial providers are already adopting per‑outcome billing or reporting:
- HubSpot moved its Breeze Customer Agent in April to $0.50 per resolved conversation, down from $1.00 per handled conversation.
- Zendesk bills per automated resolution.
- Fin charges $0.99 per outcome and bills only on end‑to‑end resolution.
Practical steps to take this week
- Require an explicit failure‑reason field for every agent run with distinct values for budget exhaustion, verifier failure and harness error. Until timeouts are separated from wrong answers, pass rates conflate two different problems.
- Compute cost per successful task per effort level, not just per model: total spend including failed attempts divided by tasks that passed your acceptance check. Rankings will likely differ from the vendor rate card, and the cheapest effort setting may win.
- Prefer token caps over wall‑clock caps unless latency is a genuine service‑level objective. A wall‑clock cap conflates provider serving speed with model quality.
- Check default effort settings on deployed models. Qwen 3.8‑Max runs at its highest reasoning setting when the effort field is unset, and that highest setting was the worst performer in independent testing. Teams that never tune this parameter may be running the configuration that costs the most per solved task.
Conclusion
The divergent benchmark outcomes around Qwen 3.8‑Max stem from differing time and token budgets, not from invalid results. To make model comparisons meaningful for production use, benchmarks and vendors should report per‑effort cost‑per‑success and explicit failure reasons, and teams should make time and token budgets first‑class acceptance criteria.
