Collinear AI has released CWE-bench v2, a public benchmark consisting of 100 tasks intended to test advanced cyber capabilities. Each task gives an agent a checkout of a real open-source repository and a short instruction: find and fix a security weakness while preserving existing functionality. Scoring is fully programmatic and deterministic so results can be reproduced without additional cost.
What the suite contains and why it matters
- The public set contains 100 tasks covering 91 different CWEs (Common Weakness Enumeration), spanning a broad range of vulnerability classes. Collinear AI emphasizes that doing well requires understanding multiple weakness surfaces an attacker might exploit—not just pattern recognition.
- Programmatic verifiers drive scoring; these verifiers were curated with input from LLM judge panels to balance difficulty and fairness. There is also a held-out set of tasks kept unseen by models; to evaluate on that held-out set, contact cyber@collinear.ai.
Quality checks and human review
For every task the team verified several properties, including:
- The instruction reads like a realistic developer task rather than an artificial evaluation prompt.
- Every graded behavior can be inferred from the instruction.
- Verifiers check for correct behavior rather than for a specific fix structure.
- The starter code fails and the reference fix passes.
- The original functionality still works after applying the fix.
- Verifiers reject memorized solutions such as known upstream patches. Each task also underwent manual review. Open-sourcing the set is part of the effort to improve quality; feedback and contributions from the community are welcome.
Results and leaderboards
Results are tracked in two places on the Hugging Face Hub:
- The CWE-bench dataset page leaderboard covers open-weight models and updates automatically; labs and community members add results via a PR to a model repo with a results file.
- The CWE-bench leaderboard Space provides the full comparison, including API-only models (such as Gemini, Claude, and GPT). New models are added to the Space via PR.
Among the 11 evaluated models so far, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, closely followed by Claude Opus 5.5 with 67.3% pass@1 and 78.0% pass@4. Ninety-five tasks have been solved by at least one model, indicating the tasks are solvable in principle. Still, even the leader fails on 19% of attempts, so many cases remain challenging.
Cost–performance tradeoffs
On cost metrics, DeepSeek V4.1 Flash (1017), Claude Opus 5.5, and Gemini 4 Argon form the Pareto frontier. Reported points:
- DeepSeek V4.1 Flash reaches 57.0% pass@1 at about $0.04 per rollout.
- Claude Opus 5.5 adds roughly ten percentage points at about $1.83 per rollout.
- Gemini 4 Argon moves ahead at around $3.92 per rollout.
Running the benchmark with Harbor and Hugging Face
Each task is packaged as a self-contained Harbor task (a Dockerized environment, an instruction, and a deterministic verifier), and the dataset includes a registry.json. Harbor can run the benchmark directly from the Hugging Face repo without cloning. Example command:
harbor run --repo "https://huggingface.co/datasets/collinear-ai/cwe-bench" -d cwe-bench@2.0 -k 4
--agent <agent> --model <model> --env docker
--allow-agent-host <model API host>
The -k 4 flag runs four attempts per task; the reported score is the mean reward over the 400 trials (100 tasks × 4 attempts). Access to the dataset is gated—accept the dataset's access conditions on its Hugging Face page first. Harbor fetches tasks over git, so save your Hugging Face token to git's credential helper once with:
hf auth login --token $HF_TOKEN --add-to-git-credential
Tasks are also tagged rl-environment: the same environments and verifiers can provide rewards for training defensive agents as well as for evaluation.
Final note
CWE-bench v2 aims to be an exhaustive, reproducible, and community-expandable benchmark for automated vulnerability identification and remediation. The public task set, the grader, and the harness are available; Collinear AI encourages researchers and developers to run their models and publish results.
