BioBasicsBench is a new benchmark of 100 routine wet‑lab tasks designed to measure how well AI agents can interpret raw experimental evidence, identify errors and recognise when data are insufficient to support a conclusion. The benchmark was developed in collaboration with Ligo, a biotech company focused on enzyme design.
Why this matters
Everyday laboratory work involves failed experiments, ambiguous images and conflicting observations. Existing benchmarks built around curated charts and images used in publications can miss the ambiguity and troubleshooting that real lab work requires. BioBasicsBench aims to make that routine analytical work measurable and improvable, because misreading images or overlooking problems with controls can lead scientists to wrong conclusions and wasted effort.
How the benchmark is organised
- The suite contains 100 tasks spanning five areas of routine wet‑lab work: protein methods, imaging and cytometry, quantification and assays, nucleic acid methods, and molecular cloning.
- Tasks range from simple readings (for example, reading a DNA sample’s molecular weight on an agarose gel) to detailed assay analysis, but all are routine tasks wet‑lab scientists should reliably perform.
- Inputs preserve the forms of evidence scientists work with: agarose and SDS‑PAGE gels, Western blots, plate photographs, microscopy z‑stacks and time‑lapse videos, raw plate‑reader exports and chromatograms, raw long‑read FASTQs, FCS flow‑cytometry files, photos of bench setups and handwritten notes.
- Supporting reference sheets (ladder charts, plasmid maps, pH charts) are provided where relevant.
Task creation and review
Tasks were authored using data from scientists’ own experiments, and each question was paired with an expert‑written grading rubric. Every task underwent internal review by a Ligo scientist who did not author it, external review by a PhD‑level wet‑lab scientist, and a final internal review. Reviewers examined the prompt, the experimental evidence, the rubric, author notes and a graded model attempt.
Answer format and scoring
Agents return a structured answer: a requested conclusion, numeric measurement or recommendation, and supporting reasoning when required. Answers are scored against rubric criteria that allocate credit to individual elements (conclusion, numeric result, reasoning, or recognition that evidence is insufficient). This allows the benchmark to distinguish a correct final answer from a partially correct analysis. Most tasks test multiple skills together: biological knowledge, image and data interpretation, error detection and protocol understanding.
Evaluation setup and aggregate results
- Seven models were evaluated: GPT‑5.2, GPT‑5.4, GPT‑5.5, GPT‑5.6 Sol, GPT‑6.1 Sol, GPT‑6 Luna and GPT‑6 Astra.
- The evaluation covered 37 model and reasoning‑effort settings. For each setting, the 100 tasks were attempted five times each, yielding a total of 18,500 results.
- The reported metric is mean reward: the average task score expressed as a percentage of available credit. Error bars show 95% confidence intervals obtained by resampling whole tasks (keeping the five attempts per task together).
At the highest reported reasoning‑effort setting (‘xhigh’), GPT‑6 Astra achieved on average 66.7% of available credit, compared with 43.8% for GPT‑5.6 Sol. These results indicate progress on routine laboratory analysis but also leave room for improvement in image interpretation, control checking and consistent application of experimental context.
Types of data included
Examples of data types present across the tasks include:
- Agarose DNA gels, SDS‑PAGE gels, Western blots
- Plate photographs and colony plates
- Raw brightfield and fluorescence images, multi‑channel sets, z‑stacks and time‑lapse movies
- Plate‑reader CSV/Excel exports, chromatograms and plots
- Raw long‑read FASTQ and FASTA sequences, GenBank maps
- Raw flow cytometry (FCS) files
- Protocols, SOPs, lab notebook and run records, photographed handwritten notes
- Photographs of bench equipment and instrument screens (NanoDrop, plate reader, balance)
Categories overlap and single tasks may include several data kinds.
Strengths and limitations observed
GPT‑6 Astra showed notable improvements in handling multimodal experimental evidence (photos, videos, sequencing outputs, plate‑reader and flow‑cytometry raw outputs, microscopy). Specific observations from the evaluation include:
- Signal ambiguity and controls: In one task GPT‑5.6 Sol classified cells as dead based on a dye signal, but the detector could also pick up light from another dye; brightness alone did not justify that conclusion. Astra recognised this ambiguity and requested better controls to separate signals.
- Labelling and control validity: In a DNA cloning task GPT‑5.6 Sol based conclusions on an apparent absence of colonies on a negative control plate. Astra noticed the plate was labeled with the wrong antibiotic (the label read “LB Kan50” indicating kanamycin instead of the ampicillin actually used) and therefore rejected the plate as a reliable control.
- Maintaining experimental context: In a sequence of microscope frames where cells appeared to come into and out of focus, GPT‑5.6 Sol sought a biological explanation for disappearance, while Astra correctly identified the set as a z‑stack (images taken at different depths).
However, Astra also demonstrated systematic weaknesses:
- Assumptions overriding evidence: In a colony PCR example Astra grouped bands correctly but inverted the experimental identities tied to those groups, leading it to nominate the wrong colonies for downstream purification.
- Discounting contradictory observations: Astra sometimes noticed contradictory regions but weighted those that fit a preferred interpretation more heavily in the final conclusion, effectively omitting valid counter‑examples.
- Counting errors: Astra’s colony counting marks did not reliably correspond to individual colonies—neighboring colonies were sometimes merged, visible colonies missed, and empty agar occasionally marked—resulting in an approximate 12% overcount relative to the benchmark reference on the illustrated plate. Such counting inaccuracies directly affect downstream estimates like transformation efficiency.
Implications
Higher benchmark scores do not imply models are ready to make reliable lab judgments autonomously. BioBasicsBench, however, makes routine wet‑lab analysis a concrete target for improvement: interpreting raw experimental data, checking controls and recognising when evidence does not support a conclusion. Progress on these tasks could reduce the time scientists spend troubleshooting and increase the time available for experimental planning. Monitoring these capabilities also helps assess potential biosafety and biosecurity implications.
