ResearchEpoch launches EBR-bench, finds little evidence of learning-from-experience; publishes new AI capability and vulnerability analysesEpoch AI introduced EBR-bench, a new benchmark that measures whether models improve on a complex board game through repeated play and reports little sign so far that models learn from experience.3 min read
ResearchSWE-Bench Pro audit conducted using model-based agents and a combination of five independent engineersThe SWE-Bench Pro audit was conducted using model-based investigative agents and individual reviews by five independent, experienced software engineers to examine a larger number of tasks in a…1 min read
ResearchThe evolution of coding models demands stricter, fairer and more reliable evaluationsResearchers and developers say that as coding models steadily improve, evaluation procedures must become more difficult, fairer and more reliable.1 min read
ResearchIndependent audit: SWE-Bench Pro unreliable due to 30% faulty tasksA recent independent audit found that SWE-Bench Pro, one of the most widely used AI coding benchmarks, does not reliably measure top-tier coding ability; the audit identified 30% of tasks as faulty,…1 min read
ResearchResearchers identify and manipulate a 'Golden Gate Bridge' feature inside Claude 3 SonnetAnthropic researchers published a paper describing how they located and modified internal activations—called features—in their Claude 3 Sonnet model, including a specific "Golden Gate Bridge" concept.2 min read
ResearchAssessing Concrete Tech Pathways in AI-Driven FuturismSome futurist claims propose that once AI research is automated, superhuman agents could rapidly invent technologies like nanotech, Dyson swarms, or near-light-speed probes.5 min read
ResearchCoordinated navigation reduces urban congestion: a large-scale routing-app experimentA six-month field experiment in 10 major US cities, described in a Nature Cities paper, tested a modified Google Maps routing that gently redirects a small share of trips away from recurring bottlenecks.4 min read
ResearchFixed Compute Budgets Skew AI Benchmark Results, UK Study FindsThe UK's AI Security Institute tested frontier models on seven benchmarks and found that fixed compute budgets baked into evaluations systematically understate model capabilities.2 min read
ResearchNonuniform Tensor Parallelism (NTP) to Preserve Goodput in Large‑Scale LLM TrainingNonuniform Tensor Parallelism (NTP) is an experimental framework that keeps large, tightly coupled LLM training jobs productive when some GPUs become transiently unavailable.5 min read
ResearchNeuroscience, philosophy and interpretability experts evaluate the researchThe research team invited experts in neuroscience, philosophy and interpretability to review their work and share their perspectives.1 min read
ResearchAnthropic: discovery of global-workspace-like operation in the Claude language modelAnthropic's new research reports that the Claude language model exhibits a 'global workspace'-like structure similar to the conscious and unconscious partitioning observed in the brain, in which only a small fraction of internal states become explicit.1 min read
ResearchOSWORLD 2.0: a benchmark for long‑horizon AI computer useOSWORLD 2.0 is a new benchmark, developed by a consortium of universities and industry partners, that measures how well AI systems perform multi‑step, multi‑program tasks on real computer environments.3 min read
