Anthropic announced on October 22, 2024 that Claude 3.5 Sonnet has been extended with the ability to interact with computer software in a human‑like way: when integrated with the appropriate software setup, the model can interpret screenshots, move a cursor, click on locations, and enter text via a virtual keyboard. This capability is available in a public beta.
Why this matters
Anthropic argues that because so much modern work is done on computers, enabling AIs to use existing software like a person could unlock many applications that current assistants cannot perform. Following recent advances in logical reasoning and image understanding, the company positions computer use as the next frontier — models that can work with arbitrary software instead of relying on bespoke tools.
The research process
The new computer‑use skills build on prior work in tool use and multimodality. Operating a computer requires seeing and interpreting images of a screen and reasoning about when and how to act. During training, Claude learned to inspect screenshots and calculate how many pixels to move the cursor vertically or horizontally to click precise locations; accurate pixel counting proved critical for reliable mouse commands.
Anthropic reports that Claude generalized quickly from training on a few simple applications (such as a calculator and a text editor). For safety, the model did not have internet access during training. Once trained, Claude could translate a user prompt into a sequence of logical steps and execute them on the computer, sometimes self‑correcting and retrying when it encountered obstacles.
According to the company, Claude is currently state of the art among models that use computers by looking at the screen and taking actions. On an evaluation created by OSWorld to test models’ computer‑use abilities, Claude scored 14.9%. That is far below human performance (generally 70–75%), but well above the next best AI model, which scored 7.7%.
Making computer use safe
Anthropic notes that each AI advance introduces new safety challenges. They view computer use chiefly as lowering the barrier for existing capabilities to be applied, so their principal concerns focus on near‑term harms rather than speculative frontier threats. The company assessed the feature against its Responsible Scaling Policy and concluded that Claude 3.5 Sonnet with computer use remains at AI Safety Level 2, so it does not require stricter safeguards than those already in place.
However, Anthropic cautions that if future models demand AI Safety Level 3 or 4 protections, computer use could amplify those risks. They believe it is preferable to introduce computer use now while models remain at Safety Level 2 so that safety issues can be addressed early.
Trust & Safety teams analyzed the new models to surface vulnerabilities. A prominent concern is prompt injection, where malicious instructions embedded in content presented to the model cause it to override prior directions or perform unintended actions. Because Claude can interpret screenshots from internet‑connected machines, it could encounter content containing prompt injection attacks.
Anthropic recommends precautions for developers using the computer‑enabled Claude in the public beta and provides a reference implementation with guidance. The company has also built classifiers and other mechanisms to detect and mitigate intentional misuse. With U.S. elections approaching, Anthropic says it is particularly alert to potential abuses that could undermine public trust in electoral processes: they monitor election‑related requests and have systems to dissuade Claude from activities such as generating and posting social media content, registering web domains, or interacting with government websites. The safety measures will be continuously evaluated and iterated during the public beta.
Early limitations and user experiences
Anthropic acknowledges that current computer use is slow and error‑prone. Many routine human actions (dragging, zooming, etc.) are not yet supported. Claude’s view of the screen is a sequence of screenshots rather than a continuous video stream, so it can miss brief notifications or short‑lived actions.
The company shared anecdotes from demo sessions that illustrate remaining fragility: in one case Claude accidentally clicked to stop a long running screen recording, causing all footage to be lost; in another, during a coding demo the model diverted to browsing photos of Yellowstone National Park.
What’s next
Anthropic characterizes computer use as a different approach to AI development: instead of making tools fit models, they are making the model fit everyday software. The aim is for Claude to use existing pieces of software as a person would.
There is still substantial work to improve speed, reliability, and richer interactions, and to make the capability easier to deploy for developers with less engineering experience. Anthropic says researchers will continue to work closely with safety teams as capabilities evolve. Developers trying the public beta are invited to submit feedback via the company’s provided form so the team can refine both usefulness and safety.
