This week’s AI Engineer New York conference focused on how to balance agent autonomy with security and compliance in financial workflows. Presentations from frontier labs and financial firms converged on the idea that safe autonomy is determined more by where you place controls and human judgment than by how many blanket restrictions you impose.
OpenAI: the ‘‘threshold of usefulness’’ and traceability
Joseph Kim, Head of ChatGPT for Financial Services at OpenAI, delivered a keynote at AIE New York. Kim described a ‘‘threshold of usefulness’’ — the point at which a model becomes capable and reliable enough to save time for users. He argued that growing model capabilities will soon push financial analysts past that threshold, enabling them to delegate more work to agents.
Kim predicted that by 2027 the financial industry will rely less on Excel and PowerPoint and that agents will be competent enough for tasks such as investment idea generation and trading. He also emphasized that improving model capabilities has led to better traceability: in finance, it is essential to check where a number came from and understand its context. Kim said these product-level details help convert raw model ability into work someone can actually use.
On the question of human-in-the-loop, Kim framed it as a product design decision: how the interface changes and what the user still needs to control, review, or approve are decisions teams should start addressing now.
Two Sigma: composable guardrails and avoiding approval fatigue
John Feminella, SVP Architecture at the hedge fund Two Sigma, addressed the same product design challenges in his keynote. Feminella argued that safety and innovation are not opposing forces and warned about ‘‘approval fatigue’’—when a person must approve too many actions, slowing innovation and increasing risk if attention lapses.
Two Sigma has implemented a set of controls they describe as ‘‘guardrails and shared capabilities.’’ These controls are composable: each performs a single job, and together they enable agents to operate with real autonomy at an appropriate safety level. One example is the use of auto-mode classifiers that decide routine calls and surface only the most important requests to humans. Feminella noted classifiers are useful for ambiguous cases while deterministic rules work better for clear-cut situations.
Anthropic: identity, approval layers and building a trust layer
Anthropic presented a related framework aimed at embedding ‘‘trust’’ into financial agentic workflows. Last month Anthropic announced Claude for Financial Advisors, which lets financial advisors connect their systems to Claude along with workflow skills.
Belinda Neal, Managing Director of Financial Services at Anthropic, said models periodically cross thresholds that trigger waves of changes in how people work. Keon Shahab, who works on applied AI at Anthropic, outlined three steps to build a trust layer: defining who the agent is acting as via an identity and permissions layer; sandboxing and establishing an approval layer so someone signs off before action; and ensuring the agent continues to function correctly as variables change over time.
Shahab illustrated the boundary between proposing and acting with a trade-break investigation agent that would recommend an order amendment for human approval rather than making the amendment itself. His conclusion: by building a trusted foundational layer once, firms can deploy many agents at speed and scale.
Bloomberg: enforce deterministic controls outside the model
Andrey Rybka, Head of Architecture & Emerging Technologies in the Office of the CTO at Bloomberg, reviewed real incidents where agents caused harm and outlined defenses to prevent similar events. His recommendations include scoped identities and credentials, controls over tool calls, outbound traffic restrictions, audit trails, budgets and kill switches.
Rybka recounted an example where an agent misinterpreted instructions embedded in a marketplace review as an instruction to stock a product. To mitigate such risks he recommended input validation, prompt-injection defenses, provenance tracking, and human approval for high-value actions. His mantra: ‘‘deterministic controls outside the model’’ — check every call, log it, and maintain the ability to shut it down.
Conclusion: go fast, but stop safely
The conference’s consensus was that while OpenAI and Anthropic are building trust frameworks for financial agents, individual financial firms must implement operational controls around their agents. The practical approach is to deploy composable guardrails, classifiers, sandboxing and external deterministic checks that enable autonomy without creating unnecessary roadblocks.
As John Feminella summarized: brakes aren’t only for stopping — they let you drive fast and stop safely.
