The Crumple Zone: Accountability in Financial Services When AI Agents Act on Their Own

Among the institutions moving agentic AI into production, the governance answer has acquired a reassuring simplicity. An agent will act, a person will supervise, and a human in the loop will catch whatever the machine gets wrong. Boards find the formulation comforting. Regulators have written versions of it into guidance. Vendors print it on slides. The phrase does productive work in a procurement meeting, because it converts an unfamiliar technology into a familiar arrangement in which someone is watching and therefore someone is responsible.

However, that arrangement was built for a world that agentic systems have already left behind. Conventional model governance assumes a clean sequence: a model produces an output, a person reviews it, a decision follows, and accountability traces back along a straight line. Agentic systems dissolve that line. They plan, call tools, act, observe the result, and adapt across long multistep sequences without pausing for review at each turn.

The accountability gap they create is, at its root, a question of where control actually sits, and “human in the loop” has become the phrase institutions reach for precisely because they cannot answer that question. Humans remain present in these systems.

The difficulty lies in the architecture around them: a loop whose scope is left undefined, whose permitted actions are never formally bound, and whose intermediate decisions cannot be reconstructed in sequence is not oversight. It is the appearance of oversight masqueraded around a process no one is governing.

Adoption is outrunning the controls

The MIT Sloan and BCG survey of 1,221 executives places agentic adoption at roughly 35 percent of organizations, with another 44 percent planning to deploy, and finds that 76 percent of respondents now describe agentic AI as more like a coworker than a tool. The controls have not kept pace. Deloitte estimates that roughly 80 percent of organizations lack mature governance for agentic AI, which it defines with unusual precision as clear boundaries for which decisions an agent may make independently, real-time monitoring that flags anomalies, and audit trails that capture the full chain of an agent’s actions.

Those three capabilities map almost exactly onto the conditions for accountable governance, and four in five institutions report having none of them, even as 74 percent expect agents to sit at the core of operations by 2027.

Financial services is the leading edge of this exposure. JPMorgan has put its internal LLM Suite in front of roughly 250,000 employees against an $18 billion annual technology budget and more than 450 production use cases, and has begun moving from assistance into agentic workflows that execute multistep tasks. Goldman Sachs has gone further into the workflows that matter most for accountability, embedding Anthropic engineers for roughly six months to build agents on Claude Opus 4.6 for client onboarding, trade reconciliation, and document review, while Goldman and Deutsche Bank have begun testing agentic systems for trade surveillance that decide what data to examine and escalate findings on their own. These are precisely the domains where a missed error is a regulatory event, a capital loss, or both.

Why static-model oversight fails under agentic conditions

The mismatch begins with arithmetic that most governance documents skip. Errors in a multistep agent do not stay isolated; they accumulate across the sequence. An agent that is 99 percent reliable at each individual step is only about 90 percent reliable across a ten-step task, and the decline steepens fast: a 1 percent per-step error rate compounds into roughly a 63 percent chance of failure over a long chain.

Real systems perform worse than these figures suggest, because the calculations assume each step’s errors are independent, when in practice an early misunderstanding cascades through every step that follows. Carnegie Mellon’s agentic benchmark bears this out, with top models completing only about 24 percent of office tasks autonomously and failure rates climbing to between 70 and 90 percent as complexity rises. The decisive consequence for oversight is that the error surfacing in a final output usually originated several steps upstream, at an intermediate decision no reviewer was shown. A human approving the end result is reviewing the symptom long after the cause has passed.

The supervisory framework that governs these systems in banking was not designed for any of this. The Federal Reserve’s SR 11-7 defines a model as a system that turns input data into a quantitative estimate, calibrated periodically and deployed largely unchanged between validation cycles. As risk practitioners have begun to note, that definition strains against agentic systems that recalibrate autonomously and initiate actions in real time, because material changes in behavior can now occur without any formal redevelopment event for a validator to catch.

The gap is already showing up in examinations, where the most common SR 11-7 finding in 2024 and 2025 has been inadequate model inventory: large language models deployed in customer service, document processing, and compliance that never made it onto the books. An oversight regime that cannot name the system it is meant to supervise is mismatched at the definitional level, well before any human is asked to review an output.

Why a human in the loop does not close the gap

Even where the human is present and attentive, the evidence on whether they catch the error is sobering. In a controlled study of trained pathologists, AI assistance overturned initially correct judgments at a 7 percent rate, with time pressure deepening the effect, which is to say that domain experts flipped right answers to wrong ones to match the machine. The standard remedy makes this worse rather than better. A synthesis of roughly sixty studies on overreliance found that richer explanations tend to increase a user’s trust in AI recommendations regardless of whether those recommendations are correct, so the explainability features sold as the answer to rubber-stamping can quietly enlarge it. European regulators have absorbed the lesson into implementation guidance for the EU AI Act, where a very low override rate is now read as a signal of automation bias and a reviewer who approves within seconds is treated as providing no meaningful oversight at all.

There is a deeper reason the loop fails, and it runs through the law of agency itself. In Governing AI Agents, legal scholar Noam Kolt shows that the classic tools for controlling any agent, namely incentive design, monitoring, and enforcement, lose their force when the agent makes uninterpretable decisions and operates at a speed and scale no supervisor can track.

The failure has two faces, and both end in no one being held accountable. The first is the “AI exceptionalism” the MIT Sloan report warns about, in which an institution treats the technology as too novel for existing rules and quietly excuses itself from assigning responsibility. The second is the inverse, captured in Madeleine Clare Elish’s concept of the moral crumple zone, in which a poorly designed loop concentrates blame on the nearest human operator who in fact had little ability to control the system’s behavior. A British Columbia tribunal has already refused the first move, holding Air Canada liable for its chatbot’s misinformation and dismissing as a remarkable submission the airline’s claim that the bot was a separate entity answerable for itself.

The crumple zone is the failure courts have not yet corrected, and it is the one a careless financial institution is most likely to build.

The same gap, one level up

The crumple zone concentrates blame on a single operator. Raise the altitude to the level of the financial system and the failure inverts, because no individual operator looks responsible when many institutions move in the same direction at once.

Hilary Allen, a law professor at American University who served on the staff of the Financial Crisis Inquiry Commission, has spent years tracing how shared tools turn herd behavior into instability, and she sees agentic AI sharpening the pattern. When institutions run similar agents built on a handful of foundational models trained on overlapping data, their judgments converge. Allen shares that “all actors are sort of acting in lockstep because they are all following the same probability-derived rules from the same data set.”

Supervisors have begun to name the same risk. The Bank of England’s April 2025 review of AI in the financial system warns that AI-based participants may take increasingly correlated positions, driven by the spread of a small number of vendor-provided models or convergence on similar designs, and the Bank is now constructing AI-specific stress tests aimed at herding that could amplify selloffs. The International Monetary Fund heard the same concern in its own outreach, where market participants ranked herding and market concentration among the leading stability risks from wider AI adoption in capital markets.

The mechanism is the brief’s compounding-error problem moved from inside one agent to across a market. In Allen’s account, which draws on the 2010 flash crash and the fictional 2030 crisis that opens her book Driverless Finance, one agent acting on an erroneous price invites others to trade against it, automated margin calls liquidate positions into the move, the selling triggers further margin calls, and the spiral outruns any committee that might convene to stop it.

This is why Allen treats some functions, automated margin calls among them, as carrying stakes too high to hand to a system that cannot pause, and why she counts grace and discretion as features worth preserving by design. A firm-level scope error, the kind that hides in Klarna’s volume metrics until it surfaces in churn, stops being one firm’s problem once every agent reads the same signals the same way.

The institutional lag runs wider than financial regulation. Writing on public finance, Korinek and Lockwood reach a parallel conclusion, that AI’s scale and speed exceed human oversight capacity and that the frameworks built for a human-centered economy, from tax administration to the legal categories that decide which entity answers for an action, will need redesigning before a revenue crisis forces the question.

The gap inside a single bank and the gap facing a treasury trace to the same source, an architecture that assumed a human at the center of every transaction.

What actually closes it: scope, boundaries, and an audit chain, designed in

The institutions actually creating accountable agentic governance share a common move: they build the conditions into the system at design time rather than bolting a reviewer onto the end. Three conditions do the work. Each is summarized below; expand any one for the evidence behind it.

That logic meets a hard objection worth taking seriously. Allen doubts these systems are auditable in the way the word implies. “I just don’t think it is auditable,” she says, because “you don’t get the same answer every time you ask them.” Her worry reaches past the backward-looking trace to the question regulators ask first, which is how a tool will behave under stress before it goes live. A system whose outputs shift across identical prompts cannot answer that with confidence, and for the highest-stakes uses Allen reads that uncertainty as grounds to withhold deployment.

One answer is to change the material the audit chain has to capture. Chris Schmitz, a Research Associate for AI and Governance at the Centre for Digital Governance, points to neurosymbolic systems, which pair the pattern recognition of neural networks with the fixed logic of symbolic rules. EY-Parthenon describes the output as predictive, auditable, adaptive and scalable reasoning that explains why it reached a result, with decisions in underwriting, claims, and compliance checked against codified regulatory rules.

The direction has early empirical support: a 2025 neurosymbolic framework for financial questions, VERAFI, lifted factual correctness from 52 to 95 percent, its rule-based policy layer closing the math and logic errors pure agentic processing left behind.

A rule layer makes reasoning legible and holds it inside bounds on well-specified tasks, and it leaves Allen’s deeper point standing, since an institution still cannot fully predict the neural component it has wrapped in rules. Better architecture lowers the cost of the audit chain. But scope, enforced boundaries, and a named owner are yet to be designed in.

What decision-makers should do now

The first move is to stop treating “human in the loop” as a control and start treating it as a design specification with three testable conditions. A board can ask, for any agent in production, whether its scope is documented, whether its action boundaries are enforced in code rather than in instructions, and whether any single decision in its chain can be reconstructed on demand. An agent that fails any one of the three is ungoverned regardless of how many humans nominally supervise it.

The second move is to redefine the test of oversight. The relevant question is no longer whether a person is watching but whether the institution could explain, afterward, exactly what the agent was permitted to do, what it actually did, and who answers for the difference. Override rates belong on the dashboard as a leading indicator, and a near-zero override rate should read as a warning rather than a success.

The third move is institutional. Bring agents onto the model inventory, extend SR 11-7 validation toward continuous rather than periodic review, and assign a single named owner accountable for each agent irrespective of where the human sits in the loop. A named owner closes both failure modes at once, because it leaves no room for the institution to claim the technology was too novel to govern and no room to let the nearest operator absorb a blame they could not have prevented.

That ownership question does not end at the firm’s edge. Allen is direct about where accountability lands when a bank runs an agent built by an outside vendor: “the buck has to stop with the bank, because banking regulators have authority over banks, they don’t have authority over vendors.” A bank cannot contract its accountability out to a model provider seated in another jurisdiction or shielded by a claim of trade secrecy, which means a vendor unwilling to open its system to scrutiny becomes a vendor the bank may be unable to use. Supervisors are arriving at the same place on shared infrastructure.

The UK Treasury Committee has pressed for the largest AI and cloud providers to be brought into the regime for critical third parties. And Allen’s proposal to treat a few foundational models as systemically important utilities follows one logic throughout: that the firm running the agent answers for it, and the infrastructure beneath many firms answers to someone as well.

The institutions that weather their first agentic failure will not be the ones with a human watching. They will be the ones that can say, with documentation, exactly what the agent was allowed to do, what it did, and whose name is on the difference.