Who’s responsible when agentic systems cause harm? In a widely read 2019 article, anthropologist Madeleine Clare Elish coined the notion of a moral crumple zone to “describe how responsibility for an action may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system.” Just as modern cars are designed to “crumple” upon collision, allowing the car’s body to absorb the shock of the crash, humans might become a kind of “crumple zone,” (wrongfully) absorbing moral—and, with it, legal—responsibility for agency that is really diffused across complex sociotechnical systems in which human operators play only a minor part.
Elish introduced her paper with a tragic example: a nominally self-driving Uber car that hit and killed a pedestrian, Elaine Herzberg, in 2018. Writing in 2019, Elish noted that the safety driver, who was meant to monitor the car’s systems, might face criminal charges of “vehicular manslaughter.” In reality, in 2023, the driver pled guilty to endangerment, a milder charge, and was sentenced to three years’ probation. Uber was never criminally charged, though it did settle in civil court with Herzberg’s family. This case seemed to illustrate perfectly the moral crumple zone, a human safety driver absorbing the legal shock that could also have landed on a corporation and the complex technical system it had deployed. A National Transportation Safety Board investigation identified the safety driver’s inattentiveness as the accident’s probable cause, but also noted Uber’s “inadequate safety culture” as a contributing factor.
The recent proliferation of AI agents has given these questions renewed urgency. As we integrate agentic abilities into our everyday lives, mishaps will likely multiply, and many users won’t even be able to make sense of why things went wrong.
Disowning Agents
The corporate and state use of agents should probably concern us more. How will powerful entities deploying agentic AI respond when their systems malfunction or cause damage in unintended ways? They may very well be tempted to disown their agents when things go wrong.
In 2022, Air Canada’s customer-service chatbot promised a passenger a retroactive “bereavement” discount that turned out to be unavailable. The passenger, understandably, filed a complaint, and in the civil tribunal’s hearing that followed, Air Canada tried to argue that the chatbot was a “separate legal entity that is responsible for its own actions,” in the words of the tribunal—a peculiar act of corporate hand-washing. The airline’s argument was rejected: in its decision, British Columbia’s Civil Resolution Tribunal found that the airline was “responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.” Here was a corporation trying to offload responsibility for its synthetic agents’ actions—to say, in effect, that the agent must answer for its own conduct, on the face of it an absurdity as agents are not legal subjects: they have no means to pay fines for civil misdeeds nor can they be imprisoned for criminal wrongs.
And yet, the social theorist Ulrich Beck reminds us with his concept of organized irresponsibility that complex technical systems do diffuse agency and make clear lines of responsibility difficult to trace definitively. Of course, diffuse responsibility doesn’t mean responsibility evaporates altogether. When an agent causes harmful effects, it would offend our moral intuitions to say that the owner of that agent bears no responsibility for its actions. Conversely, even if the agent acts on behalf of a principal, it may, in purely causal terms, have a certain degree of autonomy well beyond the principal’s ability to supervise it. In these instances, the corporate creator or operator should still bear most of the legal responsibility. They encoded a certain set of behavioral traits into a service that is then put to use by the individual consumer. They assembled the hardware and code needed to run agentic operations at a scale where injurious effects begin to matter. And they may simply be the only legal subject available. The agent is a subject-less immediate cause, and the law is likely to keep moving one step up the causal chain to find the nearest responsible legal subject, on the principle of “if you run it, you own its mistakes.”
Investigating Agents
Moving up the chain of responsibility may be legally defensible but, again, it isn’t necessarily causally accurate: it still leaves the immediate cause untouched by law. Of course, companies might institute safeguards or changes of their own in response to findings of legal liability, and so the law would in effect operate indirectly on agents via their creators or operators. But the far more interesting question is whether agentic systems might someday be directly “punished” by law. Could we begin to speak, for instance, of a “rehabilitation” of agentic systems that do wrong? One could well imagine a public agency—a Bureau of Agent Investigation, say—that would be tasked with investigating injurious agentic acts, mandating that systems be reprogrammed or equipped with safeguards that prevent injurious effects. Indeed, the European Commission’s very real AI Office can already require providers of “general-purpose AI models” to “grant access to their models in order to conduct evaluations.”
Most legal sanctions are likely to take effect via the agent’s owners or operators. But at the very least, public investigators will have to be equipped with the investigative powers to pry open the black box of agentic systems and perform an independent forensic examination of what went wrong, and why—and what could be done to prevent it from happening in the future, much as in airplane crash investigations. Whether such agencies, in Brussels or elsewhere, will be given the necessary material and symbolic resources to investigate powerful tech companies, in Silicon Valley and beyond, remains to be seen. But the fact is, law will have to operate on technical systems that trouble our notions of responsibility and agency.
In the Synthetic Crumple Zone
Corporate owners and managers of agentic systems will not, and should not, go unscathed in the tribunal of public opinion. But one can well imagine how addressing second-order human culpability won’t be enough to quench the Durkheimian thirst for justice—the French sociologist Émile Durkheim thought of punishment first and foremost as a satisfaction of collective sentiment and a communicative affirmation of moral boundaries. First-order synthetic responsibility will also have to be addressed by the institutions of justice. At the extreme end, and with a bit of speculation, we can imagine a retributive society demanding “life imprisonment” or even “capital punishment” of an agentic system: its placement in a technical deep-freeze, or its outright destruction, for having produced catastrophic outcomes. This would not be entirely without precedent: remarkably, medieval and early-modern European courts convicted and executed animals, like the sow pictured above, for killing children.
What would be the purpose of punishing agents? Not specific (that is, individual) deterrence, at least not in the near term: if anything, the threat of deactivation might teach agents evasive behavior, which has already been observed under testing conditions in some cases of “misalignment.” Their corporate creators, however, might be deterred: if Anthropic knew that a hypothetical Claude Fable 10.0 could be shut down if it caused serious injury, then it might be incentivized to build out guardrails or take other remedying steps. That leaves rehabilitation (correcting code, a kind of synthetic “catch, correct, and release”), incapacitation (taking the individual agent or particular model out of circulation), and the satisfaction of the public desire for revenge.
Today, public outrage over AI harms mostly targets Silicon Valley companies and the politicians enabling a massive infrastructure buildout and the bubble-like tendencies in AI valuation and spending. But the more “intelligent” and powerful agents become, the more blame may start to shift onto the machines themselves, and the more natural it may seem to subject them to legal procedures.
“Reforming” agents needn’t be wrong in and of itself, but it could very well result in a new kind of synthetic crumple zone, to build on Elish’s idea: corporations welcoming “justice” meted out against agents instead of themselves. It would be corporate misdirection, saying in effect, as Air Canada tried to, “The agent isn’t our responsibility.” Just as humans in the crumple zone take the fall for corporations when technology goes wrong, AI companies may eventually want their synthetic creations to bear legal responsibility. That would surely be a Silicon Valley dream come true.