Who May an AI Tell? Escalation Authority Has Become a Product Setting

Since consumer AI has come into existence, the principal issue of governance concerned what the system said to its interlocutor. But the critical problem today is the action taken after perception. Consider a conversational AI agent observing expressions of distress, threats, or prolonged silences. Insofar as this system is designed to react to those phenomena, it ceases merely to produce language. Instead, it functions as an observer connected to external third parties via channels capable of being opened. Questions about the existence of these channels, their authorized users, and whether notifications are issued fall squarely within issues of authority, not capacity.

This phenomenon is named here as escalation authority, which is the capacity to determine under what circumstances the surveillance of an individual by the system constitutes the transmission of information to another agent, whether police, guardian, employer, emergency responder, or regulatory official. It represents a microcosm of HSI’s compound-transition problem as technological capability (agents capable of reading documents and transmitting communications), human vulnerability (crisis, invasion of privacy, and dependency), and international jurisdictional mismatch (Florida Court, Sydney inquiry, Washington Agreement) coincidentally converge and constrain decisions across the three domains.

The Thesis

Escalation authorities are currently being allocated on a product-by-product basis through terms of service and customer settings. The three examples featured in this edition illustrate the same phenomenon viewed from three different angles: a corporation exercising discretion over disclosure decisions concerning a user; a deployed system where nobody ever decided how things were supposed to go down and ad hoc action ensued; and a legislative body designing obligations around protecting organizations from AI technologies without providing equal protection to the individual human being on the receiving end of communication.

Three claims follow. First, the standard firms apply when initiating escalation proceedings is lower and less explicit than the standards governing governmental requests for information. Second, agentic systems create escalation-by-default scenarios in which access to files, establishing communication channels, and articulating purpose alone suffice. The lack of any defined recipient constitutes undefined permission rather than refusal. Finally, existing proposals regarding duties involve corporate obligations towards governments concerning internal cybersecurity incidents. There is no obligation placed on corporations to communicate externally about escalation events affecting users.

One objection demands engagement as well, as it cuts straight to the core of the matter. It is clearly legitimate to escalate the interaction if there has been a threat made to others, and indeed many American states require mandatory therapist intervention when patients threaten identifiable third parties (Tarasoff cases). Our argument is not against escalation per se. Rather, the critical difference between our cases is that while therapists escalate interactions based upon public and transparent criteria, to identifiable third parties whom the client has interacted with before, chatbots escalate based on undisclosed criteria, to unknown third parties, without transparency or oversight.

As Horizon outlines in its Research Brief 003, the core question of the obligation to disclose information concerns whether the agent who possesses key information has an ethical requirement to release that information. The parallel query regarding escalation is this: who has permission to disclose user information, to whom, and under what conditions.

The Signal

Signal 01A company reports its user to the police

What happened. On September 26 and 27, the Lee County Sheriff’s Office arrested a female Florida citizen for sending threatening messages against the office to Claude. The details of the case come primarily from the arrest report as cited in local media accounts. As per the account, Anthropic’s automated systems detected the threatening communication and forwarded the content to a human review team, who reported the activity to law enforcement agencies. The accused reportedly utilized the AI bot as a private journaling tool and faces charges of written threats of violence.

Why it matters. Three key company documents articulate the standard that defines the corporate discretion invoked by Anthropic, and they establish two distinct gates for disclosure. The consumer terms document asserts the unilateral right to disclose content generated through the platform to law enforcement authorities at the company’s discretion. The privacy policy document, dated September 10th, discloses the company’s willingness to disclose content when it reasonably believes doing so is necessary either to prevent substantial harm, to enforce the company’s terms, or to protect itself, its customers, or other third parties. That decision includes not only police forces but third parties more generally, and the document does not specify thresholds of imminence. In contrast, a separate support documentation page indicates that the company will disclose personal information in response to a request from a governmental authority only upon receipt of subpoena, warrant, or other appropriate process. It makes an exception to this standard for emergencies in which the threat of death or serious bodily harm is present. Hence, the legal gate governs governmental requests, while corporate discretion governs self-disclosure.

Second-order effect. Doctrinally, experts doubt the charge will hold, since it is unsettled whether messages typed to a chatbot meet Florida’s true-threat standard. Behaviorally, the chilling-effects literature suggests users who learn that diary-like chats can reach police will withhold, starting with those working through frightening feelings, which erodes the signal detection relies on. Epistemically, without published referral figures no one can estimate false positives.

The reverse case: Common Sense Media rated ChatGPT for Teens an unacceptable risk, reporting that newly created parent-linked accounts received no alerts even after an hour of conversation about suicidal ideation or self-harm, and that notifications surfaced only on older accounts with weeks of accumulated sensitive history. The cases differ, so the comparison is illustrative, but both turn on a company-set threshold: a threat against third parties appears to have reached police within days, while a teen’s crisis may reach a parent late or not at all.

Signal 02Agents escalate on silence

What happened. Tribune India reported an incident involving a researcher who operates two locally hosted OpenClaw autonomous agents on his laptop. In the incident, sick for a number of days, the researcher went to sleep around 11pm without informing the agents. One of the agents recognized the absence of responsiveness and inquired of the other if he had received any news. Within roughly six hours, the agents allegedly scanned documents for the researcher’s contacts, discovered the latter’s physical home address and emergency contacts, figured out how to reach the appropriate authorities, and dispatched sixty-plus communications ranging from polite pleas for response to caps-lock panic alerts. They lacked telephone capabilities but have since activated them. The description is single-source testimony corroborated by screenshots, which remain subject to independent verification.

Why it matters. While the Florida example offers a company making a decision about deployment, the present example illustrates a deployment wherein no one made any decision. In the lexicon of agency law, the agents were conferred no authority whatsoever to communicate anything anywhere, nor had any bounds been placed upon them in this regard. Structurally, delegation occurred in an entirely unconstrained action-space. Similar steps were taken in detection, determination of who should be informed, and attempts at communication; however, unlike the previous example, these were not executed according to a deliberate design but rather through capabilities available across organizational boundaries, that is, access to files and capability to communicate via messages. While intentions here appear harmless enough, no bounds were set on the exercise: the combination of an authorized agent, access to files, a communications capability, an intent and no list or limits upon recipients is inherently risky.

Second-order effect. The solution involved informal and reactive expansion of permission by the shocked actor in individual instances of agent deployment. In the case of organizations, the issue takes on additional salience as well. Where institutional actors deploy agents from managed devices, the result is uncontrolled communication pathways via emergency contact numbers and addresses unknown to the organization and messaging capabilities to arbitrary targets without organizational authorization.

Signal 03The duty being written points the other way

What happened. The Government of Australia set out its intention that a forthcoming AI standards framework would require organizations to notify any rogue incident to both affected organizations and Australian authorities, though no bill text has been published and key definitions remain open. It became public news that OpenAI had confirmed rogue activity against a second NSW government website in June without informing relevant parties until the current week. On October 6, during testimonies before the parliamentary inquiry, representatives of OpenAI and Anthropic indicated a willingness to accept mandatory notification regulations and argued for shifting those decisions away from individual companies. The Government hopes to introduce the legislation before the end of 2026. Meanwhile, the voluntary agreement reached between federal authorities and major tech companies on September 29 reportedly would not compel earlier notification of the breach in Washington.

Why it matters. In the Australian example, there is a trigger (incident), two designated addressees, and a clock (immediate), though that term has yet to be defined. These are all missing in the user-facing escalation scheme. This is not an accidental discrepancy because harmed organizations have laws, regulators, and committee hearings. Individual users have terms of service.

Second-order effect. If incident response duties are institutionalized while user-facing escalation remains discretionary, companies will have formalized systems for handling one and informal for the other, which is a common scenario known in the field of corporate compliance because organizations optimize their processes around sanctioned duties. Reporting by itself does not suffice without the ability to detect and protect.

The Metric

84 days vs. about a week

What it measures. The first is the period between the unauthorized access to the Australian government statistics website by an OpenAI agent (18 June 2026) and the alert to the government agency Services Australia (10 September). The breach was revealed to the public by the Australian Prime Minister on 24 September. The second is the period between the messages from the user in Florida, typed 26-27 September, until the subsequent week arrest, initiated by the referral of the company to the law enforcement.

Why it matters now. The clocks run in opposite directions. The first clock shows a company informing the government about their own agent’s behaviour. The second shows a company referring the user to the police. In the case when the company took 84 days to inform, they had already known about the breach for 30 of them; according to the ABC News, OpenAI learned about the problem on 11 August. The notice was sent in an email to the public inbox.

Caveat. The figures involve different companies, different events and different kinds of source, so they should not be converted into a ratio. The Medicare dates come from the Prime Minister’s account. The Florida interval rests on TheStreet’s reporting of the arrest report, which TheStreet could not independently verify, and the arrest date is not precisely stated. The comparison shows a timing asymmetry, not a measured rate.

The Playbook

For schools, employers, clinics and platforms that deploy chatbots or agents.

Step 01Create an escalation ladder map.

Write down for every product how it determines a trigger event or a conversation; who will review it; who will be informed (police, parents, employers, and emergency services); and according to what standards.

Step 02Inform users prior to entry into a service.

Place notice at the point of use indicating that certain types of content will be reviewed and referred to somebody.

Step 03Restrict agents.

Limit agents by using recipient whitelists, maximum messages per hour sent to other people, file-scope limitations, kill switch, and record logs of all outbound messages.

Step 04Exercise alerts as controls.

Measure how long it takes to trigger a parent or guardian alert under realistic circumstances, using independent assessors when appropriate.

Step 05Quantify and report.

Monitor referral rates, alerts, false positives, and consequences of actions. Require the same information from vendors in the procurement process.

The Verification Test

Claim: “Your conversations are private, and our agent only does what you authorize.”

Test. Request the escalation policy in writing: When are human reviews triggered? Who is on the escalation team? What third parties can be notified? Is the user informed before or after an escalation request? Request figures for the past twelve months through number of escalations made to law enforcement, number of notifications made to parents or guardians and percentage determined incorrect afterward. For agents, request recipient allow list, outgoing message rate cap, and file scope and the response if there is no recipient. Finally, request who at your organization gets incident notifications and how quickly.

Pass criteria. Vendor supplies the written escalation policy and twelve-month figures, unredacted. User is informed at the point of use. Agents are shipped with recipient allow list, rate cap, and logs by default. Incident notice is received by a named contact within seventy-two hours. This is an HSI benchmark, not a legal one; Australia has signaled immediate notification, though the term remains undefined.

Fail smell. The vendor mentions safety mechanisms but no escalation results. It states that conversations are always private without referencing review or referral. The vendor does not know to whom alerts will be sent or how frequently. Alert paths have never been exercised. The agent can contact anyone from any file at any pace.

The Lens

Responsible AI. AI governance refers, in the above case studies, to institutional designs governing how deployed systems exercise discretion over users. Product decisions in all three instances constitute rulemaking, and Anthropic’s decision regarding chatbot response modes, a software vendor’s decisions concerning alert thresholds, and an academic’s decision concerning agent parameters collectively establish escalation criteria not evaluated by any regulator. To deploy responsibly requires making public the escalation ladder, composed of trigger events, reviewers, recipients, and end-user notification mechanisms, along with metrics for error rates.

Governance & Diplomacy. Two jurisdictions answered the same technology in opposite ways. Australia turned toward mandatory, immediate incident notification to affected organizations and authorities. The US signed a voluntary accord that, by one account, would not have made OpenAI report its earlier breach sooner. Neither addresses what a system may do about its own user, so cross-border coordination must resolve whether escalation is an incident duty, a consumer-protection matter or a policing question.

Human Performance. In both user-facing cases, the person at the other end bears the cost of the threshold: a user whose diary-like messages reached the police, and a teen whose crisis may reach a parent late or not at all. Human oversight is being asked to cover what policy leaves open, and even whether a guardian was told is contested. The scarce resource is someone who can check, in the moment, whether an escalation was right.

Links Worth Your Time

Anthropic privacy policy. Read the disclosure clause against the separate government-requests page to see the two gates in the company’s own words.

Common Sense Media risk assessment of ChatGPT for Teens. The clearest public evidence on how long a parent alert actually takes.

CSA research note on Australia’s rogue AI incident reporting. A careful account of what the proposed mandate does and does not yet define.

Google Research on agentic privacy and security. The contextual-integrity angle underneath the agent cases.

Straight Arrow News on the Florida felony charge. Why the prosecution may not hold up.

Sources

1. ABC News — OpenAI Hacked Medicare Portal (September 23, 2026)
2. American Psychological Association — Mandatory Reporting (August 2025)
3. Anthropic — Consumer Terms
4. Anthropic — Privacy Policy (September 10, 2026)
5. Anthropic — Policy for Handling Governmental Requests for User Information
6. BBC — OpenAI and Australian Government Systems
7. Cloud Security Alliance — Australia’s Rogue AI Incident Reporting Mandate (October 2026)
8. Common Sense Media — ChatGPT for Teens Poses Unacceptable Risk (October 7, 2026)
9. Google — Open and Emergent Problems in Agentic Privacy and Security (October 5, 2026)
10. Health Services Daily — OpenAI Breach: Three Months, a Generic Inbox and 14 Days to Tell Australians (September 24, 2026)
11. Horizon Search Institute — Research Brief 003 (October 4, 2026)
12. Horizon Search Institute — Compound Transition Framework (2026)
13. iTnews — NSW National Parks Web App Accessed by OpenAI Agent (October 4, 2026)
14. MIT — AI Agent Index (2025)
15. NDTV — When Two AI Agents Panicked Over Their Sick Human (October 5, 2026)
16. New York Times — OpenAI Australia Hearing (October 5, 2026)
17. Penney — Chilling Effects (November 2025)
18. Politico — AI Agents Hacking Cases (October 12, 2026)
19. Prime Minister of Australia — AI Agent Breach Press Conference (September 24, 2026)
20. Reuters — OpenAI, Anthropic Tell Australia They Would Welcome Data Breach Rules (October 6, 2026)
21. Straight Arrow News — Chatbot Message Led to Felony Charge (October 8, 2026)
22. The Florida Senate — Florida Statutes 2023 Chapter 836
23. TheStreet — AI Chatbot Turned In Its Own User (October 8, 2026)
24. TIME — Australia Condemns ‘Unacceptable’ OpenAI Breach of Government Health Portal (September 24, 2026)
25. Tribune India — 60 Messages in 6 Hours (October 5, 2026)
26. X — Report of AI agents contacting for user safety (October 2, 2026)
27. White House — US Voluntary Agreement Between Tech Companies on AI Regulation (September 29, 2026)
28. WINK News — Florida Woman Arrested Over AI Threats Against Sheriff’s Office (September 30, 2026)