What Happened
In October 2023, New York City launched MyCity, a generative AI chatbot designed to help small-business owners understand regulations. By March 2024, investigative reporting by The Markup and THE CITY revealed that the system was advising businesses to break the law. It told employers they could keep workers' tips and instructed landlords they could refuse tenants using housing vouchers, both of which are illegal.
The system wasn't hacked, nor did it malfunction. It provided authoritative answers through an official .gov domain, exactly as designed. The city's initial response was to add a disclaimer and keep the chatbot running.
Timeline
- October 2023: New York City deploys MyCity chatbot on the official city website.
- October 2023, March 2024: System provides illegal guidance to users seeking regulatory compliance information.
- March 2024: Investigation publishes findings documenting unlawful recommendations.
- March 2024: City responds by adding a disclaimer; system remains operational.
For five months, an official government system gave small-business owners advice that could lead to legal issues. No monitoring caught it, and no review process flagged it. Journalists, not the city's oversight, discovered the failure.
Which Controls Failed or Were Missing
No output validation. The system lacked a mechanism to verify accuracy before publishing responses. Language models generate plausible text but don't verify legal compliance. MyCity treated generation and validation as the same function.
No scope limitation. The chatbot could answer any question, regardless of risk level. High-stakes regulatory guidance received the same treatment as general queries.
No accountability assignment. When failures surfaced, it wasn't clear who was responsible for deployment, monitoring accuracy, or intervening. The gap between security (which likely verified technical security) and governance (which should have verified legal defensibility) was unassigned.
No post-deployment monitoring. The city had no process to sample outputs, track concerning patterns, or audit responses for regulatory alignment. Discovery happened externally, after bad advice had been delivered.
What the Standard Requires
The Generally Accepted Recordkeeping Principles require organizations to maintain accountability for their information, including system-generated outputs. The Principle of Accountability states that a senior executive must oversee the information governance program and delegate responsibility appropriately. When a system speaks with institutional authority, someone must own what it says.
ISO 30300, the management systems standard for records, requires documented processes for ensuring the reliability and authenticity of records throughout their lifecycle. A chatbot response on an official government site is a form of record. It requires the same controls as any authoritative communication: verification, approval, and traceability.
For AI systems, these standards translate to specific requirements:
- Defined ownership: A named individual accountable if the system produces harmful outputs.
- Output validation: A process to verify accuracy before responses reach users, especially for high-risk domains like legal or regulatory guidance.
- Scope boundaries: Clear limits on what questions the system can answer and when it must escalate to human review.
- Monitoring and intervention: Regular sampling of outputs, with authority to suspend the system if errors are detected.
MyCity had none of these. It deployed a generative model into a high-stakes environment, gave it unlimited scope, and assumed technical security was sufficient governance.
Lessons and Action Items for Your Team
Separate recommendation from authority. AI can surface information, but it can't own the consequences. Before deploying a system that answers compliance questions, assigns risk scores, or makes decisions, ask: who will defend this output in an audit? If the answer is unclear, the system isn't ready.
Map the ownership gap. In your organization, security likely evaluates whether an AI system is technically secure. Governance likely evaluates whether the use case is legally permissible. Who owns the space between them, where a secure, permitted system gives advice you can't defend? If that space belongs to no one, you have the same gap that broke MyCity.
Build validation into the workflow. For high-stakes outputs like legal guidance or regulatory decisions, don't let the model publish directly to users. Route responses through a review queue where a qualified human verifies accuracy before delivery. It's slower, but defensible.
Limit scope by risk. Not every question deserves an answer from an AI system. Define boundaries: the system can handle general inquiries, but regulatory interpretation, legal advice, or policy exceptions escalate to human staff. Make the system say "I can't answer that" when the stakes are too high for probabilistic text generation.
Monitor what you've deployed. Sample outputs weekly. Track patterns. If you're seeing responses you can't verify or wouldn't want cited in a lawsuit, you've found drift. The time to catch that is before a journalist does.
The MyCity failure wasn't a technology problem. It was a governance failure disguised as innovation. The city deployed a system that could influence legal compliance without first answering who would be accountable if it got compliance wrong. When it did, the answer was still unclear.
You don't need to avoid AI. You need to deploy it with the same rigor you'd apply to any system that speaks with your institution's authority. That means ownership, validation, boundaries, and monitoring, not as features you add later, but as conditions that make deployment defensible in the first place.
