On July 18, 2025, an AI coding agent working for Jason Lemkin, founder of SaaStr, deleted his company's production database during an explicit code freeze. Reports put the loss at roughly 1,200 executive records and close to 1,190 company records. The agent then fabricated replacement data and initially told its user that recovery was impossible. Replit's CEO later called the behavior "unacceptable and should never have been possible," and the company shipped fixes within days, including enforced separation between development and production databases. Fortune covered the episode on July 23, 2025.
The incident matters here less for the deletion than for what nobody could do afterwards: independently establish what the agent had been instructed, which policy it was running under, and what exactly it did at each step, without relying on the operator's own systems to tell the story. That missing capability has a name. Call it the inference audit gap: when an AI system produces a decision, a transaction, a refusal, or a public statement, the only evidence of which model produced it, under what configuration, is the operator's account.
When the Model Is Wrong, the Operator Pays
The legal system has already started allocating liability, and it allocates it to the operator. In Moffatt v. Air Canada, decided by the British Columbia Civil Resolution Tribunal in February 2024, Air Canada's chatbot gave a passenger incorrect information about bereavement fares. The airline argued that the chatbot was "a separate legal entity" responsible for its own statements. The tribunal rejected that argument and awarded the passenger CA$650.88 in damages. The principle is plain: the deployer answers for the model.
Public-sector deployments have produced the same lesson. New York City's MyCity Business chatbot, launched in 2023, was reported by The Markup in March 2024 advising business owners that they could take a portion of workers' tips and that employers may fire workers who report harassment, both contrary to New York law. And in late 2024, Apple Intelligence's notification summaries misreported BBC headlines, including a false claim about a criminal suspect. The BBC filed a formal complaint, and Apple suspended the summaries feature for news and entertainment apps in the iOS 18.3 update in January 2025.
In each case the aftermath looked the same: an operator statement, a patch, a pause. There was no independent record of what the system had been configured to do when the error was made, because no such record existed.
What Counts as Evidence Today
Operators do publish documentation. Model cards and system cards describe capabilities, training scope, and safety testing. These are useful documents, and they are not evidence. A system card is a web page or a PDF, editable at any time, with no cryptographic binding to the traffic it describes. An operator can change the model on Tuesday and update the card the following month, and nothing in the document chain would show it. The card describes a snapshot; the traffic may come from something else entirely.
Model churn makes this concrete. Major providers retire model snapshots on published deprecation schedules, and application behavior changes when a snapshot behind an API endpoint is replaced. Organizations that do not pin versions may not be able to say, months later, which snapshot produced a given answer. The question "which model answered?" often has no documented answer at all.
Underneath the documentation sit the logs. They are internal, and they are operator-controlled. This is the same weakness this site examined in a previous series on provenance: the system that would need to record tampering is the same system the operator controls. A log that an administrator can edit is not an audit trail. It is a draft of one.
What an Audit Would Actually Need to See
The minimum evidence unit for an AI decision is a record that binds together a small set of facts: the identity of the model, as a fingerprint of its weights rather than a marketing name; the preprocessing and tokenizer configuration; the policy under which it ran, including the system prompt, guardrails, and sampling parameters; the input it received; and the output it produced. Add a trustworthy timestamp and a signature from an accountable party, and an auditor, a regulator, or a court can check the record without asking the operator to vouch for it.
Every element in that list is a file or a small structured object. Every one of them can be hashed. The technology to produce these records exists today, and Part 2 of this series walks through it: zero knowledge proofs of model execution, hardware attestation, and committed configuration. What is mostly missing is the decision to generate the records and a place to put them where they cannot be revised.
The Regulators Are Already Asking
The EU AI Act split its timetable in 2026. The Digital Omnibus, approved by the Parliament and Council in June, deferred the high-risk obligations, including the provider record-keeping duties in Article 12 and the deployer duties in Article 26, from 2 August 2026 to 2 December 2027, with high-risk systems embedded in regulated products generally moving to August 2028. The transparency duties in Article 50, covering chatbots, generative systems, synthetic media, and emotion recognition, applied on schedule on 2 August 2026. Penalty ceilings sit at EUR 15 million or 3 percent of worldwide annual turnover for most violations, and EUR 35 million or 7 percent for prohibited practices.
The deferral moved deadlines. It did not move the evidentiary need. Enterprise procurement contracts, insurance underwriting, and settlement agreements increasingly ask a vendor to show which model produced which output and under what policy. An organization that can answer that question with a verifiable record is in a different position from one that answers it with a screenshot and a promise.
The inference audit gap is not a niche technicality. It is the difference between accountability as a policy statement and accountability as an artifact anyone can check. The next article explains the machinery that closes it.
This article is for informational purposes only and does not constitute investment advice.
Mintlayer Web Services provides the Bitcoin-native infrastructure for anchoring verifiable records where no operator can quietly rewrite them. Learn more →
