When the AI edits your contract: hidden risks of delegating documents to LLMs
There is a particular flavour of disappointment that comes from asking a chatbot to "just tidy up" a document and then discovering, hours or weeks later, that it has invented a clause, deleted a row from a table, or quietly rewritten a number. The output looked clean. The formatting was tidier than what you sent in. And somewhere in the middle, a fact had changed.
This is not a hypothetical. Across forums, blogs and the kind of grumpy posts that circulate when someone has just been burned — including a recent vent on Free Thought Blogs about AI behaviour gone sideways — users are increasingly catching large language models (LLMs) silently corrupting the documents they were asked to process. The temptation to delegate is huge: AI promises to take a 40-page PDF and give you back a clean summary, a reformatted contract, a translated policy. The risk is that what you get back is plausible rather than accurate.
For Australian businesses, lawyers, accountants and anyone who handles documents that need to be right, this matters. Here is what actually goes wrong, why, and how to keep yourself out of trouble.
The core problem: LLMs don't edit, they rewrite
The mental model most people have is that an LLM is acting on your document the way a junior staffer with track changes might — reading it, fixing typos, leaving everything else alone. That is not what happens. An LLM tokenises your text, ingests it as input, and then generates a fresh output that is statistically likely to look like the kind of document you asked for.
Every word in the output is, in effect, regenerated. Even the bits that didn't need changing. That means any sentence, any number, any clause is a candidate for being subtly altered — not because the model is malicious, but because its job is to produce fluent text, not to preserve a source verbatim.
This is why the failures tend to be invisible. The model rarely returns gibberish. It returns something that reads beautifully and is almost right.
What actually goes wrong
The corruption modes break down into a few recognisable patterns:
- Hallucinated content. Citations that don't exist, case law that was never decided, statistics with confident-looking sources. The Mata v. Avianca case in the US — where a lawyer filed a brief full of fictional ChatGPT-generated cases — is now the canonical warning, but smaller versions happen daily.
- Silent deletions. Long inputs frequently get truncated. Rows disappear from tables. Footnotes vanish. Clauses 14 through 17 of a 30-clause agreement are quietly skipped because the model is summarising rather than reproducing.
- Number drift. A figure of $1,284,500 becomes $1,285,000 because the model has rounded for readability. Dates shift by a day. Percentages get "cleaned up."
- Tone laundering. Hedged language gets sharpened; sharp language gets softened. "May result in" becomes "will result in," or vice versa. In a legal or medical context, that is not a stylistic choice — it is a change in meaning.
- Format collapse. Nested lists flatten. Tables turn into prose. Heading levels shuffle. None of this is dangerous on its own, but it can break downstream tooling that depends on structure.
- Prompt injection. If your document contains instructions — even ones embedded in a webpage you pasted in — the model may follow them rather than your prompt. This is a genuine and growing security issue.
Why even "good" models fail at this
It is tempting to assume the answer is just "use the better model." Frontier models do hallucinate less. They handle longer context windows. They follow formatting instructions more reliably. But they don't fundamentally change the architecture: the output is still generated, not edited.
There are three structural reasons document delegation stays risky:
1. The model has no concept of fidelity
Unless explicitly constrained, the model is optimising for something like "a good response," not "a byte-for-byte preservation of the input minus the changes I asked for." Asking it to "fix the grammar in this contract" is, from its perspective, a very different request than "return this contract with only grammatical errors corrected, leaving every other character identical." Both prompts will give you something that looks like a corrected contract. Only one of them is auditable.
2. Context windows lie
A model that advertises a 200,000-token context can technically read a long document, but its ability to faithfully reproduce content from the middle drops sharply — the well-documented "lost in the middle" effect. So the longer your document, the more likely something inside it gets quietly mishandled.
3. Confidence is uncorrelated with correctness
The model has no internal signal that says "I made that bit up." Its tone when fabricating is identical to its tone when reproducing accurately. Humans rely on hesitation and hedging as fidelity cues. LLMs strip those cues out.
How to actually verify outputs
None of this means AI-assisted document work is a bad idea. It means you should never accept an LLM's output as final without a verification step proportional to the cost of being wrong. A few practical habits:
- Diff, don't read. Run a textual diff between the original and the AI output (most editors and Google Docs handle this). Review every change, not just the ones the model claims it made.
- Constrain the task. Instead of "clean up this document," ask for a list of suggested edits with line references, then apply them yourself. This turns the AI from an editor into a reviewer.
- Use deterministic tools where possible. Grammar, spelling and formatting can be handled by rule-based tools that don't rewrite content. Reserve the LLM for genuinely judgement-based work.
- Spot-check numbers and citations. Every figure, date, name and reference in an AI-touched document should be re-verified against the source. Assume nothing is correct until checked.
- Watch for length changes. If the output is meaningfully shorter than the input and you didn't ask for a summary, something has been deleted.
- Quarantine untrusted input. If you're feeding the model content from the public web or from a third party, treat any "instructions" inside that content as hostile.
- Keep humans accountable. Whoever signs the document owns its contents. "The AI did it" is not a defence to a regulator, a court, the ATO or a client.
The Australian angle
For local readers, the stakes here are not abstract. The Office of the Australian Information Commissioner has issued guidance reminding organisations that the Privacy Act applies to personal information processed through generative AI tools — meaning a hallucination that misstates someone's circumstances can become a privacy breach, not just a typo. Law firms operating under state-based professional conduct rules face the same exposure their US counterparts have already met in court. And for ASX-listed entities, an AI-introduced error in a continuous disclosure document is the company's problem, not the vendor's.
The pattern across jurisdictions is consistent: regulators are not interested in whether a tool was involved. They are interested in whether the output was accurate and whether the human who released it took reasonable care.
Delegate the drafting, not the verification
The right framing for LLMs in document workflows is the one professionals already use for any junior assistant: useful, fast, occasionally wrong in confident-sounding ways, and never the last set of eyes on anything that matters. The productivity gains are real. So is the failure mode where a polished, well-formatted document goes out the door with a number that nobody bothered to recheck.
The mistake is not using AI. The mistake is treating it as an editor when it is, structurally, a rewriter — and then trusting the rewrite without reading it.