It was April. The Alabama Supreme Court slapped down a lawyer.
Why?
He’d filed legal briefs stuffed with citations generated by artificial intelligence. Not just any citations. Fake ones. Entirely made up by a machine that has never actually sat in a courtroom, never read the statute book, and certainly never lost sleep over precedent.
He was told. Directly. The precedent didn’t exist. He promised it wouldn’t happen again.
Then he cited “nonexistent cases at end of very next sentence,” a justice wrote in a concurring opinion later that week.
Can’t learn, won’t learn. At least one other attorney got sanctioned that same week for the exact same habit after receiving a warning. It isn’t a glitch. It is a pattern.
The plateau of lies
Damien Charlotin keeps count.
He is a senior research fellow at HEC Paris and maintains a database of legal errors tied to AI. For three years his list climbed steeply, exponentially, hitting over 1,400 instances of courts addressing hallucinated filings.
Then it flattened.
“Reached a plateau,” Charlotin said. Around 350 to 400 judicial decisions a quarter. Exasperated, yes, but steady. A new normal. He even built his own tool for this, an AI-powered checker called Pelaikan, presumably so others might avoid the trouble he documents.
“For the past two or three, we have reached plateau of around 300, 400.”
Wait, check that. 350. Or 400. The exact number doesn’t matter as much as the volume. Courts are public records. You can see when someone lies or makes a mistake. Lawyers get sanctioned for false claims.
Other sectors do not have such visible tripwires.
Silence elsewhere
Journalists got caught. Software developers broke things. Government consultants submitted work no one checked because no one could catch it in time.
On May 19 The New York Times broke a story about an author of a book called The Future of Truth.
The irony stung. His book argues how AI is shaping our reality, distorting truth, eroding fact. But the book itself? Loaded with fabricated quotes. Misattributed sources. At least half a dozen, by his own admission, produced entirely by the very technology he critiques.
He knew it could lie.
It seems knowing does not help.
Cognitive surrender
We trust the machine too much.
Alan Wagner teaches aerospace engineering at Penn State. He notes humans tend to believe machines are smarter, sturdier, more infallible than we are. “Do not break,” we assume. “Know everything.”
It’s not just blind trust. It is specific. AI generates answers that sound real. Plausible. Smooth. Humans rarely fake that level of consistency. A human guess has cracks. AI output is a mirror without smudges.
A study from February confirmed this bias. Participants played an image classification game. Half got advice they were told came from a human. The other half from AI.
The advice? Wrong 50 percent of the time. Always.
People who liked AI followed the wrong AI advice more often. They performed worse. No such effect occurred when they thought they were talking to another person.
“AI guidance has quite specific ability engender biases,” co-author Sophie Nightingale told the press. Lancaster University research, she adds.
The danger escalates outside the office.
Life or death logic
Wagner’s team ran another experiment. Darker this time. Inspired by drone warfare.
Participants saw images. Civilians? Soldiers? They had to choose to fire a missile or hold back.
A bot gave them feedback after each call.
The feedback was random. Meaningless.
If the participant guessed right and the bot disagreed, the participant often flipped their answer.
If the bot was wrong and the person was right, they changed their minds anyway.
They watched imagery of children. Of drones striking houses. Of devastation. Colin Holbrook, who co-authored the study from UC Merced, noted the subjects were trying hard.
“I think context important here,” Holbrook said. “They really thought mattered.”
In reality they would have killed innocents.
Why checking fails
Today’s AI does everything. Writes code. Drafts contracts. It invites what Wharton researchers call “cognitive surrender.”
Give the work away. Stop thinking. Let the model handle the heavy lifting.
Steven Shaw and Gideon Nave ran an experiment to test if rewards or feedback fixed this. Money for right answers? Yes, people deferred less. Item-by-item correction? Better, still, people deferred.
Not gone.
Just reduced.
Education hasn’t helped either.
Boston University researchers “inoculated” students. Told them ChatGPT lies. Told it is bad at math. Asked them to do the math.
Students who were warned verified source summaries more. Great.
Did they verify the math? No.
Time constraints in the real world mimic those in labs. People rush. Trust remains high on the numbers, low on the words.
“Awareness alone not enough,” Chi B. Vu wrote from Boston University’s Division of Emerging Media. He says competing pressures override warnings.
Deadlines loom. Advertisers tell you the tool saves hours. Why slow down to check the math if it usually works?
Nightingale calls it the ground truth problem. Users never verify deeply enough.
“They have no reason question,” she said. “Carry on in life thinking AI correct—because ‘why wouldn’t be?’”
So the briefs go out.
The errors go in.
And no one asks why.
