Vendor Hallucinations: A Field Guide to Confident Fabrication on the Exhibition Floor

I spent two days at a legal technology conference recently, speaking with vendors about their AI products. I left with a realization that has been nagging at me since: the people selling AI tools to lawyers hallucinate more than the AI does.

I don’t mean this as a throwaway line. I mean it structurally. If we applied the same diagnostic framework to vendor claims that we apply to Large Language Model (LLM) output — confident assertion of false information, gap-filling when knowledge runs out, fluent delivery that masks fundamental uncertainty — the exhibition floor would fail its own evaluation.

But the vendors are only the beginning. Because if you pull on this thread far enough, you arrive at something the legal profession would rather not examine: lawyers have been hallucinating for centuries. We just call it something else. We call it persuasive storytelling, or aggressive interpretation, or — when it goes wrong — incorrect reasoning. We have built an entire professional culture around the confident assertion of contestable propositions, and we have rewarded it so consistently, for so long, that we’ve lost the ability to see it for what it is.

And here is the thing worth remembering at a technology conference: law is itself a technology. It is one of the oldest human technologies — a system for encoding rules, resolving disputes, and managing power that predates the printing press, the steam engine, and the silicon chip by millennia. The exhibition floor is not a place where technology meets the law. It is a place where a new technology is being evaluated by an old one. And the old one — which has always rewarded its practitioners for zealously disagreeing about what “correct” means — is now demanding a standard of correctness from the new one that no party in the system has ever been held to.

Let me walk you through what I mean.

The Confident Misstatement

At one booth, a solutions consultant for a major AI-for-lawyers platform told me flatly that no AI models can watch video.

I’ll just let that sit there for a moment.

This was not a nuanced claim about the limitations of multimodal processing, or a qualified statement about production readiness. It was a categorical denial of a capability that is publicly documented, commercially available, and demonstrable in about thirty seconds on a laptop. But it was delivered with the serene confidence of someone reading from a spec sheet, and I suspect most attorneys walking that floor — attorneys who are being told to worry about AI hallucinations — would have nodded and moved on.

If an AI tool had produced that claim, we’d call it a hallucination. We’d cite it as evidence that the technology isn’t ready. When a human said it, standing three feet from a banner advertising the most advanced language models on earth, it was just a sales conversation.

The Compliance Confabulation

At another booth, I identified myself as an assistant public defender. What followed was a masterclass in what cognitive scientists call provoked confabulation — false information generated in response to a direct question, not from malice but from a compulsion to fill the gap rather than admit the gap exists.

The solutions consultant assured me that their platform could serve public defender offices because they already work with state attorney and district attorney offices and maintain Criminal Justice Information Services (CJIS) compliance. Then came the rest of the script: Federal Risk and Authorization Management Program (FedRAMP) authorization, GovCloud environments, the different providers who host them. He knew his material. He could name the infrastructure, explain the tiers, walk through the compliance architecture. None of it was wrong.

It just wasn’t relevant. Public defenders don’t need CJIS compliance. We aren’t a law enforcement agency. We represent the people law enforcement has arrested. He had pattern-matched “public defender” to “government buyer” and loaded the nearest playbook — and the playbook was accurate about everything except who he was talking to.

This is, incidentally, one of the most common failure modes in large language models: generating output that is technically correct but contextually wrong. The information is real. The relevance is hallucinated. Pierre Bourdieu would call it meconnaissance — misrecognition — the moment a participant in a social field mistakes the field’s scripts for actual knowledge.

At the same booth — or maybe it was one over, the booths blur together after a while — an account executive initially told me that every user on their platform operates in a GovCloud-secured environment by default. One follow-up question later, he clarified that there were actually different tiers of pricing and security. The first statement wasn’t a lie. It was a hallucination: a confident, contextually plausible assertion that dissolved on contact with a single clarifying question.

The Retreat from Inquiry

Then there was one of the large, established legal data companies — the kind everyone in the room already knows. They weren’t trying to dominate the floor. They didn’t have the biggest booth or the most staff. They were just there, the way they’re always there, with the quiet confidence of an institution that doesn’t feel threatened.

I asked about memory — when their AI interface would support persistent context across sessions. They said it was coming, but offered nothing further. I asked whether their database features could support local judicial analytics: data I’m collecting about judges in my circuit to help with case strategy and outcome prediction. They acknowledged that the current feature set is oriented toward federal courts. They suggested I try some other features.

I had tried the other features. What I wanted to talk about was the database capability, which is what I had asked about. But the conversation kept drifting to safer ground.

This is the most subtle form of vendor hallucination, and arguably the most pernicious. It’s not a false claim. It’s the absence of a claim where one is needed. Rather than saying “we can’t do that yet” or “that’s not our roadmap,” the response was to redirect — to generate a plausible-sounding conversational output that maintained engagement without actually addressing the query.

If a chatbot did this, we’d call it evasion or topic drift. We’d flag it as a failure mode. On the exhibition floor, it’s called consultative selling.

The Control Group

Here’s where the story turns. Because alongside the large vendors, the conference floor also had startups. And the startups — without exception — were the most epistemically honest people in the room.

This makes sense if you think about it. The startup booths were staffed by co-founders and C-suite executives who were simultaneously the product team, the engineering team, and the marketing department. They couldn’t confabulate because the distance between “what the product does” and “who is talking to you about it” was zero. And because that distance was zero, there was no field — in Bourdieu’s sense — in which misrecognition could operate. No playbook to internalize, no habitus of enterprise sales to fall back on. Just a person who built a thing, telling you what the thing does.

One startup had built a tool that worked almost entirely from your email — you could ask your “twin” (not your “agent,” a distinction I appreciated) to find emails related to a matter, draft responses, or generate documents using email threads as source material. The founder explained exactly what it could and couldn’t do. No compliance theater. No gap-filling.

Another had built a forensic accounting platform where AI was not the centerpiece but the assistant — it helped draft reports from the accounting data rather than pretending to be the accountant. Their pricing was self-serve and transparent. I’m going to bring it to a colleague this week to evaluate.

A third startup CEO was genuinely fascinated by what I was doing — building local models, writing custom tooling, coding the solutions my office actually needs. His product was, if I’m being honest, the least distinctive of the three: essentially an open-source chat interface with document drafting built in. But the pitch was simple and honest because he could see that I already understood what the underlying technology was doing. No hallucinations necessary.

None of these startups lost interest when I said “public defender.” That alone was notable.

The Oldest Technology in the Room

Here is where I’d like to turn the lens around.

Everything I’ve described so far — the confident misstatement, the compliance confabulation, the retreat from inquiry — is familiar to any attorney. Not because we’ve seen it from vendors. Because we’ve done it.

It happens in closing arguments, when an attorney constructs a coherent narrative from evidence that was, in the courtroom, fragmented, contradicted, and contested — and delivers it as though the story tells itself. It happens when a judge asks a question from the bench and the attorney answers with authority, because hesitation reads as incompetence, even when the honest answer is “I’d need to research that.” It happens when an expert witness testifies to a conclusion with a degree of certainty that the underlying methodology doesn’t fully support. It happens at cocktail parties, when someone asks about an area of law we haven’t touched in years and we answer anyway, because the social cost of saying “I’m not sure” exceeds the social cost of being confidently approximate.

None of this is misconduct. That’s precisely the point. These are examples of the legal system functioning as designed.

We don’t call any of this hallucinating. We call it practicing law. But the underlying behavior — generating fluent, confident output without adequate grounding in verified information — is structurally identical to what we condemn when a chatbot does it.

The difference is that law has had thousands of years to build institutions around managing this tendency rather than eliminating it. The adversarial system is, in a sense, a hallucination-management protocol: we assume both sides will assert confident, self-serving versions of reality, and we let the jury or judge arbitrate between them. Cross-examination is a fact-checking mechanism for human confabulation. The rules of evidence are guardrails against the most dangerous forms of gap-filling. The entire apparatus of the trial is an acknowledgment that humans cannot be trusted to assert only what they know to be true — so we built a system that works anyway.

AI has been in wide public use for roughly three years. The legal profession has had roughly three millennia to develop tolerance for its own version of the same flaw. Holding AI to a standard of epistemic perfection that law has never required of its human practitioners is not rigor. It is — to borrow Bourdieu’s framework one more time — misrecognition. It is the legal field protecting the fiction that human assertion operates on fundamentally different epistemic ground than machine-generated text, when the structural mechanics are uncomfortably similar.

The Diagnosis

I don’t think the answer is to excuse AI hallucination. I think the answer is to stop pretending that confident human assertion is a categorically different phenomenon. The cognitive science literature calls what I watched on the exhibition floor confabulation: the generation of false information without intent to deceive, driven by a compulsion to produce fluent, confident, gap-free output. In clinical settings, it’s sometimes called “honest lying.” The confabulator believes what they’re saying. They’re not trying to trick you. They just can’t tolerate the silence where “I don’t know” should go.

We have constructed a double standard. We demand that AI systems achieve a level of epistemic honesty that we do not require of the humans who sell those systems, who buy those systems, or who have been practicing the oldest technology in the room without ever submitting it to the same evaluation.

When someone tells me AI isn’t ready because it hallucinates, I think about the solutions consultant who told me no models can watch video. I think about the vendor who thought a public defender needs CJIS compliance. I think about every closing argument I’ve ever heard that was more confident than its evidentiary foundation justified.

And I wonder what we’re really afraid of. I don’t think it’s that AI makes things up. I think it’s that AI makes the making-things-up visible — and once you see it in the machine, it gets very hard not to see it everywhere else.