On confabulation, contextual inquiry, and why almost no software will make a decision.
Ask a good coach for his progression rule and watch what happens.
He will say something generic. Add weight when it feels easy. Listen to your body. Autoregulate. He is not being evasive and he is not a fraud. He genuinely cannot retrieve it, because it is not stored in the form you asked for.
Now show him an athlete. Missed the top set twice at 110, fourth week of the block, three hard sessions on the mat, shoulder flared a fortnight ago. He answers instantly, precisely, with reasons. 105, rebuild, and pull the pressing.
Same man. Same knowledge. One question retrieves it and the other doesn't.
This is the most important unsolved problem in software right now, and it is not a machine learning problem. It's a psychology problem, and the discipline that solved it forty years ago is currently not in the room.
The easy observation, dispatched
Yes: most AI products observe rather than decide. Your squat volume is trending suboptimally. Your engagement has declined. Here is an insight, surfaced for your consideration.
And yes, that's useless. Everyone agrees within ten seconds of it being said, which is why it isn't worth an essay. The interesting question is why, given universal agreement, almost nothing decides.
The usual answer is liability. If the system decides and it's wrong, the system owns it. If it suggests, and you accept, and it goes badly — you chose. It was right there. A whole category has arranged itself around not holding the wheel and named the flinch user agency.
True. Still not the reason.
A system can only decide if it has something to decide with
And when you sit down to build one, you find the thing the industry has organised itself to avoid noticing. There is nothing to put inside it.
Expert judgment is procedural. It's encoded the way a tennis serve is encoded — in the doing, not in a retrievable description of the doing. Klein's work on how experts actually decide found that they don't weigh options at all; they recognise a situation, and the action arrives with the recognition. Ask them afterwards how they chose and you are asking them to narrate a process they never ran.
So they narrate a plausible one instead.
That's the part that should frighten anyone trying to encode expertise. Nisbett and Wilson showed half a century ago that people routinely lack access to the cognitive processes driving their own judgments — and, crucially, that they don't experience this as a blank. They confabulate. They produce a fluent, confident, entirely reasonable account of a decision process that did not occur.
The expert's self-report is not a lossy version of their judgment. It is a different artefact, generated on request, and it is often wrong.
Which means the standard approach to encoding expertise — sit the expert down and ask them to state their rules — doesn't merely fail. It fails while producing something that looks exactly like success. You leave the room with two pages of confident nonsense.
And then the model papers over it
Here is where it becomes genuinely dangerous, and it is new.
Ask a language model what to do about a missed top set and it tells you. Fluently. Reasonably. Periodisation, autoregulation, fatigue management, in complete sentences, with the assurance of something that has read everything and coached nobody.
It hands you the average of the internet. And the average is plausible. That's the whole problem. It is never obviously wrong — it's the middle of the distribution, and the middle of the distribution is precisely what a competent-looking product is made of.
So the absence of judgment stops presenting as an absence. The hole where somebody's opinion should have been now returns fluent prose. You can ship a coaching product without ever deciding how you coach. The gap doesn't announce itself until month six, when the thing turns out to have no view about anything and nobody can quite say why it feels thin.
They are all drawing from the same average, which is why they all feel the same. Not badly built. Unpopulated.
UX solved this. Nobody noticed
The oldest law in user research is: don't ask people what they do, watch them do it. Self-report is unreliable. Stated preference diverges from revealed preference. You go to where the work happens and you observe the work.
Contextual inquiry. Cognitive task analysis. Forty years of methodology built on precisely the finding that experts cannot tell you what they know, and that the way to get at it is to put a real case in front of them and watch the decision come out.
The entire AI industry is currently ignoring this and asking the expert to state their rules. Or worse, skipping the expert and asking the model.
Elicitation is a design problem, not an engineering one. It is the same problem UX has always had, wearing a new hat, and it is the reason so much of what ships is confidently vacant.
What it cost to actually do it
I've been building a strength coach — one that makes the call rather than surfacing an insight. And it forced the question, because a system that decides has to have something to decide with, and I couldn't keep pretending I'd supply it later.
I tried to write my rules down. Sat with a blank page, asked myself what my deload trigger was, and produced exactly the generic slop I'd have got from any coach. Deload when you need it. Useless. My own introspection was as unreliable as anyone's.
So I did the thing my training should have told me to do in the first hour. I went and found the cases.
Six months of coaching myself through a bad shoulder, logged in conversations. Not answers to what is your rule — actual decisions, made against actual situations, with the mat and the sleep and the missed reps all present. I read the whole lot back looking only for the moments where I made a call, and what fired it.
The rules fell out. Roughly two pages.
They were the hardest two pages I have written, and the work was not writing. It was finding out what I thought, which turns out to be a different activity from having thought it. Half had never been articulated. A quarter, written plainly, was obviously wrong and had to be fixed. The rest had been sitting in my hands for a decade, unexamined, because nothing had ever required it to be legible.
The design consequence, which is not optional
Say you get there. You have the judgment, extracted properly. The system can now decide.
It still has to earn the right, and there's a well-documented reason. People abandon an erring algorithm far faster than they abandon an erring human — they'll forgive a coach a bad call and delete an app for the same one. But the same research found something useful: give people even a small ability to adjust the algorithm, and they'll keep using it.
So the override is not a courtesy. It's the mechanism by which an automated decision becomes tolerable. And the explanation isn't decoration either — an unexplained decision is one you cannot calibrate your trust against, and untrusted automation gets switched off.
The product is therefore three things at once, and none of them is separable. It decides. It shows its reasoning in plain language. And it lets you overrule it, and learns something when you do.
Decision without explanation is tyranny. Explanation without decision is a dashboard. Neither without override survives its first mistake.
Which is the same argument, arriving from a different road
Production got free. Everyone has noticed that.
What free production exposed is the part nobody says: most people never had a view. Technical difficulty was doing enormous work, concealing the vacancy beneath it. Take it away and the vacancy is what's left — and the model will now fill it with something plausible, so that nobody ever has to look.
The scarce asset in software is not engineering, and it is not taste in the decorative sense. It is somebody having taken the trouble to find out what they actually believe — properly, from cases, against the grain of their own confabulating introspection — and having written it down in a form a machine can execute and be held to.
Two pages. It's the only thing worth owning.
Nobody wrote it down. That's the opportunity, and it's the work.
