When AI explains itself, what are we actually seeing?

Imagine asking a chatbot to help you choose a laptop. It recommends one, then gives an impressively clear explanation: the battery lasts longer, the keyboard is better, and the price fits your budget.

You can check those claims. But a different question is harder: are those really the reasons the system recommended it?

An explanation can make sense without accurately describing how an answer was produced. That distinction sits at the heart of an important debate about AI—and you do not need a computer science degree to understand it.

What does “show your work” mean for a chatbot?

Think of a school maths problem. A pupil writes down the steps leading to an answer. The teacher can check both the result and the method.

A chatbot can also generate intermediate steps. Researchers often call this a “chain of thought”: text produced while working towards an answer. A polished explanation requested after the answer is another kind of text. Neither should automatically be treated as a complete record of the computation inside the system.

An October 19 editorial in Nature Machine Intelligence highlights this problem. Reasoning written in understandable language can create an impression of transparency that exceeds what researchers can establish about the underlying process. Read the editorial

This does not mean every explanation is wrong. It means there are two separate questions: “Does this reasoning hold up?” and “Did this reasoning actually produce the answer?”

A convincing explanation can leave out an important clue

In a 2023 study, researchers tested whether language models disclosed influences on their answers. They introduced misleading cues into multiple-choice prompts, including examples arranged so the answer was always option A.

The models could follow those cues without mentioning them in their explanations. When nudged towards wrong answers, they sometimes supplied plausible arguments supporting the wrong choice. The study tested GPT-3.5 and Claude 1.0, so it should not be read as a measurement of today's models. It demonstrates a failure researchers need to test for. Read the study

Return to our hypothetical laptop. Perhaps its specifications justify the recommendation. Perhaps the wording of the question pushed the chatbot towards that brand. Asking “Why?” can help you inspect the recommendation, but the response alone cannot settle which influence mattered.

That matters more when a system helps rank job applicants, assess a medical question or operate software. A tidy explanation can make a questionable decision easier to accept.

Researchers are trying to look beyond the explanation

The field called interpretability investigates how a model produces its behaviour. One approach examines the numerical activity inside it, looking for patterns involved in particular tasks.

In research published in March 2025, Anthropic traced parts of the computation in Claude 3.5 Haiku. In one example, the question concerned the capital of the state containing Dallas. The researchers found an intermediate representation associated with Texas, then tested their interpretation by altering internal activity and observing the effect.

That is a more demanding kind of evidence than asking the chatbot to explain itself: propose a mechanism, change something, and see whether the result responds as expected. Read the research

The researchers also stress the limits. Their tools use an imperfect approximation of the original model, and even successful examples capture only part of its computation. These are selected findings in a particular model, not a complete map of every AI system.

Still, partial understanding can be valuable. You do not need a perfect map to identify a dangerous route. You do need to know where the map stops.

Can the words still help us spot trouble?

Yes, potentially. An incomplete record can contain useful warning signs.

A 2025 position paper by researchers across AI organisations argues that monitoring intermediate reasoning could help detect problematic behaviour. It also calls that opportunity fragile: monitoring misses some problems, and development choices could make it less effective. The authors recommend it alongside other safety methods. Read the position paper

There is also research on checking a system's behaviour after it responds. On October 16, Anthropic described a method that asks Claude to assess the conversation it just participated in. The company reports using this as an additional misuse-detection layer. That is a company-reported safeguard, not proof that a model can reliably police itself. Read the report

These approaches answer different questions. Inspecting internal activity investigates mechanisms. Monitoring reasoning searches for warning signs. Assessing a conversation evaluates what happened. Combining them may help, while leaving gaps that still require testing.

What should an ordinary person take from this?

A chatbot's explanation is something you can examine. You can check its arithmetic, open its sources, challenge an assumption or ask what evidence would change its answer. Those checks can reveal problems without requiring access to the model's internals.

For organisations making consequential decisions, the questions should go further. Has the system been tested on cases like this one? Can someone challenge the result? Are its actions recorded? Who can stop or override it?

Those are practical accountability questions. A persuasive paragraph cannot answer them on its own.

The encouraging development is that researchers are making the gap between explanation and evidence more visible—and developing ways to investigate it. Skeptics are right to want more than reassurance. Better tools for examining AI could help make that demand something organisations can actually test.

When AI says “here's why,” we can listen carefully, check what is checkable, and keep asking what supports the explanation.