The loop at the centre of every answer
A large language model takes a sequence of tokens (pieces of words, explained in the next lesson) and assigns a probability to every token that could come next. The system then chooses one of the likely candidates, appends it, and computes the next prediction using the longer sequence, usually reusing cached intermediate values. It keeps going until it produces a token that means "stop" or reaches a length limit. [source]
That is the whole engine. A reply to "Summarize the delivery dispute" is not retrieved from a filing cabinet of answers. It is built one token at a time, each token chosen because it is a likely continuation of everything before it, including the tokens the model itself just wrote.
Two consequences follow, and they matter for legal work:
- The model is always trying to continue the text plausibly. If the most plausible continuation of a question about a case is a citation, it will produce something shaped like a citation whether or not that case exists. Fabricated authorities are not a malfunction of the loop; they are the loop working as designed without a source to anchor it. See hallucination and fabricated citation.
- Choosing among likely candidates can involve chance. The step that picks a token is called sampling. Why the same question can produce different answers explains why the same question can therefore produce different answers on different runs.
Where the probabilities come from
The model learns those probabilities by training on very large collections of text. It does not store the text as documents; it adjusts billions of numerical parameters until its predictions match the patterns in that text well (How models learn from examples covers how).
Because the probabilities are learned from a particular body of text, the model is better at some kinds of writing than others. A model trained heavily on source code becomes very good at predicting code; one trained on little Canadian case law will predict Canadian legal writing less well. And because the model has learned what text usually looks like rather than what is true, it can generate statements that read as authoritative but are not grounded in any source. [source]
The model and the application are different things
When you use a chat product, you are not talking to a bare model. You are using an application that:
- adds hidden instructions before your message (What a prompt is covers these "system prompts");
- may attach files, search results, or earlier conversation;
- may let the model call tools such as search or a calculator;
- shows you only part of what the model received and produced.
The model writes from whatever text the application hands it. The application decides what that text is. A strong model can still answer from the wrong file, from an out-of-date search result, or from nothing at all if the upload failed silently. When an answer is wrong, the fault may lie with the model, with what the application supplied, or with both.
Surprising abilities, and an honest caveat
As models became better at prediction, they started doing things no one explicitly trained them for: translating, writing working code, following a new instruction from a single example. GPT-3's authors showed that a model scaled up roughly tenfold could perform many tasks from a few examples in the prompt, without task-specific retraining. [source]
Researchers called some of these jumps "emergent abilities". [source] Later work argued that at least some apparent jumps reflect how performance is measured rather than a sudden change in the model. [source] For a practitioner, the useful lesson is narrower: what a model can do is discovered by testing it on your task, not inferred from its size or its marketing.
What to check
Suppose a model tells you: "The supplier confirmed the revised delivery date of 20 March in its email of 12 March." Three different things could have produced that sentence:
- Supplied material. The application gave the model the 12 March email, and the model reported it accurately.
- Supplied material, misread. The email said "proposed, subject to inspection", and the model compressed it into "confirmed".
- Plausible continuation. No email was supplied at all, and the model produced a sentence that fits the pattern of the conversation.
The prose looks the same in all three cases. Only the source can tell them apart. So ask for the document and passage behind each material statement, open it, and compare. Why a persuasive answer can still be wrong develops this check.