What a token is
The basic unit a language model consumes is a token, not a word. A token may be a whole common word, part of a rarer word, a single digit, a space, or a punctuation mark. Tokens can also stand for things that are not letters at all, such as the boundary between one document and the next. [source]
A tokenizer is the component that splits text into tokens and converts each one into an ID number before the model sees it. Different model families use different tokenizers, so the same paragraph can count as a different number of tokens in different products. [source]
See it on real legal text
The figure shows two sentences split by a real tokenizer (OpenAI's o200k_base, through its open-source tiktoken library). [source]
- "The Respondent shall indemnify the Applicant forthwith." is 7 words and 11 tokens. "Respondent", "indemnify", and "forthwith" each split into two pieces because they are uncommon in general text.
- "Notwithstanding s. 137.1 of the Courts of Justice Act, R.S.O. 1990, c. C.43" is 13 words and 28 tokens. Punctuation, digit groups, and abbreviations affect the split; the exact boundaries depend on the tokenizer.
Why not just use words or letters?
You could, in theory, feed a model raw bits or single characters. The model would then have to learn from scratch that characters form words and words form phrases, and in practice it performs worse. Whole words have the opposite problem: no fixed vocabulary can hold every word, name, and citation. Splitting into common pieces is the compromise: frequent words stay whole, rare words are assembled from parts. [source] [source]
Counting tokens
A widely used rule of thumb is that 1,000 tokens is about 750 words of ordinary English, or roughly four characters per token. [source] [source] For a quick estimate without a tokenizer, divide the character count by four.
Treat that as a rough estimate, not a guaranteed lower bound for legal material. The citation sentence above runs at about 2.7 characters per token. Statutes, pleadings with paragraph numbering, financial tables, and text in other languages can tokenize differently from plain English prose. A page count is not a token count: a dense 10-page schedule of figures can use more tokens than 25 pages of correspondence.
To measure rather than estimate, paste the text into a vendor's tokenizer tool or token-counting API for the model you are using. [source] Even then, vendors may add their own formatting tokens behind the scenes, so treat the result as close but not exact. [source]
You rarely need to think about tokens in everyday use, because applications take ordinary text and tokenize it for you. You do need to think about them in one situation: when material approaches the model's limit. That is the subject of the next lesson.
Attention links tokens to each other
Once text is tokenized, the transformer's attention mechanism calculates, for each token, how strongly every other token should influence it. That is how the model connects "it" to the right noun, or a defined term to its definition several paragraphs earlier. [source]
Attention is learned arithmetic. It is not human focus, and a high attention weight does not mean a fact is true or important. It means the model's training found that link useful for prediction.
Why wording still matters
Because the model links tokens by learned patterns, vague references make its job harder and your review harder too.
- "Compare it with the old one" leaves both documents unidentified. "Compare clause 7.2 of A02 (revised schedule, 12 March) with clause 7.2 of A01 (original schedule, 3 February)" names both.
- Give each source a short, unique label and use it consistently. Package source material shows how labels make citations checkable.
- Ask for the exact words from each version, not just a description of the difference.