Benchmark
Also: benchmarks
A standard test used to compare models. A high score on it does not mean the model is good at your task.
The fuller definition
A benchmark is a standardized task set and scoring approach used to compare systems under specified conditions.
Multiple-choice questions, general knowledge, clean text, scored automatically.
Scanned letters, conditional dates, two conflicting sources, scored by a lawyer.
The first number is real for the test it describes. It is not a prediction of the second. Run a small test on your own material before treating a headline score as evidence.
A concrete example
Several models answer the same defined test set and are scored under the same rules.
The distinction that matters
A headline score may not transfer to your workload, and contaminated or narrow tests can mislead.
Related terms
Evaluation (eval)
Testing an AI tool on a set of tasks to see how well it actually does.
Reliability & securityData leakage
Data reaching a place it should not. In model tests, it can also mean test answers have reached the training data.
Training & generationTest set
Examples held back from training so you can check how the model does on things it has not seen.
Where it is taught
Test an AI workflow on work you can assessSource and further reading
Anthropic · Evaluating AI agentsThe definition and example are written for this library. The link gives background, official documentation or legal authority as labelled. Legal scope depends on the jurisdiction and current text; product features depend on release and settings.