Different searches answer different needs

Keyword search finds terms and combinations of terms. Semantic search uses numerical representations to find passages with related meaning. Neither method establishes that a passage proves your proposition. [source]

Search for "termination" and a keyword system may miss "bring the agreement to an end". A semantic search may find that wording, but may also retrieve a passage about an unrelated relationship ending. Similarity is a candidate-selection signal.

What an embedding is

A map with three clusters of passages, labelled ending the contract, delivery dates and indemnity. A query point for "bring the agreement to an end" sits inside the ending-the-contract cluster. Below, the HyDE technique: a short question, a model-drafted pretend answer, and a search using that answer's embedding.
A simplified two-dimensional picture. Real embeddings have hundreds or thousands of dimensions that are not individually meaningful.

An embedding is a fixed-length list of numbers (a vector) that represents a piece of text. You can picture each number as a coordinate. Texts about similar things end up with similar coordinates, so they sit near each other. [source]

It is tempting to imagine that one number means "is about indemnity" and another means "is about delivery". That is a useful simplification but not literally true: the dimensions are learned automatically and are not individually interpretable without considerable effort. [source]

To search, the system:

  1. computes an embedding for every passage in the collection, in advance;
  2. computes an embedding for your question with the same embedding model;
  3. finds the passages whose embeddings are closest to the question's;
  4. places those passages, or summaries of them, in the model's context to answer from. [source] [source]

This is one of the most common ways to give a model more knowledge than fits in its context window, and it is the retrieval half of RAG.

A trick for short questions: write a pretend answer

This method is called HyDE. Very short questions can produce embeddings that carry little signal. "Who knew?" is close to many things. A technique from a December 2022 paper, Hypothetical Document Embeddings (HyDE), asks a model to write a plausible passage that would answer the question, then searches with the embedding of that invented passage instead. [source]

The invented passage may be factually wrong. That does not matter, because it is never shown or relied on; it only steers the search toward real passages that look like an answer. The trade-off is an extra model call per search. [source] [source] The lawyer's version of the same idea is familiar: search for the words a document proving the point would actually use.

Work a miniature search problem

Question: "Who knew about the revised delivery date before 12 March?"

A useful search plan includes exact names, document IDs, "revised schedule", "new date", and language of receipt or awareness. It also considers contrary evidence: "not received", "first saw", and references to an original schedule. Search is iterative because the record teaches you its vocabulary.

Use date and matter filters deliberately. A filter based on document date can exclude a later email that describes earlier knowledge. Whether that is appropriate depends on the question. Do not confuse the event period with the creation period of its evidence.

Combine retrieval with a second inspection

A retrieve-and-rerank system first gathers candidates, then applies another model to order them by relevance to the query. This can improve which candidates reach the answer stage, but it still depends on the first stage retrieving useful candidates. [source] Many products combine keyword and semantic results; see hybrid search.

A practical test set can include ten questions with known relevant passages. For each question, record whether a supporting passage appears in the first few results, whether contrary material appears, and whether the passage is complete enough to interpret. Do not evaluate search only by the quality of a generated answer: generation can hide a weak result list.

Watch chunk boundaries

A passage may contain the answer but omit the definition, condition, or exception immediately before it. Retrieve surrounding text where needed. Keep the source document and location attached to every passage. A chunk identifier that cannot be mapped back to the original is a poor final citation.

Improve the query before changing the system

When a known passage is missing, check extraction, filters, vocabulary, chunk boundaries, and indexing status. Only then decide whether a different embedding or reranker is the right repair. This sequence prevents an expensive model change from disguising a simple metadata mistake.