About the search

The search covers the Vespa documentation site, blog, PyVespa docs, and Vespa.ai. It offers autocomplete suggestions drawn from AI-generated example questions stored alongside each document.

The available pages are regularly updated by a web crawler. Each page is split into chunks and embedded. Chunk embeddings are stored in Vespa alongside the raw text, document title and source. The results link directly to the most relevant chunk for each page.

Embeddings, text and title are all used by Vespa for ranking the results. Each query uses hybrid scoring that combines exact word match (BM25) and semantic similarity.

Ranking

Candidates are retrieved by combining approximate nearest-neighbour search over chunk embeddings with traditional BM25 keyword matching. Every chunk then receives a hybrid score: 40% cosine similarity between the chunk and query embedding, and 60% lexical relevance. The lexical component is a BM25 blend over the document title (30%), body (60%) and URL (10%), plus a small completeness bonus when the query terms match headings. This means a chunk inherits part of its score from how relevant the document as a whole is to the query. BM25 values are arctan-scaled to keep scores in a stable range regardless of document length.

A document's final score is 70% best-chunk score and 30% average of its top-10 chunk scores. This rewards pages with at least one closely matching passage while giving a lift to documents that are broadly relevant across many chunks.

Indexing pipeline