Search System

A search system separates document acquisition from query serving. Crawling or ingestion discovers content; processing creates canonical documents and index terms; serving retrieves candidates and ranks them. Freshness, recall, latency, and index cost pull the design in different directions.

Crawl and Index Pipeline

A crawl frontier stores normalized URLs with host-level politeness and retry state. Fetchers obey access policy, content limits, and per-host budgets. Canonicalization removes fragments and normalizes known equivalents, but a canonical tag is evidence rather than permission to discard content blindly. Keep the raw fetch hash and chosen canonical ID so duplicate decisions are auditable.

Document processing extracts text, language, fields, links, and security labels. An inverted index maps a term to postings such as (document_id, field, frequency, positions). For example:

retry -> [(doc_7, title, 1), (doc_12, body, 4)]

Shard by document ID for balanced writes, then query every relevant shard and merge top candidates. Replicas improve read capacity and availability; they do not remove the need for a consistent index version during rollout.

The visual shows the high-level stages. Real systems also need crawl policy, duplicate evidence, index-version rollout, distributed top-k merging, and ranking evaluation.

Query Serving and Ranking

Parse and normalize the query, apply spelling or synonym rules with versioned dictionaries, retrieve candidates, enforce access filters, score, and return an index version with the response. Cache only after including tenant, locale, permissions, query rules, and index version in the key; otherwise a cache can leak results across policy boundaries.

Evaluate retrieval separately from ranking. Recall asks whether relevant documents entered the candidate set. Ranking metrics such as NDCG evaluate order. Clicks are biased by position and presentation, so offline labels and controlled experiments still matter.

Incremental indexing lowers freshness delay but creates more small segments and merge work. Batch rebuilds are simpler and reproducible but stale. Most systems combine an immutable base index with a small fresh tier, then compact.

References

• Google Search Central crawling documentation — official crawler discovery, fetching, rendering, indexing, and canonicalization boundaries.
• Apache Lucene index file formats — official segment, postings, stored-field, and index metadata structures.
• Introduction to Information Retrieval — Stanford reference for inverted indexes, scoring, evaluation, and distributed retrieval.
• ByteByteGo: how search engines work — provenance for the crawl, index, query, ranking, and feedback topology.