DEVBOOK

Elasticsearch

6 min read

Edit this pageReport / suggest

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It earns a place beside the system of record when the read path needs relevance-ranked full-text search, faceting, geospatial predicates, or aggregations over indexed events. The application writes JSON documents; mappings define field types, analyzers turn text into terms, and Lucene builds immutable segments whose inverted indexes map terms to matching document identifiers.

Elasticsearch is normally a derived read store. If the cluster is lost, an ingestion pipeline should be able to rebuild it from PostgreSQL, an object store, or a retained log. Treating the index as authoritative changes the backup, durability, concurrency, and recovery problem substantially.

Mapping and Analysis

A mapping decides how every field is indexed and queried. text fields are analyzed for full-text search; keyword fields retain one exact value for filters, sorting, and aggregations. Numeric, date, boolean, geo_point, nested, and vector fields each have different storage and query behavior. Dynamic mapping is convenient during exploration but can turn an accidental field shape or unbounded field names into a production mapping explosion.

PUT products
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "name": { "type": "text", "fields": { "raw": { "type": "keyword" } } },
      "tenant_id": { "type": "keyword" },
      "price": { "type": "scaled_float", "scaling_factor": 100 },
      "updated_at": { "type": "date" },
      "location": { "type": "geo_point" }
    }
  }
}

An analyzer runs a character-filter, tokenizer, and token-filter pipeline at index time and again at search time. For "Running Shoes", a standard analyzer may emit running and shoes; a language analyzer may also lowercase, remove stop words, or stem terms. Analyzer changes alter the indexed term space, so they usually require a new index and reindexing behind an alias rather than an in-place mapping edit.

Segments, Refresh, and Shards

Each Elasticsearch index has primary shards; every primary is one Lucene index. A document ID is routed to one primary shard, indexed there, then copied to configured replica shards. More shards increase parallelism and distribution options, but every shard has heap, file-handle, metadata, recovery, and merge cost. Choose shard count from measured data size, indexing rate, recovery target, and node capacity—not from node count alone.

Lucene segments are immutable. New and updated documents first enter in-memory indexing buffers and the transaction log. A refresh opens newly written segments for search, which is why Elasticsearch is near real-time rather than instantaneously searchable. A flush commits Lucene state and starts a new transaction-log generation. Background merges combine segments and discard obsolete document versions; aggressive refreshes create many small segments and increase merge pressure.

OperationWhat it changesFailure/cost boundary
Index documentBuffer, translog, and eventually a new segmentAcknowledgement depends on primary/replica write settings, not refresh visibility
RefreshMakes recent segments searchableLower interval improves freshness but raises segment and merge overhead
MergeRewrites immutable segmentsConsumes CPU, disk bandwidth, and temporary disk space
Replica recoveryCopies shard history/state to another nodeLarge or numerous shards extend recovery and rebalance time

The source learning visual is not embedded because it describes the term dictionary as an LSM tree. Lucene uses immutable segments and specialized term dictionaries/postings structures; the lifecycle resembles append-and-merge storage, but calling the term dictionary itself an LSM tree obscures the actual boundary.

Query, Filter, and Aggregation

Query context computes a relevance score, for example a match query over analyzed text. Filter context asks a yes/no question without scoring, for example tenant, status, or date predicates, and is the right default for exact constraints. Aggregations bucket and summarize the matching documents, but high-cardinality terms, large bucket counts, and cross-shard reduction can consume substantial heap and network bandwidth.

GET products/_search
{
  "size": 20,
  "query": {
    "bool": {
      "must": [{ "match": { "name": "running shoes" } }],
      "filter": [
        { "term": { "tenant_id": "t9" } },
        { "range": { "price": { "lte": 15000 } } }
      ]
    }
  },
  "aggs": {
    "price_bands": { "histogram": { "field": "price", "interval": 2500 } }
  }
}

The tenant filter is a correctness boundary, not only a performance hint. Every search, aggregation, autocomplete, and export path must apply it, or documents can leak across accounts.

Use Cases

Near-real-time and product-specific features

Elasticsearch search visibility follows refresh, so the visual’s “real-time” label means near real-time. Flink, Beats, Logstash, Kibana, machine-learning, and SIEM capabilities are separate components or licensed features whose availability and subscription terms must be checked for the deployed distribution and version.

Use caseAccess pattern that earns ElasticsearchPipeline and freshnessCost to accept
Product or knowledge searchRelevance, stemming, synonyms, facets, typo toleranceCDC/outbox or batch rebuild; seconds of index lag may be visibleAnalyzer/mapping evolution, reindexing, and relevance tuning
Logs and eventsTime-bounded filtering, free-text investigation, and aggregationsData stream with rollover and lifecycle retentionHigh ingest/storage volume, mapping/cardinality control, and tier management
Operational analyticsDashboards over recent indexed eventsRefresh interval defines visibilityAggregation heap, shard fan-out, and sampled/pre-aggregated alternatives
Geospatial discoveryBounding, distance, and shape queriesApplication events index geo_point/geo_shape fieldsSpecialized mappings and expensive broad geometry queries
Security analyticsSearch and correlation over normalized security eventsRetained ingestion plus rules and case workflowSensitive-data controls, long retention, and feature licensing
Anomaly detectionTime-series feature jobs over indexed dataModel/job cadence adds another freshness boundaryLicensed capabilities, model operations, and false-positive review

Use PostgreSQL full-text search when one relational data set needs modest search and transactionally current results. Use Elasticsearch when relevance, language analysis, faceting, log/event scale, or geospatial search justifies a separate derived system. Keep a rebuild path either way.

Operational Boundaries

  • Mapping explosion: unbounded dynamic fields consume cluster state and heap. Use explicit templates, dynamic: strict where practical, and flatten truly arbitrary key/value payloads.
  • Oversharding: many tiny shards waste heap and make recovery slow. Rollover by measured size/age and consolidate cold data.
  • Refresh pressure: calling _refresh after every write creates small segments and merge load. Use refresh-on-demand only for bounded workflows that truly need it.
  • Unbounded aggregations: high-cardinality terms queries can exhaust memory. Bound bucket counts, use composite pagination, or pre-aggregate.
  • Disk watermarks: merge and recovery need free space. A cluster near disk capacity can stop allocating shards or block writes before raw bytes reach 100%.
  • Schema changes: field types generally cannot be changed in place. Create a new versioned index, reindex, validate, then switch an alias.

Questions

References

Graph View

Backlinks