Elasticsearch

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It belongs beside the system of record when a read path needs relevance-ranked full-text search, faceting, geospatial predicates, or aggregations over indexed events. Applications write JSON documents, mappings define field types, analyzers turn text into terms, and Lucene stores those terms and document references in immutable segments.

Elasticsearch is normally a derived read store. If the cluster is lost, an ingestion pipeline should be able to rebuild it from PostgreSQL, an object store, or a retained log. Treating the index as authoritative changes the backup, durability, concurrency, and recovery problem substantially.

Mapping and Analysis

A mapping decides how each field is indexed and queried. text fields are analyzed for full-text search. keyword fields retain one exact value for filters, sorting, and aggregations. Numeric, date, boolean, geo_point, nested, and vector fields each carry different storage and query behavior. Dynamic mapping helps during exploration, but an accidental field shape or unbounded set of field names can become a production mapping explosion.

PUT products
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "name": { "type": "text", "fields": { "raw": { "type": "keyword" } } },
      "tenant_id": { "type": "keyword" },
      "price": { "type": "scaled_float", "scaling_factor": 100 },
      "updated_at": { "type": "date" },
      "location": { "type": "geo_point" }
    }
  }
}

An analyzer runs a character-filter, tokenizer, and token-filter pipeline at index time. Elasticsearch uses the same analyzer for search unless the field defines a separate search analyzer. For "Running Shoes", a standard analyzer may emit running and shoes. A language analyzer may also remove stop words or stem terms. Analyzer changes alter the indexed term space, so they usually require a new index and reindexing behind an alias rather than an in-place mapping edit.

Segments, Refresh, and Shards

Each Elasticsearch index has primary shards. Every primary is one Lucene index. A document ID is routed to one primary shard, indexed there, then copied to configured replica shards. More shards add parallelism and distribution options, but every shard also consumes heap, file handles, metadata, recovery time, and merge capacity. Shard count follows measured data size, indexing rate, recovery targets, and node capacity—not node count alone.

Lucene segments are immutable. New and updated documents first enter in-memory indexing buffers and the transaction log. A refresh opens newly written segments for search, which is why Elasticsearch is near real-time rather than instantaneously searchable. A flush commits Lucene state and starts a new transaction-log generation. Background merges combine segments and discard obsolete document versions. Aggressive refreshes create many small segments and increase merge pressure.

Operation	What it changes	Failure/cost boundary
Index document	Buffer, translog, and eventually a new segment	Acknowledgement depends on primary/replica write settings, not refresh visibility
Refresh	Makes recent segments searchable	Lower interval improves freshness but raises segment and merge overhead
Merge	Rewrites immutable segments	Consumes CPU, disk bandwidth, and temporary disk space
Replica recovery	Copies shard history/state to another node	Large or numerous shards extend recovery and rebalance time

Lucene uses immutable segments and specialized term dictionaries and postings structures. Its segment lifecycle resembles append-and-merge storage, but the term dictionary itself is not an LSM tree.

Query, Filter, and Aggregation

Query context computes a relevance score, for example for a match query over analyzed text. Filter context evaluates a yes/no condition without scoring, as with tenant, status, or date predicates. Exact constraints belong in filter context. Aggregations bucket and summarize the matching documents, but high-cardinality terms, large bucket counts, and cross-shard reduction can consume substantial heap and network bandwidth.

GET products/_search
{
  "size": 20,
  "query": {
    "bool": {
      "must": [{ "match": { "name": "running shoes" } }],
      "filter": [
        { "term": { "tenant_id": "t9" } },
        { "range": { "price": { "lte": 15000 } } }
      ]
    }
  },
  "aggs": {
    "price_bands": { "histogram": { "field": "price", "interval": 2500 } }
  }
}

The tenant filter is a correctness boundary, not only a performance hint. Every search, aggregation, autocomplete, and export path must apply it, or documents can leak across accounts.

Use Cases

Near-real-time and product-specific features

Elasticsearch search visibility follows refresh, so the visual’s “real-time” label means near real-time. Flink, Beats, Logstash, Kibana, machine-learning, and SIEM capabilities are separate components or licensed features whose availability and subscription terms must be checked for the deployed distribution and version.

Use case	Access pattern that earns Elasticsearch	Pipeline and freshness	Cost to accept
Product or knowledge search	Relevance, stemming, synonyms, facets, typo tolerance	CDC/outbox or batch rebuild. Seconds of index lag may be visible	Analyzer/mapping evolution, reindexing, and relevance tuning
Logs and events	Time-bounded filtering, free-text investigation, and aggregations	Data stream with rollover and lifecycle retention	High ingest/storage volume, mapping/cardinality control, and tier management
Operational analytics	Dashboards over recent indexed events	Refresh interval defines visibility	Aggregation heap, shard fan-out, and sampled/pre-aggregated alternatives
Geospatial discovery	Bounding, distance, and shape queries	Application events index geo_point/geo_shape fields	Specialized mappings and expensive broad geometry queries
Security analytics	Search and correlation over normalized security events	Retained ingestion plus rules and case workflow	Sensitive-data controls, long retention, and feature licensing
Anomaly detection	Time-series feature jobs over indexed data	Model/job cadence adds another freshness boundary	Licensed capabilities, model operations, and false-positive review

PostgreSQL full-text search fits when one relational data set needs modest search and transactionally current results. Elasticsearch fits when relevance, language analysis, faceting, log/event scale, or geospatial search justifies a separate derived system. A rebuild path is needed either way.

Operational Boundaries

• Mapping explosion: unbounded dynamic fields consume cluster state and heap. Explicit templates, dynamic: strict where practical, and flattening of truly arbitrary key/value payloads keep the field set bounded.
• Oversharding: many tiny shards waste heap and make recovery slow. Rollover creates a new write index when size or age conditions set from measured data are met. Consolidating or removing cold data is a separate operation.
• Refresh pressure: calling _refresh after every write creates small segments and merge load. Refresh-on-demand fits only bounded workflows that truly need it.
• Unbounded aggregations: high-cardinality terms queries can exhaust memory. Bounded bucket counts, composite pagination, or pre-aggregation limit that memory use.
• Disk watermarks: merge and recovery need free space. A cluster near disk capacity can stop allocating shards or block writes before raw bytes reach 100%.
• Schema changes: field types generally cannot be changed in place. The change path is a new versioned index, a reindex, validation, and then an alias switch.

Questions

Why might Elasticsearch map one logical value as both text and keyword?

Full-text search and exact-value operations need different index forms, and one field type serves only one of them well. A multi-field indexes the same source value both ways, so each query can use the form that matches its intent.

• text passes the value through an analyzer that splits and normalizes it into terms. A standard analyzer may index Running Shoes as running and shoes, so a search for shoes matches. The index holds separate normalized terms instead of the whole original value, so text is a poor basis for exact filters, sorting and aggregations.
• keyword indexes the whole value unchanged. A term filter, a sort or an aggregation such as a facet count works on the exact value Running Shoes. A full-text search for shoes does not match it.

{
  "properties": {
    "name": { "type": "text", "fields": { "raw": { "type": "keyword" } } }
  }
}

With this mapping, full-text queries target name, while exact filters, sorting and aggregations target name.raw.

Elasticsearch: text fields · Elasticsearch: keyword fields

Why can a newly indexed Elasticsearch document be missing from search results for a short time?

Elasticsearch search is near real-time. A document becomes searchable only after a refresh opens the segment that contains it, so an acknowledged write can be stored before it appears in search results.

1. Index: the document enters an in-memory indexing buffer and the transaction log on its shard. Acknowledgement depends on the primary and replica write settings, not on search visibility.
2. Refresh: on a periodic interval, or when explicitly requested, buffered documents become a new immutable Lucene segment that is opened for search.
3. Merge: background merges combine small segments and discard obsolete document versions.

A shorter refresh interval reduces the delay but creates more small segments and more merge work. That work consumes CPU, disk bandwidth and temporary disk space. Forcing a refresh after every write is therefore a costly default and fits only bounded workflows that truly need immediate visibility.

Elasticsearch near-real-time search · Elasticsearch refresh parameter · Elasticsearch translog settings

References

• Elasticsearch reference