Unit 4.1 · Search and Log Analytics: Elasticsearch, OpenSearch, ELK

Indexing, Mappings, Shards, and Queries

The inverted index, documents and mappings (text vs keyword), analyzers, shards and replicas, near-real-time refresh, and the queries and aggregations you'll use daily.

Beginner 50 min 5 lab steps 2 failure drills

Start here

The mental model

A database finds rows by exact values. A search engine works like the index at the back of a book: for every WORD it lists the pages where it appears. To answer 'red mug', it looks up 'red' and 'mug', intersects the page lists, and ranks pages by how well they match. That structure, the INVERTED INDEX, is why search engines are fast at text and aggregations and bad at transactions.

An INDEX is a collection of JSON DOCUMENTS (like a table). It's split into SHARDS (independent mini-indexes, each a Lucene index) spread across nodes, and each shard has REPLICAS on other nodes for safety and read throughput.

Go deeper

How it works inside

01Analysis and the inverted index

When a document is indexed, each text field goes through an ANALYZER: a tokenizer splits it into terms ('Red Ceramic Mugs!' → red, ceramic, mugs) and filters lowercase, stem, or remove stop words (mugs → mug). The same analyzer runs on the search query, so 'MUG' matches 'mugs'. The terms are stored in the inverted index with the documents they appear in; relevance is scored with BM25 (term frequency, rarity across documents, and field length).

Analysis and the inverted indexdiagram

Rendering diagram…

02Mappings: text vs keyword

The MAPPING is the schema: each field's type. text is analyzed for full-text search. keyword is stored as one exact term, used for filters, sorting, and aggregations (status, SKU, log level, Kubernetes namespace). Numbers, dates, booleans, ip, and geo_point have their own types. DYNAMIC MAPPING guesses types from the first document it sees, which is convenient and dangerous: a field first seen as "42" becomes text/keyword, and later numeric range queries don't work. Once a field is mapped, its type can't change without REINDEXING into a new index. Define explicit mappings (or index templates) for anything important.

03Shards, replicas, and near-real-time

The number of PRIMARY shards is fixed at index creation (changing it means _split/_shrink or reindexing); replicas can be changed any time. A document's shard is chosen by hashing its _id (or routing value). Searches fan out to one copy of every shard and merge results, so many tiny shards make every search slower and cost memory on every node (Unit 4.2 covers sizing).

Indexing is NEAR-REAL-TIME: new documents become searchable after the next REFRESH (every 1 s by default, only on indexes that were searched recently). A per-shard TRANSLOG makes writes durable before Lucene commits to disk. For bulk loads, raising refresh_interval (e.g. 30s, or -1 during a one-off load) dramatically speeds up indexing.

04Queries and aggregations

The query DSL has two contexts. QUERY context scores relevance (match, multi_match); FILTER context is yes/no, cached, and faster (term, range, exists). A bool query combines them: must (scored), filter (not scored), should (boosts), must_not. AGGREGATIONS compute analytics over matching documents: terms (top values), date_histogram (per minute/hour), avg/percentiles. They're how Kibana draws charts of errors per service over time.

Do it

Hands-on lab

1
Start a single-node OpenSearch
Security plugin disabled for the lab only. (Elasticsearch 9 works the same with docker.elastic.co/elasticsearch/elasticsearch:9.1.0 and xpack.security.enabled=false.)
terminal
$ docker run -d --name os -p 9200:9200 -e discovery.type=single-node -e DISABLE_SECURITY_PLUGIN=true -e OPENSEARCH_INITIAL_ADMIN_PASSWORD='Lab-Only-Pw1!' opensearchproject/opensearch:3
curl -s localhost:9200 | grep -E '"number"|distribution'
── expected output ──
"distribution" : "opensearch",
"number" : "3.2.0",
2
Create an index with an explicit mapping
name is full-text with a keyword sub-field for sorting; category and sku are exact; price is numeric.
terminal
$ curl -s -XPUT localhost:9200/products -H 'content-type: application/json' -d '{
"settings": { "number_of_shards": 1, "number_of_replicas": 0 },
"mappings": { "properties": {
"name": { "type": "text", "fields": { "raw": { "type": "keyword" } } },
"category": { "type": "keyword" },
"sku": { "type": "keyword" },
"price": { "type": "integer" },
"created": { "type": "date" } } } }'
── expected output ──
{"acknowledged":true,"shards_acknowledged":true,"index":"products"}
3
Bulk-index some products
The _bulk API takes newline-delimited JSON: an action line, then the document. Always bulk in real pipelines; one request per document is far slower.
terminal
$ cat > products.ndjson <<'EOF'
{"index":{"_index":"products","_id":"1"}}
{"name":"Red Ceramic Mug","category":"kitchen","sku":"MUG-RED","price":399,"created":"2026-09-01"}
{"index":{"_index":"products","_id":"2"}}
{"name":"Blue Mugs Set of 4","category":"kitchen","sku":"MUG-BLU4","price":1199,"created":"2026-09-10"}
{"index":{"_index":"products","_id":"3"}}
{"name":"Red Cotton T-Shirt","category":"clothing","sku":"TS-RED","price":599,"created":"2026-09-15"}
EOF
curl -s -XPOST 'localhost:9200/_bulk?refresh=true' -H 'content-type: application/x-ndjson' --data-binary @products.ndjson | grep -o '"errors":[a-z]*'
── expected output ──
"errors":false
4
Search: full text + filter + aggregation
'red mug' in the name (scored), price under 1000 (filter), plus a count per category. Note 'mug' matches 'Mugs' thanks to analysis.
terminal
$ curl -s localhost:9200/products/_search -H 'content-type: application/json' -d '{
"query": { "bool": {
"must": [ { "match": { "name": "red mug" } } ],
"filter": [ { "range": { "price": { "lt": 1000 } } } ] } },
"aggs": { "by_category": { "terms": { "field": "category" } } },
"_source": ["name","price"] }' | jq '{hits: [.hits.hits[] | {score: ._score, name: ._source.name}], categories: .aggregations.by_category.buckets}'
── expected output ──
{
"hits": [
{ "score": 1.2, "name": "Red Ceramic Mug" },
{ "score": 0.46, "name": "Red Cotton T-Shirt" }
],
"categories": [
{ "key": "clothing", "doc_count": 1 },
{ "key": "kitchen", "doc_count": 1 }
]
}
5
See the analyzer at work
_analyze shows exactly which terms a string becomes. It's the first tool when 'search doesn't find X'.
terminal
$ curl -s localhost:9200/_analyze -H 'content-type: application/json' -d '{"analyzer":"english","text":"Blue Mugs Set of 4"}' | jq -c '[.tokens[].token]'
── expected output ──
["blue","mug","set","4"]

Operate it

Knobs that matter

Setting	Default	What it does	When to change it
number_of_shards	1	Primary shards (fixed at creation).	Aim for shards of roughly 10–50 GB; use rollover for growing data rather than many shards up front.
number_of_replicas	1	Copies of each primary.	1 in production (2 for critical search); 0 temporarily during a big initial load.
refresh_interval	1s	How often new docs become searchable.	30s for logs (much faster indexing); -1 during bulk loads, then restore.
dynamic (mapping)	true	Auto-add new fields.	`strict` for product data; `false` or templates with limits for logs (see Unit 4.3).

3am practice

Failure drills

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.

Drill #1

Filter on status returns nothing

A dashboard filters orders with {"term": {"status": "Shipped"}} and gets zero hits, although documents clearly contain "status": "Shipped".

terminal

$ curl -s localhost:9200/orders/_mapping | jq '.orders.mappings.properties.status'

── what you'll see ──

{ "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }

Drill #2

Can't change a field type

price was dynamically mapped as text in the live index. You try to update the mapping to integer.

terminal

$ curl -s -XPUT localhost:9200/products_v1/_mapping -H 'content-type: application/json' -d '{"properties":{"price":{"type":"integer"}}}' | jq -r .error.reason

── what you'll see ──

mapper [price] cannot be changed from type [text] to [integer]

The bigger picture

Connects to

System Design · System 12.20 — Search Engine

The one HLD that's genuinely different: crawling, indexing, ranking, and serving billions of queries.

System Design · Sharding

Partitioning across separate physical databases — the real 'put more money in the machine' of writes.

Unit 0.1 · Replication

Primary and replica shards follow the leader-follower model.

Observability · Parse what you didn't write

Turning unstructured log lines into fields, which is the same job as mappings and ingest pipelines.

Part 1 · PostgreSQL

Why the database stays the system of record and search is a derived copy.

Prove it

Interview questions

What's an inverted index?

text vs keyword?

How do you change the mapping of a field in production?

0/3 · 0%

Command Palette