Part 4

Search and Log Analytics: Elasticsearch, OpenSearch, ELK

How search engines index text, shards and replicas, cluster health, index lifecycle management, and building a log pipeline with Fluent Bit, Logstash, and Kibana/OpenSearch Dashboards.

ShopLite uses a search engine twice: product search ('red ceramic mug under 500') and centralised logs from every pod. Both run on Elasticsearch or its open-source fork OpenSearch (AWS's managed service uses OpenSearch). The APIs in this part work on both unless noted. Search clusters are powerful and notoriously easy to break: too many shards, full disks, and mapping explosions cause most of their outages.

1. 4.1

   Indexing, Mappings, Shards, and Queries

   Beginner

   The inverted index, documents and mappings (text vs keyword), analyzers, shards and replicas, near-real-time refresh, and the queries and aggregations you'll use daily.

   50 min · 5 lab steps · 2 drills

2. 4.2

   Operating a Cluster: Health, Sizing, Lifecycle, Snapshots

   Advanced

   Green/yellow/red and how to fix each, node roles, heap and shard sizing, disk watermarks, rollover with ILM/ISM (hot → warm → cold → delete), and snapshots to S3.

   55 min · 4 lab steps · 2 drills

3. 4.3

   The Log Pipeline: Fluent Bit, Logstash, Kibana, and Friends

   Intermediate

   ELK vs EFK vs Loki, shipping container logs with Fluent Bit, parsing with ingest pipelines or Logstash, buffering and backpressure, index templates for logs, mapping explosions, and dashboards.

   50 min · 4 lab steps · 2 drills