Operating a Cluster: Health, Sizing, Lifecycle, Snapshots

1. 1

   Read cluster health and explain yellow

   On a single node, any index with replicas is yellow: a replica may never live on the same node as its primary. _cluster/allocation/explain tells you why a shard is unassigned, and is the first command in any red/yellow incident.

   terminalcopy cmd

   $ curl -s -XPUT localhost:9200/logs-test -H 'content-type: application/json' -d '{"settings":{"number_of_replicas":1}}' > /dev/null

   curl -s 'localhost:9200/_cluster/health?pretty' | grep -E 'status|unassigned_shards"'

   curl -s localhost:9200/_cluster/allocation/explain -H 'content-type: application/json' -d '{"index":"logs-test","shard":0,"primary":false}' | jq -r '.node_allocation_decisions[0].deciders[0].explanation'

   ── expected output ──

   "status" : "yellow",

   "unassigned_shards" : 1,

   a copy of this shard is already allocated to this node [[logs-test][0], node[...], [P], s[STARTED]]

2. 2

   The _cat APIs: your dashboard in a terminal

   Human-readable views of nodes, shards, indexes, and disk. ?v adds headers and s= sorts.

   terminalcopy cmd

   $ curl -s 'localhost:9200/_cat/allocation?v'

   curl -s 'localhost:9200/_cat/indices?v&s=store.size:desc&h=health,index,pri,rep,docs.count,store.size' | head -4

   ── expected output ──

   shards disk.indices disk.used disk.avail disk.total disk.percent host node

   4 12.4mb 41.2gb 58.7gb 99.9gb 41 172.17... 3f1c...

   health index pri rep docs.count store.size

   yellow logs-test 1 1 0 208b

   green products 1 0 3 9.8kb

3. 3

   An ISM policy: hot → warm → delete

   OpenSearch's ISM syntax (Elasticsearch ILM is similar in spirit, different in JSON). The ism_template attaches the policy to every new index matching logs-shop-*.

   {
     "policy": {
       "description": "ShopLite logs: rollover daily, warm after 3d, delete after 30d",
       "default_state": "hot",
       "states": [
         { "name": "hot",
           "actions": [ { "rollover": { "min_size": "30gb", "min_index_age": "1d" } } ],
           "transitions": [ { "state_name": "warm", "conditions": { "min_index_age": "3d" } } ] },
         { "name": "warm",
           "actions": [ { "read_only": {} }, { "force_merge": { "max_num_segments": 1 } }, { "replica_count": { "number_of_replicas": 1 } } ],
           "transitions": [ { "state_name": "delete", "conditions": { "min_index_age": "30d" } } ] },
         { "name": "delete", "actions": [ { "delete": {} } ] }
       ],
       "ism_template": [ { "index_patterns": ["logs-shop-*"], "priority": 100 } ]
     }
   }

   terminalcopy cmd

   $ curl -s -XPUT localhost:9200/_plugins/_ism/policies/logs-shop -H 'content-type: application/json' -d @logs-policy.json | jq -r ._id

   curl -s -XPUT localhost:9200/logs-shop-000001 -H 'content-type: application/json' -d '{"settings":{"plugins.index_state_management.rollover_alias":"logs-shop"},"aliases":{"logs-shop":{"is_write_index":true}}}' | jq .acknowledged

   ── expected output ──

   logs-shop

   true

4. 4

   Register an S3 snapshot repository and snapshot

   Needs the repository-s3 plugin (built into the managed services) and credentials via IAM role. Then snapshot on a schedule and test restores.

   terminalcopy cmd

   $ curl -s -XPUT localhost:9200/_snapshot/s3-backups -H 'content-type: application/json' -d '{"type":"s3","settings":{"bucket":"shoplite-search-snapshots","base_path":"prod"}}'

   curl -s -XPUT 'localhost:9200/_snapshot/s3-backups/snap-2026-09-27?wait_for_completion=true' | jq -r .snapshot.state

   ── expected output ──

   {"acknowledged":true}

   SUCCESS

Each drill is a real failure mode. Read the scenario and the output, decide what's wrong, then reveal the diagnosis.