Clean up Bad Indices

Posted on | In

The ES became red due to multiple data nodes got deleted accidentally, the data loss was irreversible as 99% of the indices was set with only 1 replica, and both primary and replica were gone.

To bring the cluster to green again, we needed to deleted the bad unassigned shards which were marked as unassigned_shards from GET _cluster/health response.

First, we need to find the target shards which were reasoned by NODE_LEFT:

1
2
curl -s "http://localhost:9200/_cat/shards?h=index,prirep,unassigned.reason" \
| grep NODE_LEFT

You can filter more to exclude system indices or others.

Set 0 replica

This only works for replica shard, we just set 0 replica of the index:

1
2
3
4
5
6
curl -XPUT "http://localhost:9200/<index name>/_settings" \
-H 'Content-Type: application/json' \
-d \
'{
  "index.number_of_replicas": 0
}'

Delete index

If the unassigned shard is primary type, we need to delete the whole index of the shard, this usually mean this index is old, so no writing on it:

1
curl -XDELETE "http://localhost:9200/<index name>"

Delete data stream

If the unassigned primary shard is from writing index of the data stream, we need to delete the data stream, usually the data stream only contains this single bad index:

1
curl -XDELETE "http://localhost:9200/_data_stream/<data stream name>"
0%