Python Curator for Elasticsearch

Posted on | Edited on | In

Recently I was working on optimizating deletion of ES stale indices and hot shard rebalancing.

Every weekend there is plenty of outdated index needs to be deleted, the heavy workload contributes to busy ES cluster along with Kafka topics lag, end up with PD alerts.

Also the lack of intelligence of elasticsearch could occasionally allocate lots of hot/active shards to one data node result in a hotspot with high CPU load average and IOPS.

1
2
3
4
5
6
7
# query hot shards distribution on data nodes
curl -s "localhost:9200/_cat/shards/<hot index pattern>?s=node:asc" > shards \
&& cat shards | awk {'print $8'} | uniq -c | sort -rn

# get hot shard number average
cat shards | awk {'print $8'} | uniq -c | sort -rn | \
awk 'BEGIN { sum = 0; count = 0 } { sum += $1; count += 1 } END { print sum / count }'

There are 2 basic Python modules can help:

ILM and Curator, Curator will not act on any index associated with an ILM policy without setting.

For writing your own specific curator, see this quick start, for instance to delete indices:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
import elasticsearch
import curator

es_host = "localhost:9200"
client = elasticsearch.Elasticsearch(hosts = es_host)

ilo = curator.IndexList(client)

# filter indices based on conditions
ilo.filter_by_regex(kind='regex',
                    value='^.*info.*_\d{2}\.\d{5}|^.*-\d{2}\.\d{5}')
ilo.filter_by_age(source='creation_date',
                  direction='older',
                  timestring='%Y.%m.%d',
                  unit='days',
                  unit_count=30)

delete_indices = curator.DeleteIndices(ilo)
delete_indices.do_action()

BTW, to list indices sorted by creation date on Kibana:

1
2
3
4
5
6
7
8
9
# view url parameters
GET _cat/indices?help

# specify columns and sort for all indices
GET _cat/indices?h=h,s,i,id,p,r,dc,dd,ss,creation.date.string&s=creation.date

# sort only specified indices
# the index list cannot be too long (<= 4096 bytes)
GET _cat/indices/[indices list]?s=creation.date

For using existing curator docker container, the curator CLI is ready to use with command syntax (the repo is organized bad…):

1
curator [--config CONFIG.YML] [--dry-run] ACTION_FILE.YML

There are the examples of config.yml and action.yml. I find the well formated and published document is here

K8s Cronjob is easy to have with docker container mentioned above (see others example), put config and action yaml file in configMap, BTW, to manually trigger cronjob for testing purpose:

1
kubectl create job --from=cronjob/<cj name> <cj name>-manual-0001

Some useful cronJob settings:

1
2
3
4
5
spec:
  # cron explain: https://crontab.guru/#15_6-23/2_*_*_*
  schedule: "15 3-23/2 * * *"
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 3
0%