Kibana: Clean Large Files and Indices in Elasticsearch

Before starting, you need to have a few prerequisites to execute the different commands.

Prerequisites

Curl: To interact with the Elasticsearch REST API.
Access to the Elasticsearch instance: You will need the URL, username, and password to access your Elasticsearch instance.

Objective
The objective is to delete indices that are either:

Larger than 500 MB.
Larger than 1 GB.

Automation Script
The following script lists the indices of your Elasticsearch instance and deletes those that exceed a certain size.

for index in $(curl -u "username:password" -s -X GET "https://url:9200/_cat/indices?h=index,store.size" --insecure | awk '$2 ~ /gb/ || $2+0 > 500 {print $1}'); do
  echo "Deleting index: $index"
  curl -u "username:password" -s -X DELETE "https://url:9200/$index" --insecure
done

Step-by-Step Explanation

List the indices with their sizes:
Use the curl command to get a list of indices and their sizes.

   curl -u "username:password" -s -X GET "https://url:9200/_cat/indices?h=index,store.size" --insecure

This command returns a table of indices and their sizes.

Filter large indices:
Use awk to filter indices whose size is greater than 500 MB or 1 GB.

   awk '$2 ~ /gb/ || $2+0 > 500 {print $1}'

This part of the script checks if the index size contains “gb” (gigabytes) or if it is larger than 500 MB.

Deletion loop:
For each filtered index, use curl to send a DELETE request to the Elasticsearch API.

   for index in $(...); do
     echo "Deleting index: $index"
     curl -u "username:password" -s -X DELETE "https://url:9200/$index" --insecure
   done

Explanation of Parameters

-u "username:password": Authentication to access Elasticsearch.
-s: Silent mode to suppress the progress bar.
-X GET: GET request to retrieve index information.
--insecure: Allows bypassing SSL certificate issues.

This simple and effective script allows you to automatically clean up large indices in your Elasticsearch instance. By running it regularly, you can maintain efficient disk space usage and optimize the performance of your Elasticsearch cluster.

For more complex tasks, you can extend this script or use more advanced Elasticsearch management tools like Curator.

See you soon on Tech To Geek.