check_es_system

Last update: September 27, 2019

This is a monitoring plugin to check the status of an ElasticSearch cluster node. Besides the classical status check (green, yellow, red) this plugin also allows to monitor disk or memory usage of Elasticsearch. This is especially helpful when running Elasticsearch in the cloud (e.g. Elasticsearch as a service) because, as ES does not run on your own server, you cannot monitor the disk or memory usage. This is where this plugin comes in. Just tell the plugin how much resources (diskspace, memory capacity) you have available (-d) and it will alarm you when you reach a threshold.

Download

Download check_es_system.sh

check_es_system.sh

1464 downloads so far...

Download plugin and save it in your Nagios/Monitoring plugin folder (usually /usr/lib/nagios/plugins, depends on your distribution). Afterwards adjust the permissions (usually chmod 755).

Community contributions welcome on GitHub repo.

Version history / Changelog

20160429: Started programming plugin
20160601: Continued programming. Working now as it should =)
20160906: Added memory usage check, check types option (-t)
20160906: Renamed plugin from check_es_store to check_es_system
20160907: Change internal referenced variable name for available size
20160907: Output now contains both used and available sizes
20161017: Add missing -t in usage output
20180105: Fix if statement for authentication (@deric)
20180105: Fix authentication when wrong credentials were used
20180313: Configure max_time for Elastic to respond (@deric)
20190219: Fix alternative subject name in ssl (issue 4), direct to auth
20190220 1.1: Added status check type
20190403 1.2: Check for mandatory parameter checktype, adjust help
20190403 1.3: Catch connection refused error
20190426 1.4: Catch unauthorized (403) error
20190626 1.5: Added readonly check type
20190905 1.5.1: Catch empty cluster health status (issue #13)
20190909 1.6: Added jthreads and tps (thread pool statistics) check types, handle correct curl return codes
20190924 1.6.1: Missing 'than' in tps output

Requirements

  • curl (SUSE: zypper in curl, Debian/Ubuntu: apt-get install curl, CentOS/RHEL: yum install curl)
  • jshon command (SUSE: search for jshon, Debian/Ubuntu: apt-get install jshon
  • Bash internal commands/functions (plugin checks for its existance)

Definition of the parameters

Parameter Description
-H* Hostname or ip address of ElasticSearch Node
-P Port (defaults to 9200)
-S Use secure http (https)
-u Username if authentication is required
-p Password if authentication is required
-t* Type of check to run (disk|mem|status)
-d+ Available disk or memory size (ex. 20)
-o Size unit (K|M|G) (defaults to G)
-i Space separated list of indexes to be checked for readonly (default: '_all')
-w Warning threshold
Threshold format for 'disk' and 'mem': int (for percent), defaults to 80 (warn) and 95 (crit)
Threshold format for 'tps': int,int,int (active, queued, rejected), no defaults
Threshold format for all other check types': int, no defaults
-c Critical threshold
Threshold format for 'disk' and 'mem': int (for percent), defaults to 80 (warn) and 95 (crit)
Threshold format for 'tps': int,int,int (active, queued, rejected), no defaults
Threshold format for all other check types': int, no defaults
-m Maximum time in seconds to wait for Elasticsearch server response (default: 30)
-h Help!

* mandatory parameter

+ mandatory for check types disk and mem

Definition of the check types

Type Description
status Check the current status of the cluster (green, yellow, red). Besides that shows additional information (nodes, shards, docs). When the status is yellow or red, the relevant shard information is shown (e.g. initializing or unassigned)..
mem Check the current memory usage and compare it with the available memory defined with -d parameter. Thresholds possible.
disk Check the current disk usage and compare it with the available disk capacity defined with -d parameter. Thresholds possible.
readonly Check all (default: _all) or index(es) listed with -i parameter for read-only flag.
jthreads Monitor the number of Java threads across the ES cluster. Thresholds possible.
tps Monitor the thread pool statistics across ES cluster. For each thread pool the 'active', 'queued' and 'rejected' queues are checked. A high and increasing number of certain queues can indicate problems with your Elasticsearch cluster. Thresholds possible.

Usage / running the plugin on the command line

Usage:

./check_es_system.sh -H hostname [-P port] [-S] [-u user] [-p password] -t checktype [-d capacity] [-o unit] [-i index(es)] [-w warning] [-c critical] [-m time]

Example 1: Classical status check. Here the Elasticsearch cluster runs on escluster.example.com and is accessed using HTTPS (-S to enable https) on port 9243 using the basic auth credentials user and password. The output shows the cluster status (green) and some additional information.
As performance data the node numbers, shard information and total number of documents are used.
Note: When the status changes to yellow (= WARNING) or red (=CRITICAL) the output also contains information of relocating, initializing and unassigned shards. The performance data stays the same to not confuse graphing databases.

./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t status
ES SYSTEM OK - Elasticsearch Cluster is green (3 nodes, 2 data nodes, 114 shards, 8426885 docs)|total_nodes=3;;;; data_nodes=2;;;; total_shards=114;;;; relocating_shards=0;;;; initializing_shards=0;;;; unassigned_shards=0;;;; docs=8426885;;;;

Example 2: Disk usage check. The same Elasticsearch cluster as before is accessed. As this ES runs in the cloud, we do not have host monitoring available (meaning: We cannot do file system monitoring). But we know, we have 128GB available disk space. We tell the plugin we have a capacity of 128GB (-d 128). Let the plugin do the rest:

./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t disk -d 128
ES SYSTEM OK - Disk usage is at 14% (18 G from 128 G)|es_disk=19637018938B;109951162777;130567005798;;

Example 3: Memory usage check. Same as before, ES runs in the cloud and we cannot do memory monitoring on the host itself. But we have booked our Elasticsearch service with 24GB RAM/memory.

./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t mem -d 24
ES SYSTEM OK - Memory usage is at 58% (14 G from 24 G)|es_memory=15107616304B;20615843020;24481313587;;

Example 4: Readonly index check. The plugin checks the mentioned indexes, using the -i parameter, for a read-only flag. If no -i parameter was used, all indexes will be checked.

./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t readonly -i "filebeat- *logstash-*"
CRITICAL - Elasticsearch Index filebeat- *is read-only (found 53 index(es) set to read-only) Elasticsearch Index logstash-* is read-only (found 125 index(es) set to read-only)

Example 5: JVM Threads check. The plugin checks the number of JVM threads across the cluster. The plugin should warn when 200 or more threads are running, critical at 300 or more threads.

./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t jthreads -w 200 -c 300
ES SYSTEM CRITICAL - Number of JVM threads is 319|es_jvm_threads=319;200;300;;

Example 6: TPS (Thread Pool Statistics). The plugin goes through all the detected thread pools of the cluster. Without thresholds, the plugin just outputs the number of detected thread pools and adds performance data. With thresholds (note the special format!), the plugin will alert if one of the thread pools is equal to or above a threshold.

./check_es_system.sh -H escluster.example.com -P 9243 -S -u user -p password -t tps -w 200 -c 300
ES SYSTEM OK - Found 46 thread pools in cluster|tp_es02-analyze_active=0;10;50;; tp_es02-analyze_queue=0;50;200;; tp_es02-analyze_rejected=0;1000;2000;; tp_es02-ccr_active=0;10;50;; tp_es02-ccr_queue=0;50;200;; tp_es02-ccr_rejected=0;1000;2000;; tp_es02-fetch_shard_started_active=0;10;50;; tp_es02-fetch_shard_started_queue=0;50;200;; tp_es02-fetch_shard_started_rejected=0;1000;2000;; tp_es02-fetch_shard_store_active=0;10;50;; tp_es02-fetch_shard_store_queue=0;50;200;; tp_es02-fetch_shard_store_rejected=0;1000;2000;; tp_es02-flush_active=0;10;50;; tp_es02-flush_queue=0;50;200;; tp_es02-flush_rejected=0;1000;2000;; tp_es02-force_merge_active=0;10;50;; tp_es02-force_merge_queue=0;50;200;; tp_es02-force_merge_rejected=0;1000;2000;; tp_es02-generic_active=0;10;50;; tp_es02-generic_queue=0;50;200;; tp_es02-generic_rejected=0;1000;2000;; tp_es02-get_active=0;10;50;; tp_es02-get_queue=0;50;200;; tp_es02-get_rejected=0;1000;2000;; tp_es02-index_active=0;10;50;; tp_es02-index_queue=0;50;200;; tp_es02-index_rejected=0;1000;2000;; tp_es02-listener_active=0;10;50;; tp_es02-listener_queue=0;50;200;; tp_es02-listener_rejected=0;1000;2000;; tp_es02-management_active=1;10;50;; tp_es02-management_queue=0;50;200;; tp_es02-management_rejected=0;1000;2000;; tp_es02-ml_autodetect_active=0;10;50;; tp_es02-ml_autodetect_queue=0;50;200;; tp_es02-ml_autodetect_rejected=0;1000;2000;; tp_es02-ml_datafeed_active=0;10;50;; tp_es02-ml_datafeed_queue=0;50;200;; tp_es02-ml_datafeed_rejected=0;1000;2000;; tp_es02-ml_utility_active=0;10;50;; tp_es02-ml_utility_queue=0;50;200;; tp_es02-ml_utility_rejected=0;1000;2000;; tp_es02-refresh_active=1;10;50;; tp_es02-refresh_queue=0;50;200;; tp_es02-refresh_rejected=0;1000;2000;; tp_es02-rollup_indexing_active=0;10;50;; tp_es02-rollup_indexing_queue=0;50;200;; tp_es02-rollup_indexing_rejected=0;1000;2000;; tp_es02-search_active=0;10;50;; tp_es02-search_queue=0;50;200;; tp_es02-search_rejected=0;1000;2000;; tp_es02-search_throttled_active=0;10;50;; tp_es02-search_throttled_queue=0;50;200;; tp_es02-search_throttled_rejected=0;1000;2000;; tp_es02-security-token-key_active=0;10;50;; tp_es02-security-token-key_queue=0;50;200;; tp_es02-security-token-key_rejected=0;1000;2000;; tp_es02-snapshot_active=0;10;50;; tp_es02-snapshot_queue=0;50;200;; tp_es02-snapshot_rejected=0;1000;2000;; tp_es02-warmer_active=0;10;50;; tp_es02-warmer_queue=0;50;200;; tp_es02-warmer_rejected=0;1000;2000;; tp_es02-watcher_active=0;10;50;; tp_es02-watcher_queue=0;50;200;; tp_es02-watcher_rejected=0;1000;2000;; tp_es02-write_active=8;10;50;; tp_es02-write_queue=10;50;200;; tp_es02-write_rejected=0;1000;2000;; tp_es01-analyze_active=0;10;50;; tp_es01-analyze_queue=0;50;200;; tp_es01-analyze_rejected=0;1000;2000;; tp_es01-ccr_active=0;10;50;; tp_es01-ccr_queue=0;50;200;; tp_es01-ccr_rejected=0;1000;2000;; tp_es01-fetch_shard_started_active=0;10;50;; tp_es01-fetch_shard_started_queue=0;50;200;; tp_es01-fetch_shard_started_rejected=0;1000;2000;; tp_es01-fetch_shard_store_active=0;10;50;; tp_es01-fetch_shard_store_queue=0;50;200;; tp_es01-fetch_shard_store_rejected=0;1000;2000;; tp_es01-flush_active=0;10;50;; tp_es01-flush_queue=0;50;200;; tp_es01-flush_rejected=0;1000;2000;; tp_es01-force_merge_active=0;10;50;; tp_es01-force_merge_queue=0;50;200;; tp_es01-force_merge_rejected=0;1000;2000;; tp_es01-generic_active=0;10;50;; tp_es01-generic_queue=0;50;200;; tp_es01-generic_rejected=0;1000;2000;; tp_es01-get_active=0;10;50;; tp_es01-get_queue=0;50;200;; tp_es01-get_rejected=0;1000;2000;; tp_es01-index_active=0;10;50;; tp_es01-index_queue=0;50;200;; tp_es01-index_rejected=0;1000;2000;; tp_es01-listener_active=0;10;50;; tp_es01-listener_queue=0;50;200;; tp_es01-listener_rejected=0;1000;2000;; tp_es01-management_active=1;10;50;; tp_es01-management_queue=0;50;200;; tp_es01-management_rejected=0;1000;2000;; tp_es01-ml_autodetect_active=0;10;50;; tp_es01-ml_autodetect_queue=0;50;200;; tp_es01-ml_autodetect_rejected=0;1000;2000;; tp_es01-ml_datafeed_active=0;10;50;; tp_es01-ml_datafeed_queue=0;50;200;; tp_es01-ml_datafeed_rejected=0;1000;2000;; tp_es01-ml_utility_active=0;10;50;; tp_es01-ml_utility_queue=0;50;200;; tp_es01-ml_utility_rejected=0;1000;2000;; tp_es01-refresh_active=0;10;50;; tp_es01-refresh_queue=0;50;200;; tp_es01-refresh_rejected=0;1000;2000;; tp_es01-rollup_indexing_active=0;10;50;; tp_es01-rollup_indexing_queue=0;50;200;; tp_es01-rollup_indexing_rejected=0;1000;2000;; tp_es01-search_active=0;10;50;; tp_es01-search_queue=0;50;200;; tp_es01-search_rejected=78;1000;2000;; tp_es01-search_throttled_active=0;10;50;; tp_es01-search_throttled_queue=0;50;200;; tp_es01-search_throttled_rejected=0;1000;2000;; tp_es01-security-token-key_active=0;10;50;; tp_es01-security-token-key_queue=0;50;200;; tp_es01-security-token-key_rejected=0;1000;2000;; tp_es01-snapshot_active=0;10;50;; tp_es01-snapshot_queue=0;50;200;; tp_es01-snapshot_rejected=0;1000;2000;; tp_es01-warmer_active=0;10;50;; tp_es01-warmer_queue=0;50;200;; tp_es01-warmer_rejected=0;1000;2000;; tp_es01-watcher_active=0;10;50;; tp_es01-watcher_queue=0;50;200;; tp_es01-watcher_rejected=0;1000;2000;; tp_es01-write_active=8;10;50;; tp_es01-write_queue=20;50;200;; tp_es01-write_rejected=0;1000;2000;;

Command definition

Command definition in Nagios, Icinga 1.x, Shinken, Naemon

The following command definition allows optional parameters all defined within ARG4.

# 'check_es_system' command definition
define command{
  command_name check_es_system
  command_line $USER1$/check_es_system.sh -H $ARG1$ -t $ARG3$ $ARG4$
}

Command definition in Icinga 2.x

object CheckCommand "check_es_system" {
  import "plugin-check-command"
  command = [ PluginContribDir + "/check_es_system.sh" ]

   arguments = {
      "-H" = {
        value = "$es_address$"
        description = "Hostname or IP Address of ElasticSearch node"
      }

      "-P" = {
        value = "$es_port$"
        description = "Port number (default: 9200)"
      }

      "-S" = {
        set_if = "$es_ssl$"
        description = "Use https"
      }

      "-u" = {
        value = "$es_user$"
        description = "Username if authentication is required"
      }

      "-p" = {
        value = "$es_password$"
        description = "Password if authentication is required"
      }

      "-d" = {
        value = "$es_available$"
        description = "Define the available disk or memory size of your ES cluster"
      }

      "-t" = {
        value = "$es_checktype$"
        description = "Define the type of check (disk|mem|status)"
      }

      "-o" = {
        value = "$es_unit$"
        description = "Choose sizing unit (K|M|G) - defaults to G for GigaByte"
      }

      "-i" = {
        value = "$es_index$"
        description = "Space separated list of indexes to be checked for readonly (default: '_all')"
      }

      "-w" = {
        value = "$es_warn$"
        description = "Warning threshold"
      }

      "-c" = {
        value = "$es_crit$"
        description = "Critical threshold"
      }

      "-m" = {
        value = "$es_max_time$"
        description = "Maximum time in seconds (timeout) for Elasticsearch to respond (default: 30)"
      }
  }

  vars.es_address = "$address$"
  vars.es_ssl = false
}

Service definition

Service definition in Nagios, Icinga 1.x, Shinken, Naemon

In this example, the disk check happens on myexcluster.in.the.cloud and assumes a 50GB available disk space. There is authentication required to access the cluster statistics so here the login happens with user "read" and password "only".

# Check ElasticSearch Disk Usage
define service{
  use generic-service
  host_name myesnode
  service_description ElasticSearch Disk Usage
  check_command check_es_system!myescluster.in.the.cloud!disk!-d 50 -u read -p only
}

In the next example, the status of the Elasticsearch cluster is checked on myexcluster.in.the.cloud:

# Check ElasticSearch Status
define service{
  use generic-service
  host_name myesnode
  service_description ElasticSearch Status
  check_command check_es_system!myescluster.in.the.cloud!status
}

Service object definition Icinga 2.x

In this example, the disk check happens on myexcluster.in.the.cloud and assumes a 50GB available disk space. There is authentication required to access the cluster statistics so here the login happens with user "read" and password "only".

# Check Elasticsearch Disk Usage
object Service "ElasticSearch Disk Usage" {
  import "generic-service"
  host_name = "myesnode"
  check_command = "check_es_system"
  vars.es_address = "myescluster.in.the.cloud"
  vars.es_user = "read"
  vars.es_password = "only"
  vars.es_checktype = "disk"
  vars.es_available = "50"
}

In this example, the status of Elasticsearch running on myexcluster.in.the.cloud is checked. There is authentication required to access the cluster statistics so here the login happens with user "read" and password "only".

# Check Elasticsearch Status
object Service "ElasticSearch Status" {
  import "generic-service"
  host_name = "myesnode"
  check_command = "check_es_system"
  vars.es_address = "myescluster.in.the.cloud"
  vars.es_user = "read"
  vars.es_password = "only"
  vars.es_checktype = "status"
}

In this example, the thread pool statistics of Elasticsearch running on myexcluster.in.the.cloud are checked. If any of the thread pool stats (active,queued,rejected) triggers a treshold, the plugin will exit with warning or critical status.

# Check Elasticsearch Status
object Service "ElasticSearch Status" {
  import "generic-service"
  host_name = "myesnode"
  check_command = "check_es_system"
  vars.es_address = "myescluster.in.the.cloud"
  vars.es_user = "read"
  vars.es_password = "only"
  vars.es_checktype = "tps"
  vars.es_warn = "50,170,1000"
  vars.es_crit = "100,200,5000"
}

Screenshots

check_es_system Elasticsearch Queues Critical
check_es_system Elasticsearch cluster yellow status
check_es_system Elasticsearch yellow status
check_es_system capacity checks
Grafana show performance data Elasticsearch unassigned shards check_es_system plugin
Grafana shows watcher queue of Elasticsearch rising
Elasticsearch indexes are read-only alert