A 101 on ElastAlert & How To Set It Up

Simple Framework for Alerting anomalies, spikes and other patterns from data in elasticsearch.

We use ElasticSearch, Log stash, Kibana for managing over increasing amount of data and logs, where kibana is great visualizing & querying data,
but it need companion tool for alerting when inconsistencies in our
data — where Elastalert is created.

Overview — Reliable , Modular ,Ease Set Up & Configure.

Why is it Reliable? — in event of restart elastic search or in case of unavailable elasticsearch.

• ElastAlert— save its state to elasticsearch (when it started & resume previously stopped).
• If Elasticsearch is unresponsive, ElastAlert waits until it recovers before continuing.
• ElastAlert (alerts) which throws Errors may be automatically retried for Period of time.

Why is it Modular? — Main due to 3 Components.

• Rule Types : Responsible for Processing Data returned from elasticsearch(basically by initializing the rule configuration & parse data -return data by querying elasticsearch).
• Alerts : Responsible for Taking action based on match.
• Configuration : Global Configuration file (config.yaml) — defines several aspects of its operation.(Ex: es_host,es_port,type)

How it works? — by combining elasticsearch with two types of components, rule types and alerts.

Elasticsearch is periodically queried and the data is passed to the rule type, which determines when a match is found. When a match occurs, it is given to one or more alerts, which take action based on the match.

This is configured by a set of rules, each of which defines a query, a rule type, and a set of alerts.

Several Rule types with common monitoring are included with ElastAlert:

1. Match when there are X events in Y time — Frequency
2. Match when the rate of events increase/decrease — Spike
3. Match when there are less than X events in Y time — flatline
4. Match when there a certain field matches a blacklist/whitelist — blacklist/whitelist
5. Match on any event matching a given a filter — any
6. Match when a field has two different values within some time — Change

Support In Built are — Jira ,email, Google Chat,the Hive,Telegram,Slack…

In Addition to this basic usage elastalert supports :

• Alert links to Kibana Dashboard Aggregate Counts for arbitrary fields
• Combine alerts into periodic reports
• Separate Alerts by a unique field
• Intercept/enhance match data.

Installation:

1. Clone Repo : git clone https://github.com/Yelp/elastalert.git or pip install elastalert
2. See Requirements in Cloned Repo and install using pip.
3. Elasticsearch
4. Python3
5. python3 setup.py install

Yes! we have installed ElastAlert.

In the next part, we'll talk about configuring elasticsearch to elastalert and configuring built in type queries.

In case of queries please feel free to comment if any issues!.

Thanks for reading

Read Part II Here