Monitoring Development Environment with Monit

These days a lot of effort is being put into designing systems that are highly available, reliable, fault tolerant etc. A lot of alerting tools are also being used to notify about any sort of mishaps like servers crashing, performance degradation etc. There are dedicated DevOps teams, SRE teams and a bunch of other folks who ensure production systems are always running smoothly & without any downtime. More often than not a lot of paid tools are being used for achieving this.

This is a must have setup for any service running in production environment. It is not feasible to replicate the same setup for your local dev environment where most of the feature development work is done and a single developer is responsible for maintaining and running it.

Lets take a small example where an application uses Redis and DynamoDB for storing and caching data. So Redis and DynamoDb needs to be running locally for developing this application.

Most common approach is to open multiple terminal windows and run each local server in a separate dedicated tab. But adding a new tab for each server becomes complex and difficult when the number of servers increase.

To simplify this we are going to configure Monit.

Setup Guide:

This is a setup guide for MacOS.

Setup Terminal Notifier brew install terminal-notifier

Enable Notifications from System Preferences:

Run the following command on your terminal:

terminal-notifier -message "Test Notification"

Output:

This is how the notification looks.

Note: Looks like notify-send can be used on linux, haven’t tried it myself.

**Setup Redis**brew install redis

Lets make some changes in the redis config file:vim /usr/local/etc/redis.conf

Set default pid path to/usr/local/var/run/redis-server.pid To run redis as daemon:daemonize yes

Create a wrapper script for starting and stopping redis-server file:vim /usr/local/etc/redis-init

With the following content:

#!/bin/bashcase $1 instart)exec redis-server /usr/local/etc/redis.conf;;stop)redis-cli shutdown ;;*)echo "usage: redis-init {start|stop}" ;;esacexit 0

Lets make it executable:

chmod +x redis-init

Setup DynamoDB

Download the latest version of DynamoDb local.

cd /usr/local/etcmkdir dynamo

copy DynamoDBLocal.jar and DynamoDBLocal_lib/ in/usr/local/etc/dynamo/

Create a wrapper script for DynamoDB like we did for redis:

vim /usr/local/etc/dynamo/dynamo-init

With the following content:

case $1 instart)cd /usr/local/etc/dynamo;nohup java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -port $2 -sharedDb 1> dynamo_$2.out & echo $! > /usr/local/var/run/dynamo_$2.pid;;stop)kill `cat /usr/local/var/run/dynamo_$2.pid`;rm /usr/local/var/run/dynamo_$2.pid;;;*)echo "usage: dynamo-init {start|stop} port";;esacexit 0

Lets make it executable:

chmod +x dynamo-init

**Setup httpd(Apache http server program):**sudo suapachectl startTo Verify try accessing http://localhost

Setup Monit:

Create monitrc file

vim /usr/local/etc/monitrc

With the following content:

set daemon 30set log /usr/local/etc/monit.log # monit logsset httpd port 2812 anduse address localhost # only accept connection from localhostallow localhost # allow localhost to connect to the server and

check process redis-server with pidfile /usr/local/var/run/redis-server.pidstart program = "/usr/local/etc/redis-init start"stop program = "/usr/local/etc/redis-init stop"if does not exist then exec "/bin/bash terminal-notifier -message 'redis does not exist'"

check process dynamo8001 with pidfile /usr/local/var/run/dynamo_8001.pidstart program = "/bin/bash /usr/local/etc/dynamo/dynamo-init start 8001"stop program = "/bin/bash /usr/local/etc/dynamo/dynamo-init stop 8001"if does not exist then exec "/bin/bash terminal-notifier -message 'Dynamo 8001 does not exist'"

This runs dynamoDb on port 8001. Change it to some other value if you want to run it on some other port.

Run monit -t to valid monitrc file.

To start monit:

monit

Open localhost:2812 in your browser and you will see something like this

Type monit summary in terminal

To start the processes type

monit start all

Check the status using monit summary

monit summary

Clicking on a process will give a detailed info about that process:redis-server:

Monit Commands:

monit -h # usage

monit summarymonit status

monit start all # start all processesmonit stop all # stop all processesmonit restart all # restart all processes

# For single process

monit start redis-servermonit stop redis-servermonit restart redis-server

monit start dynamo8001monit stop dynamo8001monit restart dynamo8001

Here is how the Alerts look:

Now if any of the process crashes or gets killed monit will send us an alert.To try it lets kill the redis-server process.

kill -9 `cat /usr/local/var/run/redis-server.pid`

And There it is:

To start redis-server again run:

monit start redis-server

Note: Monitrc file can be modified to restart the process automatically, if it does not exist.

Conclusion

If a system is being booted/rebooted monit start all can be used to start all the services.To restart all services monit restart all is enough. There is no need to individually restart all the services.

Monit can be used to monitor process on development environment. This can be useful when there are a lot of servers or micro-services running.If a standard installation way is devised, can be used in a team.

More where this came from

This story is published in Noteworthy, where thousands come every day to learn about the people & ideas shaping the products we love.

Follow our publication to see more stories featured by the Journal team.