How do your servers talk to you?
Most of us monitor what our servers do, in some way or another. We might use Icinga/Nagios for some tasks, or use any of what feels like a myriad different tools for other tasks. I assume, that in most cases, people use e-mail, SMS, etc. for sending out warning or critical alerts when they occur.
Each of these monitoring services needs to be configured to send alerts via a particular channel (e.g. SMTP, SMS) to particular users, and if the need to change these associations arises, most people have to reconfigure their software in order to do so. For Icinga/Nagios, there are "recipes" on getting particular notification systems configured; that works, of course, but it‘s painful, error-prone, and the configuration is very static.
Some specialized tools have other mechanisms. Ansible for example, has a slew of notification modules (some of which I even wrote), which can announce to your deployment team what‘s going on while it runs. Here again, each time you use a particular notification module in Ansible, you have to configure it to notify a particular service‘s account. What if that changes? Do you really want to change all the Ansible playbooks ...? No.
Let me show you what I mean by listing the source of an alert, its type, and its destination:
icinga disks [email protected] icinga mail queue [email protected] icinga dnsbl SMS:1234567890 ansible playbook twilio:john18 ...
So, when Icinga reports a problem with a DNS blacklist, an SMS will be sent, and when Icina warns about a mail queue, an e-mail to Jane is fired off.
Suppose we want to change that: in addition to having an e-mail sent off, we wish to alert a bunch of mobile phones via Pushover. Or we want to change the content of the message, etc. For most of us that means chasing down all instances of the configured notification method, and modifying / adding to that. No thanks. Really not.
Can we set up some sort of notification "bus" which services use to send notifications to, and which then decides, using a central config, what to do with those messages, maybe even reformat them on a per-destination basis? Yes we can.
We created that recently. It‘s called mqttwarn, and it‘s pretty easy to set up. All you need is an MQTT broker, which is a lightweight "bus" for your network. Your services then "talk" to that broker (i.e. send it messages). mqttwarn grabs those messages, and dispatches them on to a notification service (SMTP, NNTP, Twilio, Pushover, HTTP, ...), stores them in files or in MySQL, publishes them to Redis, pipes them to programs, etc. All messages, some only (filtered by content, say).
I‘ll show you a small example to hopefully whet your appetite.
mqttwarn is configured in an ini-type file. The difficult part is setting up the so-called service targets which are notification services. I‘ll configure mqttwarn to provide a target called smtp:jplocal which delivers via SMTP, and another one called pushover:icinga which delivers a message to my mobile phone via Pushover.net.
[defaults] hostname = ‘localhost‘ port = 1883 ; name the service providers you will be using. launch = smtp, pushover [config:pushover] targets = { ‘icinga‘ : [‘xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx‘, ‘yyyyyyyyyyyyyyyyyyyyyyyyyyyyyy‘], } [config:smtp] server = ‘localhost:25‘ sender = "MQTTwarn <[email protected]>" starttls = False targets = { ‘jplocal‘ : [ ‘[email protected]‘ ], } [monitoring/+] targets = smtp:jplocal, pushover:icinga
When I launch mqttwarn, it will subscribe to the MQTT topic monitoring/+. (The + is a single-level wildcard which matches any one subtree.)
If I then publish a message to my MQTT broker, mqttwarn will receive that message and act upon it. In this example I‘ll use a command-line utility to do so, but we‘d normally configure our monitoring system (or Ansible, etc.) to publish via MQTT, and there are all manner of language bindings to accomplish that.
mosquitto_pub -t monitoring/warning -m ‘Disk utilization: 94%‘
As you‘ve probably guessed, -t specifies the topic name, and -m the payload to publish.
Do we have e-mail?
Date: Thu, 03 Apr 2014 09:26:36 +0200 From: MQTTwarn <[email protected]> To: [email protected] Subject: mqttwarn X-Mailer: mqttwarn Disk utilization: 94%
Simultaneously, mqttwarn notified the message via Pushover, and my phone alerts me accordingly:
Coming back to Ansible, if I wanted to post that same notification from a playbook, I would use the mqtt notification module. (The strange-looking quotes are necessary to protect the colon in the string.)
--- - hosts: all tasks: ... - local_action: ‘mqtt topic=monitoring/warning payload="Disk utilization: 94%"‘
If I later change my mind and want to modify one of the targets, say, I want to add another e-mail recipient, store messages, etc., I just modify mqttwarn‘s configuration file; no need to change the system which actually initiated the notification alert (i.e. I don‘t have to alter my Icinga configuration).
Looking at Icinga (or Nagios), we can very easily add something similar. Consider your fugly notification module
define command{ command_name notify-service-by-email command_line /usr/bin/printf "%b" "***** Icinga *****\n\nNotification Type: $NOTIFICATIONTYPE$\n\nService: $SERVICEDESC$\nHost: $HOSTALIAS$\nAddress: $HOSTADDRESS$\nState: $SERVICESTATE$\n\nDate/Time: $LONGDATETIME$\n\nAdditional Info:\n\n$SERVICEOUTPUT$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ Service Alert: $HOSTALIAS$/$SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$ }
and now look at mine:
define command{ command_name notify-service-by-mqtt command_line /usr/bin/notify-by-mqtt.py }
The rather simple notify-by-mqtt program extracts the values I‘m interested in from the environment (note: requires setting enable_environment_macros=1 in icinga.cfg) and produces a JSON payload which is published via MQTT to mqttwarn. The payload looks like this:
{ "_type": "icinga", "date": "2014-04-04", "eventstarttime": "1396621975", "hostdisplayname": "localhost", "hostname": "localhost", "hoststatetype": "HARD", "servicedesc": "JPtest", "servicedisplayname": "JPtest", "serviceoutput": "file /tmp/f1: ENOENT", "servicestate": "CRITICAL", "servicestateid": "2", "shortdatetime": "2014-04-04 16:38:25", "time": "16:38:25", "timet": "1396622305" }
On the receiving end, mqttwarn can filter messages, and I change their formatting on a per/service basis (e.g. I can be more verbose in an e-mail than in a tweet). mqttwarn can transform data on the fly, and it can apply templates for format the message.
Let me show you a simple example. I‘ll take the above mqttwarn configuration and add the title and format lines to it:
[monitoring/+] targets = smtp:jplocal, pushover:icinga title = {servicestate} {servicedisplayname} format = {time}: {hostname}:{servicedisplayname} is {servicestate}: {serviceoutput}
mqttwarn attempts to decode JSON from the payload received over MQTT, and it can use the JSON elements to populate format-type strings. The result as seen on my phone via Pushover is like this:
Seen from bottom to top:
- our first example
- an Icinga alert with the raw JSON in it
- the final result with formatted title and body
We‘ve had a lot of very positive feedback on mqttwarn, and I‘m confident you‘ll be able to put it to good use!