#SwarmWeek: Realtime Cluster Monitoring with Docker Swarm and Riemann

Lately I’ve become fascinated with Riemann, a piece of software written in Clojure for the purpose of supervising the health of distributed systems. Created by Kyle Kingsbury (a.k.a. aphyr) of Jepsen fame, Riemann is a powerful tool for montioring, processing, and responding to information from your cattle herd.

Today we’re going to discuss:

1. Why should Docker users care about monitoring?
2. Why Riemann in particular?
3. How do you Dockerize Riemann’s components?
4. How do you distribute Riemann’s components using Docker Swarm?
5. How do you use the Riemann dashboard?

---

 

Why should Docker users care about monitoring?

For some developers, nothing is going to put them to sleep faster than having a hearty discussion about “monitoring”. Indeed, it’s easy to see why introducing health monitoring or any other kind of application telemetry is often an afterthought: it’s yet another yak shave in the endless pile of yak shaves that most modern developers have grown accustomed to as obstacles in the quest to deliver our application to the end user. But proper monitoring can help us to secure the holy grail for developers: fast, reliable applications and happy users.

Without integrating monitoring as a first-class citizen in your infrastructure and application development lifecycle, it’s likely that you’ll find yourself in an awkward situation sooner or later: Angry users, via Twitter or otherwise, will become your actual monitoring system as they notify you that XYZ feature isn’t working or that the website has gone down. Nothing is likely to erode confidence in your brand or business faster than these types of incidents, so if you can prevent them, you absolutely should.

Additionally, once monitoring tools have been set up and made accessible, the insight that they provide is incredibly valuable. I’m confident you’ll find that going from manually checking up on hosts and services to having good monitoring software installed and configured is like going from surfing the web before search engines were invented to a world where Google search exists.

Docker users are in a particularly unique position to gain a lot of value from monitoring, simply by virtue of the fact that the number of things which need to be monitored in a Docker-based world is an order of magnitude larger than it was before. Not only are you likely to have a fleet of VMs which need to be supervised, you will now have a variety of containers running on each VM. There are likely to be additional moving pieces which need to be monitored such as a container orchestrator, a key-value store, the Docker daemon itself and so on. That’s a lot of moving pieces! The rewards of those who are disciplined in focusing on application monitoring and debuggability in a Dockerized world will be self-evident.

 

Why Riemann in particular?

There are a variety of reasons to use Riemann. Here are a few of the most obvious.

Riemann’s push-based model lets you detect problems and verify that fixes are working in near real-time.
Traditional monitoring systems such as Nagios usually work with a poll-based model: They wake up periodically (say, every 5 minutes), run a series of health checks to verify that everything in the system is in the desired state (e.g. your services are up), and report the results to you via e-mail or another method. Riemann requires that services send their own events, and consequently outages or network partitions can be detected as soon as the events stop flowing. Likewise, we can collect and process events of any type, opening the doors to proactively providing value through monitoring instead of simply fighting fires.

Riemann’s stream processing allows for powerful transformations of incoming data.
For instance, even if an exception is logged 1000 times in an hour, Riemann can roll up all exceptions of that type into just a few e-mails with information about how many times the exception occured. This helps combat alarm fatigue. Another example- Riemann makes measuring metrics in percentiles quite trivial. This allows to gain true insight into the operations of our systems without having important information masked behind aggregations such as averages.

Riemann is easy to get data into or out of, allowing interesting use cases such as storing event data for analysis later on.
Clients for Riemann can send events using a thin protocol buffers layer over TCP or UDP. Riemann can emit events and metrics from its index into a variety of backends including Graphite, InfluxDB, Librato, and more. Riemann also has built-in support for notification via SMS, e-mail, Slack, etc.

Riemann’s architectural model is simple, preventing the potentially ironic situation of having monitoring software that is difficult to operate and/or failing often.
Riemann is designed from the ground up to be straightforward to operate and reason about. While this has some tradeoffs (for instance, it makes it impossible to distribute a single Riemann instance safely), in practice Riemann’s philosophy is largely that imperfect information right now is better than perfect information which never arrives. If the Riemann server crashes, usually a simple restart will remedy the problem; no finicky reconciling of distributed state needs to occur.

Likewise, the push-based model helps to alleviate the “who monitors the monitoring software” question (i.e. how do we detect when our monitoring itself has issues). We can simply set up a few Riemann servers and forward events to the downstream servers. If the upstream server goes down, the downstream servers will notice this and alert you. Especially running Docker across clouds, this could be particularly valuable.

It’s rather fast.
From the landing page: “Throughput depends on what your streams do with events, but a stock Riemann config on commodity x86 hardware can handle millions of events per second at sub-ms latencies, with 99ths around 5ms. Riemann is fully parallel and leverages Clojure and JVM concurrency primitives throughout.”

 

How do you Dockerize Riemann’s components?

Today we are going to look at Dockerizing three components which, when combined, will allow us to use Riemann effectively:

1. The Riemann server process, written in Clojure, which is the main stream processing engine
2. The riemann-health program, written in Ruby, which reports health / usage metrics to the central Riemann server
3. The riemann-dash program, written in Ruby, which is a small Sinatra application providing a web dashboard for Riemann

We will run them using Docker Swarm and they will be able to talk to one another using the overlay driver of libnetwork. Here is a sample architecture diagram of the components, and how they fit into the 3-node cluster we will be demonstrating. Each node has one instance of riemann-health to report its metrics to the server. The other two containers will be scheduled arbitrarily, as which host they end up on does not matter.

For the purpose of this article I’m going to assume that you have a Swarm cluster of at least 3 nodes up and running. Here’s an example of how to get started with Swarm in the official Docker documentation.

 

Docker Images

The Dockerfile for the Riemann server is as follows:

FROM debian:jessie
 
ENV RIEMANN_VERSION 0.2.10
 
RUN apt-get update && apt-get install -y default-jre
ADD https://aphyr.com/riemann/riemann_${RIEMANN_VERSION}_all.deb /riemann_${RIEMANN_VERSION}_all.deb
RUN dpkg -i riemann_${RIEMANN_VERSION}_all.deb
EXPOSE 5556/tcp 5555/udp
COPY ./riemann.config /etc/riemann/riemann.config
CMD ["riemann", "/etc/riemann/riemann.config"]

As you can see, it simply installs the Java runtime environment, grabs the .deb package from Kyle’s website, and installs the central Riemann server. It also inserts the following configuration file into the image:

; -*- mode: clojure; -*-
; vim: filetype=clojure

(logging/init {:file "/var/log/riemann/riemann.log"})

; Listen on the local interface over TCP (5555), UDP (5555), and websockets
; (5556)
(let [host "0.0.0.0"]
(tcp-server {:host host})
(udp-server {:host host})
(ws-server  {:host host}))

; Expire old events from the index every 5 seconds.
(periodically-expire 5)

(let [index (index)]
; Inbound events will be passed to these streams:
(streams
  (default :ttl 60
    ; Index all events immediately.
    index

    ; Log expired events.
    (expired
      (fn [event] (info "expired" event))))))

This is a very simple configuration file (written in Clojure) which simply inserts incoming events into Riemann’s index and expires them after 5 seconds. When an event expires, Riemann will log that this event is expiring. The ability to define behavior around events going stale is a big advantage of Riemann. Using Riemann’s primitives, we can actually treat the absence of information as new information: if a service or host hasn’t sent events in a while, we know that it’s either down or something is interfering with its ability to get there, e.g. networking. Since we can define actions to take when these types of state transitions happen, we can do things like alerting in a Slack channel when a host has gone down (or gone back up).

This could theoretically be expanded into a fully self-contained system which is self-healing. For instance, when we detect that metrics for a host or service have expired, we could launch a new service to compensate, restart the host, and so on.

Now we have a way to build our Riemann server image (to collect metrics), but we need a way to actually send information to it. The riemann-health program will send CPU, memory, and load information to the centralized server, so let’s Dockerize that:

FROM debian

RUN apt-get update && apt-get install -y ruby ruby-dev build-essential
    zlib1g-dev && \
    gem install --no-ri --no-rdoc riemann-tools

ENV RIEMANN_HEALTH_SERVER_HOSTNAME riemann-server
COPY ./entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

We use this entrypoint.sh script to ensure that we can configure the location of the Riemann server the health program should connect to (by default, we will this container will be called riemann-server and we will use libnetwork default DNS).

entrypoint.sh looks like this:

#!/bin/bash

riemann-health --host ${RIEMANN_HEALTH_SERVER_HOSTNAME} --event-host $(cat /etc/hostname)

When we eventually run this container, /etc/hostname will be mounted from the external system into it to ensure that riemann-health sends the correct host name to the centralized server.

Last but not least, we will have an image for the Riemann dashboard:

FROM debian

RUN apt-get update && apt-get install -y ruby && \
    gem install --no-ri --no-rdoc riemann-dash
ENV RIEMANN_DASH_CONFIG /config.rb
EXPOSE 4567/tcp
COPY ./config.rb /config.rb
CMD ["riemann-dash"]

config.rb:

      set :bind, "0.0.0.0"

 

How do you distribute Riemann’s components using Docker Swarm?

We will use a Docker Compose file to express these containers’ runtime information, their scheduling parameters (to ensure that an instance of riemann-health can end up on every node), and the corresponding overlay network to ensure that they can all communicate with each other across multiple hosts. Because we are specifying a network to create, we will use the version 2 (latest, at the time of writing) configuration format for Compose.

Our Docker Compose file looks like this:

version: "2"

networks:
  riemann:
    driver: overlay

services:
  riemannserver:
    container_name: riemann-server
    image: "nathanleclaire/riemann-server:article"
    net: riemann
    ports:
    - "127.0.0.1:5556:5556"
    restart: always

  riemannhealth:
    image: "nathanleclaire/riemann-health:article"
    net: riemann
    pid: host
    environment:
      - "affinity:container!=*riemannhealth*"
    volumes:
      - "/etc/hostname:/etc/hostname:ro"
    restart: always

  riemanndash:
    image: "nathanleclaire/riemann-dash:article"
    ports:
      - "127.0.0.1:4567:4567"
    restart: always

Some noteworthy aspects of this:

• riemannhealth is run in pid namespace of host to accurately gather per-process usage metrics
• riemannserver and riemannhealth are on the riemann overlay network to ensure that events can be sent to the central server from the health daemon
• scheduling constraints (for Swarm) are set using environment, ensuring that riemannhealth service will never run on more than one instance per host
• container_name of riemannserver service is set manually to avoid relying on Compose’s automatic naming convention for DNS discovery via libnetwork

Once we have this file in place, we can simply:

$ docker-compose up -d
Creating admin_riemannhealth_1
Creating riemann-server
Creating admin_riemanndash_1

We can see the created containers with docker-compose ps:

$ docker-compose ps
        Name                       Command               State                 Ports
---------------------------------------------------------------------------------------------------
admin_riemanndash_1     riemann-dash                     Up      127.0.0.1:4567->4567/tcp
admin_riemannhealth_1   /entrypoint.sh                   Up
riemann-server          riemann /etc/riemann/riema ...   Up      5555/udp, 127.0.0.1:5556->5556/tcp

Then, scale out the health service to the number of nodes we have available:

$ docker-compose ps
        Name                       Command               State                 Ports
---------------------------------------------------------------------------------------------------
admin_riemanndash_1     riemann-dash                     Up      127.0.0.1:4567->4567/tcp
admin_riemannhealth_1   /entrypoint.sh                   Up
admin_riemannhealth_2   /entrypoint.sh                   Up
admin_riemannhealth_3   /entrypoint.sh                   Up
riemann-server          riemann /etc/riemann/riema ...   Up      5555/udp, 127.0.0.1:5556->5556/tcp

$ docker ps
CONTAINER ID        IMAGE                                   COMMAND
      CREATED              STATUS              PORTS                                NAMES
5ba3c435c8b6        nathanleclaire/riemann-health:article   "/entrypoint.sh"         
      19 seconds ago       Up 18 seconds                                            swarmnode-0/admin_riemannhealth_3
07b6f10ac711        nathanleclaire/riemann-health:article   "/entrypoint.sh"         
      20 seconds ago       Up 18 seconds                                            swarmnode-2/admin_riemannhealth_2
b21655baecc5        nathanleclaire/riemann-dash:article     "riemann-dash"           
      About a minute ago   Up About a minute   127.0.0.1:4567->4567/tcp             swarmnode-0/admin_riemanndash_1
1ad65a1a384d        nathanleclaire/riemann-server:article   "riemann /etc/riemann"   
      About a minute ago   Up About a minute   5555/udp, 127.0.0.1:5556->5556/tcp   swarmnode-2/riemann-server
da4ce7199489        nathanleclaire/riemann-health:article   "/entrypoint.sh"         
      About a minute ago   Up About a minute                                        swarmnode-1/admin_riemannhealth_1

Note that if you try to scale it higher, it won’t work due to our scheduling constaints:

$ docker-compose scale riemannhealth=4
Creating and starting 4 ... error

ERROR: for 4  unable to find a node that satisfies container!=*riemannhealth*

Great, now we have an instance of the Riemann server and dashboard running, as well as one health daemon reporting metrics back to the central server per host. Now let’s take a look at using the Riemann dashboard.

How do you use the Riemann dashboard?

Now we are collecting metrics, but we need to visualize what we are aggregating. To do this we will access our Riemann dashboard instance.

My suggestion based on the Compose file above is to use SSH port forwarding to forward the dashboard panel port (4567) and the central Riemann server websocket port (5556) to localhost on your workstation. This will allow you to access the dashboard securely and simply. For instance, if the server is on swarmnode-0, and the dashboard is on swarmnode-1:

$ ssh -fN -L 5556:127.0.0.1:5556 [email protected]<swarmnode-0 IP>

$ ssh -fN -L 4567:127.0.0.1:4567 [email protected]<swarmnode-1 IP>

Be advised that this will fork SSH processes into the background.

You are greeted by a blank Riemann dashboard at first.

To create visualizations, we modify the tiles on the dashboard to display Riemann queries in a particular format such as Chart, Grid, etc. To start creating them, CMD-click the top pane (which says “Riemann” in big letters) and press your e key to edit it.

We’ll create a grid-based view first. Select “Grid” from the dropdown, and set the query to simply “true”. Then, click “Apply”.

You’ll see the “grid” view pop up. These green rectangles represent metrics coming in to the central Riemann server from various hosts and services. If these metrics start to hit the “danger zone”, the rectangles will turn yellow, and then red.

You can split panes like so:

1. CMD+click on them.
2. CTRL+<arrow key> in the direction you wish to split, e.g. to the left.
3. ESCAPE to defocus the selected pane.

Let’s try splitting this grid view and making a graph for memory usage of each host like so. Just like the original pane, we can CMD+click and then press e to set the view this pane should display. The ‘Flot’ type shows interesting line charts, so let’s make one of those. Our query will be service = "memory".

The query language takes a little getting used to, but is pretty straightforward once learned. CTRL+S will save the dashboard to /var/lib/gems/2.1.0/gems/riemann-dash-0.2.12/config/config.json in the container. If you want to, you could docker cp the saved config back to your build host and then bake it into future dashboard images:

$ docker cp admin_riemanndash_1:/var/lib/gems/2.1.0/gems/riemann-dash-0.2.12/config/config.json .

The dashboard JS could definitely use a little love, so hopefully some ambitious front-end developer out there will read this article and contribute some changes back upstream.

Oh, and by the way, you can also configure Riemann to send metrics to a time series database (optionally hooked up to a visualization front-end such as Grafana), so that you can go and look at the history of what’s been monitored and make pretty graphs like these ones:

Combined with centralized logging using something like ELK, the amount of insight you can start gaining into the motions of your infrastructure is quite exciting. With such tools at your disposal you can proactively ensure the health of your systems before they become problematic. For instance, you can see that using percentile-based monitoring we can identify very clear spikes in Riemann stream latency occuring about once per hour. Given this information we can proactively fix the cause of this issue and prevent it from becoming a part of a future cascading system failure.

Thanks

I’d like to give a special shout-out to Kyle Kingsbury for writing Riemann originally and for tending to the community with persistence and patience. You should hire him if you have a database or other distributed system which would benefit from safety testing (and most of them can). Additionally, James Turnbull is a very helpful figure in the community and some of the ideas here are directly learned from him. He is writing a book called The Art of Monitoring which I’d recommend picking up if you are interested in such topics.

In Conclusion

Docker, Swarm, and Riemann are some very fantastic technologies. I suggest that you go forth and use them. There are many available Riemann tools for sending information about whatever you happen to be running to the central server, and I hope to see many more. In particular, tools to monitor the Docker daemon and orchestration systems themselves would be really great. You could be the one to write them, so go forth and code!

---

Thanks Nathan for that awesome blog post! Don’t forget to participate in our DockerCon ticket raffle! Share a picture or description of your Swarm with us on Twitter and tag @docker and #SwarmWeek for a chance to win a free ticket to DockerCon 2016 on June 19-21 in Seattle, WA.

Here are some additional resources on Docker Swarm:

• Get started by downloading Docker Swarm and read the docs
• Try Docker Swarm as part of Docker Datacenter
• Submit questions to Docker Forums or file issues in Github
• Contribute to the Docker Swarm project
• View the other Swarm Week Posts

 

---

 

Learn More about Docker

• New to Docker? Try our 10 min online tutorial
• Share images, automate builds, and more with a free Docker Hub account
• Read the Docker 1.10 Release Notes
• Subscribe to Docker Weekly
• Sign up for upcoming Docker Online Meetups
• Attend upcoming Docker Meetups
• Register for DockerCon 2016
• Watch DockerCon EU 2015 videos
• Start contributing to Docker