Category: Monitoring

In this post I am going to demonstrate how to monitor your docker volumes with Sensu.
I came across a problem when our Jenkins instances were running out of space and no jobs could be scheduled because of this,
so before it was too late, it would be very useful to have something in place that will show if some container is too greedy, eating all the space on the volume.

In general we are going to look at the next things today:
1. Some Ruby scripting
2. Identifying disk and docker volume usage commands
3. Configuring Sensu server and client for monitoring
4. Making script run as a root
5. Running a simple Uchiwa dashboard

1. Some Ruby scripting
So first we are going to write the script which will check the volume and report if usage is higher than we configured.
Following best Sensu practices we will write it in Ruby, we probably could also use bash, but it really gets messy once we add more logic and lines.

#!/usr/bin/env /opt/sensu/embedded/bin/ruby

max_size = ARGV[0].to_i
container_name_filter = ARGV[1]
message = ""

procs=`du -sk  /var/lib/docker/volumes/* | sort -rn`
procs.each_line do | process|
  result = process.split(" ")
  vol_usage = result[0].to_i/1024
  vol_name = result[1].gsub "/var/lib/docker/volumes/", ''

  if vol_usage > max_size
    cont_name = `docker ps --filter=volume=#{vol_name} --filter=name=#{container_name_filter} --format {{.Names}}`
    if !cont_name.empty?
      message = message + "container: #{cont_name.delete!("\n")} volume exceeds max disk usage(#{max_size}MB): #{vol_usage}MB; \n"
    end
  end
end

unless message.empty?
  puts message
  exit 1
end

2. Identifying disk and docker volume usage commands

Comments closed