Here is a Pager Duty alert for a persistent disk being greater than 80% full on an etcd node:

pagerduty

It turned out to be relatively simple to "turn on" disk monitoring on all jobs that contained persistent disks, across all deployments, across all stages of our client's pipeline.

We created a simple BOSH release that just adds some monit monitoring; and BOSH health monitor does the rest. It was all built-in.

Introducing monitor-server

https://github.com/cloudfoundry-community/monitor-server-boshrelease

This BOSH release initially includes monitoring of ephemeral and persistent disks on a BOSH VM. Conceptually it could monitor for any global attribute of a VM that good ol' monit can monitor

[See Quick guide to using Monit in BOSH for an introduction to Monit in BOSH]

By default it alerts on either disks reaching 80% capacity.

See the monitor-server README for how to co-locate it with all your BOSH deployments for instant disk monitoring.

BOSH health alerts

BOSH already includes a Health Monitor that forwards any alerts from any VM's monit monitor. All you have to do is turn it on.

Currently the following backends are supported:

  • SMTP/email
  • Pager Duty
  • DataDog
  • AWS CloudWatch

Read the BOSH docs for more information on monitoring: