Stark & Wayne
  • by James Hunt

This morning, I was trying to wrangle our CI/CD pipeline for the Containers BOSH Release so that I could cut a 1.1.0 release and generally forget about the process of integration testing.

Our stock pipeline architecture for BOSH releases runs a deployment test by taking a manifest — either the example manifest or a CI-specific manifest — and deploying it to a BOSH director.  If the deployment takes, the BOSH release is considered fit for a release.

To conserve space, we usually target a BOSH director running the Warden CPI, which just spins up Linux containers for each BOSH instance group.  For most things, this is sufficient; BOSH releases rarely care about the infrastructure they are deployed on, after all.

But the Containers BOSH Release is a bit different.  It runs Docker, which needs a whole slew of container-y things like cgroups to function.  When I tried deploying it onto a containerized CPI, it blew up, as things on the cutting edge are wont to do.

Task 352 | 18:16:01 | Updating instance docker:
   docker/b3ab4f49-2f05-4865-b46c-eb248ebded5b (0) (canary) (00:02:26)
   L Error:
       'docker/b3ab4f49-2f05-4865-b46c-eb248ebded5b (0)' is not running
       after update. Review logs for failed jobs: docker, compose

Looking into the logs, I found that, at least under Warden, you don't have access to mount new things (including cgroup hierarchies) under /sys/fs/cgroup.  It's just plain not allowed.

A simple workaround is to move the mountpoint elsewhere.  Docker still finds the cgroups wherever they happen to be mounted. However, since this is a workaround specifically for Warden, I was hoping to find a way to limit the behavior to only occur there.  Other IaaS deployments (vSphere, AWS, GCP, etc.) are perfectly happy to mount things where they belong.

Luckily, I found just what I was looking for in /var/vcap/bosh/etc/infrastructure.  On Warden, this contains the text warden, making it dead easy to recognize a Warden CPI from a mile away:

if grep -q warden /var/vcap/bosh/etc/infrastructure ]]; then
  echo 'Hello, Warden!'
  # ... do warden-y things ...

  # ...


In the past, I've resorted to such tricks as the im_in_warden_dont_do_X property trick:

  im_in_warden_dont_do_nfs: yes

or the just-ignore-the-failure trick, which is less than ideal.

Now, armed with /var/vcap/bosh/etc/infrastructure, I can make IaaS-specific decisions to tailor my BOSH releases to the situation at hand.

Find more great articles with similar tags BOSH docker containers warden author-jhunt