Writing your own BOSH release/configuration management is relatively simple – you get monit, and monit is relatively simple.
There are some interesting basics and interesting top tips for using monit with BOSH.
When is monit used?
When you are writing a BOSH release, each job template (the folders within the jobs/ folder of a release) contains a single monit file.
On each running BOSH job server, BOSH delegates to the monit daemon to use these monit files to start/stop processes.
So to write BOSH releases you need to learn some monit syntax and to write monit wrapper scripts.
How to do nothing?
Monit doesn’t support comments. And BOSH releases must have a monit file. So can a BOSH job template, via monit, do absolutely nothing? Yes.
Copy and paste the following into your job template’s monit file and when you deploy it it will do nothing.
That’s right. To do nothing, make your monit file empty.
One way to "comment out" the contents of a monit file is to use ERb templating.
<% # this is a comment %>
<% if false %>
This text will be removed by BOSH because of the "if" statement above.
<% end %>
(Thanks Abhi for the first example!)
Basic example
The most common syntax used by BOSH job templates is to start/stop a process.
Here is the monit file for running the gorouter in cf-release (source).
check process gorouter
with pidfile /var/vcap/sys/run/gorouter/gorouter.pid
start program "/var/vcap/jobs/gorouter/bin/gorouter_ctl start"
with timeout 60 seconds
stop program "/var/vcap/jobs/gorouter/bin/gorouter_ctl stop"
group vcap
Removing the file paths & timeout to make it easier to read:
check process gorouter
with pidfile gorouter.pid
start program "gorouter_ctl start"
stop program "gorouter_ctl stop"
group vcap
When monit is told to start all processes, monit will run the gorouter_ctl script with the argument start.
If monit ever detects that the process has stopped running, then it will again run gorouter_ctl start to restart the process.
When monit is told to stop processes it is configured above to again run the gorouter_ctl script, but with the argument stop.
That is, this monit file assumes there is a single gorouter_ctl script that can be run with either start or stop as the first command-line argument.
Alternately, two scripts could have been written and the monit file could have used them.
It is common in BOSH job templates to use a single monit wrapper script for both start and stop instructions.
Monit wrapper scripts
In the example above, gorouter_ctl is an executable shell script (source).
The allow the start and stop argument it includes a case statement; and the pseudo code for starting/stop the gorouter process is:
case $1 in
start)
echo $$ > $PIDFILE
/var/vcap/packages/gorouter/bin/gorouter
;;
stop)
kill_and_wait $PIDFILE
;;
*)
echo "Usage: router_ctl {start|stop}"
;;
esac
The start section of the real script does a lot more setup prior to running the /var/vcap/packages/gorouter/bin/gorouter command (source.
Job templates and runtime files
Quick recap of the relationship between BOSH job templates and the resulting files on a BOSH job instance:
The job template gorouter_ctl.erb file is located within the BOSH release at jobs/gorouter/templates/gorouter_ctl.erb.
When the job template is deployed, it will be available at: /var/vcap/jobs/gorouter/bin/gorouter_ctl.
Why does templates/gorouter_ctl.erb become bin/gorouter_ctl? Because the job template’s spec file said so.
Personally, in all my BOSH releases I prefer to keep the same structure within the templates/ folder as it will be in production. That is, I would move templates/gorouter_ctl.erb into templates/bin/gorouter_ctl.erb. I find the flat templates of Pivotal’s BOSH releases to be harder to comprehend at a glance (e.g. https://github.com/cloudfoundry/cf-release/tree/master/jobs/gorouter/templates)
So we know that the gorouter_ctl script will ALWAYS be available at /var/vcap/jobs/gorouter/bin/gorouter_ctl, which is why we can put this hardcoded full path in the monit file:
check process gorouter
start program "/var/vcap/jobs/gorouter/bin/gorouter_ctl start"
stop program "/var/vcap/jobs/gorouter/bin/gorouter_ctl stop"
Blank slate
The start/stop commands are run as the root user and they start with a very very minimal environment.
Assume that $PATH and all other common environment variables are blank or do not contain the contents you expect.
Your wrapper script must setup all environment variables.
Additionally, for security it is a good idea to change from root user to another user to run processes.
PID files
In addition to taking instruction from BOSH about starting/stopping processes, Monit wants to be able to detect when a process has died and restart it. It does this by watching process IDs, aka PIDs.
Each process that runs on Linux has a process ID. In the monit example above, we tell monit how to find the expected process ID:
check process gorouter
with pidfile /var/vcap/sys/run/gorouter/gorouter.pid
start program "/var/vcap/jobs/gorouter/bin/gorouter_ctl start"
...
Monit expects that after gorouter_ctl start completes there will be a file created /var/vcap/sys/run/gorouter/gorouter.pid that contains the process ID of the running process. This file is called a PID file or pidfile.
The best way to create a pidfile is to delegate it to the process you are running. Many applications will allow you to run them in "daemonized" mode and to specify the path to the pidfile.
For example, to run the Redis key-value store you run the redis-server command. You can provide this command with a configuration file path. And that configuration file can specify to run redis-server in daemonized mode and the pidfile location:
daemonize yes
pidfile /var/vcap/sys/run/redis/redis.pid
What if the command does not support daemonizing?
The gorouter is run in blocking mode (it does not daemonize itself and write a pidfile). So instead we wrote out the monit wrapper script’s own PID to the pidfile.
echo $$ > $PIDFILE
Dynamic monit file
The monit file in each BOSH job template can include templating code. For example the <% if false %> hack earlier. The template code used is Ruby’s ERb library.
This means you can dynamically create the contents of the resulting monitrc file (example):
<% 4.each do |index| %>
check process cloud_controller_worker_<%= index %>
with pidfile...
start program...
<% end %>
The value 4 above could be dynamically provided at bosh deploy time by BOSH Job Properties.
Other configuration examples?
Whilst the primary purpose of monit within BOSH is to start/stop processes, there are many things that you can do with monit. All these are available to you within your BOSH releases.
http://mmonit.com/wiki/Monit/ConfigurationExamples
How do I get started creating BOSH releases?
I recommend starting with the bosh-gen generator tool that we use at Stark & Wayne. You can create new releases, add packages (pulling in source files, debian packages), add jobs, and more.
I also recommend using bosh-lite for your initial development of a release. Everything is local and it uses Warden containers to quickly create/destroy BOSH jobs.
Spread the word
