Post 2, Electric Boogaloo

This is the second part of a series of posts about monitoring your CloudFoundry installations. You can find the first post in the series here. This post will cover some of the basics to encourage non Ruby developers to help us interface with services they've written.

So... You want a BOSH Monitor plugin?

We briefly touched on BOSH monitor in the first post. But the tl;dr is that BOSH Monitor runs plugins that fire whenever a NATS message is read from the bus. This allows us to forward those messages to a 3rd service. At this time there are documented solutions for

  • Event Logger - Logs events
  • PagerDuty - Sends events to PagerDuty.com
  • DataDog - Sends various events to DataDog.com
  • AWS CloudWatch - Sends various events to Amazon's CloudWatch
  • Emailer - Sends configurable Emails on events reciept

There are also a few undocumented plugins such as the NATS forwarder we mentioned in the previous post.

A Plugin by any other name

While there are many services to which you could reasonably forward this information there are only a few options at the moment. This may be because BOSH Monitor plugins are actually resident within the BOSH Monitor itself. There is not currently a plugin architecture that allows for including your feature from a separate source in to a running BOSH. Being that this is just a ruby application you can surely monkey patch your way in to Cloudfoundry monitoring glory but lets all agree not to do that. Thanks.

Your road to Open Source nirvana

Adding a plugin requires working directly on BOSH and hopefullly providing the team with a tidy pull request. Before getting started you'll want to review the BOSH developer docs. A point to keep in mind is that they favor pull requests with small, single commits with obvious purpose. Adding a bosh monitor plugin requires touching a couple files and adding a couple more. So you should be able to keep your commit history nice and neat. For example, this is what my current commit looks like. And this is probably close to what yours should look like.

modified:   lib/bosh/monitor.rb  
new file:   lib/bosh/monitor/plugins/<plugin>.rb  
new file:   spec/unit/bosh/monitor/plugins/<plugin>_spec.rb  
modified:   ../release/jobs/health_monitor/spec  
modified:   ../release/jobs/health_monitor/templates/health_monitor.yml.erb  

Shut up and build something already!

With the pleasantries out of the way, lets get to work. If you are familiar with Ruby, this process should feel pretty familiar to you. If not, don't run away, I think we can get through this.

First lets get some code and make sure it works. By checking out the bosh source and running it's test suite.

git clone https://github.com/cloudfoundry/bosh.git  
cd bosh  
gem install bundler  
bundle install  
cd bosh-monitor  
bundle exec rspec  

If the test suite does not pass, you are welcome to do some spelunking in the code but if you think your environment is set up correctly it really should be passing. See the BOSH Developers user group for help.

Welcome, Ruby Expert!

The lines above will get you far enough to poke around and get familiar with BOSH Monitor. If you don't know Git, Ruby or Bundler this is your chance to consult Google. Go ahead.. We'll wait.

To get started we'll need to create two files in BOSH-Monitor.

lib/bosh/monitor/plugins/<plugin>.rb and spec/unit/bosh/monitor/plugins/<plugin>_spec.rb This is the bulk of your work.

Anatomy of a Plugin

The first thing to keep in mind is that your plugin will run in Event Machine and the basic template below will get you started.

module Bosh::Monitor  
  module Plugins
    class MyPlugin < Base

      #You should override validate options in your plugin
      #It should return true or false depending on the validity of options
      #provided in the manifest
      def validate_options
        true
      end

      #Your run method controls the overall state of the event loop and 
      #it will get called after the plugin instance is initialized. 
      #Here we can parse out options or set up global plugin state
      def run
         #Example, initialize option in to instance 
         @option_one = options['option_one']

         #Example, Announce your plugin's existence
         logger.info("My plugin is running...")
      end

      #Your process method will be called whenever a new event is found
      #on the Nats message bus. 
      def process(event)

        #Example. You can use the event type to specify how you'd like to parse
        #a given message
        case event
          when Bosh::Monitor::Events::Heartbeat
            #Forward to your service
          when Bosh::Monitor::Events::Alert
            #Forward to your service
          else
            #Do something or Do nothing, It's Anarchy!
        end
      end
    end
  end
end  

Note that if you are writing a plugin that needs to put or post to a 3rd party service there is a library included in bosh-monitor/lib/bosh/monitor/plugins/http_request_helper.rb You probably want to use or extend that module instead rolling your own.

Testing 1,2,3..

Testing may the most complicated part of writing your plugin. Fortunately there are lots of examples in the bosh-monitor/spec/unit/bosh/monitor/plugins. The example below provides a couple of tests to ensure your plugin hands an event off to your service and that it validates options appropriately. It assumes you are using the httprequesthelper module. These are the very basic tests you will likely need.

require 'spec_helper'  
describe Bhm::Plugins::MyPlugin do

  #set subject to a new instantiated bosh monitor plugin of your type
  subject{ described_class.new(options)  }

  #When you instantiate a plugin you will pass it required options
  let(:options){ { 'option1' => 'foo', 'option2' => 'bar' }}

  #make_heartbeat is a helper that creats a fake event for you to process
  let(:heartbeat){ make_heartbeat(timestamp: 1320196099) }

  #A fake uri for your service endpoint
  let(:uri){ URI.parse("http://fake-service-endpoint/path/to/service") }

  #What is the body of the request to  your service going to look like
  #In this case we are just fowarding the heartbeat directly
  let(:request){ { :body => heartbeat.to_json } }


  context "with invalid options" do
    let(:options){{ }}
    it "is not valid" do
        subject.run
        expect(subject.validate_options).to eq(false)
    end
  end

  context "with valid options" do

    it "is valid" do
        subject.run
        expect(subject.validate_options).to eq(true)
    end

    #Ensure that your plugin hands the package off to your service
    it "forwards events to service" do
      #calling run on your plugin instance will begin its event loop
      subject.run
      expect(subject).to_not receive(:send_http_put_request).with(uri, request)
        subject.process(heartbeat)
      end
    end
  end

end

Trading Options

The two files we havent touched yet are configuration and template files that tell BOSH how to pass our options in to the plugin.

First we have /release/jobs/health_monitor/templates/health_monitor.yml.erb which is a template that tells BOSH how to apply options from spiff templates. Note that option1 is required as it is not wrapped in an if_p block. Deploying BOSH with consul_enabled, without this set will result in an error. But option2 will be skipped if it does not exist in your manifest.

  ...
  <% if p("hm.my_plugin_enabled") %>
- name: my_plugin
  events:
    - alert
    - heartbeat
  options:
    option1: <%= p("hm.my_plugin.option1") %>
    <% if_p("hm.my_plugin.option2") %>
    option2: <%= p("hm.my_plugin.option2") %>
    <%end%>
...
<% end %>  
 ...

And lastly we have /release/jobs/health_monitor/spec which defines all of your options for BOSH.

  hm.my_plugin_enabled:
    description: Enable Your Service
    default: false
  hm.my_plugin.option1:
    description: What is option 1?
  hm.my_plugin.option2:
    description: What is option2?

That should be enough to get you started. You can run your tests from the bosh-monitor directory with bundle exec rspec and you can get more help here

Finally, once you have a nice simple plugin with passing tests you'll want to test your plugin in a real world example.

Run the rake task to create a bosh release of your modified bosh

bundle exec rake release:create_dev_release  

This will muddy up your local branch with release details so you may want to commit your current branch and create a new playground branch for the deployment.

Now upload to your microbosh

bundle exec rake release:upload_dev_release  

Now you can get your dev release version by running bosh releases

Create a manifest file and add your plugin options to the health monitor section

properties:  
  hm:
    my_plugin_enabled: true
    my_plugin:
      option1: 'foo'
      option2: 'bar'

Now deploy your BOSH and test your plugin in a real setting.

bosh deployment path/to/manifest.yml  
bosh deploy  

You'll want to deploy something non trivial to the BOSH in order to generate some nats traffic for monitoring. Since this is a series about monitoring Cloud Foundry. We'd recommend starting there.

Next up we'll be releasing announcing a new plugin. Stay tuned.