This article follows along from our series of articles on deploying BOSH and Cloud Foundry using the wonderful new bosh2 CLI.

Now that you have Cloud Foundry running, you'll discover it doesn't do much without some stateful services into which you will store data. Cloud Foundry supports the Open Service Broker API to allow you to integrate any compatible service broker.

Fortunately there is one service broker that you can deploy easily with BOSH 2.0 - the Docker Broker.

We will now deploy the Docker Broker to include PostgreSQL, Redis and MySQL service offerings. It is mouth watering:

$ cf marketplace
service        plans   description  
mysql56        free    MySQL 5.6 service for application development and testing  
postgresql96   free    PostgreSQL 9.6 service for application development and testing  
redis32        free    Redis 3.2 service for application development and testing  

Quick steps

First clone the docker-broker-deployment project.

git clone https://github.com/cloudfoundry-community/docker-broker-deployment  
cd docker-broker-deployment  

Let's introduce a new idea - targeting your BOSH and setting your deployment name using environment variables, instead of the requiring the -e vbox -d docker-broker flags for all bosh commands.

export BOSH_ENVIRONMENT=vbox  
export BOSH_DEPLOYMENT=docker-broker  

Next, if you've installed the bosh2 CLI, then for the sake of future proofing this article, we will alias it to bosh (which will be its future name):

alias bosh=bosh2  

It is now very simple to get the Docker Broker running:

bosh deploy docker-broker.yml --vars-store tmp/creds.yml \  
  -o services/op-postgresql96.yml \
  -o services/op-redis32.yml \
  -o services/op-mysql56.yml

That's it! Up and running.

What is running? Run bosh instances to find out:

Instance                                                 Process State  AZ  IPs  
docker/140b0e37-f0e1-4928-b538-b2e1a829bef9              running        z2  10.244.0.152  
docker/8a3ae2e3-7843-4f3e-a0b3-79e56469267e              running        z1  10.244.0.151  
docker/dde7b97e-3625-491e-815d-ecb3ab084d24              running        z3  10.244.0.153  
sanity-test/69eb7045-58c2-49fc-8727-3fb1c6aed010         -              z1  10.244.0.154  
subway/4bca6846-b0c3-4f84-8d34-d3872b064af9              running        z1  10.244.0.150  

Three docker instances, with a front-facing subway instance which contains the Open Service Broker API interface.

If you've looked in bosh-deployment and cf-deployment repos you'll see a pattern in these three repos - one "just works" base manifest, and a collection optional modifier manifests. In this article's docker-broker-deployment repo, this base manifest is docker-broker.yml.

Does it work?

You can sanity test the deployment to confirm that each of the services is working as expected:

bosh run-errand sanity-test  

When this is finished look carefully at the output. You'll see that postgresql96 and redis32 services actually self-tested - the Docker images that describe each service also know how to sanity test themselves.

For example:

Running frodenas/postgresql:9.6 sanity-test...  
Waiting for postgres://ty4kuimcjem4ma9j:7pxnhxynzorbzana@10.244.0.152:32768/2yubhxqlcm5ivigo to be ready (max 60s)  
10.244.0.152:32768 - accepting connections  
Postgres is ready  
10.244.0.152:32768 - accepting connections  
+ psql postgres://ty4kuimcjem4ma9j:7pxnhxynzorbzana@10.244.0.152:32768/2yubhxqlcm5ivigo -c 'DROP TABLE IF EXISTS sanitytest;'
NOTICE:  table "sanitytest" does not exist, skipping  
DROP TABLE  
+ psql postgres://ty4kuimcjem4ma9j:7pxnhxynzorbzana@10.244.0.152:32768/2yubhxqlcm5ivigo -c 'CREATE TABLE sanitytest(value text);'
CREATE TABLE  
+ psql postgres://ty4kuimcjem4ma9j:7pxnhxynzorbzana@10.244.0.152:32768/2yubhxqlcm5ivigo -c 'INSERT INTO sanitytest VALUES ('\''storage-test'\'');'
INSERT 0 1  
+ psql postgres://ty4kuimcjem4ma9j:7pxnhxynzorbzana@10.244.0.152:32768/2yubhxqlcm5ivigo -c 'SELECT value FROM sanitytest;'
+ grep storage-test
 storage-test

Nice. Any Docker image you use in Docker Broker can also be used to self-test that the service is working as expected. See frodenas/docker-postgresql as an example.

Is it secure?

Yes. The Docker daemons are all protected by TLS certificates and the Broker API is protected by random password. In this article at no time will you personally need to see these passwords or certificates.

Easy helpful generation of certificates and passwords is one of the new features of bosh2 CLI, as well as the up-coming Credhub project.

Integration with Cloud Foundry

This article is about integrating the Docker Broker with Cloud Foundry. For that we redeploy the command above, but include the op-cf-integration.yml file and register the API as a Cloud Foundry Service Broker.

First, fetch the system_domain and the cf_admin_password data from the cf deployment:

bosh -d cf manifest > tmp/cf.yml  
system_domain=$(bosh int tmp/cf.yml --path /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/system_domain)  
cf_admin_password=$(bosh int tmp/cf.yml --path /instance_groups/name=uaa/jobs/name=uaa/properties/uaa/scim/users/name=admin/password)  

Next, add the -o op-cf-integration.yml operator patch file, plus the additional variables required so we know into which Cloud Foundry to register our Docker Broker.

bosh deploy docker-broker.yml --vars-store tmp/creds.yml \  
  -o services/op-postgresql96.yml \
  -o services/op-redis32.yml \
  -o services/op-mysql56.yml \
  -o op-cf-integration.yml \
  -v cf-api-url=https://api.$system_domain \
  -v broker-route-name=docker-broker \
  -v broker-route-uri=docker-broker.$system_domain \
  -v cf-admin-password=$cf_admin_password \
  -v cf-admin-username=admin \
  -v cf-skip-ssl-validation=true

Once completed, you can register the broker:

bosh run-errand broker-registrar  

The output will include this magic:

Enabling access to all plans of service postgresql96 for all orgs as admin...  
OK  
Enabling access to all plans of service redis32 for all orgs as admin...  
OK  
Enabling access to all plans of service mysql56 for all orgs as admin...  
OK  
Getting service access as admin...  
broker: docker-broker  
   service        plan   access   orgs
   mysql56        free   all
   redis32        free   all
   postgresql96   free   all

With your cf CLI you can now see the three services ready for action:

cf marketplace  

The output will as shown at the top of the article. The three -o services/op-*.yml files appear as services:

service        plans   description  
mysql56        free    MySQL 5.6 service for application development and testing  
postgresql96   free    PostgreSQL 9.6 service for application development and testing  
redis32        free    Redis 3.2 service for application development and testing  

Three brand new service offerings for everyone to use!

cf create-service postgresql96 free pg1  
cf create-service-key pg1 pg1-key  
cf service-key pg1 pg1-key  

The output will rock your world:

Getting key pg1-key for service instance pg1 as admin...

{
 "dbname": "zvenlj1owjcyj4ot",
 "hostname": "10.244.0.152",
 "password": "8elayohhjptrfqzo",
 "port": "32768",
 "ports": {
  "5432/tcp": "32768"
 },
 "uri": "postgres://mkfqd5yzxbfqitjh:8elayohhjptrfqzo@10.244.0.152:32768/zvenlj1owjcyj4ot",
 "username": "mkfqd5yzxbfqitjh"
}

This is a PostgreSQL service with randomly generated auth credentials that is ready to use.

Backup & restore

I am repulsed by data services that do not have functioning, tested, well known backup & restore systems in place. Yuck.

But today, what you've deployed above doesn't have backup & restore. So... don't share it with anyone you love just yet.

We will introduce Shield to this deployment in a future article to provide backup & restore to each service instance.