Pre-Flight Checks: Sprucing up Concourse with Test Concourse

As part of a project, a client wants to have a self-deploying Concourse. Basically that means that once everything is set up, the alpha Concourse will deploy the beta Concourse and, if that completes successfully, the beta Concourse will then update/deploy the alpha Concourse. Because automation is shiny.

Current Goal

Ensure that the the beta ("test") Concourse mirrors the alpha ("production") Concourse.

What This Means

Basically, both Concourses need to be configured identically except for the necessary deviations for networking.

Important Note

By the way, since the client is using vSphere I am building this off Concourse’s vsphere.yml example manifest.

CDC Pipeline: Concourse Deploys Concourse

Sprucify

For efficiency, I spruced the concourse.yml manifest. What is spruce?

A quick note about spruce

spruce is a CLI tool primarily being developed by Geoff Franks. The goal is to have spruce be the next generation replacement for spiff, which is the current tool used to generate BOSH manifests. Here‘s its inaugural blog post and here is where the code is located on Github.

A Quick How To

First things first: I defined the static IPs in the concourse network like this:

static:
- x.x.x.50 - x.x.x.60

Now with spruce I can reference the IP address x.x.x.50 as static_ips(0) in the concourse network, likewise x.x.x.51 is static_ips(1), etc. So this:

networks:
  - name: concourse
    static_ips: x.x.x.50

Is the same as this:

networks:
  - name: concourse
    static_ips: (( static_ips(0) ))

Instead of using anchor syntax for the ATC credentials like the Concourse example manifest, I created meta at the top level and grabbed both values:

meta:
  atc_db_name: atc
  atc_db_role:
    name: atc
    password: ATCPASSWORD
...
properties:
  atc:
    postgresql:
      database: (( grab meta.atc_db_name ))
      role: (( grab meta.atc_db_role ))

I grabbed IP addresses referenced outside of the concourse network like so:

consul:
  agent:
    servers:
      lan: (( grab jobs.discovery.networks.[0].static_ips ))

To create alpha.yml, the manifest for the alpha Concourse’s BOSH deploy, run:

spruce --prune meta alpha-concourse.yml > alpha.yml

Note that the sections in the output will be alphabetized, which can be a little disorienting if you don’t expect it.

Now For The Beta Concourse

Creating the beta Concourse manifest is easy as pi(e) since I will only be changing the networking. In fact, there are only two top level keys: the name and networks. That’s it!

To generate beta.yml, the manifest for the beta Concourse’s BOSH deploy, run:

spruce merge --prune meta alpha-concourse.yml beta-concourse.yml > beta.yml

Order matters with spruce – the order above means that the second file overrides the first. This priority holds true no matter how many files you merge.

Some Cool Things

spruce is very helpful in this project for a couple reasons:

• If I ever need to change the static IP range in the future I don’t need to go through the manifest and fix the IPs in multiple places.
  • Relatedly, this is why the test manifest is so small: since all the static IPs are generated, all I had to do was change the IP ranges in the network. No fishing around for stray "old" IP addresses here!
• Tucking the ATC credentials under meta was convenient for referencing the credentials throughout the file and since spruce has a prune feature I can eliminate that section from the resulting manifest.

S&W CDC Templates

In our CDC repository I have added:

• The vSphere spruced concourse templates – AWS and BOSH Lite to follow
• Scripts to make the manifests and to deploy the pipeline are in bin/
• The Concourse pipeline files for each pipeline.