Oct 29, 2015 Pre-Flight Checks: Sprucing up Concourse with Test Concourse
As part of a project, a client wants to have a self-deploying Concourse. Basically that means that once everything is set up, the alpha Concourse will deploy the beta Concourse and, if that completes successfully, the beta Concourse will then update/deploy the alpha Concourse. Because automation is shiny.
Ensure that the the beta ("test") Concourse mirrors the alpha ("production") Concourse.
What This Means
Basically, both Concourses need to be configured identically except for the necessary deviations for networking.
By the way, since the client is using vSphere I am building this off Concourse's
vsphere.yml example manifest.
CDC Pipeline: Concourse Deploys Concourse
For efficiency, I
concourse.yml manifest. What is
A quick note about
spruce is a CLI tool primarily being developed by Geoff Franks. The goal is to have
spruce be the next generation replacement for
spiff, which is the current tool used to generate BOSH manifests. Here's its inaugural blog post and here is where the code is located on Github.
A Quick How To
First things first: I defined the static IPs in the
concourse network like this:
static: - x.x.x.50 - x.x.x.60
Now with spruce I can reference the IP address
static_ips(0) in the
concourse network, likewise
static_ips(1), etc. So this:
networks: - name: concourse static_ips: x.x.x.50
Is the same as this:
networks: - name: concourse static_ips: (( static_ips(0) ))
Instead of using anchor syntax for the ATC credentials like the Concourse example manifest, I created
meta at the top level and
grabbed both values:
meta: atc_db_name: atc atc_db_role: name: atc password: ATCPASSWORD ... properties: atc: postgresql: database: (( grab meta.atc_db_name )) role: (( grab meta.atc_db_role ))
grabbed IP addresses referenced outside of the
concourse network like so:
consul: agent: servers: lan: (( grab jobs.discovery.networks..static_ips ))
alpha.yml, the manifest for the alpha Concourse's BOSH deploy, run:
spruce --prune meta alpha-concourse.yml > alpha.yml
Note that the sections in the output will be alphabetized, which can be a little disorienting if you don't expect it.
Now For The Beta Concourse
Creating the beta Concourse manifest is easy as pi(e) since I will only be changing the networking. In fact, there are only two top level keys: the name and networks. That's it!
beta.yml, the manifest for the beta Concourse's BOSH deploy, run:
spruce merge --prune meta alpha-concourse.yml beta-concourse.yml > beta.yml
Order matters with
spruce - the order above means that the second file overrides the first. This priority holds true no matter how many files you merge.
Some Cool Things
spruce is very helpful in this project for a couple reasons:
- If I ever need to change the static IP range in the future I don't need to go through the manifest and fix the IPs in multiple places.
- Relatedly, this is why the test manifest is so small: since all the static IPs are generated, all I had to do was change the IP ranges in the network. No fishing around for stray "old" IP addresses here!
- Tucking the ATC credentials under
metawas convenient for referencing the credentials throughout the file and since
prunefeature I can eliminate that section from the resulting manifest.
S&W CDC Templates
In our CDC repository I have added:
- The vSphere
spruced concourse templates - AWS and BOSH Lite to follow
- Scripts to make the manifests and to deploy the pipeline are in
- The Concourse pipeline files for each pipeline.