Company A

Client Background

Company A is a large industrial farm equipment manufacturer.

Company A had three separate Cloud Foundry environments up and running; two production environments, both with assigned data centers that had been running for a year, and a third development environment that was not yet linked to a data center.

The production environments were critically important to the day-to-day operation of the business, as they hosted dealership-related services.

The Challenge

With the loss of a key employee and the institutional knowledge that went with him, the remaining team members at Company A weren't actively monitoring certificates or the respective re-issuance dates.

Due to the problems during this transition of responsibility, some key re-issuance dates had already passed, leading to the expiration of BOSH-related certificates for the development environment, thus making it inoperable.

Soon after the development environment went down, the BOSH-related certificates for the first production environment also expired, leading to another outage.

Fortunately, Company A's high-availability design forced all traffic to go through the second production environment, enabling Company A's operations to continue running smoothly.

These outages acted as a clear warning to the Company A team, alerting them to the problem of expiring certificates along with the potentially disastrous consequences of having a second production become inoperable.

Because of the sequence of certificate expiration's, it also gave the team a three-day warning period before the BOSH certificate for the second production would expire.

The clock was ticking and, with the possibility of having day-to-day operations grinding to a halt, the team at Company A called on Stark & Wayne for assistance.

That's when Stark & Wayne's experience with Cloud Foundry and our collective approach enabled us to find a solution.

The Solution & Next Steps

Stark & Wayne's engineers quickly gained access to BOSH and identified each certificate that had expired. With reissuing the certificates ruled out as a viable option, the experts at Stark & Wayne were undeterred and devised another plan for what to do.

Relying on the experience, the Stark & Wayne team decided to extract the certificates from the existing deployments and create renewal requests.

By taking this approach, Stark & Wayne was able to extend the expiration dates on the existing certificates, leveraging BOSH deployment paradigms to update the certificates for each VM through purpose-built software using Company A's last standing production environment.

This secondary approach, unfamiliar to most people with little Cloud Foundry experience, helped Company A avoid what could have been a catastrophic period of downtime, allowing Stark & Wayne to keep the one production environment and restore the two inoperable environments.

During this engagement, Stark & Wayne leveraged their Collective approach, which advised the onsite team to use the Stark & Wayne Safe certificate management tool for the job.

This choice tool, based on all of Stark & Wayne's considerable experience, was an important factor in having the onsite team manage the process more effectively, allowing them to complete the job of renewing certificates a full day ahead of the projects three-day deadline.

To help prevent future outages, Stark & Wayne also taught Company A's operation team how to use Doomsday, a Stark & Wayne-developed tool used to alert personnel automatically when certificates are nearing their expiration date.

This reduces the team's reliance on tracking certificate expiration manually.

Stark & Wayne supplied best practices architecture recommendations to Company avoid BOSH certificate problems going forward.