I have been investigating using BOSH with Windows stemcells to help Windows developers have all the delights of BOSH for their deployment lifecycle.

Very quickly in the journey I created bugs that needed debugging, and it wasn't the same debugging experience for a BOSH/Linux deployment. I got help. Hopefully the debugging tips below are helpful to you when you've got an .exe that needs to run on Windows and you want to use BOSH to do it.

Summary documentation for BOSH on Windows is available which was helpful.

The screen images in this blog post are from Windows Server 2012R2 stemcells which I found at https://bosh.io/stemcells

To experiment with BOSH on Windows I created a sample BOSH release https://github.com/cloudfoundry-community/sample-go-windows-boshrelease which borrowed heavily from https://github.com/cloudfoundry-incubator/garden-windows-bosh-release

It compiles a Hello World Golang web server that runs on port 3000. But it timed out when I tried to access the app:

$ curl 10.0.0.5:3000 -v
* Rebuilt URL to: 10.0.0.5:3000/
* Hostname was NOT found in DNS cache
*   Trying 10.0.0.5...
* connect to 10.0.0.5 port 3000 failed: Connection timed out
* Failed to connect to 10.0.0.5 port 3000: Connection timed out
* Closing connection 0
curl: (7) Failed to connect to 10.0.0.5 port 3000: Connection timed out  

This was deployed on Google Compute Platform (GCP) and I knew my GCP firewalls allowed any app on the bosh-bastion to access any VM in the subnet, including my Windows VM running at 10.0.0.5.

Debugging time.

Fetching BOSH job logs

At the time of this writing, bosh2 logs --follow does not work on Windows VMs to stream the logs of running jobs, however, we still have the static dump version of bosh2 logs:

$ bosh2 logs -d simple-go-web-app
...
22:40:34 | Fetching logs for webapp/b95a2a12-b7c4-4c46-9003-713d64c013c0 (0): Finding and packing log files (00:00:01)  
...
Downloading resource 'b174c5b7-da7c-40aa-5926-2af98a6332b2' to '/home/drnic/workspace/sample-go-windows-boshrelease/simple-go-web-app-20170723-224036-02661666.tgz'...  

To unpack and view all the logs:

$ tar xfz simple-go-web-app-*; tail -n 200 simple-go-web-app/simple-go-web-app/* simple-go-web-app/*
...
==> simple-go-web-app/simple-go-web-app/job-service-wrapper.out.log <==
[martini] listening on :3000 (development)
...

It seems that the simple-go-web-app.exe compiled Golang app is successfully running and listening on port 3000.

Delete the logs:

rm -rf simple-go-web-app*  

A one-line combo I used was:

bosh2 logs -d simple-go-web-app && tar xfz simple-go-web-app-* && tail -n 200 simple-go-web-app/simple-go-web-app/* simple-go-web-app/* | less && rm -rf simple-go-web-app*  

These logs showed that the simple-go-web-app.exe was running and it thought it was listening on 3000.

Using RDP to access a Windows VM

Normally to access a BOSH VM we would use bosh2 ssh to get a secure terminal. This does not work with Windows servers. Instead, we must use the Microsoft Remote Desktop Protocol (RDP) [download links].

Currently RDP access and authentication is managed by the cloud provider, not via BOSH.

With luck, you can find the VM on GCP dashboard and click "RDP" button:

rdp-dashboard

I wasn't lucky.

Using RDP via an SSH tunnel

No matter how many firewalls or gateways or RDP clients I played with I could not get RDP working.

My eternal thanks go to David Jahn who suggested running an SSH tunnel from my local machine to the Windows VM thru the bastion.

To setup an SSH/RDP tunnel with GCP, where 10.0.0.5 is the GCP IP of the VM to be debugged:

gcloud compute ssh bosh-bastion -- -L 3389:10.0.0.5:3389  

This will open a normal ssh session into bosh-bastion but will transparently include a tunnel from localhost:3389 to 10.0.0.5:3389.

Using Microsoft Remote Desktop, create a new "desktop":

desktop-via-tunnel

The username and password will change for each VM. The localhost:3389 will be available whenever you the gcloud compute ssh bosh-bastion -- -L ... command above is running.

To setup a username and password on the Windows VM, click "Set Windows password" and follow the prompts.

windows-auth

Initial look at how BOSH runs on Windows

As a quick sanity test, I confirmed that files and paths were as I expected.

In a PowerShell terminal, run:

cmd /c start /b set  

The Path env var were displayed in full:

Path=C:\windows\system32;C:\windows;C:\windows\System32\Wbem;C:\windows\System32\WindowsPowerShell\v1.0\;C:\ProgramData\  
GooGet;C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files\Google\Compute Engine\metadata_scri  
pts;C:\Program Files\Google\Compute Engine\sysprep;C:\var\vcap\bosh\bin  

The BOSH packages were accessible via the normal folder aliases, within the c:\ drive:

C:\Users\drnicwilliams>dir c:\var\vcap\packages

    Directory: C:\var\vcap\packages

Mode                LastWriteTime         Length Name  
----                -------------         ------ ----
d----l        7/23/2017   9:38 PM                simple-go-web-app  

Firewalls were enabled by default

Using Internet Explorer, after disabling IE default security, I could view the http://localhost:3000 application locally:

localhost3000

This confirmed that the application was running as expected and can be accessed locally.

Therefore the Windows Server must have additional firewall rules that aren't managed by GCP or BOSH that are blocking ingress traffic to port 3000.

To test, I disabled the private firewalls:

disable-private-firewalls

Success. The curl request from bosh-bastion now worked:

$ curl 10.0.0.5:3000
Hello world  

Configuring Windows Firewall from BOSH

Manually changing the firewall rules only demonstrated where the issue was. The BOSH release needed a specific firewall rule to allow TCP traffic into port 3000 (or whatever port the application is bound to) every time the release was deployed.

Windows Firewalls can be scripted using PowerShell New-NetFirewallRule command. To run PowerShell using BOSH, one option is to include a bin/pre-start.ps1 script.

New-NetFirewallRule -LocalPort <%= p("port") %> `  
  -Protocol TCP `
  -Direction Inbound `
  -Name simple-go-web-app `
  -DisplayName simple-go-web-app

See pre-start.ps1 in the final BOSH job.

Other PowerShell hooks are documented https://bosh.io/docs/windows.html

Summary

When deploying Windows Server VMs with BOSH we do lose a couple of standard debugging options we have with traditional BOSH Linux VMs. Instead we can use Microsoft RDP to get a Windows desktop session, as well as download the BOSH job logs using bosh2 logs.

The Windows stemcell has default firewall rules. If your BOSH jobs require additional ingress or egress rules, use a pre-start.ps1 script to run PowerShell commands.