Capture HTTP Request Content in Cloud Foundry Using “Gotcha”

I was working on an application, which talks to a few API’s to set up a lab environment on a system I’m working on. This system happens to be using Cloud Foundry to not only host its UI, but also its underlying API’s. The API’s I’m using in this case, while the source code is open source, don’t have a lot of documentation, as they aren’t intended to be publicly facing.

Some of the time I’m able to infer the structure of my API call by watching cf logs as I interact with the UI, which in turn calls the API’s. This works great for things like GET and DELETE. However – what I was not seeing in cf logs was HTTP request content, as you would see in a PUT or POST.

Since we’re dealing with open source, I could trace through their code and figure out how to structure the body. However – there is now a faster way!

Enter Gotcha, another project by one of Stark & Wayne’s resident innovators – James Hunt. Gotcha is a MiTM proxy that intercepts traffic meant for your HTTP API and logs exactly what is sent to it. Here’s how you use it in Cloud Foundry:

Switch Up The Routes

In order to intercept traffic meant for your API, Gotcha needs to listen on the same hostname that is currently being used by your API. We’re then going to use an alternate route to get to the API, so Gotcha can proxy traffic to it.

First, unmap the route from your API:

cf unmap-route my-api testbed.budnack.net --hostname api

Next, map an alternate route:

cf map-route my-api testbed.budnack.net --hostname api-actual
Prep Gotcha for CF Deployment

Now, we’re ready to insert Gotcha into the middle of all this.

First, clone the repository:

git clone [email protected]:starkandwayne/gotcha.git

Next, go into the gotcha directory and create a file named Procfile with the following contents:

web: gotcha http://api-actual.testbed.budnack.net $PORT

Notice the $PORT variable here – this allows Gotcha to listen on the port number assigned to it by Cloud Foundry.

Next, push the app:

cf push gotcha-my-api -d testbed.budnack.net -n api
Tail The Logs and Send Some Traffic

Once Gotcha is running, you can use cf logs to tail the logs on both the application and on Gotcha.

cf logs my-api
cf logs gotcha-my-api

In this next example, I initiate a data transfer via the application’s UI. In the api’s logs, I see the POST request come through:

2016-07-14T18:56:01.70-0400 [RTR/0]      OUT api-actual.testbed.budnack.net - [14/07/2016:22:56:01 +0000] "POST /rest/downloader/requests HTTP/1.1" 200 270 1488 "-" "Java/1.8.0_65-" 10.0.0.162:35686 x_forwarded_for:"52.202.197.223" x_forwarded_proto:"http" vcap_request_id:2ff5fdd5-bbfc-4ba6-58fa-bf53d1bb4d5f response_time:0.391775647 app_id:4c972f0e-84bf-4437-ad91-e89545868720

Now let’s see that request in Gotcha’s logs. As you can see, we get a wealth of information here, including the POST request format we’re looking for (in bold):

2016-07-14T18:56:01.31-0400 [App/0]      ERR POST /rest/downloader/requests HTTP/1.1
2016-07-14T18:56:01.31-0400 [App/0]      ERR Host: api-actual.testbed.budnack.net
2016-07-14T18:56:01.31-0400 [App/0]      ERR User-Agent: Java/1.8.0_65-
2016-07-14T18:56:01.31-0400 [App/0]      ERR Transfer-Encoding: chunked
2016-07-14T18:56:01.31-0400 [App/0]      ERR Accept: application/json, application/*+json
2016-07-14T18:56:01.31-0400 [App/0]      ERR Authorization: bearer REDACTEDFORBLOGPOST
2016-07-14T18:56:01.31-0400 [App/0]      ERR Content-Type: application/json;charset=UTF-8
2016-07-14T18:56:01.31-0400 [App/0]      ERR X-Cf-Applicationid: f1f2b239-d26c-4fd9-9a7a-8d0aca91e17d
2016-07-14T18:56:01.31-0400 [App/0]      ERR X-Cf-Instanceid: 81b4f26753aa4253844f03b8276cb84496a748f7570f4b7ba6c04bce028e1f5b
2016-07-14T18:56:01.31-0400 [App/0]      ERR X-Cf-Requestid: bd01af62-8e4b-4c0e-7fe8-4afd108582f5
2016-07-14T18:56:01.31-0400 [App/0]      ERR X-Forwarded-For: 52.202.197.223, 10.0.0.162
2016-07-14T18:56:01.31-0400 [App/0]      ERR X-Forwarded-Proto: http
2016-07-14T18:56:01.31-0400 [App/0]      ERR X-Request-Start: 1468536961307
2016-07-14T18:56:01.31-0400 [App/0]      ERR X-Vcap-Request-Id: b2d43ea3-3537-43a5-7ac0-02ed197d001f
2016-07-14T18:56:01.31-0400 [App/0]      ERR Accept-Encoding: gzip
2016-07-14T18:56:01.31-0400 [App/0]      ERR 1
2016-07-14T18:56:01.31-0400 [App/0]      ERR {
2016-07-14T18:56:01.31-0400 [App/0]      ERR 10d
2016-07-14T18:56:01.31-0400 [App/0]      ERR "orgUUID":"2bc11e2a-5b04-4fb1-967c-9d78b529a670","source":"https://github.com/trustedanalytics/dataset-reader-sample/raw/master/data/nf-data-application.csv","callback":"http://das.testbed.budnack.net/rest/das/callbacks/downloader/d326167b-edf7-4b20-8d77-ea553efe4df4"}
2016-07-14T18:56:01.31-0400 [App/0]      ERR 0
2016-07-14T18:56:01.71-0400 [RTR/0]      OUT api.testbed.budnack.net - [14/07/2016:22:56:01 +0000] "POST /rest/downloader/requests HTTP/1.1" 200 270 1488 "-" "Java/1.8.0_65-" 10.0.0.162:35685 x_forwarded_for:"52.202.197.223" x_forwarded_proto:"http" vcap_request_id:b2d43ea3-3537-43a5-7ac0-02ed197d001f response_time:0.404644956 app_id:f1f2b239-d26c-4fd9-9a7a-8d0aca91e17d
2016-07-14T18:56:01.71-0400 [App/0]      ERR request took 399.586 ms
2016-07-14T18:56:01.71-0400 [App/0]      ERR HTTP/1.1 200 OK
2016-07-14T18:56:01.71-0400 [App/0]      ERR Content-Length: 1488
2016-07-14T18:56:01.71-0400 [App/0]      ERR Cache-Control: no-cache, no-store, max-age=0, must-revalidate
2016-07-14T18:56:01.71-0400 [App/0]      ERR Connection: keep-alive
2016-07-14T18:56:01.71-0400 [App/0]      ERR Content-Type: application/json;charset=UTF-8
2016-07-14T18:56:01.71-0400 [App/0]      ERR Date: Thu, 14 Jul 2016 22:56:15 GMT
2016-07-14T18:56:01.71-0400 [App/0]      ERR Expires: 0
2016-07-14T18:56:01.71-0400 [App/0]      ERR Pragma: no-cache
2016-07-14T18:56:01.71-0400 [App/0]      ERR Server: nginx/1.11.1
2016-07-14T18:56:01.71-0400 [App/0]      ERR X-Application-Context: api:cloud,multitenant-hdfs,proxy:0
2016-07-14T18:56:01.71-0400 [App/0]      ERR X-Cf-Requestid: bd01af62-8e4b-4c0e-7fe8-4afd108582f5
2016-07-14T18:56:01.71-0400 [App/0]      ERR X-Content-Type-Options: nosniff
2016-07-14T18:56:01.71-0400 [App/0]      ERR X-Frame-Options: DENY
2016-07-14T18:56:01.71-0400 [App/0]      ERR X-Xss-Protection: 1; mode=block
2016-07-14T18:56:01.71-0400 [App/0]      ERR {"source":"https://github.com/trustedanalytics/dataset-reader-sample/raw/master/data/nf-data-application.csv","callback":"http://das.testbed.budnack.net/rest/das/callbacks/downloader/d326167b-edf7-4b20-8d77-ea553efe4df4","id":"51","state":"IN_PROGRESS","downloadedBytes":0,"savedObjectId":null,"objectStoreId":null,"token":"REDACTEDFORBLOGPOST"}
2016-07-14T18:56:01.71-0400 [App/0]      ERR receive response took 0.007 ms
2016-07-14T18:56:01.71-0400 [App/0]      ERR send response took 2.039 ms

There you have it – You can leave this running as-is if it performs well enough, or you can even just re-map the original route back to your API, leaving Gotcha waiting in limbo until you need him again for more MiTM hijinks!

Spread the word

twitter icon facebook icon linkedin icon