I was working on an application, which talks to a few API’s to set up a lab environment on a system I’m working on. This system happens to be using Cloud Foundry to not only host its UI, but also its underlying API’s. The API’s I’m using in this case, while the source code is open source, don’t have a lot of documentation, as they aren’t intended to be publicly facing.
Some of the time I’m able to infer the structure of my API call by watching cf logs
as I interact with the UI, which in turn calls the API’s. This works great for things like GET and DELETE. However – what I was not seeing in cf logs
was HTTP request content, as you would see in a PUT or POST.
Since we’re dealing with open source, I could trace through their code and figure out how to structure the body. However – there is now a faster way!
Enter Gotcha, another project by one of Stark & Wayne’s resident innovators – James Hunt. Gotcha is a MiTM proxy that intercepts traffic meant for your HTTP API and logs exactly what is sent to it. Here’s how you use it in Cloud Foundry:
Switch Up The Routes
In order to intercept traffic meant for your API, Gotcha needs to listen on the same hostname that is currently being used by your API. We’re then going to use an alternate route to get to the API, so Gotcha can proxy traffic to it.
First, unmap the route from your API:
cf unmap-route my-api testbed.budnack.net --hostname api
Next, map an alternate route:
cf map-route my-api testbed.budnack.net --hostname api-actual
Prep Gotcha for CF Deployment
Now, we’re ready to insert Gotcha into the middle of all this.
First, clone the repository:
git clone [email protected]:starkandwayne/gotcha.git
Next, go into the gotcha
directory and create a file named Procfile
with the following contents:
web: gotcha http://api-actual.testbed.budnack.net $PORT
Notice the $PORT variable here – this allows Gotcha to listen on the port number assigned to it by Cloud Foundry.
Next, push the app:
cf push gotcha-my-api -d testbed.budnack.net -n api
Tail The Logs and Send Some Traffic
Once Gotcha is running, you can use cf logs
to tail the logs on both the application and on Gotcha.
cf logs my-api
cf logs gotcha-my-api
In this next example, I initiate a data transfer via the application’s UI. In the api’s logs, I see the POST request come through:
2016-07-14T18:56:01.70-0400 [RTR/0] OUT api-actual.testbed.budnack.net - [14/07/2016:22:56:01 +0000] "POST /rest/downloader/requests HTTP/1.1" 200 270 1488 "-" "Java/1.8.0_65-" 10.0.0.162:35686 x_forwarded_for:"52.202.197.223" x_forwarded_proto:"http" vcap_request_id:2ff5fdd5-bbfc-4ba6-58fa-bf53d1bb4d5f response_time:0.391775647 app_id:4c972f0e-84bf-4437-ad91-e89545868720
Now let’s see that request in Gotcha’s logs. As you can see, we get a wealth of information here, including the POST request format we’re looking for (in bold):
2016-07-14T18:56:01.31-0400 [App/0] ERR POST /rest/downloader/requests HTTP/1.1 2016-07-14T18:56:01.31-0400 [App/0] ERR Host: api-actual.testbed.budnack.net 2016-07-14T18:56:01.31-0400 [App/0] ERR User-Agent: Java/1.8.0_65- 2016-07-14T18:56:01.31-0400 [App/0] ERR Transfer-Encoding: chunked 2016-07-14T18:56:01.31-0400 [App/0] ERR Accept: application/json, application/*+json 2016-07-14T18:56:01.31-0400 [App/0] ERR Authorization: bearer REDACTEDFORBLOGPOST 2016-07-14T18:56:01.31-0400 [App/0] ERR Content-Type: application/json;charset=UTF-8 2016-07-14T18:56:01.31-0400 [App/0] ERR X-Cf-Applicationid: f1f2b239-d26c-4fd9-9a7a-8d0aca91e17d 2016-07-14T18:56:01.31-0400 [App/0] ERR X-Cf-Instanceid: 81b4f26753aa4253844f03b8276cb84496a748f7570f4b7ba6c04bce028e1f5b 2016-07-14T18:56:01.31-0400 [App/0] ERR X-Cf-Requestid: bd01af62-8e4b-4c0e-7fe8-4afd108582f5 2016-07-14T18:56:01.31-0400 [App/0] ERR X-Forwarded-For: 52.202.197.223, 10.0.0.162 2016-07-14T18:56:01.31-0400 [App/0] ERR X-Forwarded-Proto: http 2016-07-14T18:56:01.31-0400 [App/0] ERR X-Request-Start: 1468536961307 2016-07-14T18:56:01.31-0400 [App/0] ERR X-Vcap-Request-Id: b2d43ea3-3537-43a5-7ac0-02ed197d001f 2016-07-14T18:56:01.31-0400 [App/0] ERR Accept-Encoding: gzip 2016-07-14T18:56:01.31-0400 [App/0] ERR 1 2016-07-14T18:56:01.31-0400 [App/0] ERR { 2016-07-14T18:56:01.31-0400 [App/0] ERR 10d 2016-07-14T18:56:01.31-0400 [App/0] ERR "orgUUID":"2bc11e2a-5b04-4fb1-967c-9d78b529a670","source":"https://github.com/trustedanalytics/dataset-reader-sample/raw/master/data/nf-data-application.csv","callback":"http://das.testbed.budnack.net/rest/das/callbacks/downloader/d326167b-edf7-4b20-8d77-ea553efe4df4"} 2016-07-14T18:56:01.31-0400 [App/0] ERR 0 2016-07-14T18:56:01.71-0400 [RTR/0] OUT api.testbed.budnack.net - [14/07/2016:22:56:01 +0000] "POST /rest/downloader/requests HTTP/1.1" 200 270 1488 "-" "Java/1.8.0_65-" 10.0.0.162:35685 x_forwarded_for:"52.202.197.223" x_forwarded_proto:"http" vcap_request_id:b2d43ea3-3537-43a5-7ac0-02ed197d001f response_time:0.404644956 app_id:f1f2b239-d26c-4fd9-9a7a-8d0aca91e17d 2016-07-14T18:56:01.71-0400 [App/0] ERR request took 399.586 ms 2016-07-14T18:56:01.71-0400 [App/0] ERR HTTP/1.1 200 OK 2016-07-14T18:56:01.71-0400 [App/0] ERR Content-Length: 1488 2016-07-14T18:56:01.71-0400 [App/0] ERR Cache-Control: no-cache, no-store, max-age=0, must-revalidate 2016-07-14T18:56:01.71-0400 [App/0] ERR Connection: keep-alive 2016-07-14T18:56:01.71-0400 [App/0] ERR Content-Type: application/json;charset=UTF-8 2016-07-14T18:56:01.71-0400 [App/0] ERR Date: Thu, 14 Jul 2016 22:56:15 GMT 2016-07-14T18:56:01.71-0400 [App/0] ERR Expires: 0 2016-07-14T18:56:01.71-0400 [App/0] ERR Pragma: no-cache 2016-07-14T18:56:01.71-0400 [App/0] ERR Server: nginx/1.11.1 2016-07-14T18:56:01.71-0400 [App/0] ERR X-Application-Context: api:cloud,multitenant-hdfs,proxy:0 2016-07-14T18:56:01.71-0400 [App/0] ERR X-Cf-Requestid: bd01af62-8e4b-4c0e-7fe8-4afd108582f5 2016-07-14T18:56:01.71-0400 [App/0] ERR X-Content-Type-Options: nosniff 2016-07-14T18:56:01.71-0400 [App/0] ERR X-Frame-Options: DENY 2016-07-14T18:56:01.71-0400 [App/0] ERR X-Xss-Protection: 1; mode=block 2016-07-14T18:56:01.71-0400 [App/0] ERR {"source":"https://github.com/trustedanalytics/dataset-reader-sample/raw/master/data/nf-data-application.csv","callback":"http://das.testbed.budnack.net/rest/das/callbacks/downloader/d326167b-edf7-4b20-8d77-ea553efe4df4","id":"51","state":"IN_PROGRESS","downloadedBytes":0,"savedObjectId":null,"objectStoreId":null,"token":"REDACTEDFORBLOGPOST"} 2016-07-14T18:56:01.71-0400 [App/0] ERR receive response took 0.007 ms 2016-07-14T18:56:01.71-0400 [App/0] ERR send response took 2.039 ms
There you have it – You can leave this running as-is if it performs well enough, or you can even just re-map the original route back to your API, leaving Gotcha waiting in limbo until you need him again for more MiTM hijinks!