Multi-tenant ELK for your private Cloud Foundry

If you’re using a public Cloud Foundry – Pivotal Web Services, IBM Blue Mix – then you are probably ok with the idea of using a public log storage service like Papertrail or Logentries. What about if you are using your own Cloud Foundry – open source or PivotalCF – and you’re not allowed to use a hosted log service? What are your options?

Our various clients have needed something and I thought I’d share one idea. It has pros and cons.

Pretty?

Events over time:

events-over-time

Line-by-line logs

line-by-line

Pros and cons

The pros are:

  • using well known Elastic Search/Logstash/Kibana stack [ELK]
  • it exists as a solution
  • it allows multi-tenancy isolation of logs
  • easy deployment via bosh-workspace for deploying docker/logstash as a service
  • all components are open source

The cons are:

  • it uses the relatively unmaintained cf-containers-broker project
  • not easily horizontally scalable – the cf-containers-broker project only supports a single Docker server
  • each logstash service instance is running in a single Docker container and cannot be scaled out
  • limited ops tooling to manage the logstash docker containers once they are instantiated
  • the kibana app includes an insecure proxy through to elastic search

But, above all the cons are the pros – it exists and it is multi-tenant.

Perhaps the solution will work for you long enough until a better solution is proposed (hopefully by us, but we’d love any solution).

Why not logsearch?

http://www.logsearch.io/ is a distribution of ELK that include a BOSH release and BOSH workspace and serves as a great backend for Cloud Foundry component logs.

And with a firehose nozzle [see firehose-to-syslog] for loggregator it could also store all applications’ logs.

But there is no multi-tenancy for Kibana – the UI portion of ELK – to restrict what a user can see. We only want a user to see the logs that a) they want to see; b) they have permission to see.

Elastic, the company, does have a product Shield that might solve this problem. I learnt about this yesterday and will investigate it later.

How do we do multi-tenant ELK?

Two components:

  • cf-containers-broker is a Cloud Foundry service broker that locally provisions Docker containers – we can use it to run ELK in a Docker container
  • kibana-me-logs is an app that shows the Kibana UI to display your application’s logs

A platform engineer will deploy the former component once and it will be used by all users as a Cloud Foundry service.

Each application developer will deploy the latter app so they can view their logstash service instance.

This solution means that you application’s logs are isolated inside a dedicated Docker container that runs logstash and elastic search.

To view the logs via Kibana you run the kibana-me-logs app as another Cloud Foundry app.

The images above are examples.

Coming soon

Currently I’m working on a way to make it very simple for platform engineers to deploy the cf-container-broker/Docker component.

To get a head start, your homework is to review:

Spread the word

twitter icon facebook icon linkedin icon