And so KubeCon 2018 begins. The biggest gathering of the community so far, and the rise of Kubernetes continues. In the course of my career I’ve now seen three major open source projects reach dizzying heights that we in the community would have considered to be impossible for many years before: Linux now runs the world; OpenStack built the largest collaborative software project the world had ever seen; and now Kubernetes looks to be on a similar trajectory.

I’ve recently been working with PySpark, building a natural language processing pipeline demo for DC/OS. This has been a great learning experience, and PySpark provides an easier entry point into the world of Spark programming for a systems guy like myself than having to learn Java or Scala When you’re developing Spark jobs, testing locally is a very different environment from deploying to a cluster, so it’s not always straightforward working out how to deploy to a Spark cluster on DC/OS once you think you’ve got a working job.

Deploying microservices in Docker containers is a great way of ensuring immutability in the operating environment that your microservice needs, giving you exactly the dependencies and runtime versions that your application expects. However, as microservices get smaller, the overhead of running additional filesystems, libraries, binaries and runtimes just to run a few lines of code can seem like overkill. In many cases, Docker containers can end up very large, which somewhat defeats the point of containerisation in the first place.