I think I've tried most of the CI tools out there at some point. It took me a while to get around to Buildkite because at first I didn't see much point given it seemed like you had to run the agent yourself. Eventually it dawned on me why this approach was more ingenious than I realised:

Running my app in a production (or production-like) environment was already a solved problem, because everything was already in some form of "everything as code". Having a test environment where the only difference was adding the Buildkite agent was a trivial addition.

It means that dev/test/prod parity is simple to achieve and maintain. It's also proven to be much easier to support than trying to deal with the problems that come with trying to force an app to fit into the nuances and constraints that are imposed by the containers/runtime of a CI service. When you completely control all of the environment the tests are running in you define those constraints too. It's been a great balance between a managed service and the flexibility of running it yourself.

And while none of my needs have hit the scale of Shopify (I saw one of their engineers speak about it at a conference once, I can't find the video now though 😞) it's good to know I can scale out my worker nodes to hundreds of thousands of workers to reduce the time it takes for my tests to run.

Glenn Gillen