How we open-sourced (some of) our monorepo
When we decided to open-source some of our code, our goal was to bring many of the benefits of developing in a monorepo to our open-source projects, while keeping each project relatively small and purpose-driven. We wanted to find a way to make a similar guarantee of cross-project compatibility, without forcing contributors into a large repo with a custom tool chain. We wanted contributors to be able to build, modify, and test individual projects easily, and to spread out into neighboring projects just as easily. In short, we wanted to have our cake and eat it too. This post describes how we bake that cake.
There Can Be Only One
The source of truth for code at Turbine Labs is our internal monorepo. This is not really a blog post about monorepos, and I will avoid defending them beyond stating that, for us, a monorepo provides atomicity of commit across multiple projects, which lets us make a strong guarantee about the integrity of our source code at any point in time. We like this guarantee. A lot.
We partition and push commits from our monorepo as patches to smaller open-source projects, at a regular cadence. This is a common pattern — so common that we were able to use Facebook’s FBShipIt more or less off the shelf to do the actual partitioning and pushing. With a few tweaks to the workflow and the addition of an important invariant, we’re able to regularly ship a collection of open-sourced projects that can be used individually or in concert, with strong guarantees of cross-project compatibility, and a full commit history.
We release and version all Turbine Labs open-source projects and public artifacts in lock-step, guaranteeing the following invariant:
All Turbine Labs projects and artifacts tagged with the same version can be safely used together.
The version tags follow Semantic Versioning rules, but applied to the aggregation of all projects. We push all projects in every open-source release (though some will be unchanged), after which we tag them all with the new version. We push new docker images of tbncollect, tbnproxy, and our all-in-one demo as well. Everything works well together, because it already works together in our monorepo.
The main benefit of this approach is that contributors can work comfortably in individual projects, while enjoying the compatibility guarantees of our monorepo as they extend their reach. The downside is that the version increments will seem pretty chatty, especially for lower-velocity projects. We may introduce project-specific semantic version numbers in the future, but for now we are avoiding the operational overhead.
Because our open source export is one-way, pull requests from contributors are not merged directly into the open-source repositories; Instead, approved pull requests are applied as a patch to our monorepo, and then pushed back out. A side benefit of this approach is that it’s possible to submit a suite of pull requests across several projects to make a broader change. We’ll merge them as a single commit, knowing we can build and test our entire tree with the new code.
Tell Me More!
For a detailed run-down of the projects we’ve open-sourced, and more about how we did it and why, please head over to our developer repository. You can also read our documentation at docs.turbinelabs.io, which is itself open-sourced. If you’re interesting in enabling your team to ship more often with greater confidence, visit our website and sign up for a free 30 day trial.