Tuesday, July 04, 2023

Trunk based dev and merging

The unfortunate reality is that new code contains new bugs. Unless you want to stop all feature development while a release candidate is being tested and fixed, you're going to need a mechanism for selectively including code in releases.

There are two different ways to do this: trunk based development and release branch merging.

If you can get away with having only a single version of your software running live at a time, you can use trunk based development very simply: Have only a single release branch live at any time. Always add changes to trunk, and never directly into the release branch. Cherry pick bug fixes from trunk to the release branch. Manually test bug fixes in the release branch.

If you need to support different groups of users using different versions at the same time, trunk based development gets a little trickier. For example, Stakeholder A desperately want a new feature, but delivering that feature risks destabilizing the the product for Stakeholder B. You'll need multiple release branches that include different sets of changes. There are only two known solutions: feature toggles and release branch merging.

Feature toggles involve guarding new code with a runtime flag, such that the risky new code can be disabled for stakeholders that don't care about it. One additional advantage of feature toggles is that, if the new code ends up having a major bug which is discovered in production, it can even be easily disabled for everyone. The two big disadvantages are: 1) feature toggles open up a brand new category of technical debt, and 2) you need to have whole new mechanisms for managing them. For example, how do features get toggled based on testing? How do you handle feature combinations and interactions?

The alternative is to leverage the merge support in modern revision control systems like git. Instead of trunk based development, fix bugs directly against the release branch. Then always merge all changes from more stable branches to less stable branches, from release branch to mainline. There is a bit written about this under the name "gitflow", but the approach existed before gitflow, and gitflow really describes a larger standard. Some things are not complicated: if you're doing code reviews, you should be using "topic" branches. Many web pages just confuse matters in their comparisons of gitflow, github flow, gitlab flow, and trunk based development. The real question is: would you prefer to use feature toggles to enable trunk based development, or would you prefer to merge from release branches?

Release branch merging is not a panacea, and it can be a pain. Arguably it is simpler than feature toggles though; it is trivial to understand whether master is caught up with all of the bug fixes from release branches. The installed version number tells you exactly which code is in your system, and you easily compare it with other versions of the code that were or will be running at any other time.

Quotes:

Raymond Chen, long time Microsoft developer: “Cherry-picking is a common operation in git, and it’s not a good idea. Sometimes it’s a neutral idea, but I haven’t yet found a case where it’s actually good.

Google according to Winter, Manshreck, and Wright: “A key to reliable continuous releases is to make sure engineers "flag guard" all changes.” (emphasis mine)