Continuous Delivery

I was looking for an article/talk about Flickr's crazy ideas of thousands deployments per day that my co-worker told me years ago. I want to write about Continuous Delivery. That would be my best reference. I am not sure how my LTM works, somehow I only remember Flickr, 7000, deployments per day. My best Guru, Google wasn't able to give me the answer. So, I posted my question to L, "I remember long long time ago, you mentioned some company, I think it was Flickr, aim for thousands of release in a day. Do you still have any idea about it?" After L pointed me to Martin Fowler's website, and I did further research on Google, finally I found what I was looking for!

I think any company or organisation that is moving into devops or CICD trend, should watch this presentation. It happened more than 10 years ago, but it is still very relevant.

The presentation is by John Allpaws and Paul Hammond at Velocity 2009.



If you don't have the ~ 46 minutes, you could read the write up about the presentation at Talk Notes: John Allspaw and Paul Hammond: "10+ Deploys Per Day: Dev and Ops Cooperation at Flickr": Velocity 2009 by Gene Kim

I attended some devops meet ups in 2018. I stopped doing that, probably is because there was no further meet up? Or I stop watching the events from Peatix last year to give myself some time to switch my focus. I wanted to work in devops, but I saw a lot of devops engineer JDs are mostly release engineer wearing the disguise, I got discouraged. So, I never really work on actual devops role or task. I was just happened to be a developer, then switch to operation role. I have a strong will to see developers to develop pieces of code that not only to support new feature, but also not to introduce more workload for operation team to support the platform. With my limited working experience, I have not got into that vision. :) I am sure some organisations out there have reached this equilibrium in devops.

It is a long way to go, for a development which got used to waterfall methodology with minimum interaction with the operation and system team, and deployments have been done manually with Excel checklist...

You can start with practicing or mimicking agile/scrum, start to have CICD work flow, using Jira to track backlogs and features, 4-week sprint, using Puppet or Chef or Jenkins for build and deploy, autotest to perform smoke test... It is definitely a big leap towards devops from traditional development era.

If you watched what John Allpaws and Paul Hammond shared in that presentation, this actually is not even half way there. If you still have pieces of codes, or a quick fix, only available for production after a whole sprint ended and tested. This is not continuous delivery. This is not CICD - continuous improvement, continuous delivery. It is another version of CICD - continuous improvement, and continuous development only.

To enable continuous delivery, which allows quick fix being deployed to production, the source code repository branching management is very vital. The master trunk should always be whatever is in the production. If you are working on any new feature or bug fix, it should be branch out from the trunk. When a feature or fix is ready to be released, ensure the latest code from trunk is already merged to the release candidate branch and tested. On the other hand, whenever there's new release successfully deployed to the production, the code must be merged to trunk.

This approach allows any change/fix on production made possible any time, or any change/fix can be discarded any time. It is normal to have some branches to be discarded. It could be the feature no longer needed (due to it was on hold for too long?). It could be the bug fix introduced more bugs, and you need to fix it in a different way.

I see some teams are having the release candidate as the master trunk. When the release is on hold due to new bugs found during testing in stage environment or probably pending on pen-test, then any urgent fix to the production would be impeded. Yes, you can find the release branch from the history, perform the fix, then release a sub-branch. That would be contradict with the original plan, i.e. to have the master branch as the release candidate.

Furthermore, when the master branch as the release candidate is impeded, and next sprint started, how shall we proceed? Make a branch from the master, and continue development? Another team, or developers work on the master branch to do fixes? Then merge the changes in master to the new development branch? What if the next sprint ends before the "master" branch is ready for release? Merge in the new development branch to master, so both sprint work can be released at the same time? What if continue to stack up? And the questions go on...

There is a reason why these wise guys suggest to have production code always on master branch.

I am trying to illustrate the trunk and branches scenario here. I hope this picture speaks the thousand words. :)

P/S: I guess the 7000 (or thousands) probably is due to my memory messed up with the movie Ferrari v Ford. (I watched it twice!) "There's a point at 7000 RPMs where everything fades. The machine becomes weightless. It disappears. All that's left, a body moving through space, and time."