Deployment Pipeline (Let's make Continuous Deployment painless)
In the life of Deployment Manager of an agile aggressive team1, there comes a day when he/she needs to manage releases and communicate with stake holders about features being released and also control what goes live on production. Why control? because in any practical scenario, feature which is ready and accepted by product managers doesn't guarantee its readiness for the business and marketing and quite often non-technical teams need a buffer time before a feature is released. Deployment Pipeline is here to help you manage these releases and keep non-technical teams and users better informed about whats going on.
- Your Product Managers write feature/bug stories and prioritize them anytime during the day.
- Your engineering team believes in Continuous Integration and all engineers commit to Master branch (or Trunk) several times a day.
- If the build (CI) is green, that tag from Master Branch might get pushed to QA environment for stories to be delivered.
- Once all stories are accepted, (QA) Staging Tag is deployed to production, Happy Ending of the day!
- You have 5 Product Managers who request feature/bug stories and prioritize them anytime during the day.
- You have 10 engineers working on 5 stories and commits to Master branch (or Trunk) several times a day.
- Build is green only 60% of time during the day.
- By the time build is green there are commits to 4 finished & intermediate commits to an 1 un-finished story.
- QA delivers the stories(4) which has been finished and 3 are accepted and 1 is rejected.
- Among 3 accepted stories Marketing Team takes a call to hold 1 story even though its ready.
- Now we have 2 production ready, 1 held by marketing, 1 rejected, 1 un-finished.
- Commits related to 2 prod ready are shuffled between commits related to all non-ready stories.
- Among 2 ready-to-deploy , 1 is marked urgent but it's commits are part of the day when CI build was RED (not necessarily due to this commit).
- What and how would you deploy today? (cherry-pick commits for a feature with no green build? FAIL) You post-pone release!
- ==Day Rolls Over==
- 3 new stories requested by Product Managers.
- An engineer finishes 2 of new stories quickly whose commits goes to master. 3rd new story might take long to finish, but gets its commits pushed to master.
- Build is green and a tag on master is pushed to QA environment.
- 2 new stories are delivered and accepted.
- Which staging tag on master would you deploy to production? At any given time there are commits from un-finished/un-delivered stories.
- Sure you can use feature toggle, but it only makes sense for long running (for weeks) set of stories. When every story starts to have a feature toggle, then system gets polluted with if-else everywhere, which again is difficult to manage and error prone
- You can also ask engineers to have separate feature branches for each stories and rebase with master often. This brings in its own over heads...
- Time spent in merging changes
- It needs a single controller of release branch who pulls the changes and makes sure what goes live is vetted. (This controller can soon become a bottleneck in the process)
- Engineers work in isolation and can not commit intermediate commits unless feature is complete.
- Use All-Accepted Marker : This is a commit on master below which all stories have been accepted and there are few commits (shuffled with other un-finished) above the marker that can be cleanly cherry-picked.
Workflow for All-Accepted Marker Deployment:
- Find a suitable commit below which all stories are accepted
- Branch out to new "Release" Branch
- Inform stake-holders about what features are being released and locked down release marker
- Cherry-pick related commits from above the marker to release branch
- Build the release branch and wait for it to be green
- Deploy release branch to production
- Automate this entire process
Deployment Pipeline @ Work :
developers-machine:~/workspace/repository (master)$ <b>pipeline help</b> Tasks: pipeline help [TASK] # Describe available tasks or one specific task pipeline release_plan # Prepares a release plan pipeline setup # Setup Deployment Pipeline Tool pipeline status # lists all stories with their status pipeline suitable_release # Suggests a release commit to be picked and also includes a release plan Options: [--config=CONFIG] # A ruby file that defines relevant constants & configs. accepts ENV $PIPELINE_CONFIG # Default: /Users/dev_home/.pipeline_config
Find Suitable Commit for All-Accepted Marker:
developers-machine:~/workspace/repository (master)$ <b>pipeline suitable_release --last-release-commit="c0512b2cebf86e96baa1fe783ba4c01b55b40000"</b> Analysing... c0512b2cebf86e96baa1fe783ba4c01b55b40000 Commit is untagged hence considered Analysing... aaa756c1c79829aa6680df66dda62f0a59c336a1 Stories accepted for aaa756c1c79829aa6680df66dda62f0a59c336a1, hence considered Analysing... 7b9cf2863e35add6663a7e964440b4f75befe572 Stories accepted for 7b9cf2863e35add6663a7e964440b4f75befe572, hence considered Analysing... eb94f53ac8969fee134fe6677f7a6bf457b155c5 Stories accepted for eb94f53ac8969fee134fe6677f7a6bf457b155c5, hence considered Analysing... 44d1324d0899cba238a3a53d470f655fcf4b64e2 Stories accepted for 44d1324d0899cba238a3a53d470f655fcf4b64e2, hence considered Analysing... ef9d1a6a1e8bac5112c6ff491c97c083769bd553 Stories accepted for ef9d1a6a1e8bac5112c6ff491c97c083769bd553, hence considered Analysing... ee0112cad3f58b362281c786a197d27148b13f12 Stories accepted for ee0112cad3f58b362281c786a197d27148b13f12, hence considered Analysing... 331cdaca1cfd3672d901ca56218cef9330540241 Commit is untagged hence considered Analysing... a2380cf7f3ad3ab660927bac6177073b123b002e Commit is untagged hence considered Analysing... b1c281030100d40f46057b5b8463ee7d16952b29 Commit is untagged hence considered Analysing... e65d84b68ff038a96769bd9b10c1cb3997f625bb Commit is untagged hence considered Analysing... 20c4a74ffd298f0d9bf60679b23823fc32d4ec26 Unsuitable commit because story status is not accepted <b>Commit that can be pickup up for release is: e65d84b68ff038a96769bd9b10c1cb3997f625bb</b>
Find status of stories between given commits on a branch:
developers-machine:~/workspace/repository (master)$ <b>pipeline status --commit-range="3191320e8d7c5a402d17a560b12677b44344b300..9471d8794a28cc3a6888fa3a42c03edb4f4458812" --html </b> Fetching...: 100% |============================================| Time: 0:00:11 STATUS:
- (bug) Users unable to reset their password requested by Rajnikanth owned by Tom Cruise
- (bug) Banner ad spacing issue requested by Bill G owned by Jacky Chan
- (chore) Reduce memory foot print requested by Big B owned by Tony
- (feature) Users can foresee their future requested by Rajnikanth owned by Tom Cruise
Following commits have not been tagged, (**why?**)
- 3191320e8d7c5a402d17a560b12677b44d11b3d1 Super crazy commit by Tom
- 49297c0b1e28cc63d0117c62b08420a4465b6b8f lazy to add tag by Dick
- 401a997f402666a148a8e54582b6feedfb7a9148 Fixed Build by Harry
Find suitable release plan (find out exact stuff that will be seen on production):
developers-machine:~/workspace/repository (master)$ <b>pipeline release_plan --last-release-commit="3191320e8d7c5a402g17a561b12777b44344b3d5" --target-release-commit="9471d8794a28cc3a6888fa3a42c03eab4f445823" --html </b> Fetching...: 100% |============================================| Time: 0:00:22
OUTPUT: Release can be locked at commit 9471d8794a28cc3a6888fa3a42c03edb4f4458a6
Stories being released are:
- (chore) Reduce memory foot print requested by Big B owned by Tony
- (feature) Users can foresee their future requested by requested by Rajnikanth owned by Tom Cruise
Commits that needs to be cherry-picked as they are part of above stories:
- 401a997f402666a148a8e54582b6feedfb7a9149 [#1234] Related to memory by Harry
developers-machine:~/workspace/repository (master)$ <b>gem install deployment_pipeline</b> developers-machine:~/workspace/repository (master)$ <b>pipeline setup</b> Let's setup Deployment Pipeline Pivotal Tracker Token: <i>2d2b1f708d2b1f708cd2b1f708c</i> Pivotal Tracker Project Ids (comma separated): <i>1234,5678,9876</i> Setup complete! (Configuration updated in <b>~/.pipeline_config</b>) Thanks for choosing Deployment Pipeline
[**1**]: Team which is motivated for release-often philosophy so much that it releases to production multiple times a day. It uses DVCS like Git and agile story tracker like PIVOTAL TRACKER. It has adopted TDD & Continious Integration as way of life. Every engineer commits to master all the time.