FOUND: 7 lost principles of continuous delivery
“You Americans, you’re all the same. Always overdressing for the wrong occasions” said Major Toht (Raiders of the Lost Ark). Sometimes… that’s how I feel about the continuous delivery process. We overdo the delivery process for the wrong reasons. Here are 7 things to keep in mind while practicing continuous delivery:
It’s not about “can we build it?”, it’s about “should we build it?”
1. Create a repeatable and reliable process for releasing software
Ask yourself, “How long would it take your organization to deploy a change that involves just one single line of code?”
We live in a world where shortened release cycles are a necessity. With agile development practices and continuous integration (CI), it is imperative for teams to have access to a stable tool which can manage the process of building, testing and deploying software as changes are made.
For a long time, Jenkins has been the tool of choice among developers, but as many companies have moved into the cloud, and in particular, AWS ecosystem, other tools such as Bamboo, AWS CodePipline, Go CD are gaining traction. The underlying purpose of these tools remains the same — to release your software.
Every change committed to source control triggers the creation of a new instance of the pipeline. We call this a release candidate. If the release candidate passes all the tests, it can be released.
Deploying through the pipeline involves three main phases:
- Building the infrastructure — provisioning and managing the environment in which your software will run
- Installing the software and its dependencies — installing the correct version of the software into the infrastructure
- Setting up configuration for software — configuring your software and database, including any data or state it requires
The deployment of your software can be implemented using a fully automated process from version control. Hardware cannot be kept in version control, but you can use provisioning tools, like Chef, Puppet or Terraform. With these provisioning tools you are able to codify your configuration and commit them into your source control.
“The key in such a transition to continuous delivery is to expect things to get worse before you’ll be able to make them better.” — Matthias Marschal
2. Automate, if possible, everything
Automation is a prerequisite for the deployment pipeline because it is only through automation that we can guarantee that people will get what they need at the push of the button.
“The most powerful tool we have as developers is automation.” — Scott Hanselman
When you’re not automating the process, you’re creating manual one. These manual steps usually require communication from a person to another. As the path of communication gets wider, the original set of instructions or documentation are misinterpreted, miscommunicated, and not maintained.
Computers, on the other hand, are way better at running repetitious tasks accurately. We should write scripts to automate any manual process in our release process. Commit the automation script in source control for other developers to collaborate and improve the process.
Automation is about taking once manual processes and placing technology around them so they’re inherently repeatable. Before we automate a process, we should go back to the drawing board and list the steps that would make the automated process shorter and faster compared to the manual process. Otherwise, if your processes are bad or flawed, then you’re just making bad processes happen faster.
Automation will ultimately improve production velocity and quality.
3. Keep everything under version control
Everything should be committed into source control, not just your software code. Version control isn’t all about having backups. That is only a side effect. Version control is about documenting and annotating the process of creation, and it is uniquely necessary for programming because of how eternally iterative programming is. Code that is not annotated with its history and its motivations suffers to an extreme degree of entropy, becoming more and more difficult to understand.
Configuration settings have a lifecycle completely different from that of code, while passwords and other sensitive information should not be checked into version control. Ensure that your configuration information is modular and encapsulated so that changes in one place don’t have knock-on effects for other, unrelated pieces of configuration. Be minimalistic. Keep the configuration information as simple and focused as possible. This includes ‘requirement’ documents, test scripts, automated test cases, network configuration scripts, deployment scripts, database creation, database maintenance scripts, technical documentation and so on. These scripts should be version-controlled, and the relevant version should be identifiable for any given build — these change sets should have a single identifier, such as a build number or version control change set number, that references every piece.
Ask yourself:
Could you completely re-create your live environment (like production), excluding production date, from scratch with the version-controlled assets that you store?
Can you regress to an earlier, known good state of your application?
4. If it hurts, do it more often — bring the pain forward
If testing is a painful process that occurs just before release, don’t do it at the end. Instead, do it continually from the beginning of the project. In the same way that your application and build scripts need testing, so do your configuration settings. Ensure that references to external services in your configuration settings are good.
Ronny Kohavi, who directed Amazon’s Data Mining and Personalization group before joining Microsoft as General Manager of its Experimentation Platform, reveal that 60%–90% of ideas do not improve the metric they were intended to improve. Thus if we’re not running experiments to test the value of new ideas before completely developing them, the chances are that about 2/3 of the work we are doing is of either zero or negative value to our customers — and certainly of negative value to our organization, since this work costs us in three ways. In addition to the cost of developing the features, there is an opportunity cost associated with more valuable work we could have done instead, and the cost of the new complexity they add to our systems (which manifests itself as the cost of maintaining the code, a drag on the rate at which we can develop new functionality, and often, reduced operational stability and performance).” — Lean Enterprise: How High Performance Organizations Innovate at Scale
Create a comprehensive automated test suite. At a minimum, you should have:
- Unit tests —this involves breaking your software into pieces and subjecting each piece to a series of tests. Tests are usually run periodically, often after every change to the source code. The more often the better, because the sooner you will catch problems.
- Component test — this involves testing of specific module or program. It may be done in isolation from rest of the system depending on the life cycle model selected for that particular application. Component testing is like unit testing with the difference that the developer uses real data instead of dummy data for testing of the written code.
- Acceptance test — is the test that the application meets. The acceptance criteria decided by the business — capacity, functionality, availability, scalability, and security
Don’t comment out failing test. Delete it if it’s not relevant or refactor the test.
5. Keep the build and test process short
The longer the release cycle, the longer the development team has to make incorrect assumptions before the deployment occurs, and the longer it will take them to fix it.
Speed is essential because there is an opportunity cost associated with not delivering software. Delivering fast is also important because it allows you to verify whether our features and bug fixes are really useful.
An important part of usefulness is quality. Our software should be fit for its purpose. Quality does not equal perfection.
“The perfect is the enemy of the good” — Voltaire.
Our goal should always be to deliver software of sufficient quality to bring value to its users. Check-in frequently. You should at least check-in your code a couple of times a day.
If the build takes too long, multiple commits would have taken place — you won’t know which check-in broke the build. Your build output should inform you the commit that triggered the build.
If the build process is too long, developers will check in less often because they have to sit around for ages waiting for the software to build, test and deploy.
We should wait for tests to pass before moving forward. That way we don’t compound the failure with more problems. Remember, build a little, test a little, deploy a little.
Don’t check-in on a broken build. This is the cardinal sin of continuous integration. If you continue to do so, it will take much longer for the build to be fixed, because you added more complexity to the problem.
Test locally on your machine before checking in your code.
Never go home on a broken build. For example, at 5:30 pm on a Friday, all your colleagues are leaving the office and you are itching to leave as well. You decide to check-in your code and leave. Your checked in code triggers a build and it fails. What do you do next? You have two options:
- revert the commit to a working version. Always be prepared to revert to the previous revision.
- fix the broken test and build before you leave the door.
If the build is broken, the feedback loop should be fast and efficient. It should notify the appropriate people to look at the build’s failure. Feedback is at the heart of any software delivery process. The best way to improve feedback is to make the feedback cycle short and the result visible.
6. Make every part of the process of building, deploying, testing and releasing software visible to everybody involved
Improve feedback so that problems are identified, and is resolved, as early as in the process as possible. Enable teams to deploy and release any version of their software to any environment at will through a fully automated process.
The entire team must act on a feedback — it is essential that everybody involved in the process of delivering software is involved in the feedback process. This includes software developers, release engineers, DevOps engineers, database administrators, security team, tester and product managers.
Is it easy for team members to get the information they need and to make the changes they need to make? Or does the strategy get in the way of efficient delivery, leading to increased cycle time and reduced feedback?
Often the poor collaboration that causes so many problems in deployment to staging is shored up with ad-hoc phone calls, emails, and quick fixes.
Any changes or commits should trigger a feedback process — anything that changes between environment should be captured as configuration information. Feedback should be delivered as soon as possible. The key to a fast feedback is automation (see point 2). People are expensive and valuable, they should be focused on producing software.
7. Done Means Released
“Continuous is more often than you think” — Mike Roberts
Too often in software development, “done” doesn’t really mean “DONE!”. It doesn’t mean tested. It doesn’t necessarily mean styled. And it certainly doesn’t usually mean accepted by the product owner. It just means developed.
It’s also important to really complete each feature before moving on to the next. Of course, multiple features can be developed in parallel in a team situation. But within the work of each developer, do not move on to a new feature until the last one is shippable. This is important to ensure the overall product is in a shippable state at the end of the Sprint, not in a state where multiple features are 90% complete or untested, as is more usual in traditional development projects.
We should make sure that each feature is fully developed, tested, styled, and accepted by the product owner before counting it as “DONE!”. And if there’s any doubt about what activities should or shouldn’t be completed within the Sprint for each feature, “DONE!” should mean shippable — by Kelly Waters
- Done means monitoring is set up and working.
- Done means there is an on-call schedule to attend any alerts if something goes wrong.
- Done means the end users should be able to open a ticket to report a bug.
- Done means the software is well tested.
- Done means the software is able to scale up to handle high loads — completed load testing.
- Done means your software is secure —completed security testing.
DONE means shippable and functional.
Conclusion
Continuous integration requires that every time somebody commits any change, the entire application is built and a comprehensive test is run against it. Crucially, if the build or test fails, the development team stops whatever they are doing and fixes the problem immediately. The goal of continuous integration is that the software is in a working state all the time.
In order to create a reliable and repeatable process for releasing your software, we need to automate the release process as much as possible. Everything should be kept under source control. A software without adequate testing is not complete, therefore should not be released. Strive to keep the build and test process short. Build a little, test a little and deploy a little — repeat. When multiple teams are involved in the delivery process, the feedback loop is crucial and should be visible to everyone involved. When a software is considered “done”, it means it is shippable, functional and fully tested.
Interested in learning more about software development? Read my previous article on: Agile Architecture
This article is inspired by Dave Farley and Jez Humbel’s articles and book- Continuous Delivery, and Kelly Waters’s book- All About Agile. I had the privilege of attending Dave’s presentation in Chicago.
