Bottlenecks are usually created because of a fear — real or imagined — that something will go wrong in production. For example, a bad deployment to production that caused major problems in the past often resulted in executive leadership putting checks in place to safeguard against an issue like that happening again.
Unfortunately, this means that a large and strict bureaucratic process likely was built at some point for an understandable reason, but then never reevaluated. The problems that led to a bad deployment, in the first place, probably don’t even exist anymore, but the deployment process is still cumbersome.
Ironically, this leads to making things even less secure over time. When releases become a big event, any failure is a huge deal.
Start by identifying the problem you’re trying to solve
When the time between code being complete and release is always at least 2-3 weeks because so much process has to be executed (e.g., performance testing, regression testing, security review, etc.), technology is not the problem.
I worked at an enterprise company where there were seven separate production environments between where the developers wrote code and where it was pushed to production. Seven separate environments to get to actual user value!
For the various teams involved, the bulk of the time spent was on managing and coordinating the processes between each of these environments. This left no time for executing any risk mitigation work like testing. In a case like this, implementing cutting-edge technology to speed up deployment would not have accomplished the desired outcome. The problem lay in the process, not the tech.
That does not mean technology is not necessary. It is absolutely needed and you do have to invest in tools to implement CI/CD successfully. However, to choose the right technology, it is first necessary to understand exactly what the problem is that you’re trying to solve.
Understanding the value proposition
The first step I advise companies to take when dealing with the process bottlenecks is to conduct a value stream mapping (VSM) of the path to production.
Value stream mapping is a lean tool that uses a flow chart to visualize and quantify all the waste and pain points in a process. In this initial exercise, it is important to really think in detail through the end-to-end process, asking questions such as:
- What is the value of this work?
- Who are the people involved?
- What do they do?
- Why do they do it?
A critical part of the value stream process is to set a collaborative tone. Make sure to involve everyone in the value stream mapping exercise so that you fully understand the reasoning and value behind each of the steps in the process. Don’t assume your counterparts are trying to slow you down on purpose. There’s often history behind their processes that they feel bound by.
Once the value stream map is complete, this is when the redesign work begins. Using a clean sheet approach, create a North Star of your ideal path to production leveraging true CI/CD. Being able to compare what you have (VSM) with what you want (North Star) will help define the requirements for any technology solutions you might need to invest in to achieve your goals.
I have found that taking the same tools and thinking when building software for customers and leveraging them in discussions around teams who work for software companies can really help. For example, ask QA teams who might be supporting development teams to create personas of those development teams, defining pain points and goals for each persona.
Ask teams to define what other teams would say about them. This doesn’t mean that what the other side says is completely true, but often there is some truth in complaints. The issues identified through these exercises can be very useful in shifting organizational thinking, to show that changing processes does not remove safeguards, but rather provides an opportunity for improvement.
Moving from problems to solutions
Breaking down bottlenecks requires a combination of people, process, and technology solutions. Improving CI/CD is best achieved through an iterative approach: change one thing, remove one step, at a time. As you begin considering process changes, this is the point where technology considerations really come into play.
Assuming that you have invested in aligning groups around a common value proposition, it will be clear how technology provides value to the people and the processes. For example, bringing in a tool like Selenium, which supports an iterative, automated testing development process alongside product development, cuts down on testing time..
The success of a technology requires a change in process. In this case the tech will enable that process by providing integrated development and automation capabilities.
CI/CD requires ongoing, proactive maintenance
CI/CD is applicable to all levels of scale, from e-commerce applications accessed by thousands of people, to enterprise software solutions used solely by an accounting team. But getting new code to production and at the fingertips of users as frequently as possible becomes practically impossible when there are too many bottlenecks slowing down your CI/CD technology.
For larger organizations that successfully remove process bottlenecks, and for SMBs who might not yet have these challenges, it’s important to proactively set up systems that regularly maintain and improve upon CI/CD processes.
This might take the form of a quarterly or year-end review. Teams can work together to map out data and conduct a retrospective on how the process is performing. This includes any persistent pain points.
Consider establishing an annual Hack Week, during which employees are allowed to work on whatever they want. Product teams are the ones dealing firsthand with many of these problems. Giving developers the opportunity to dive in and focus on solving process problems often leads to truly remarkable benefits to the business.
CI/CD requires a shift in mindset
CI/CD requires a shift in mindset. This can be a significant undertaking and might require you to bring everyone together to talk in a heads-down way for a considerable amount of time. Do not be afraid to do that. This will not only break the teams away from siloed thinking, but also get everyone on the same page around the broader context of user experience and value.
Rob O'Brien
