Lessons From The Phoenix Project
A novel about IT, DevOps, and helping your business win
Introduction
The Phoenix Project is a business novel by Gene Kim, Kevin Behr, and George Spafford, pioneers in the DevOps movement. The book follows the story of Bill, an IT manager at a parts manufacturer, struggling in a competitive market. Parts Unlimited, once a dominant player in the automotive parts manufacturing and online retail space, has lost ground to its competitors regarding growth and profitability. Despite a long-standing pledge from the company's leadership to revive its fortunes through a new online initiative, dubbed "Phoenix," the project is significantly delayed. In response, investors are calling for a significant shake-up in leadership to steer the company back on track. The board of directors has removed the CEO, Steve Masters, from his position as chairman and given him six months to effect substantial improvements, failing which, they may resort to more drastic measures such as breaking up the company. After several failed initiatives, Bill is promoted, and the reluctant protagonist is responsible for straightening the mess.
The story provides the backdrop for discovering and applying the concepts of DevOps, an evolutionary combination of developers, IT operations, business units, and product owners infused with ideas from manufacturing, Lean, leadership, security, and other disciplines. With DevOps, organizations use the same Agile methodology in every domain, like product managers, marketing, quality, operations, network engineering, and security, as they do with their software development teams.
DevOps began as a disparate set of ideas coalescing in the late aughts, aiming to change IT operations under the rubric of pragmatism, modeling operational efficiency on concepts like Lean Manufacturing and the Toyota Production System (TPS) [1].
The core idea presented in The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win, and the overall DevOps philosophy, is that managing the development, quality assurance, deployment, maintenance, and retirement of IT systems, including security updates, is akin to managing a production line for any other product. The TPS was a remarkable post-WWII advancement in managing production lines, and DevOps is the IT corollary. DevOps practitioners endeavor to minimize technical debt by constraining the amount of work in progress, thereby regulating the flow of the entire system.
Synopsis Of Bill's (and Erik’s) Adventures
In the story, Bill is tasked with leading the Phoenix Project, an ambitious IT initiative to modernize the company's technology infrastructure and improve its business processes. However, the project is plagued by delays, technical issues, and miscommunications between different departments.
Bill is introduced to Erik, a recently appointed board member, who ultimately assumes a “Master Shifu” role guiding and coaching Bill to independently discover how principles used in manufacturing can be applied to IT.
To help Bill understand these ideas, Erik takes him to one of the company’s plants, noting that while IT may not primarily use physical components, IT risks, and successful initiatives are strategic to any business and can impact the bottom line as much as other critical areas such as operations, sales, and marketing.
Bill learns valuable leadership, teamwork, and problem-solving lessons as he struggles to keep the project on track. He can only overcome these challenges with his colleagues' help by learning and applying core concepts of DevOps principles and practices. These ideas involve collaboration and communication between development and operations teams to improve software delivery's speed, quality, and reliability. The book also highlights the importance of continuous improvement, automation, and metrics in modern IT organizations.
Main Concepts
Let’s take a closer look at some of the main concepts from the book and see if they can still be applied in today’s IT organizations.
The Three Ways
One of the book's core guiding principles focuses on delivering customer value. You can think of The Three Ways as describing a series of steps for any work or process, flowing from ideation to final product, representing the value stream, or delivering customer value. This approach to maximizing the value stream is based on a framework or set of principles for managing IT operations in a way that fosters continuous improvement and innovation while promoting collaboration across teams:
The First Way: The First Way emphasizes the importance of creating a fast, efficient, and reliable system of work. This means ensuring that work flows smoothly from one stage to another without delays or bottlenecks. The goal is to optimize the flow of work so that it moves quickly and efficiently from concept to production, improving the delivery of value to the customer. The idea of flow is critical, understanding how work moves through the system and knowing that changes will have random effects. Optimization includes continuously working to increase flow, never passing defects downstream.
The Second Way: The Second Way focuses on creating a culture of continuous feedback. This means creating mechanisms for capturing input at every stage of the process and using that feedback from internal teams and customers to improve the process.
The Third Way: The Third Way emphasizes creating a culture of experimentation and learning. This means fostering an environment where experimentation is encouraged and supported, and taking risks and making mistakes is safe. The goal is to promote a culture of continuous improvement so that the organization can stay ahead of the competition and meet customers' changing needs.
Types of Work
Erik hints at ways to improve the efficiency of IT operations by sending Bill on a journey to learn the four Types of Work, a framework that helps teams prioritize and manage their workload. By understanding the different types of work and their relative importance, teams can focus on the most critical initiatives while promptly addressing urgent requests and unexpected events.
The four types of work are:
Business Projects: Business Projects are an organization's initiatives to create new products, services, or features. They are typically driven by business objectives, such as increasing revenue, improving customer satisfaction, or expanding into new markets.
IT Internal Projects: These are generally smaller initiatives that accumulate over time and must be executed to keep the organization functioning. They may include upgrading hardware or software, improving network security, or implementing new systems.
Changes: Changes are modifications, bug fixes, or improvements made to existing systems, infrastructure, or processes generated from the first two types of work. They may include software, hardware, or network configuration changes to procedures or workloads.
Unplanned Work: Unplanned Work results from errors or required re-work that builds up over time. It may include urgent requests for support or maintenance, troubleshooting of systems or infrastructure, or responding to unexpected events such as system failures or security breaches. Unplanned Work can be stealthy and harmful because it keeps teams from progressing with needed projects. Unplanned Work can be difficult for other business units and stakeholders to understand, hampers productivity, and makes hitting planned dates and milestones challenging.
Work in Process
Work in Process or WIP comes from Lean Manufacturing and the TPS concepts, referring to all the tasks, projects, and activities that are in various stages of completion but have not yet been delivered or deployed.
WIP is a significant problem in many organizations because it can cause delays, bottlenecks, and waste. WIP can accumulate when teams take on too many projects at once, when there are communication breakdowns between teams, or when there are inefficiencies in the workflow.
WIP can be addressed using the concept of flow from The Three Ways framework. This means optimizing the flow of work through the system by identifying and removing bottlenecks, reducing handoffs between teams, and minimizing work in progress at any given time. By reducing WIP, teams can improve their efficiency and throughput while reducing the risk of delays and errors.
The book also recommends using visual management tools, such as Kanban boards or task boards, to help teams monitor and manage their WIP. These tools provide a clear and visible representation of the work in progress, which can help teams identify and address any issues or bottlenecks in the workflow.
While The Phoenix Project touches on Lean Manufacturing, it owes a deep debt of gratitude to the ideas around continuously reducing waste so that the saved resources can be redirected to other work. Lean practitioners can help organizations by making processes more efficient, improving tools and procedures, and experimenting with creating new features or products.
Development Pipeline Improvements
Development pipeline improvements refer to streamlining and optimizing the software development process to deliver higher-quality software faster and more efficiently. The development pipeline includes all the steps in building and delivering software, from the initial concept to final deployment and maintenance. A company can maximize throughput and flexibility to customer demand by using small increments of work delivered frequently.
Continuous delivery refers to the software engineering practice of automatically building, testing, and deploying codebase changes to production environments. Continuous delivery is a prime example of The Three Ways, embodying their core principles by emphasizing small batch sizes, such as checking into trunk daily. Additionally, it mandates halting all work in the face of problems, like build, test, and deployment failures, elevating the system's integrity over the work itself. Finally, it emphasizes the importance of continually building validation tests to prevent production failures or, at the very least, detect and correct them swiftly. This is exemplified by the transition from manual process reviews to automated tests, particularly in IT service management, such as release, change, and configuration processes.
While the authors lightly touch on these concepts, we can extrapolate several ways to improve the development pipeline:
Automation: Automating as many steps in the development pipeline as possible, such as building, testing, and deployment, can reduce errors and improve efficiency.
Continuous Integration and Delivery: Implementing continuous integration and delivery (CI/CD) practices can reduce the time it takes to get new features and improvements into the hands of users. CI/CD involves automatically building, testing, and deploying changes as soon as they are made rather than waiting for large releases.
Testing: Implementing automated testing practices, including unit testing, integration testing, and end-to-end testing, can help catch bugs, errors, and security problems earlier in the development process, reducing the time and cost of fixing issues later.
Collaboration: Encouraging collaboration between teams, including developers, testers, security, and operations, can help identify and address issues earlier in the development process.
Feedback Loops: Creating feedback loops between teams and users can help teams better understand user needs and improve the software quality they deliver.
Metrics and Monitoring: Collecting and analyzing metrics related to the development process and the software itself can help identify areas for improvement and ensure that the software meets user needs.
By implementing these improvements, organizations can reduce the time and cost of developing and delivering software while improving the quality and reliability of the software they produce.
Theory of Constraints
The Theory of Constraints (TOC) is a management philosophy Dr. Eliyahu M. Goldratt developed [2], suggesting that any system, no matter how complex, is only as efficient as its most limiting factor or constraint.
Bill begins to unwind the organization’s problems in the book as his understanding of TOC develops. Erik explains that in any system, a bottleneck or constraint limits the system's throughput. The TOC aims to identify and manage this constraint to maximize the system's output.
To apply the TOC, Bill and his team identify the bottleneck in their IT system. They discover that the development team is overwhelmed with work, causing delays in the project's delivery. They also find that the cause of the bottleneck is a lack of testing environments, which leads to long wait times for developers to access the testing environments, slowing down the entire process.
Using the TOC, the team comes up with several solutions to address the bottleneck. They set up a process to prioritize the most critical projects, reduce work in progress, and create additional testing environments. As a result, the development team's productivity increases, and the entire IT system becomes more efficient.
Conclusion
The Phoenix Project is a compelling and educational read for anyone interested in IT management, software development, or business process improvement. It has had a lasting impact on how I view organizational work with continued relevance for the issues we face today. Changing focus to consider any project and development effort from the lens of the customer’s interpretation of value, you can begin to help navigate toward strategic competitive advantage. Similarly, only by understanding the Types of Work, the Theory of Constraints, and limiting Work in Process can teams start to improve quality, reduce waste, and improve an organization’s customer value stream.
References
[1] https://global.toyota/en/company/vision-and-philosophy/production-system/
[2] "The Goal: A Process of Ongoing Improvement," by Eliyahu M. Goldratt, and Jeff Cox, Published 1982 by North River