The Phoenix Project – Part 7

This post continues the book club discussion of The Phoenix Project. See also: The Phoenix Project – Part 6

Here are my notes for chapters 31-35…


  • The team is heading toward ten deployments each day, which the team initially finds unreasonable.
  • The ability to deploy is likely greater than the number of deployments.
  • How is this threshold of ten deployments set? Is there a genuine need to do so, or is this another example of “more is better”? Another question our group considered was the cost of getting from one deployment to ten deployments. This may be an example of the five nines of availability issue, where the cost to get you an order of magnitude difference may be infeasible.
  • Related to the previous item, having a specific target in mind can help rally the troops. The folks doing the work need to understand the why (i.e., how it helps/supports the company). Top-down ownership of the work give people buy-in and boost productivity.
  • Geoff mentioned a TED talk about the difference between efficiency and robustness and how efficiency can be at odds with agility.
  • IT work is full of invisible work centers. If one step goes wrong, big problems could result. Having Brent diagram the deployment workflow helps to visualize tribal knowledge.
  • Having a separate environment to try development or operations actions without risk of bringing down production systems is key. Otherwise, you have no safe place to fail, learn, and experiment.
  • Project Unicorn is crucial to making sure Brent’s knowledge gets distributed to those who need it. If you have a critical piece of your system that only few people understand because it’s complicated means that you may have developers responsible for too much of the business logic and its dependencies.
  • Vague requirements can lead to bad assumptions on the developer’s or product owner’s side. Rapid prototyping and good communication help mitigate this.
  • Bill doesn’t have much experience in cloud computing and is initially skeptical; yet, he trusts his team to make good decisions. Our group had a lengthy follow-on discussion about this…
  • It’s no longer practical to know and do everything; this is why we have teams of people to accomplish these projects. Labor doesn’t always scale. Knowledge doesn’t always scale. Perhaps the “full stack developer” is a myth — maybe they touch each layer, but only at the 80/20 level.
  • Bill’s team is starting to have “good problems”, where they’ve discovered they need tools like load testing and feature flags.
  • One of the last problems the team bumps into is a legacy mainframe system that can’t move as quickly as their competitors. All too often a temporary solution is put in place that becomes the permanent one, while the desired permanent one goes to the bottom of the backlog. Entropy and rot always catches up with you if not properly managed.
  • At the end, the team finally got to the point where they can breathe and practice healthy habits such as penetration testing and reliability/resiliency testing.
  • Especially for Parts Unlimited, the COO needs a balance of technical skills and business knowledge; otherwise, both sections will suffer.
  • Erik made a comment that the health of the IT department is a critical predictor of company performance, as IT is the nervous/circulatory system of a company.