Geoff Mazeroff

This post continues the book club discussion of The Phoenix Project. See also: The Phoenix Project – Part 4

Here are my notes for chapters 21-25…

• Security audits can have multiple sets of rules. Unless auditors want to punish you, they tend not to. Geoff has experienced that with FDA audits: There are auditors who look for compliance, and those who look for non-compliance. Even restaurant health inspections are subject to who’s doing the inspection.
• An example of John’s meaningless security work was given at the beginning of the book… encrypting the Social Security numbers when it wasn’t needed, and doing so brought down critical systems.
• “We can’t standardize our work like manufacturing.” This will probably turn out to be incorrect. The goal is to standardize, and then automate.
• Having all work requests go through a single channel (e.g., not email, not chat, not hallway conversations) is critical to get visibility of the work.
• Checklists… This gets to the definition of done. It’s important to get agreement on the acceptance criteria and not miss things. It’s easy to get complacent and miss steps. Geoff mentioned a related book: The Checklist Manifesto by Dr. Atul Gawande.
• The Plan-Do-Check-Act cycle is from the ISO 9001 model.
• The IT department has assembled a kanban board for their department to help visualize work in process (WIP). Color-coding helps ensure they have the right percentage of types of work in play. It also helps keep work synchronized, as you may have teams working together or systems that need downtime concurrently.
• Knowing about work blockers is key for monitoring flow; Scrum’s daily standup ceremony aims to address this.
• The team makes three lists: things requiring the bottleneck (Brent in this case), things that speed up the bottleneck, and everything else.
• We discussed the scenario where Brent claimed it was a 45-minute task to stand up a server for testing. Why did this take way longer; what was the root cause? Jameson said this could be a grooming versus sizing issue, which led to further discussion of what backlog grooming really means. Houston and I have worked at places where grooming = prioritization, not work breakdown.
• The busier (utilization percentage) the resource, the longer you must wait in line to use it.
• Brent built infrastructure that only he understands. This is why you need “deputies” that also understand it. Jameson said QA is often expected to know how everything works because otherwise how can you test it? This led to more discussion about a controversial idea… As developers, does it make sense to concern yourself with business workflows, as they are typically not useful outside that company. Houston and I disagreed with that sentiment, as the experience you gain helps shape further decisions. If nothing else, at least write the business logic down so others can understand it later.
• John and Bill should have had the discussion about company survival vs. security mindset much earlier in their jobs. John points out that no one involves him. Jameson believes the reason for that is the “cry wolf” mentality (e.g., everything is a must-fix vulnerability) which most of the time isn’t true, which erodes trust. How do you know when it’s serious? You burn political capital so no one takes you seriously.
• Until now, IT has been a silo instead of integrating with other business units.
• Big companies are too complex for the person at the top to understand how everything works. For scale, they have to delegate that responsibility to others.
• The conversation with the CFO (Dick) brought to light that it’s difficult to determine what individual or set of things is the root cause. He even claims it could be him. Complex problems are difficult to solve, as you don’t know what part affects others.
• We enjoyed the clarity of focus Dick’s two slides bring. He has a set of areas where his attention is needed, and a set of questions to let him know whether things are working. This is probably easier to see at the C-suite level.
• The classic project triangle involves cost, time, and quality. Good process will help you get better at all of those things.
• Maintenance can be seen as a nuisance; not doing maintenance is expensive. As Houston said, “It’s hard to notice when everything is going right.” Pro tip: When it is, offer some thanks for people that make that happen!