Modernizing systems is hard work
26 July 2023
I highly recommend Kill it with Fire by Marianne Bellotti for data professionals building or inheriting a system. It’s just over 200 pages long and took me about 8 hours to read.
Modernizing systems will become a key part of many data professionals’ work.
When I was working in the construction industry, the elephant in the room of all the sustainable building design work I was doing on new buildings was that 99% of building stock had already been built, and refurbishing those to be more sustainable was going to be difficult and costly, but necessary.
Likewise, in the not too distant future, there will be less scope to be working on brand new data systems – we’ll be inheriting them. This book pairs very well with Switch: How to change when change is hard by Chip and Dan Heath which talks about strategies & tactics to precipitate large-scale changes and many of those approaches I see applied in this book.
A little epiphany I had when reading this is that many data system modernization projects actually start with a digitization project (at least in my world), and that digitization projects are also modernizations projects and all of these lessons are relevant for those.
The book deals both with starting a new modernization project but also how to handle coming in mid-stream to an existing one.
I’ve pulled out some main ideas below, and then captured all my highlighted sections and some commentary below that.
- The hard problems around modernization are people problems.
- The existing system is not a spec for the new one.
- Just because something works without thought or effort doesn’t mean it’s simple.
- Architects should beware the pitfall of building their second system with too much confidence. It results in unnecessary complexity.
- Future proofing is to make small incremental steps that add value (not just upgrades for the sake of newness)
- Pick measurable outcomes, not feature releases (because the existing system already has those features and no one in the business will celebrate features they already have).
- The importance of building momentum by gearing all progress towards measurable outcomes cannot be understated.
Collection of highlighted sections of the book.
Introduction Xvi “People from my father’s generation wrote a lot of programs and every year they are shocked by how much of their work survives….
My generation has programmed exponentially more… We will similarly be shocked when those systems are in place 30, 40 or 50 years from now.”
Mainframe and the Cloud pg9 – “All advancements with data processors come down to one of two things: either you make the machine faster or you make the pipes delivering data to the machine faster. These forces cannot grow independently of each other. If the pipes pumping the data in get too far ahead of the chips processing the data, the machine crashes. If the machine gets too far ahead of the network, the user experiences no actual value add from the increases in speed.”
OB: And this shift back and forth is one of the external drivers of change forced upon systems and their ecosystems.
Migrating for Value, Not for Trends pg 14 – “And yet, information technology that never changes is doomed. It’s important to understand that we advance in cycles… Changing technology should be about real value and trade-offs, not faulty assumptions that newer is by default more advanced.”
OB: The repeated message is that development happens in cycles that repeat themselves, borrowing from other fields and previously underserved markets. Some of the historical examples are really fascinating to read.
“Value propositions are often complicated questions for this reason. It’s hard enough for a purely technical organization to get it right; it’s even harder at organizations where the only people with enough knowledge to advise on these issues are vendors”
OB: I feel it’s perfectly reasonable (and even wise) to hire consultants to help you vet vendors.
Cannibal code pg 17 – “Not all cycles are in sync”
OB: Multiple components of systems are all going through development cycles, and these are not in sync which makes upgrading etc difficult.
“So, the takeaway from understanding that technology advances in cycles isn’t that upgrades are easier the longer you wait, it’s that you should avoid upgrading to new technology simply because it’s new.”
Alignable differences and User Interfaces pg 21 – “Truly new systems often cannibalize the interfaces of older systems to create alignable differences.
This is why maintaining software is so difficult.
…not keeping up is also dangerous.
As technology advances, it collects more and more interfaces and patterns.
….
Keep your systems they way they are for too long, and you get caught trying to migrate decades of assumptions”
OB: This whole section was super interesting. In summary, developments borrow interfaces and elements from other fields and applies them with deeply ingrained characteristics that might not make sense in the new application but are there nonetheless. If you wait too long for an upgrade, then the leap you’ll need to make to the new system/paradigm can be very large because it has to cross the chasm of all the accumulated inheritances that don’t make sense but nonetheless were part of the total advancement.
Inheritance Paths pg 31 – “The lesson to learn here is the systems that feel familiar to people always provide more value than systems that have structural elegance but run contrary to expectations.”
Leveraging interfaces when approaching legacy systems pg 33 – “Second System Syndrome… The second system syndrome was not the second version of an existing system, it was the second system the architect had produced. Brooks’s feeling was that architects are stricter with their first systems because they had never built software before, but for their second systems, they became overconfident and tack on all kinds of flourishes and features that ultimately overcomplicate things. By their third system, they have learnt their lesson.”
OB: Something to keep in mind when my second system comes about.
“The goal of full rewrites is to restore ambiguity and, therefore, enthusiasm. They fail because the assumption that the old system can be used as a spec and can be trusted to have diagnosed accurately and eliminated every risk is wrong.”
OB: My own new system build was actually a rewrite in disguise which I only realized much later. I also mistakenly assumed we were digitizing a working process, but we were digitizing and fixing the process at the same time.
The biggest takeaway for me from this whole section was not to be fooled into thinking the old system is sufficient as a specification for the new system. Seems obvious in hindsight.
Evaluating your architecture pg 36 – “A big red flag for me is when people talk about the phases of their modernization plans in terms of which technologies they are going to use rather than what value they will add.
Teams tend to move in the direction they are looking. If we talk about what we’re doing in terms of technical choices, user’s needs get lost.”
The curse of hindsight pg 61- “We struggle to modernize legacy systems because we fail to pay the proper attention and respect to the real challenge of legacy systems: the context has been lost.”
OB: I think this goes hand-in-hand with the warning not to treat the existing system as a specification for the new one
Easy and also impossible pg 63/64 – “We assume that successful systems solved their core problems well, but we also assume things that just work without any thought or effort are simple when in fact bear the complexity of years of iteration we’ve forgotten about.”
OB: This is a biggie – I’ve fallen in this trap. Things that work without effort are not necessarily simple.
Building and protecting momentum pg 76 – “In truth what works when rebuilding a system is not all that different from what worked to build it in the first place. You need to keep the scope small, and you need to iterate on your successes.
…
Assuming you fully understand the requirements because an existing system is operational is a critical mistake.”
Momentum builders: The bliss of measurable problems pg 77 – “..I prefer to restrict the scope by defining one measurable problem. Building a modern infrastructure is not a goal.”
OB: The author uses quite a nice example of improving the number of one specific type of refugee application processed at a large organization. Talks about the challenge of defining measurable problems.
Anatomy of a measurable problem pg 78 – “The strength of the measurable problem is that it is objective and irrefutable, and therefore, it helps the team navigate the internal politics they have inherited from the existing system.
…
The last benefit of measurable problems is that positive results are not linked to feature launches.
…
In all likelihood, the business side of the organization does not understand what’s wrong with the existing system. Rolling out features they already have is not something they will celebrate.”
OB: So true.
“If you’re thinking about rearchitecting a system and cannot tie the effort back to some kind of business goal, you probably shouldn’t be doing it”.
OB: The tricky one with this is something like documentation or refactoring code. Creating a measurable outcome on the benefits of these is not obvious to me, nor tying it to a business goal – even though the benefits to the business for the team maintaining it are clear.
“At first glance, this approach might seem unwise or even irresponsible, but the number one killer of big efforts is not technical failure. It’s loss of momentum”
OB: Here the author was saying one should focus on solving problems that specifically work towards that measurable problem, even passing up other opportunities for much needed infrastructure changes.
Step 2: Check for conflicting optimization strategies pg 83 – “A quick trick when capable engineers cannot seem to agree on a decision is to ask yourself what each one is optimizing for with their suggested approach”
..
“Becoming good at experiments is valuable for practically any organization”
Momentum Killer: A history of failure pg 84 – “The first deliverables of a modernization effort have to take this history of failure into account. People aren’t pessimistic and uninspired by legacy modernization projects because they don’t care or don’t realize that modernization is important. They often feel that way because they are convinced success is impossible after experiencing a number of failures.
At the same time, I have yet to find a group of engineers who didn’t want to believe they could reach a better state.”
…
“The hard problems around legacy modernization are not technical problems: they’re people problems. The technology is usually pretty straight-forward. Keeping people motivated through the months or years it takes to finish the job is hard.
Protecting momentum: A quota on Big Decisions pg 89 – “Therefore, the more big decisions your proposal seems to include, the more likely people are going to want to slow down or delay it a quarter.”
Mess: Leadership has lost the room pg 123 – “One of the ways people process trauma is by wondering if they deserved it. Removing rejected leaders might solve superficial problems, but it doesn’t restore the team back to excellence.
People without confidence self-sabotage.”
…
The only thing that convinces people to stop belittling themselves is knowing they have the trust and acceptance of their peers.”
Building trust through failure pg 169 – “It’s not the lack of failure that makes a system more likely to fail, it’s the inattention in the maintenance schedule or the failure to test appropriately or other cut corners.”
…
“For this reason, the occasional outage and problem with a system – particularly if its resolved quickly and cleanly – can actually boost the user’s trust and confidence. The technical term for this is the service recovery paradox.”
Building something wrong the correct way pg 207 – “Throughout this book and in this chapter especially, the message has been don’t build for scale before you have scale.”
…
“Build something wrong and update it often”
Conclusion pg 217 – “..to move the product, you must do it by turning the gars of this other complex, undocumented system”
OB: Author again highlights the organization and people-management aspect of the migration project. The undocumented system is the organization itself.
Keep up to date with new Data posts and/or Big Book of R updates by signing up to my newsletter. Subscribers get a free copy of Project Management Fundamentals for Data Analysts worth $12.
Once you’ve subscribed, you’ll get a follow up email with a link to your free copy.