Debugging Architects – The Architect Elevator

The importance of the architect’s role appears unhampered by the lack of a precise definition. Failing an all-encompassing depiction, we resort to individual aspects, skill sets, or tasks. However, even those are much debated and opined upon. Take the perennial question: should architects code? And if so, how much?

I prefer to base my opinions on actual data points, so after my recent engine room visit , I concluded that visiting the engine room is indeed essential for architects, but for a different reason:

Architects must visit the engine room. Not to deliver code but to re-emerge with new insights.

If gaining insights in the engine room isn’t linked to cranking out lines of code, perhaps architects can achieve the same benefit with other activities?

The engine room: Opposite the ivory tower

Why is it so important for architects to be connected to the engine room? In short, to validate (or invalidate) their assumptions. Architects make decision based on mental models and heuristics. However, the systems we deal with evolve and change rapidly, forcing us to revisit our past assumptions. For example, many architects still aim to eliminate any duplication in their systems. However, modern development, testing, and automation approaches have made it much easier for us to handle duplication while Economies of Speed highlight the cost of de-duplication. That’s why many organizations now tolerate duplication and some even favor it (consider the famous Amazon “2 > 0” mantra).

Visiting the engine room keeps architects from the proverbial ivory tower: making decisions and passing them down via a one-way command-and-control setup, ignoring downstream costs and challenges. In the past, assumptions and heuristics held longer thanks to slower evolution of the underlying technologies, allowing architects to get away with some amount of ivory tower-ness. At the current pace of technical evolution and the rate at which disruptive technologies arrive, not visiting the engine room regularly would be considered gross negligence for architects.

Not visiting the engine room regularly is considered gross negligence for IT architects.

I often describe the key benefit of the cloud (coupled with modern software delivery approaches) as removing past constraints. Those past constraints drove our architecture decisions, so changing constraints force us to revisit our assumptions. My book Cloud Strategy gives a concrete example when it discusses the end of multitenancy thanks to cloud automation, elastic infrastructure, and observability (an early version of that chapter is still on LinkedIn).

Architects enjoy being wrong

Architects must test their assumptions regularly and must be happy if they find out that some of them were wrong—that’s how we remain relevant!

I recently recorded a podcast with my old friend and ex-coworker Dave Farley for his Engineering Room series of interviews with industry luminaries (plus me, apparently). We were jesting that boredom isn’t a problem we have in our industry, because something new comes along all the time. Occasionally that new thing pulls out the conceptual rug from underneath you, invalidating your hypotheses, assumptions, and heuristics. Both Dave and I felt that this makes up the fun part of our job and that as an architect you should enjoy finding out that you are wrong – that’s how learning takes place.

Being wrong with style: Debugging

A fantastic way in which systems tell us that we are wrong is debugging (including trouble-shooting operational issues). Debugging happens when reality deviates from what you intended it to be, i.e., you were wrong in some way. The task now is to figure out where you were wrong. That means the happiest moment of debugging is when you find out where you were wrong:

The happiest moment of debugging is finding out where you were wrong.

Instead of (or in addition to) coding, architects gain a lot by debugging systems:

Low overhead

Helping a developer debug carries little setup or overhead tasks. The developer will have an IDE and all tooling setup and most observability tools due to their nature are running subsystems that don’t require individual setup or configuration. That means you can jump into a debugging session quickly and provide value to the teams.

Focus on relationships

Much of architecture is concerned with the lines, i.e. the connections and dependencies between elements. The “boxes”, i.e., individual components, are typically well tested and less likely to be the source of the problem. That’s why many bugs, and especially run-time issues, are the result of complex interplay between elements. Unsurprisingly, this is also the domain of system architecture. Modern architects don’t need to tell developers how many lines of code are allowed per method or when to use a Singleton. Rather, architects look at the overall system structure and assure reliable operation in face of rapid change in load, requirements, or context.

Small change, large effect

The result of a debugging exercise is generally a lot of thinking, followed by a small code or configuration change. That’s why it’s an ideal task for architects. Cranking out lines of code isn’t the most value-add activity for architects. But understanding system structures and hidden dependencies is, and debugging is all about that.

Transcend levels

The key notion of the Architect Elevator is to move across levels, from IT and business strategy down into technical details. Debugging does require you to dive deep. One of the greatest debugging stories is Jeff Dean and Sanjay Ghemawat looking at data in binary format to detect hardware-related failures.

Serendipity

As architects we like to be structured in our approach and thinking. Debugging works the other way around: the system takes you to a place that you didn’t pay enough attention to. Therefore, debugging is a great way to get to know corners of the system that you might have otherwise neglected.

Immediate value-add

Coding is a useful exercise for architects, but it also overlaps with developers’ tasks and performance metrics. A coding architect can be seen as a help to the team, but also as stepping over his or her boundaries. Help with debugging or production issues, in contrast, is virtually always welcome. And it certainly beats management asking for updates on the outage. Helping developers trouble-shoot can earn you some very helpful friendships in the engine room.

Lazy web workers

One of my favorite debugging stories is helping a colleague resolve a scalability issue for a browser-based end-to-end encryption. To utilize modern multi-core CPUs, he was using the Web Worker framework to decrypt data in the browser in parallel. Sadly, whatever level of parallelism he set, things didn’t speed up. To me, the exercise was fun because I had little experience in JavaScript and had no idea about Web Workers until I realized that it’s a descendant of Google Gears which I spoke about in Japan in 2007 (I’ve always enjoyed parallel and distributed things). So, I learned a good bit in the process as well – a great takeaway from the engine room.

Despite my technical ignorance, my contribution to the debugging exercise came in two forms. For one, to look for blind spots, areas that a developer would normally not look. We are all bound to the Streetlight Effect, so a second (and unencumbered) set of eyes can be very helpful. Second, I focused on building a mental model of the application in my head to establish hypotheses that we subsequently tested.

One such hypothesis was that the bottleneck might not actually be in the parallel section handled by the Workers. We verified this assumption by setting the number of threads to one, noticing that the process didn’t slow down at all. Knowing that the bottleneck is in the single-threaded part of the application, we followed a divide-and-conquer approach, aided by a profiler. Not going as deep as Jeff and Sanjay, we found the problem in a parsing library that contained the mother of all RegEx expressions to accommodate a myriad of UTF-8 blank and end-of-line character variations. Since we controlled the backend that generated the data to be parsed and decrypted, no such flexibility was needed. Replacing the RegEx expression with a simple split statement essentially solved all our problems—a classic 1-line fix.

Similarly, much of my learning during my serverless engine room deep-dive came took place when I had to debug and trouble-shoot my system, ranging from permissions issues to quirks with JSONPath.

Architects must debug

I am still in favor of coding architects, but debugging architects potentially get more bang for the bug, pun intended.