Beware the Complexity Merchants

Something I’ve come to appreciate over my engineering career is simplicity and how it is an enabler of speed and creation.

Scott Carey wrote a great essay on this: Complexity is killing software developers:

“Complexity kills,” Lotus Notes creator and Microsoft veteran Ray Ozzie famously wrote in a 2005 internal memo. “It sucks the life out of developers; it makes products difficult to plan, build, and test.”

We have all felt this friction at some point in our careers where seemingly simple tasks are met with painful papercuts wrought by complexity. (It often reminds me of that scene in the movie Saw where one of the captives has to dive into a pit of needles to find a key.)

Fred Brooks famously made the distinction between complexity of the “accidental” and “essential” kind thinking, at the time, that many advancements had rooted out much of the accidental complexity in software engineering. Brooks might be quite disappointed at the reality that we face today.

From that same article, Carey quotes:

Justin Etheredge, cofounder of the software agency Simple Thread, helpfully differentiates between essential and accidental complexity. He told InfoWorld, “Essential is the complexity in the business domain you are working in, the fact that enterprises are extremely complicated environments, so the problems they are trying to solve are inherently complex. The other area is accidental; this is the complexity that comes with our tooling and what we layer on top when solving a problem.”

This distinction probably spells out the reason you’ve read this far: because you too, know the frustration of having to deal with accidental complexity while trying to solve the valuable problem of the essential complexity.

The challenge is that when a group of smart engineers are gathered, it can often be a difficult exercise of discipline to keep them from reaching for complexity of the accidental kind to get their highs.

Complexity is the Enemy of Value

When foundational systems have been addled by accidental complexity, the net result is that it:

• Unnecessarily creates drag on overall team velocity
• Creates an unstable foundation to build upon
• Focuses the efforts of smart engineers on work that does not correlate to business value
• Creates a culture of gatekeeping by making it difficult if not impossible (or at least highly discouraged) for the broader team to contribute

These systems are often brittle and difficult to own, understand, and scale because well, that was actually the intent from the start…

The Darker Side of Accidental Complexity

While accidental complexity can arise innocently enough from well-intentioned engineers, the darker side is that it can manifest from ego and self-preservation. For you see: building a simple, easy-to-use, easy-to-own, and easy-to-operate system means that such a system does not need nearly as much manpower, attention, and of course budget to own. If there are no fires to fight, then there is no need for a firefighter.

To make things simple from the get-go is not exciting to the complexity merchant and is also not profitable. How else shall the complexity merchant demonstrate their value and build their fiefdom within the enterprise but to create the context for which ever more complexity is layered to solve the previously added complexity? Of course the complexity merchant seeks to command more resources because can’t you see? How else will all of this complexity be solved?

For the complexity merchant to operate, they must always introduce a bit of friction and pain (this self-inflicted injury is how you will know that you are dealing with a complexity merchant). It is the presence of this friction and pain that grants them the justification to peddle their solutions (without it, how else could they expand their influence if everything were easy and smooth?). If you only bring on this tool or hire more engineers or add this vendor, then everything will be solved!

All along, they will tell you that such complexity is unavoidable and essential; such problems, they will say, have no simple solutions. They will tell tales of Airbnb and Facebook and Google and how yes, you — your team — also needs this complexity. You want to be Facebook, don’t you?

Even in cases where there is no malicious intent, it can be driven by the need to feel accomplished by summiting that peak of complexity.

These are individuals that have let their ego drive their decisionmaking.

A Path to Undoing Complexity

As parent, I often grumble and reprimand my kids if they start playing with something new without cleaning up and putting away what they were playing with previously. It is a simple rule to keep one’s space clean and organized and also imparts a basic sense of responsibility upon the child.

One simple solution to solving for new complexity? Make the engineers clean up after themselves first. No new complexity until the old complexity has been stabilized or simplified so that anyone can operate and maintain it and keep it in a stable state. No toys left in a state of disarray; all things orderly in their place where they belong at the end of the day. No half-finished promises and half-delivered (of course undocumented) solutions.

Even better to be skeptical of Silver Bullets and favor tried, true, and boring solutions and approaches for these rob the complexity merchant of their grip. Where complexity is necessary or actually adds business value, it must be contained and there must be a path to transition ownership of that complexity that includes handing it off in a stable, well-documented state.

Demand documentation and transferability of systems since obfuscation and gatekeeping are key tools of complexity merchants. The more push-back encountered for these basic demands, the more sure you can be that you’ve encountered a complexity merchant.

Closing Thoughts

I get it: simple, boring platform solutions just aren’t exciting or thrilling to build. They do not create the conditions for those concerned with self-importance, self-preservation, and simultaneously the desire to lord over a fiefdom. If the system is simple, well-documented, and transferable, then anyone can own and operate it, the complexity merchant has no foundation on which to build their empire.

Why build following boring, tried, true, and established patterns and solutions following paths where the hard challenges have been solved for? Where’s the thrill in that? How shall the moat of self-importance be created if these systems are boring, easy, and straightforward to build upon, operate, and maintain?

So a word of warning to you, dear traveler: beware of the complexity merchants as you endeavor to create value and seek to gain velocity towards whatever your destination shall be. You will know them by the trail of once new and shiny toys, only half played with, that they leave in their wake — packaging strewn to the side. You will feel the friction that they create and the control that they wish to exert. Do not fall, time and again, for their promises of a Silver Bullet only half kept.