Welcome to Polymathic Being, a place to explore counterintuitive insights across multiple domains. These essays take common topics and explore them from different perspectives and disciplines and, in doing so, come up with unique insights and solutions. Fundamentally, a Polymath is a type of thinker who spans diverse specialties and weaves together insights that the domain experts often don’t see.
Today's topic investigates the failure of so many proofs of concept, minimum viable products, innovation initiatives, and systems in general. The underlying reason might actually surprise you because very often the details look great, the people worked together really well, and all of the processes and artifacts were tip top. It’s only when we step back and look at the whole system that we discover that the main failure happened long beforehand, back in design, or the lack thereof.
The problem with proofs of concept isn’t that they fail, it’s that they don’t scale
What if I told you there was a way to increase your project’s chance of success by over 100%? That you could change the script on failure to that of learning? That you could address risks proactively so that your team achieves success that scales? All of this is possible, leverages known enablers, and borrows from ancient Taoist philosophy.
Many of us have been there: a great idea, a motivated team, and a sponsor who has just approved the project when we weren’t expecting it. We’ve got some visibility, there are some vested interests to prove you can do it, and so you jump right in and start executing. Soon the team realizes that the initial ideas of what was possible begin to run into reality. The data is dirty, the other tools aren’t quite there, and other teams aren’t collaborating as well. But you strive for success and so you narrow your focus, add on a few more assumptions, and continue to execute.
Eventually, after a lot longer than expected, and nowhere near the goal you had started with, you cross the finish line with your proof of concept. Maybe it even works pretty well. But does it scale? And thus, we run into one of the key failures I’ve seen with projects. The problem with proofs of concept isn’t that they fail, it’s that they don’t scale! In helping teams conduct hundreds of retrospectives, what continues to emerge in this failure seems shockingly simple. They didn’t design first. They fell victim to the programmers’ credo: “We do these things not because they are easy, but because we thought they were going to be easy.”
This brings us to the focus of our topic. What’s in a design, what is often overlooked that can dramatically improve outcomes, and how we can set ourselves up for success from the start?
(Note: While this can be any sort of project, I’m going to focus on Software for this essay)
Polymathic Being is a reader-supported publication. To receive new posts and support our work, please subscribe!
What’s in a Design:
In a quote attributed to Einstein, the importance of design is stated as:
“If I had an hour to save the world, I would spend 55 minutes defining the problem, and 5 minutes coming up with solutions”
In the case of our proof of concept, 55 minutes should be on design and 5 minutes on the code itself. Yes, this does seem a bit extreme but it highlights several concepts that are important:
Define the problem
Systems Modeling and Simulation
We’ll break each of these down in the next sections to explore some unique idiosyncrasies of each, how they tie together, and why that time spent in what appears to be non-action, will actually save an incredible amount of time
1: Define the Problem
One of the key elements of Operations Research, the somewhat niche discipline I call my intellectual home, is to really spend a lot of time defining the problem. We aren’t designers in the sense of traditional engineering, we are solutioners, working in really messy, complicated areas in an attempt to provide clarity, illuminate key measures of success, and often challenge the very foundation of what we thought the problem was.
An adage of Operations Research we had at my role in Lockheed Martin was:
“Answering the Why before the What and How.”
As we covered in a previous essay on leadership; “Providing Direction, Energy, and Accountability” the direction requires defining the problem, and defining the problem starts with '“Why.” As simple as this concept seems, you might be shocked at how much contention this can actually cause. (I’ve been told asking “Why?” is disruptive and to just execute) Fundamentally starting with why is a great way to identify what we do and or do not know about the problem.
The next step, in defining the problem, is to make sure it doesn’t change on us. As we discussed in Systems Thinking, there are two key things to focus on here.
If you think you understand the problem, step back and see if you understand the system.
Intentionally reframe the problem from multiple perspectives and if the problem statement changes as well, you may not have the right definition yet.
Asking why, stepping back and looking at the system, and reframing are three essential steps to truly defining the problem. There is one more I’d like to add that we really haven’t covered yet, and this is the addition of urgency. So often, the definition of a problem is tightly coupled to a “Do it Now!” sense of reactive urgency. It elevates a problem’s sense of importance and is probably the number one reason why teams fail to spend time on design. I’ve seen this manifest from root cause analysis to software development, to systems analysis, where the impetus to execute, to just get something down, overrides all other measures.
But this time constraint is rarely actually required and can typically be ignored. In dozens of retrospectives, the excuse of urgency looks more like poor insight, reactive leadership, and myopic thinking than any form of legitimate decision-making. Borrowing a finding from the Lazy Leader, often industriousness in execution creates failure. Instead, applying the Taoist concept of Wu Wei helps break the insistence on action and provides the opportunity for flow. Wu Wei is about as counterintuitive as you can get because this is not inaction, it is intentional NON-action.
A great example of Wu Wei is what we call a Tactical Pause, in the military. It means, in the height of a combat situation, you might intentionally do nothing in order to let the situation develop more. For example, if I am on a mission to conduct a recon, and an enemy unit is attempting to find me, they may perform their own ‘recon by fire’ which means they shoot into an area that they think I might be at. If I return fire, which seems logical at first, I confirm my location and lose my advantage of stealth. If instead, I hunker down and take intentional non-action, the enemy has no confirmation and I can let the situation develop more while maintaining an ability to bring in other resources such as aircraft or artillery.
Applied back to defining the problem, just like we reframe the problem from multiple perspectives to see if the problem statement changes, we can also shift the problem in time to see if it changes to determine its effect on the proposed solution. A great question to answer is whether not taking action now has any effect on the problem, or worse, a negative effect. Sometimes we frame this question as “What if I do nothing?” because with a solid understanding of the problem, in time, we can confidently step into the design.
2: Design Rigor
Even with a solid problem definition, we often like to jump into the design and start putting together ideas to work toward a solution. Yet, in line with Wu Wei again, we need to understand the systems and structures we’d like to support our design with.
For instance, there are a few key ideas that can completely change the way in which you execute concepts like applying Risk-Driven and Client-Driven Iterative Planning, or Decision / Learning Point Planning, or Design Structure Matrices, and many more. What’s important to consider is that these techniques must be applied proactively, and have significant capabilities to unlock incredible value in your design such as:
Risk and Client Driven Iterative Planning - This is a design process by which high-risk items are front-loaded into the plan to ensure faster feedback in order to adapt or change the proof of concept.
Decision / Learning Point Planning - This is a foundational reframing of execution that completely inverts the risk of failure into learning and decision opportunities. If we think of a design as a process of learning, not milestones, it changes the way we think of executing completely.
Design Structure Matrices - This tool is not new but it is very overlooked. It is a core foundation for creating modular and composable systems while looking at how features and infrastructure work together to avoid “bubble gum and bailing twine” designs.
All three can be applied to a design and to achieve full success, they require understanding what you want the design to look like to start with. As silly as it sounds to say it like this, so many of our proofs of concept look like a builder starting a house with no blueprints. Ensuring design rigor through proper processes and tools is essential to true success.
A last thought on Design Rigor is to ensure you are capturing your systems' architecture and requirements at the systems level. Following a process like The Open Group Architecture Framework (TOGAF) or the Department of Defense Architecture Framework (DODAF), among others sets the stage for the final element of design which is Modeling and Simulation.
3: Modeling and Simulation
There’s one final consideration that is essential to systems design and proofs of concept and that is modeling and simulation (M&S). This is an essential step for designs in ambiguous solutions space with more unknowns in both design and performance metrics. M&S will not be successful without applying the first two aspects of design that we’ve discussed as you’ll create a pseudo design based on a properly formed problem definition and crafted from the documented architecture.
Properly applied, an M&S environment allows the hypothesis testing of ideas in support of the decision/learning point planning discussed earlier. It allows experimentation of new ideas, facilitates defining improved performance indicators, and helps ensure that features avoid common bugs that can emerge from a lack of systems understanding.
M&S is a much deeper topic on its own so, suffice it to say, that designs that apply M&S dramatically out-perform designs that don’t, they are more likely to stay on schedule, scope, and budget, and are much more likely to be integrated into multiple systems on completion. M&S doesn’t have to be incredibly high fidelity either. It can be tailored to the decisions and learning points that were previously defined and create a path for rapid iterative execution before designers are too tied up in coding.
Designing first, while obvious when saying it, actually happens much less than you'd think. The push to produce code, demonstrate value, and keep moving create incentives to compress, or even skip the process entirely. In almost all of the retrospectives I’ve conducted, whether it be code, hardware, factory improvements, business development processes, or military combat missions, a common theme that emerges is that they were all biased toward action, and they all failed because of it. We’d often be told that this was an opportunity to “Build the plane while flying it” as if this was both intelligent and logical.
Tying all these concepts together, the solution begins to take form where through proper problem definition, we can create design rigor, changing the script from milestones to learning and creating strong systems designs. With these designs, we can quickly iterate through modeling and simulation to test our hypotheses and create effective code. All this is wrapped up in what we learned with Wu Wei, where the right answer is to spend that 55 minutes defining the design, and only 5 minutes writing highly effective and composable code modules that can solve the right problem thereby creating proofs of concepts and designs that don’t fail and scale long into the future.
I’d love your thoughts, feedback, and experiences with this topic. How often do you run into this problem? What have you done to resolve it? Please comment below so we can continue the collaborative learning!
Please share Polymathic Being! If there is anyone you know who could benefit from these essays, send them the link! Share it to your workplace slack channels to start up a new discussion! Post on social media! This is a great way for me to expand my feedback and continue to create impactful essays.
Further Reading from Authors I really appreciate
I highly recommend the following Substacks for their great content and complementary explorations of topics that Polymathic Being shares
Looking for other great newsletters and blogs? Try The Sample
Every morning, The Sample sends you an article from a blog or newsletter that matches up with your interests. When you get one you like, you can subscribe to the writer with one click. Sign up here.
Also, if you have a newsletter you would like to promote, they offer a great service that gets your writing out there to a new audience.
Decision and Learning Point Planning isn’t a concept that has caught on completely mostly because it is counter-intuitive. The Department of Defense, through DARPA, has developed a concept called Decision Centric Warfare from which I borrow extensively. Just as with project management, it is incredibly powerful, yet still nascent in application.
1. Correct me if I missed it, but I did not see anywhere the striving for "elegance" in design (many other references and documentation abound discussing "elegant design")
2. Here's a few 'heuristics' to consider:
Some heuristics for building a system
The conceptual phase:
• The choice between architectures may well depend upon which set of drawbacks the client can handle best
• Extreme requirements should remain under challenge throughout system design, implementation, and operation
• Don't assume that the original statement of the problem is necessarily the best or even the right one
• No complex system can be optimum to all parties concerned, nor all functions optimized
• A model is not reality
• Complex systems will develop and evolve within an overall architecture much more rapidly if there are stable intermediate forms than if there are not
• Build in and maintain options as long as possible
• Don't make an architecture too smart for its own good
The build and test phases:
• The product and process must match
• An element good enough in a small system is unlikely to be good enough in a more complex one
• Within the same class of products and processes, the failure rate of a product is linearly proportional to its cost
• High-quality, reliable systems are produced by high-quality architecting, engineering, design, and manufacture, not by inspection, test, and rework
• Regardless of what has gone before, the acceptance criteria determine what is actually built
• To be tested, a system must be designed to be tested
• Qualification and acceptance tests must be both definitive and passable
• The cost to find and fix a failed part (or software bug) increases by an order of magnitude as that part is successively incorporated into higher levels in the system
The operations phase:
• Before the flight, it's opinion; After the flight, it's obvious
• The first quick-look failure analyses are often wrong
• For every competitive system, there is a countersystem
• Success is defined by the beholder, not the architect
• There's nothing like being the first success
Systems Architecting: Creating and Building Complex Systems, by Eberhardt Rechtin, 1991
Love this. My favorite line is: The problem with proofs of concept isn’t that they fail, it’s that they don’t scale
I also love your 55/5 design-to-doing ratio, which I just wrote about. HOWEVER I might add one dimension, which you certainly document nicely here: SHIPPING and STARTING OVER. That is, I think the danger of the 55/5 idea is that it might be interpreted as causing analysis paralysis, that you just sketch all day.
The longer view of creativity is more like a long assembly line of 55/5 sprints of work:
Something like that!