Confidently Answering the Wrong Question
Failing to Understand the Why
Welcome to Polymathic Being, a place to explore counterintuitive insights across multiple domains. These essays take common topics and investigate them from different perspectives and disciplines to come up with unique insights and solutions.
Today's topic pulls the thread on a specific behavior within the workforce that I’ve seen quite a lot from people who should know better. It’s particularly problematic in Data Science and Operations Research which are supposed to avoid this and it also permeates research in science, politics, and more. We’ll explore when people confidently answer the wrong question and how to avoid it.
Intro
I had a manager who wanted to provide analysis on selling more Javelin anti-tank missiles to the government. The question he wanted us to answer was whether we should mount this anti-tank missile on top of the Stryker Armored Personnel Carrier.
On first blush, this seems logical since a Stryker does look a bit like a tank to the average person. There is one major challenge, however; this vehicle was not designed to be a direct fire platform.
The Stryker was designed to quickly move infantry soldiers from one point to another while protecting them from small arms fire and shrapnel. The weapon system you see is to suppress enemy infantry and small vehicles, not heavy armored tanks. It was designed to avoid direct combat, especially against an actual tank!
More importantly, it was created to replace the Bradley Fighting Vehicle which, as the docu-comedy Pentagon Wars aptly shows, started as a troop transport before evolving (devolving?) into a pseudo-tank that doesn’t do well for any missions.
We were being asked to add direct-fire weapons onto a vehicle to perform a mission that it was never designed for. Basically a repeat of the Bradley catastrophe.
My manager selected a simple Lanchester’s Force Exchange model, set up a Battalion of Strykers (without infantry) against a Russian Battalion Tactical Group (BTG) and ran it. Unsurprisingly, when you take a Battalion of Strykers with no direct-fire weapons and put it in a firefight with a BTG full of tanks and heavy weapons, it got wiped out. Then, when we put Javelins on top, they did better.
With no mounted Javelins and no infantry, the Strykers took out 0% of a BTG
With mounted Javelins the unit took out 15% of a BTG. Clearly an improvement
To make a simple analogy on how dumb that analysis really was, it’s like seeing what happens when you put a kid in a fight against Mike Tyson and then seeing if he does any better when you had him a baseball bat.
Clearly, the kid will do slightly better the second time but it’s still absurd. This analogy did not deter my boss who went right back to the customer and told them that yes, you’d do better if you put Javelins on Strykers and provided the detailed report.
We confidently answered the wrong question and had facts and data to back it up.
Asking the Right Question
The thing is, I can answer that question without modeling and simulations:
It’s not an apples-to-apples fight. It’s a kid vs. Mike Tyson
You’d never fight with a Stryker-mounted Infantry Battalion that way.
The simple fact is that the biggest threat to the Russian BTG is dismounted infantry armed with anti-tank missiles like the Javelin. This is because BTGs don’t maneuver as well as infantry supported by high-mobility transport like a Stryker. Further, dismounted infantry are really hard to see when you’re in a tank and they can ‘shoot and scoot’ making them a huge threat.
We were ignoring the natural strength of the Stryker unit and, instead, putting them into a fight they weren’t designed for and would never tactically engage in.
I decided to set up a different model using a more complex simulation environment to look at what would happen if the infantry were deployed per standard tactics and we doubled, tripled, or quadrupled the number of Javelin missiles they had available.
With no mounted Javelins and using infantry with their normal Javelin allotment, the unit took out 30% of a BTG (100% better than the mounted option from earlier and also the baseline for how that unit would tactically fight in real life)1
That same unit, with 2X the number of Javelins would take out 60% of a BTG
If we increased the Javelins to 4X that infantry unit could take out 1.5 BTGs!2
What I found is that we could sell a lot more missiles and have a massive improvement in performance if we didn’t mount it on top of the Stryker to be fired. Instead, we simply needed to provide additional missile reloads on ready racks mounted on the side of the Stryker. In this configuration, infantry soldiers can grab a missile, run forward to shoot it, get back to the Stryker, and move to a new location and repeat.
However, when I brought this back to my manager, he didn’t want to bring it forward because that wasn’t the question we were asked. He also got offended because he felt I was challenging the veracity of his simulation which he defended as being correct.
The answer was correct. It’s the question that was wrong.
Path Forward
“Answer the why before the what or how.”
There’s an adage I like to follow that goes: “Answer the why before the what or how.”
If we had stepped back and defined the goal more clearly, and then mapped out the methods we currently use to achieve that goal, we would have a better understanding of what questions need to be answered. In Operations Research, we call these Measures of Outcome, Measures of Effectiveness, and Measures of Performance.
For this example I’ll do a really rough outline:
Measures of Outcome: We want to increase our lethality against a BTG. Our hypothesis is that Javelin missiles can help.
Measures of Effectiveness: The Stryker is not a direct-fire platform. It’s used to haul infantry. Infantry is exceptionally lethal against a BTG but they only have enough Javelins to affect 30% of the targets.
Measures of Performance: What happens when I increase the number of Javelins available to the infantry while keeping the tactics constant?
Too often leaders come to teams and ask “Can we do this?” referring to the Measures of Performance. The right response to that is “Why would we do that?” and draw them back up to defining the Measures of Outcome.
I’ve seen so many efforts creating solutions to problems that don’t exist that I have material for years’ worth of essays. Worse, sometimes these confident answers to the wrong questions create new problems that are even worse than what they hope to fix.
In the example we’ve been using, if the customer implemented our solution and mounted Javelins on Strykers, it would put our troops at greater risk and significantly sped up our defeat because once you have a solution you feel compelled to use it even if it is worse!
Answering the “Why” first, and ensuring you have defined the right Measures of Outcome is imperative to avoid confidently answering the wrong question.
For further reading on this topic, I recommend What Data Science Forgot where we explored this topic from a slightly different angle and found a similar solution.
Enjoyed this post? Hit the ❤️ button above or below because it helps more people discover Substacks like this one and that’s a great thing. Also please share here or in your network to help us grow.
Polymathic Being is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.
Check Out Refind: Brain food, delivered daily
Every day, Refind analyzes thousands of articles and sends you only the best, tailored to your interests. Loved by 503,336 curious minds. Subscribe Here
Further Reading from Authors I Appreciate
I highly recommend the following Substacks for their great content and complementary explorations of topics that Polymathic Being shares.
Goatfury Writes All-around great daily essays
Never Stop Learning Insightful Life Tips and Tricks
Cyborgs Writing Highly useful insights into using AI for writing
Educating AI Integrating AI into education
Mostly Harmless Ideas Computer Science for Everyone
The other challenge with mounting a Javelin is that its range is only about 2.5km on a ‘good shot.’ Yes, it can go further but it’s better in a 1-2KM engagement. A BTG though, has weapons that can reach out to 6km easily. So a Stryker would have to get much much closer to even engage the BTG all while the BTG is able to shoot at the Strykers. This is why the ‘shoot and scoot’ and maneuverability of the infantry make them better.
We are talking combat effectiveness here, not complete annihilation.
Haha, "docu-comedy." So true.
"To make a simple analogy on how dumb that analysis really was, it’s like seeing what happens when you put a kid in a fight against Mike Tyson and then seeing if he does any better when you hand him a baseball bat."
Now you're just feeding Netflix ideas for a new reality show.