Rethink the Wisdom of Crowds

Keys to Innovation Part 3

Welcome to Polymathic Being, a place to explore counterintuitive insights across multiple domains.  These essays take common topics and explore them from different perspectives and disciplines and, in doing so, come up with unique insights and solutions.  Fundamentally, a Polymath is a type of thinker who spans diverse specialties and weaves together insights that the domain experts often don’t see. 

Today's topic evaluates a trite and therefore, often wrong concept known as the Wisdom of Crowds. This concept suggests that large groups of people are smarter than an elite few, no matter how brilliant, and are better at solving problems, fostering innovation, coming to wise decisions, and even predicting the future. While this appears to be very aligned with the Polymathic mindset, in application it misses the nuance of how this idea is grossly misapplied to our detriment. We’ll detail the concept, explain the limitations, and introduce other options to extract wisdom in a true Polymathic manner.

This is Part 3 of the Keys to Innovation Series. You can find Parts 1 and 2 here:

• Part 1: Embrace the Divergents

• Part 2: Face the Fear

Image by 22612 from Pixabay

The Wisdom of Crowds was popularized by the circa 2004 book authored by James Surowiecki and contained a simple idea: experts aren’t always right and leveraging a broader group can yield better results. This concept soon exploded into the business world and showed up in motivational speeches and innovation and strategy sessions. On the surface, this seems like a really good idea. It is true that focusing on a larger field of view is better than relying on a few experts. This is a core fundamental of Polymathic Disciplines and you’d think we’d readily embrace this concept. Yet, while we do embrace the larger view, we have to add heavy a caveat to put it all into perspective so we understand the strengths and weaknesses in application. Let’s start with the origination of the idea behind The Wisdom of Crowds.

History

In 1907, statistician Francis Galton compiled hundreds of estimates from an Ox weight-judging competition and found that while individual guesses varied wildly, the median of the entries was within one percent of the ox's actual weight. This ushered in the theory of collective intelligence, which ended up being called The Wisdom of Crowds.

A great example of this concept today is when a facilitator has a large container of gumballs and asks people from within the crowd to guess the number. As expected, numbers range from high to low but the average is very close to the actual number. As captured in the video below the average was only 22 gumballs different than the actual number. Seems like a great opportunity doesn’t it?

The Caveat

While I fundamentally agree that we need to break away from the experts, what we are looking at isn’t the Wisdom of Crowds, it’s merely the statistical probability that people with a numbering system and a general idea of volume, given a large enough sample, will come close to the actual number even though most won’t be close individually; it’s just the law of large numbers.

This system entirely falls apart depending on the audience. The first example with the Ox occurred in 1907 at a county fair. The type of people who would guess the weight of an Ox in order to win an Ox are likely to be people who have an interest in said Ox. While some random people might throw their guesses in, by and large, the population of the guesses will be those who know what an ox is and have some idea of general weight. It is important to highlight the question of whether people today even know that an Ox is a castrated male cow trained to pull loads and this clarification clearly shows the essential nature of context.

For instance, if I brought in a large number of people from the city with no knowledge of bovines it would potentially skew the outcome. But even these city folk still have an idea of weight and volume so they’d probably guess terribly high or low. But what if I brought in a group of people with a limited numbering system like the Pirahã tribe? The Pirahã are a small, isolated hunter-gatherer tribe from the Amazon with a counting system of only a few words: “one” (which can also mean “roughly one”), “two”, and “many.” Clearly, if we added a sample of these to the mix, our numbers will become even more skewed.

What you see, from a statistical sampling perspective, is that the experts are represented by the red line in the image below. While not individually perfect, their distribution is tightly clustered around the average, known as the mean. If we add in a larger sample of less knowledgeable people, you’d get the blue line. The mean is still the same, but the errors of under and over-estimation are greater. Lastly, if you brought in people with a limited numbering concept, you’d probably get something like the orange or green line where the mean ends up shifting either higher or lower.

Figure 1 (Source)

What clearly emerges is that the concept of the Wisdom of Crowds is based on a common understanding, a simple target to measure, and a common numerical system that can control the output. Put a different way, this application requires taking into account the crowd and the goal.

What also emerges is that the Wisdom of Crowds doesn’t mean the accuracy of crowds because landing on the average isn’t a skill, it’s the statistical probability of counter-acting errors. Said that way, it doesn’t sound very wise at all!

This concept works for simple problems like guessing gumballs, so why spend all this time rethinking the concept to begin with? What’s the risk of applying this concept elsewhere?

Misapplication of the Wisdom of Crowds

The first major reason to write this essay is that there is a huge risk in applying something that works on a simple and controlled problem to the increasingly complex, and emerging wicked problems we face in our industries today. Bottom-line, the Wisdom of Crowds works in the finitely discrete and does not scale to systems.

For example, one of my biggest frustrations is conducting innovation or strategy by popular vote. All too often this involves some motivational presentation at the start, typically talking about the gumball solution with the presented idea that “With our powers combined we’ll come up with the best ideas,” channeling Captain Planet.

Captain Planet “With our powers combined!”

The issue here is that the average of a lot of ideas is an average idea. While a large sample of guesses might identify the number of gumballs, it won’t help you harness wisdom to create strategy or innovation. We’ll pause here and define the word “Wisdom”:

noun

• the quality of having experience, knowledge, and good judgment; the quality of being wise.

• the soundness of an action or decision with regard to the application of experience, knowledge, and good judgment.

• the body of knowledge and principles that develops within a specified society or period.

When it comes to the concept of the Wisdom of Crowds, it clearly pertains only to that third definition within a specific society or period. Simply put, you must have crowd control for any useful outcome. Painfully put, not everyone’s idea or opinion is valid or useful until you understand what outcome you are attempting to achieve.

Further shortcomings include the requirement for peoples’ decisions to be independent of one another. If you allow people to answer in a group, the first answer results in a ‘herding’ of the other’s guesses towards a relatively arbitrary output. Fundamentally, the Wisdom of Crowds should be thought of as a tool with limited usefulness in accomplishing tasks, as it can lead to herd behavior and limit new ideas.

Jaron Lanier in You Are Not A Gadget articulates this point as well:

The “wisdom of crowds” effect should be thought of as a tool. The value of a tool is its usefulness in accomplishing a task. The point should never be the glorification of the tool. Unfortunately, simplistic free market ideologues and noospherians tend to reinforce one another’s unjustified sentimentalities about their chosen tools.

Where else is there wisdom?

The key element that is missed in the general application of the Wisdom of Crowds is actually the Polymathic mindset. That is the ability to understand broad areas of knowledge and recognize how to tie concepts together. It’s not experts vs. crowds. It can’t be distilled down that myopically and work in either direction. What’s important to note is that there is a position between the experts and the crowds and that is what I capture as the Wisdom IN Crowds.

A great example of this is how almost all the best high school sports teams come from large schools. This is because they have a very large population of students and if we were to measure them on sports talent, the tail end of that distribution contains a much larger population. Working off of Figure 3 and assuming you select from the top 2.5% of athletes within a specific sport:

• Big school with 2,000 students x 2.5% = 50 high performing athletes

• Small school in the country of 200 students x 2.5% = 5 high performing athletes

Now, this is a gross simplification but it highlights the point. A bigger school has a larger population from which to draw top sports talent. This is why those larger schools dominate the smaller ones in a given sport.

Figure 3 (Source)

The key element to keep in mind is that these sports teams are looking to draw in the BEST talent, not the average talent. This is the Wisdom IN the Crowd.

You can juxtapose this against the Wisdom of Crowds which merely represents a random selection from the population. If we were to measure the sports prowess of both populations, the mean might remain the same, but the distributions would be different. While the large school might have more elite athletes, they also have a larger population of really poor athletes whereas that small country school will likely have a tighter distribution of capability. While they have fewer elites, almost everyone at the country school knows how to play sports.

Therefore, if we did a random selection for each sports team from their general population, meaning we are as likely to end up with the best AND the worst, the probability that the smaller country team would dominate the city team is almost inevitable. (see Figure 2) Understanding the Wisdom IN Crowds allows us to recognize the nuances of the crowd against our goals, and illuminates the need to manage the effort of extracting the wisdom.

Figure 2 (Source)

Comparing Wisdom

This isn’t to say that the Wisdom of Crowds is completely wrong, but it needs to be best understood in context. Just like the Quantum Superposition Problem in Politics, there isn’t a binary; it’s just the proper application of a tool and this tool can be made better.

Researchers Albert Kao (Harvard University), Andrew Berdahl (Santa Fe Institute), et. al., in the Journal of the Royal Society Interface, explored the accuracy of our collective intelligence. Specifically how individual bias and information sharing skew aggregate estimates. Their findings led to a mathematical correction to the Wisdom of Crowds by taking bias and social information into account so that they could produce an improved estimate.

Another way to make the Wisdom of Crowds work better was captured by Lucio Buffalmano as 5 elements that are necessary to consider when using this tool:

1. Diversity of Opinion: individuals must draw different background and beliefs and base their opinions on private information

2. Independence: individual’s opinion must be independent and not swayed by the group

3. Decentralization: decentralized systems allow an individual to draw on local knowledge and react faster

The top three are the conditions for a group to be “intelligent” and reach wise conclusions.

To make the group functional and to keep working, the system also needs:

4. Aggregation: there must a system that from the decentralized individuals aggregates the private judgment into collective wisdom

5. Trust: each individual must trust that the group is fair and that punishment is handed out for rule-breakers

These are things that improve the capability of the Wisdom OF Crowds as well as the Wisdom IN Crowds.

These caveats don’t mean that the Wisdom in Crowds is also the only way to go. The bottom line, to borrow a phrase from Systems Engineering, is that “it depends.” What quickly becomes highlighted is that extracting the Wisdom OF, or IN Crowds requires additional analysis. We have to fully understand the needs of wisdom, recognize the strengths and limitations of our collection methods, and not fall victim to a false binary of one over the other.

This is exactly what the Polymathic mindset prepares us for as the very definition of a Polymath is:

An individual whose knowledge spans a substantial number of subjects, known to draw on complex bodies of knowledge to solve specific problems.

Embracing this focus allows us to balance the idiosyncrasies of extracting the most appropriate wisdom in the most appropriate way.

Unlocking Wisdom

There are some keys to help manage unlocking wisdom. For example, in brainstorming ideas, it is important to put selection criteria on submissions. Ideas for new product ideas may have to meet time to implement and potential revenue expectations. Ideas that don’t pass those criteria should be put aside where they can be modified, matured, or combined with other ideas.

Similarly, we have to be critically discerning about the expertise we bring in. The current mantra of ‘trust the experts’ arising during COVID has quickly demonstrated their myopic and often wrong decisions because health officials were not complemented by psychologists, educationalists, economists, behavioralists, and more! The challenge here is that the Wisdom of Crowds was tainted by these incorrect ‘experts’ and so the only way to extract a solid analysis was to carefully select an appropriate pool from IN the Crowd. Going back to the Captain Planet imagery from before, each of the characters in that TV show were uniquely different experts whose cross-domain knowledge was combined to solve the challenge and not just randomly selected from the crowd or a specific silo of expertise.

Lastly, teams need to pre-commit to a strategy for combining their opinions and aim for strategies that remove human judgment from the aggregation process when facing quantifiable questions. We have to look at the problem and the solution from a broader perspective to understand how to properly, and accurately provide solutions.

The strength of the Polymath is being a type of thinker who can help discern the needs, information, and people needed to look at the problem from a Systems Mindset. This helps to unlock ideas, solutions, and innovation that neither the crowd nor the experts are able to.

Conclusion

I feel like this topic went down a nuanced path that might seem trivial. I wrote it because I’ve seen this trite and feel-good concept applied far out of the scope of its capabilities often with detrimental results. As with many topics we’ve explored, the Wisdom OF or IN Crowds needs a much more systematic review to understand what we are expecting to extract from the engagement. It isn’t a binary either/or, and it requires a nuanced approach with an intentional structure to extract wisdom to solve our increasingly complex problems. It isn’t terribly hard once you understand the strengths and limitations of these concepts but if you are still unsure how to pull it all together, it might be time to Embrace the Divergents and leverage the Polymaths in your organization.

---

Thanks for reading and listening to this essay. We are continuing to grow week after week as we add new subscribers. Please take a moment to like, comment, and share these essays to further expand the Polymathic network!