AREA Method: How Flawed Data Trumped A Hillary Presidency

By November 12, 2016Blog

Donald Trump’s victory was a surprise. How come? With all of the legions of pundits, commentators and pollsters, how did so many experts make the same mistake, and how can we as decision makers do a better job of understanding the data before us?

The AREA Method teaches that in research and decision making we need to carefully distinguish between what is salient and what is significant. Salient information is only worthy of note, but what is significant carries meaning.

The word salient is originally a military term: It’s a line of defense. In the 1560’s it was thought to come from the word “leaping” in Latin, salientem. The Greek source, hallesthi also means “to leap.” From the Middle Irish, saltrain, is translated as “to trample.” And that’s what happened: either a failure of data, a failure of analysis –or both- that inadvertently created a sense of certainty where certainty wasn’t possible. The result was that many Americans were enticed into a complacency before the election that was false.

Might there have been problems of sample size or selection bias, where a sample is not indicative of the broader complexion of the nation? How do we protect ourselves from assumptions related to our evidence since skill at statistical analysis and skill at drawing findings from the data are often two different things? To prevent such problems, consider:

  1. Comparing Apples to Apples: Review how the data is selected. Make sure you understand how a study was designed and conducted.
  2. The Rule of Three: does your data come from at least three different sources? By collecting data from three unrelated sources you can be more certain that your information is valid. Be aware of groupthink. When sourcing contacts, it isn’t enough to have three sources if they are too closely related or depend upon the same data set.
  3. Base Rates: We all tend to ignore base rates, which are the underlying percentages or the actual likelihood of an event occurring. This tendency can lead to poor decision-making. For example, millions of people play the lottery every day in spite of overwhelming odds against winning the jackpot because media stories about winners — the exceptions to the odds — are more salient and memorable than the odds themselves. Despite having the lowest payout rate, the lottery is the most successful form of commercial gambling because we all rely too heavily on memorable events rather than base rates. We also all tend to overestimate our own ability to beat the odds.
  4. Data Fishing: The findings from one data set don’t automatically apply to another data set. While it’s useful to identify patterns and relationships between data, be careful to avoid data fishing, or taking more information from a data set than it actually contains. For example, weather patterns in Minnesota don’t necessarily apply in Texas.

These simple rules can help us uncover mistakes in our data and in our thinking. The power of discovering a mistake is not only that it gives you something concrete and detailed to fix, but that is also allows you to advance your knowledge.

This election reinforces that polls are a lot like balance sheets; they are a single snapshot in time that miss information that if, or when recognized, would have a significant effect on the future. The problem though, is that there are times when the missing information can’t be captured at all, when the unknown is unsolvable. And that too may have been the case in this election. The result is that we could have known the flaws in the data even though we may not have been able to rectify them.

Knowing that the data is flawed is valuable. Americans might have been less confident that Clinton would win. Armed with that knowledge they could have adjusted their expectations and, if they were Clinton supporters, perhaps they might have been more motivated to turn out more voters to better influence the outcome.  Or maybe those who felt that she would win handily and therefore didn’t vote for one reason or another might have turned out themselves.  If people really knew how precarious the situation was, maybe the election would have turned out differently, after all while votes are still being tallied around the country, many state races came down to razor thin margins in favor of Trump.

Since we all need to work with, and through ambiguity, we need systems now like the AREA Method that help us understand flaws in our data, our analysis and in our thinking. For while our instinct to piece things together from incomplete information can be both good and bad, in making a high stakes decision, we want to avoid the bad.

May a Trump presidency guide this great nation with humility, empathy, thoughtful reflection and open-mindedness; may he too recognize that data, analysis and thus thinking may be flawed before making decisions.

Leave a Reply