Opinion

Here’s what went wrong with the Clinton-Trump polls

Donald Trump takes on “Crooked Hillary”

Donald Trump refers to Hillary Clinton as "Crooked Hillary" when speaking to the National Rifle Association's Leadership forum in Louisville, Ky on Friday, May 20. Top NRA executives officially endorsed him for president at this annual convention.
Up Next
Donald Trump refers to Hillary Clinton as "Crooked Hillary" when speaking to the National Rifle Association's Leadership forum in Louisville, Ky on Friday, May 20. Top NRA executives officially endorsed him for president at this annual convention.

The election of Donald Trump to the U.S. presidency surprised almost everyone, including apparently Trump himself.

On the morning after the 2016 election, my teenage son made snarky comments about the state of polling and statistical science. As a trained statistician, I took offense. However, I had no background in political science and really no idea what had gone “wrong.”

So I decided to put him to work, gathering and entering vote totals and poll data from 2016 and past elections, to judge for ourselves. In our analysis, we examined the performance of presidential poll-based predictions and proposed a new, improved model.

The 2016 election caused considerable hand-wringing over the state of opinion polling. However, the best evidence is that current polling is generally sound, but tune-ups to one particular aspect of how polls are collected — notably the practice of aggregating poll data — would be helpful.

Most observers concluded that the polls, on average, underestimated support for Trump. Such a systematic error is known as “polling bias.” Numerous reports and think pieces have tried to explain the bias, pinpointing specific problems in how polls assessed likely voters and were weighted by voters’ education levels.

To understand the issues, it’s important to recognize the distinction between polls, which represent samples of individuals at a particular time using a particular methodology, and poll aggregation.

Poll results can vary greatly, depending on who is sampled and what methods the pollsters use to weight respondents to account for nonrepresentative sampling, or assess who is likely to vote. With more and more people not answering pollsters and using cellphones, it’s an amazing testament to the field that opinion polling still works as well as it does.

Our own research suggests that the polling bias was actually not very large — that is, pollsters may have underestimated the support for Trump but not to a large degree.

We examined the state-level predictions across all 50 states, plus D.C., in 2016, as well as their stated uncertainty. We found that FiveThirtyEight and The Upshot showed statistical bias and overestimated support for Clinton, but with enough uncertainty that their probabilities left room for a Trump victory. HuffPost had similar state-level predictions, but was overconfident in these predictions.

When the polling data up the eve of the election were fully taken into account, we estimated the chance of a Trump victory as at least 47 percent. The main novelty in our approach was to use polling data from multiple states to “fill in” the sparse information from state-level polling. In contrast, the popular poll aggregation sites gave much lower chances, ranging from 2 percent on HuffPost to about 29 percent on FiveThirtyEight.

Our own analysis: The polls weren’t highly biased and were roughly correct at the time. However, pollsters conducted too few state-level polls just prior to the election.

Our rationale also explains why the estimate of the popular vote – 3.3 percent estimated margin for Clinton versus 2.1 percent actual – was largely accurate. National polls were conducted more frequently, and so the national averaging could include polls closer to the election.

The voter environment in 2016 election was unusual, with a sharp drop in support for the leading candidate prior to the election. The drop in support for Clinton started in mid-October.

Although the 2016 experience was unusual, we proposed a statistical model designed to be sensitive to a national trend. The model combines information across numerous states, instead of relying only on polling within each state. The model estimates the Democratic-Republican vote spread and overall win probabilities for the 90 days leading to an election.

Although our analysis was conducted after the election, we plan to try it out in 2020 in real time.

Fred Wright is director of the Bioinformatics Research Center at N.C. State University.
  Comments