ADVERTISEMENT

A question for Ladoga re: Lessons Learned from 2012

DougS

Hall of Famer
May 29, 2001
17,075
716
113
I'm interested in knowing how the Republican pollsters went from being so horribly wrong in their prognostications to be virtually dead-center correct in the space of 4 years. Since this is your kind of gig - the analysis of polling data that is - I'm curious as to what changed in polling techniques and analysis of the data.

Or am I wrong and it wasn't all that different this time around. I seem to recall that in 2012 there were a significant number of folks, I think you among them, who had a high level of confidence that Obama would not be re-elected, based on polling data (and not on wishful thinking). Am I wrong about that?
 
Nate Silver says the polls this time

were skewed toward Democrats, i.e. had a Democrat bias.

Here's a cut and paste of his article on the matter.

Locally every poll here - Indiana Senate, Indiana House, statewide offices - showed a level of support for Republicans much lower than the outcome. I'll give detail on that in another post.

Here's Silver:








2014 Midterms 9:08 AM Nov 5, 2014
The Polls Were Skewed Toward Democrats

By Nate Silver



For much of this election cycle, Democrats claimed early-voting data was proving the polls wrong. They cited the fact that polls were biased against Democrats in 2012.

The Democrats' complaints may have been more sophisticated-seeming than the "skewed polls" arguments made by Republicans in 2012. But in the end, they were just as wrong. The polls did have a strong bias this year - but it was toward Democrats and not against them.

Based on results as reported through early Wednesday morning - I'll detail our method for calculating this in a moment - the average Senate poll conducted in the final three weeks of this year's campaign overestimated the Democrat's performance by 4 percentage points. The average gubernatorial poll was nearly as bad, overestimating the Democrat's performance by 3.4 points.

silver-feature-pollbias-1.png


The problem with Democrats' claims is that they were one-sided. I don't mean that in the typical false equivalence way. I mean that they were ignoring some important empirical evidence.

This evidence suggests that polling bias has been largely unpredictable from election to election. Beyond the shadow of a doubt, the polling was biased against Democrats in 1998, 2006 and 2012. However, just as certainly, it was biased against Republicans in 1994, 2002 and now 2014. It can be dangerous to apply the "lessons" from one election cycle to the next one.

Our estimates of polling bias are evaluated in 1 released in the final three weeks of the campaign. Bias is calculated as the difference between the polled margin and the actual result. (For instance, a poll that projects the Democrat to win by 7 points when she actually wins by just 3 has a 4-point Democratic bias.) We exclude races in which one of the top two finishers was an independent candidate.

We'll publish 2014 data in full soon2 but here are the preliminary results - first, for this year's Senate polls:

silver-feature-pollbias-table-11.png


As you can see, the polls were biased toward the Democratic candidate in almost all key races. They were fortunate to have "called" the winner correctly in certain cases: Some of the worst misses came in states like Kentucky and Arkansas where the Republican won, but by a considerably larger margin than polls projected. There was also a near-disaster in Virginia. It looks like Democratic incumbent Mark Warner will pull out the race, but the polls had him up by 9 points rather than being headed for a photo finish.3

Meanwhile, the average bias in gubernatorial polls was 3.4 points in favor of Democrats.

silver-feature-pollbias-table-23.png
In the governor's races there were a higher number of missed "calls" - including in Illinois and Kansas and especially in Maryland, where Republican Larry Hogan wound up winning by 9 percentage points despite trailing in every nonpartisan poll released all year.

This type of error is not unprecedented - instead it's rather common. As I mentioned, a similar error occurred in 1994, 1998, 2002, 2006 and 2012. It's been about as likely as not, historically. That the polls had relatively little bias in a number of recent election years - including 2004, 2008 and 2010 - may have lulled some analysts into a false sense of security about the polls.

Interestingly, this year's polls were not especially inaccurate. Between gubernatorial and Senate races, the average poll missed the final result by an average of about 5 percentage points - psephologists to recognize that the error in polls is often correlated. It's correlated both within states (literally every nonpartisan poll called the Maryland governor's race wrong, for example) and amongst them (misses often do come in the same direction in most or all close races across the country).

This is something we've studied a lot in Real Clear Politics polling consensus and tweak their assumptions so as to match it - but sometimes the polling consensus is wrong.

It's equally important for polling analysts to recognize that this bias can just as easily run in either direction. It probably isn't predictable ahead of time.

To the extent polling bias is predictable, it may call for assessing nonpolling factors - the so-called "fundamentals" - along with the polls in each race. One simple factor is the overall partisanship of a state as measured by its past voting history. In the past, Republicans have tended to outperform their polls in red states while Democrats have done so in blue states.

won't be true in 2016, and we'll undoubtedly see some pollsters and polling aggregators draw the wrong lessons from 2014. The polls may be biased again in 2016; we just won't know much about the direction of it until votes have been cast and counted.

CORRECTION (Nov. 5, 2:52 p.m.): An earlier version of this article misstated how much the average gubernatorial poll overestimated the Democrat's performance. It was 3.4 points, not 4 points.
 
this is backfiring on Dems

Traditionally polls from the MSM have intentionally skewed Democrat so the casual observer will jump on the bandwagon (everybody loves a front runner and winner). Rachel Maddow did a bit last nite questioning why the polling was so bad? Apparently she didn't like the cold water splash in the face on election nite.

I don't know why but the last two election cycles the polling has been way off. Ed Gillespie is a classic example. Is it because the polling services aren't doing LIKELY voters?

In any event, nobody likes ugly surprises and I think even Maddow would like more accurate polling.
 
It could be that voters are like

Alison Grimes and just refuse to say who they are/did vote for. :) I know I always hang up on them.

Turnout is the key to every election, and the polls can't show that.
 
Utter nonsense.

If you read the article Ladoga pasted, or a similar article Silver wrote about a week ago, you'd know that polls skew Dem about half the time and GOP about half the time.
 
And, unfortunately

we don't know which way it was until the counting occurs.

There was a hint in the sharp upturn of Republican probabilities for victory in the last two days of charts by aggregators such as 538, NYT, HuffPo and a couple others. But, the upturn stopped with their final postings on November 4. Those graphs turn almost straight up for Rs and straight down correspondingly for Dems, but, being lagging indicators, never fully revealed the extent of the wave.

It surprised us in Indiana races. We had polls showing people losing by a couple of points 7-10 days before the election who won by 10% or more.

After the polls had closed, Indiana House Republicans, holding a 69-31 majority, thought they might have lost 5 seats. They gained 2 (lost 1 incumbent and beat 3 Dems) and now sit at 71-29.
 
I won't dispute the use of the term "wave"

here in the Hoosier state. It was a shellacking.

I had seen some evidence that the polls a week ago were skewed very slightly to the Repubs. The late GOP surge in the models could have been the lagging indication of a surge, as you suggest (and as it apparently was), but it could also have been confirmation that the polls were artificially pushed to the right.

Needless to say, I think the outcome proves that folks like Nate Silver are going to have plenty of work for the next two years, because the polls in a lot of states were way off, even late in the game.

goat
 
There may be many lessons

First the obvious, nothing below is meant to attempt to downplay the drubbing the D's took. The Republicans ripped their arm off and beat them with the wet end. (sorry, that was a popular phrase in an old gaming group).

But if one wanted to suggest something positive it might be turnout. Turnout appears bad. (source). 2014 may be a record low turnout. Now here is what that is good and bad for the Ds.

It is good because they do better with higher turnout. Unless we are entering a period of record low turnout, the Ds will probably do better in 2016. Illinois had about a 4.5% drop in turnout. Who were those people? If Dems, that might explain the governor race compared to polls. Maryland was down 1.2%, same possible issue. I don't know if there are studies on who stayed home but that would be interesting. I know Lagoda was talking about Indiana, Indiana was down 10%. That's pretty significant. I have no idea who that 10% was, or why they stayed home. But I imagine the smart operatives are looking into that.

Now here is why it is bad for dems, and there are two reasons. If dems stayed home out of a lack of enthusiasm, are they going to gain that enthusiasm by 2016? There are no guarantees. 2016 could have a far better turnout than 2014 and still fall short of the big numbers Dems need.

Second, North Carolina and Colorado had higher turnouts than 2010. NC was up .9% and CO up .7%. The D's still lost both. Now maybe, just maybe, all those higher turnout people were R's. But as I recall, the R's won independents in NC. There is a problem for the Ds that isn't just explained by a lack of excitement.

But all these trends are transitory. I recall Pat Caddell saying during Reagan's second term that we wouldn't see another D as president the rest of the century as the move south and west created a hostile electoral college. Eventually we got 8 years of Clinton. Even if Tuesday was a tidal wave, and it pretty much was, I don't know that we have evidence a major realignment took place.

The lesson I see is that it helps to give people a positive reason to vote for you. "We aren't as much of a bum as those guys" isn't a clarion call for action. The exit polling shows both parties with VERY high unfavorables. It seemed to me the Dems kept trying to drive up the GOP negatives, and it may have worked, but they never were able to articulate why one should vote for them and I think that cost them. People either stayed home, or held their nose and voted Republican (the last exit polling I saw showed higher R unfavorables than D).
 
Larry Sabato calls for investigation

into the polls by the polling industry along with some video including R and D pollsters.

Every poll I know of, public and private, from US Senate seats down to state legislative races missed the outcomes - some by huge margins. But they all - every single one I know of - missed in the same direction. They underestimated Republican outcomes.

Now Sabato is one of the nationally respected gurus and teaches at the University of Virginia. His concern is to be taken seriously.

I do not believe that the errors are a result of the difficulty of contacting cell phones, or models on race, age or other demographic sectors. The pollsters have that science under control.

So, what is it. I'm not the only person here who knows something about this. Whatcha think?

This post was edited on 11/7 10:41 AM by Ladoga

Sabato et al
 
Why no mention of turnout...

...as in Dems and Pubs skew predictions because they simply stay at home at rates which weren't predicted?
 
I think they are data

gathering on the issue. It is not universal that turnout was unique. In Indiana I know for sure that its was not the same across the board. It was 40% in Wayne County -Richmond - for example and around 10% in the north half of Lake in some places. Wayne had no real races and northern Lake had some - legislative races. I'm waiting for the guru's who can access the data to write about it.
This post was edited on 11/7 12:56 PM by Ladoga
 
Do you think that

maybe a lot of Democrats weren't going to vote for a R so but they didn't like their D so they just stayed home?
 
I think that is some of it

I wouldn't say it was a major factor, but it was a factor. I never saw excitement in the extended Democratic family. The people who live/breathe/eat/sleep Democratic politics were, as always, excited. But those people who don't care until election day, didn't seem to care. Again, the Dems explained why voting for Republican X was bad, but that isn't the same as getting someone in to vote for Democrat Y.

Republicans meanwhile had the reason to vote, to stop Obama. It didn't matter who was the choice, they were pulling any lever they could to block the president. I think this happened in 2006 in reverse when the Dems made some good gains.

Dems didn't want to vote for the R, and didn't see a big need to go vote for the D. Republicans talk often about low-information voters. Both parties have them, and they tend to be people who need a reason to vote. I'm just not sure the Ds offered a reason.
 
ADVERTISEMENT