Subscribe via RSS Feed

Social Desirability and Response Validity in Current Polling

[ 22 ] September 28, 2012 |

Several days ago in these very pages, discussion ensued regarding the latest conservative attempt to rewrite reality through re-weighting polls to one guy’s liking.  Of course, polling is not an exact science, but it is a science, and the latest wingnut delusion has no grounding in theory or empirical evidence.  Like any science, survey research continuously attempts to improve upon the validity and reliability of its measures and findings.  While I’m not at all concerned about some nefarious (and successful) attempt by the MSM and that paragon of power, the Democratic Party, to turn otherwise professional and reputable polling houses into duplicitous shills.

However, I have been somewhat interested (note, not concerned) if there might be something else going on that causes the polls to over estimate support for Obama.  Social desirability bias is something I’ve published on in the past (direct link to the paper here).  While that article suggests a contextual effect that causes variance in social desirability across countries (regarding accurately reported turnout in survey research), relevant here is what is colloquially known as the Bradley Effect.  It’s possible (though I consider it unlikely in the specific context of the 2012 Presidential election) that this helps explain Obama’s consistent polling advantage in an election where many if not most structural conditions suggest an incumbent defeat.

It’s difficult with the data I have available to examine this hypothesis to any satisfaction, but that’s not going to stop me from trying.

To begin with, we have the current state of the polls.

Obama Romney Advantage
RCP 28 Sep. 48.6 44.6 4.0
538 28 Sep. 52.2 46.5 5.7
538 6 Nov. 51.5 47.4 4.1


RCP’s running average has Obama up 4 points, Nate Silver’s “nowcast” model up 5.7, and his current prediction for election day 4.1 points.

If social desirability is at work here, a poll respondent will state that she or he supports the President because internally, our not entirely sincere respondent is seeking the socially desirable response, and not supporting the black guy might be racist.  However, this is done knowing that they will ultimately support the white guy.  Practically, this would mean that Obama’s support in these polls is inflated.

I’m approaching this from several directions.  First, I’ve averaged the final month of polls for Presidential elections going back to 1976 (an arbitrary cut off) to examine how accurate the polls were in predicting the final outcome between two white men, with 2008 to serve as a benchmark for 2012.  Shift represents how wide of the mark the final polls were, to the benefit or detriment of the incumbent party.

Poll Result Shift
2008 D 7.6 D 7.3 0.3
2004 R 1.5 R 2.4 0.9
2000 R 3.0 D 0.5 3.5
1996 D 11.0 D 8.0 -3.0
1992 D 12.0 D 6.0 -6.0
1988 R 12.0 R 7.0 -5.0
1984 R 18.5 R 18.0 -0.5
1980 R 4.0 R +10.0 -6.0
1976 D 2.0 D 2.0 0.0


Social desirability response bias in an election can take on many forms, not just race.  To wit, the 1992 general election in the United Kingdom is a good case study (one I lectured on here at Plymouth about six or seven years ago, shame I have no clue where those lecture notes now reside) as the polls largely predicted a narrow Labour outright victory or a hung parliament with Labour having the plurality of seats, yet the Conservatives under John Major easily won by 7.5%.  This is called the Shy Tory Factor on this island, which is simply another manifestation of social desirability.  But in 2012, I’m primarily considering race, and comparing 2008 to past elections does not support the hypothesis that this might be a problem for Obama’s numbers in 2012.

I also considered several of the primary elections in January, 2008.  This was the beginning of a primary where Obama was a somewhat unknown junior senator only four years into his Congressional career, going up against the assumed nominee.  For these data, I average all polls from the last week of the campaign in the given state (there were 20 in New Hampshire alone).

Poll Result Shift
NH Obama 35 37 2
NH Clinton 30 39 9
NV Obama 33.25 45 11.75
NV Clinton 37 51 14
SC Obama 41 55 14
SC Clinton 26 27 1


This evidence is more ambiguous than the examination of previous general elections.  Both Clinton and Obama received shifts in their favor, which isn’t surprising considering the undecideds presumably made a decision of some sort once voting.  However, in both New Hampshire and Nevada, the shift was stronger towards Clinton than Obama: a 7 and 2.25 point advantage respectively.  Both are dwarfed by Obama’s advantage in South Carolina.

These are the wrong data to be analysing this with, of course; ideally we’d have individual level data.  While not individual level data, the following figure, by Greenwald and Albertson, offers a more holistic view of the 2008 primaries.

The above shows that among 32 states where data were available, the “Bradley effect” was only evident in three states, yet 12 states demonstrated what has been termed (erroneously, in my opinion) the “reverse Bradley effect”: states where Obama’s support in the primaries was under, not over, estimated (see South Carolina above).  I consider this an erroneous classification because where the theoretical explanation for the Bradley effect hinges on social desirability, the reverse has been hypothesized as a function of systematic sample bias, through either the under-representation of African Americans in polling samples, or the cell-phone effect.  However, some have hypothesized that “black voters might have been reluctant to declare to pollsters their support for Obama”, and the link above does discuss that

After the Super Tuesday elections of February 5, 2008, political science researchers from the University of Washington found trends suggesting the possibility that with regard to Obama, the effect’s presence or absence may be dependent on the percentage of the electorate that is black. The researchers noted that to that point in the election season, opinion polls taken just prior to an election tended to overestimate Obama in states with a black population below eight percent, to track him within the polls’ margins of error in states with a black population between ten and twenty percent, and to underestimate him in states with a black population exceeding twenty-five percent. The first finding suggested the possibility of the Bradley effect, while the last finding suggested the possibility of a “reverse” Bradley effect in which black voters might have been reluctant to declare to pollsters their support for Obama or are under polled.

There are numerous possible explanations for the “reverse” effect, including faulty likely voter models, under sampling of blacks, sampling bias due to cell phones, to name a few.  There might be some sort of contextual effect at work here, but to ascribe it to behavioural motivations (rather than factors exogenous to the individual, such as sampling bias) such as blacks being shy to state their support for Obama fails the face validity test to me.

Ultimately, given the wide array of mediocre data presented here, I am not concerned about social desirability biasing the estimates of support for Obama in any significant, substantive manner.  However, much as I’d like to, I wouldn’t say that the Republican conspiracy is more likely, if only because that is so creatively ludicrous I initially thought it was an Onion piece.

  • Joshua

    Yea, I’m not buying the “Bradley effect” this time around. Fact is, not supporting Obama is pretty damn mainstream (as it is with any sitting President). Unless you go into crazy wingnut territory, there’s really no implication of racism by saying you’re not gonna vote for Obama. And if you’re already knee-deep in birth certificates and Kenyan anti-colonialist history, you probably don’t care about sounding not-racist to the polling guy on the phone.

    • Just Dropping By

      +1 The evidence for the Bradley Effect prior to 2008 was already very weak. It seems incredibly implausible that there would be a statistically significant Bradley Effect for an incumbent.

  • undecideds presumably made a decision of some sort once voting.

    I love that line.

    But it’s not true. Undecideds might have stayed home.

  • The number of defeated elected incumbents after the 19th century is so low, I’m very suspicious of broad claims that “structural” elements support incumbent defeat.

    Aside from being extremely rare, every incumbent defeat in the 20th century looked much different than this election — a Worsening depression, internal party strife, a major third-party challenger.

    • JMP

      Also, every incumbent defeat since 1968 involved a serious primary challenge to the sitting President. The fact that there was none to Obama make his reelection chances look historically good.

  • L2P

    So basically it doesn’t look like there’s much evidence of any Bradley effect for Obama?

    • Craigo

      There isn’t a whole lot of evidence that a Bradley effect exists for anyone, or ever did – even Bradley. If polls overestimate support for a black candidate with a white opponent, people shout “Bradley effect!” If polls overestimate support for a white candidate with a white opponent, people say “Random/methodological error” and move on with their lives.

  • Incontinentia Buttocks

    What are examples of US elections in which there actually was a measurable Bradley Effect? I’ve read that the California gubernatorial election after which the effect was named was not, in fact, one of them. So what’s left? Helms v. Gantt? Anything else?

    • Richard

      I havent seen the later analyses of the Bradley run for the governship but the polls were certainly wrong for that one. The main California poll, the Field Poll, showed Bradley with a lead of over five percent as of a couple days before the election. I remember all the polls showing him with a lead. And even on the night of the election, as the votes showed that he was getting beat pretty bad (10% or so if my memory is right), Mervin Field was still on television saying that it was still too close to call. If it wasn’t the Bradley effect, was there something else that caused the polls to be so wrong in that election?

      • Incontinentia Buttocks

        My understanding was that the discrepancy between the Field Poll’s exit polls and the final result was best explained by the fact that Field ignored absentee voters (who obviously cannot be exit polls) who were overwhelmingly voting for George Deukmejian, his Republican opponent. Field’s exit polling proved to be accurate, but Mervin Field underestimated the size of the absentee vote (especially relative to the low in person turnout) or how overwhelmingly it would favor Deukmejian. Bradley actually won more in-person votes cast on election day that Deukmejian did. (Link)

  • Bill Murray

    why do use the shift rather than the shift per undecided fraction? If there were only 2% undecided in 2008 and 20% in 1992, Obama would have done much better than Clinton despite the much larger shift to Clinton. I suppose if the original data did not include undecideds in determining the percentage some of the effect would go away, but I’m not sure it all would.

    • dave brockington

      Some of the polls did, some didn’t include undecideds. Ultimately, the data were gathered and this post written in the hour I had at available at the office before having to make it home in time to meet my daughter for the weekend, so “it’s a little rough around the edges” would be charitable in writing, design, and analysis.

      • Bill Murray

        expediency is a good choice for a blog post, especially if the choice is blog post vs. time with daughter

  • Hypothesis: Independent and usually-Republican-leaning voters who voted for Obama in 2012 and who switch their vote to the Rs in 2012 are more likely to demonstrate the social-desirability behavior in polling than the population as a whole.

    If true, this would make 2008 a problematic benchmark.

    • Bill Murray

      a) I assume you meant the first 2012 to be 2008

      b) certainly if your hypothesis is true, then your statement is correct, but the question is whether or not your hypothesis is true.

      I don’t know many people in this category and the couple I do know seem are not shy about saying it to me. I suppose they could be outspoken to a friend that defends the President (mainly because they pick conventional wisdom stupidities like the deficit and taking a log time to pass a budget) and not to an unknown pollster but that seems unlikely. OTOH, who know how well the 2-3 people I know generalize to the population.

  • Obama’s consistent polling advantage in an election where many if not most structural conditions suggest an incumbent defeat.

    I thought this was clearly not true. Even naively, Obama is an incumbent which is a hugely beneficial structural condition. But more generally, this Klein piece is illuminating:

    Political scientist James Campbell has an article in the upcoming edition of the journal PS: Political Science and Politics that surveys 13 models that attempt to forecast the presidential election. Larry Sabato got an early peek at the table, and the bottom line is that eight of them foresee Barack Obama’s reelection and five of them predict that Mitt Romney will be our next president.

    I think that prima facie refutes the idea that the structural features are against Obama.

    • Craigo

      There is also the slight problem of econometric models being absolute disasters, on average.

  • baltassoc

    The Bradley effect is interesting and all, but doesn’t this skip over the rather more obvious issue that (at least according to the table in the post) polls have underestimated Republican performance seven out of the last nine times? OTOH, they seem to be getting better.

  • erik

    I could imagine that some percentage of black voters who honestly intend to vote for the white candidate, might have a change of heart at the last minute. That would me my most plausible expaination for a reverse Bradley effect, if there is such an effect.

  • Dave Brockington

    FWIW, none of these data — the quick tables I mashed up, the figure by Greenwald and Albertson, are well suited to test the hypothesis that the Bradley effect exists in these elections, or exists at all. The difference between the NH, NV, and SC numbers could be random noise or attributed to any number of alternative explanations that we can not rule out. There is a strong theoretical explanation for such an effect, and social desirability has certainly been demonstrated in other contexts, but I’m not at all convinced that the Bradley effect empirically exists. What the above data do demonstrate is that while such an effect is theoretically possible, that hypothesis is not supported here.

    • tomstickler

      The difference between the NH, NV and SC numbers may best be explained by the observation that South Carolina is too small to be a Republic, and too large to be an asylum.

      Remember, Gingrich beat Romney handily in the 2012 primary.

  • melior

    You briefly allude to non-cellphone sample bias, but only in the context of “reverse” racial correlation, which I think leads to muddling of effects.

    Here’s Nate Silver has to say:

    Overall, this version of the model [excluding polls that only include land lines] gives Mr. Obama an 83 percent chance of winning the Electoral College, a full 10 percentage points higher than the 73 percent chance that the official FiveThirtyEight forecast gave him as of Monday night.

    It seems most parsimonious to me to read this as excluding cellphones via auto-dialing that’s reverse-biasing Rasmussen, rather than the other way around.