September 30, 2012

Comparing Polls: Varied results arise from differences in methods

Findings from four surveys differ in part because of live interviews and including cellphone users.

By John Richardson jrichardson@pressherald.com
Staff Writer

Four polls focusing on Maine's Senate race are conducted within two weeks.

Charles Franklin

Related headlines

ABOUT THE POLLS

HERE IS a quick look at the four recent Maine polls.

CRITICAL INSIGHTS: Sept. 12-16, live interviews with people on landlines and cellphones, 618 likely voters with a 4 percent margin of error.

MAINE PEOPLE’S RESOURCE CENTER: Sept. 15-17, automated phone calls, landlines only, 856 voters in likely voter households with a 3.35 percent margin of error.

PUBLIC POLICY POLLING: Sept. 17-18, automated phone calls, landlines only, 804 likely voters with a 3.5 percent margin of error.

RASMUSSEN REPORTS: Sept. 25, automated phone calls on landlines only and some Internet-based surveys, 500 likely voters with a 4.5 percent margin of error.

POLL VARIATIONS

Charles Franklin is the founder of PollsAndVotes.com, a website devoted to nonpartisan analysis of the details of polling and election outcomes, mostly in the U.S. He also is a University of Wisconsin at Madison political science professor and the director of the Marquette University Law School Poll.

Franklin is no stranger to poll variation and controversy.

His polls before the June 5 gubernatorial recall vote in Wisconsin led Democrats and bloggers to accuse him of having Republican ties and favoring Republican Gov. Scott Walker. While most polls had the recall effort much closer, Franklin's final poll showed that Walker was winning the recall election by 7 percentage points. Walker survived by 6.8 points.

Last week, Franklin said, he was under fire from Republicans because of his latest presidential poll results.

Three have Angus King at 43 to 44 percent. One has him at 50.

Charlie Summers' support ranges from 28 percent to 35 percent.

King's lead is as big as 22 percentage points and as small as 8.

So, what's wrong with this picture?

Actually, nothing, according to Charles Franklin, founder of PollsAndVotes.com and a nationally known polling analyst.

The Portland Press Herald asked Franklin to analyze three independent statewide polls conducted in mid-September. One was a poll by Critical Insights commissioned by the Press Herald. A fourth poll released late last week by Rasmussen Reports was not part of Franklin's review.

We asked Franklin if something was wrong with our poll, or the others, and why there seemed to be so much variation.

"I've spent the last eight years actively comparing polls to each other," Franklin said after looking at the methods, questions and results of each poll. "Based on that experience, it doesn't surprise or bother me when I see this level of variation, and it doesn't mean that any one of these polls is particularly out of line with the others."

Franklin said the Maine polls all appeared to be valid, and the variation was generally within or very near the margins of error. He also identified basic differences between the polls that can be expected to lead to variations in the results.

Because all polls are estimates, pollsters typically say they are 95 percent confident in the numbers, give or take a few percentage points.

In the Maine polls, the margins of error were 3 percent to 4.5 percent. That means that the 43 percent to 50 percent range for King is at the combined margin of error in the two polls. The same is true for Summers' range -- 28 percent to 35 percent.

Franklin also said the variation is to be expected given differences in the polls themselves.

The Critical Insights poll, the one that has the largest lead for King, was the only poll to conduct live interviews, include cellphone users, in its sample and ask undecided voters if they are leaning toward any particular candidate. The other polls used automated phone calls to landlines only -- automated polls are not allowed to call cellphones -- and did not separate leaners from undecided voters.

Those so-called "robopolls" are far less expensive and can produce basic horse-race numbers much faster than live interview polls such as the Critical Insights survey, which asked deeper questions to allow the newspaper to explore voter sentiment and political trends.

Different pollsters can make valid cases for their methodologies, Franklin said, but it's no surprise that they get variation in the numbers.

For example, if leaners are removed from the Critical Insights poll and counted as undecided the way they are in other polls, King's support drops to 47 percent and his lead shrinks from 22 points to 19 points.

The impact of other differences is harder to measure.

"Clearly, your cellphone sample is going to pull in more young voters," Franklin said. "I believe that adding cellphones probably helps King a little bit."

The polls show King has especially strong support among the youngest Maine voters, so it makes sense that the poll including cellphone users would have higher numbers for King. The cellphone bump for King was probably not more than a percent or two, he said.

Nationally, cellphone users make up as much as 30 percent of political poll samples and can have a big impact on results. However, Critical Insights uses cellphones for only 10 to 12 percent of its polling sample, because that is the estimated percentage of Mainers who are now cellphone-only users. The state's spotty cell coverage is likely the biggest reason for lower rates of cell use here.

Cellphones also may have contributed to stronger support for the same-sex marriage referendum in the Critical Insights poll. Young voters, the ones most likely to have cellphones and no landlines, are strongly behind the legalization effort.

The use of live interviewers instead of automated voice recordings might also have contributed to more same-sex marriage support in the Critical Insights poll.

Pollsters have clearly found that voters are more likely to say they support gay marriage than to actually vote that way in the privacy of a voting booth. It may be that voters opposed to same-sex marriage are more willing to say so with computerized calls than live interviewers, Franklin said.

"I think that there could be some effect here." But, he said, "the bottom line for me is that you have three polls here that are all pointing in the same direction."

One other difference is worth considering, he said.

Critical Insights is the only one of the three polls Franklin reviewed that mathematically adjusted the results to match the party affiliations of Maine's voting population.

While pollsters all tend to "weight" their samples by age and gender, they are split on whether to statistically adjust by party affiliation.

The polls by Public Policy Polling and Maine People's Resource Center did not weight by party, although the surveys included a higher percentage of Democrats and a lower percentage of independent voters than the state population in general.

Both methods are valid and neither is perfect, Franklin said. "Different pollsters might make good principled arguments both ways."

Party affiliations may have had some effect on the polls, but it doesn't appear to be a large one.

Critical Insights' unweighted data shows its adjustment for party affiliation slightly increased King's lead but had almost no effect on the same-sex marriage question.

Meanwhile, the Rasmussen Reports poll released Thursday also was weighted by party affiliation, and its results were closely in line with the polls by Maine People's Resource Center and Public Policy Polling.

In the end, Franklin said, all four polls are estimates, and they're telling the same story.

"We certainly shouldn't expect strong agreement on all of these numbers," he said, "but my sense is they line up pretty good."

Publishing poll results two weeks after a survey is conducted -- as the Press Herald did today -- is unusual for horse-race polls and potentially risky because events could change the race dynamics in the meantime, Franklin said. It also opens a poll to more criticism than usual from campaigns that don't like the numbers, he said.

But such a delay in publishing results is not unethical if disclosed, and it is less unusual in the case of value-added polls used to explore issues and trends, Franklin said. 

Staff Writer John Richardson can be contacted at 791-6324 or at:

jrichardson@pressherald.com

Were you interviewed for this story? If so, please fill out our accuracy form

Send question/comment to the editors




Further Discussion

Here at PressHerald.com we value our readers and are committed to growing our community by encouraging you to add to the discussion. To ensure conscientious dialogue we have implemented a strict no-bullying policy. To participate, you must follow our Terms of Use.

Questions about the article? Add them below and we’ll try to answer them or do a follow-up post as soon as we can. Technical problems? Email them to us with an exact description of the problem. Make sure to include:
  • Type of computer or mobile device your are using
  • Exact operating system and browser you are viewing the site on (TIP: You can easily determine your operating system here.)