September 30, 2012

Comparing Polls: Varied results arise from differences in methods

Findings from four surveys differ in part because of live interviews and including cellphone users.

By John Richardson jrichardson@pressherald.com
Staff Writer

Four polls focusing on Maine's Senate race are conducted within two weeks.

Charles Franklin

Related headlines

ABOUT THE POLLS

HERE IS a quick look at the four recent Maine polls.

CRITICAL INSIGHTS: Sept. 12-16, live interviews with people on landlines and cellphones, 618 likely voters with a 4 percent margin of error.

MAINE PEOPLE’S RESOURCE CENTER: Sept. 15-17, automated phone calls, landlines only, 856 voters in likely voter households with a 3.35 percent margin of error.

PUBLIC POLICY POLLING: Sept. 17-18, automated phone calls, landlines only, 804 likely voters with a 3.5 percent margin of error.

RASMUSSEN REPORTS: Sept. 25, automated phone calls on landlines only and some Internet-based surveys, 500 likely voters with a 4.5 percent margin of error.

POLL VARIATIONS

Charles Franklin is the founder of PollsAndVotes.com, a website devoted to nonpartisan analysis of the details of polling and election outcomes, mostly in the U.S. He also is a University of Wisconsin at Madison political science professor and the director of the Marquette University Law School Poll.

Franklin is no stranger to poll variation and controversy.

His polls before the June 5 gubernatorial recall vote in Wisconsin led Democrats and bloggers to accuse him of having Republican ties and favoring Republican Gov. Scott Walker. While most polls had the recall effort much closer, Franklin's final poll showed that Walker was winning the recall election by 7 percentage points. Walker survived by 6.8 points.

Last week, Franklin said, he was under fire from Republicans because of his latest presidential poll results.

Three have Angus King at 43 to 44 percent. One has him at 50.

Charlie Summers' support ranges from 28 percent to 35 percent.

King's lead is as big as 22 percentage points and as small as 8.

So, what's wrong with this picture?

Actually, nothing, according to Charles Franklin, founder of PollsAndVotes.com and a nationally known polling analyst.

The Portland Press Herald asked Franklin to analyze three independent statewide polls conducted in mid-September. One was a poll by Critical Insights commissioned by the Press Herald. A fourth poll released late last week by Rasmussen Reports was not part of Franklin's review.

We asked Franklin if something was wrong with our poll, or the others, and why there seemed to be so much variation.

"I've spent the last eight years actively comparing polls to each other," Franklin said after looking at the methods, questions and results of each poll. "Based on that experience, it doesn't surprise or bother me when I see this level of variation, and it doesn't mean that any one of these polls is particularly out of line with the others."

Franklin said the Maine polls all appeared to be valid, and the variation was generally within or very near the margins of error. He also identified basic differences between the polls that can be expected to lead to variations in the results.

Because all polls are estimates, pollsters typically say they are 95 percent confident in the numbers, give or take a few percentage points.

In the Maine polls, the margins of error were 3 percent to 4.5 percent. That means that the 43 percent to 50 percent range for King is at the combined margin of error in the two polls. The same is true for Summers' range -- 28 percent to 35 percent.

Franklin also said the variation is to be expected given differences in the polls themselves.

The Critical Insights poll, the one that has the largest lead for King, was the only poll to conduct live interviews, include cellphone users, in its sample and ask undecided voters if they are leaning toward any particular candidate. The other polls used automated phone calls to landlines only -- automated polls are not allowed to call cellphones -- and did not separate leaners from undecided voters.

Those so-called "robopolls" are far less expensive and can produce basic horse-race numbers much faster than live interview polls such as the Critical Insights survey, which asked deeper questions to allow the newspaper to explore voter sentiment and political trends.

Different pollsters can make valid cases for their methodologies, Franklin said, but it's no surprise that they get variation in the numbers.

For example, if leaners are removed from the Critical Insights poll and counted as undecided the way they are in other polls, King's support drops to 47 percent and his lead shrinks from 22 points to 19 points.

The impact of other differences is harder to measure.

"Clearly, your cellphone sample is going to pull in more young voters," Franklin said. "I believe that adding cellphones probably helps King a little bit."

The polls show King has especially strong support among the youngest Maine voters, so it makes sense that the poll including cellphone users would have higher numbers for King. The cellphone bump for King was probably not more than a percent or two, he said.

Nationally, cellphone users make up as much as 30 percent of political poll samples and can have a big impact on results. However, Critical Insights uses cellphones for only 10 to 12 percent of its polling sample, because that is the estimated percentage of Mainers who are now cellphone-only users. The state's spotty cell coverage is likely the biggest reason for lower rates of cell use here.

(Continued on page 2)

Were you interviewed for this story? If so, please fill out our accuracy form

Send question/comment to the editors




Further Discussion

Here at PressHerald.com we value our readers and are committed to growing our community by encouraging you to add to the discussion. To ensure conscientious dialogue we have implemented a strict no-bullying policy. To participate, you must follow our Terms of Use.

Questions about the article? Add them below and we’ll try to answer them or do a follow-up post as soon as we can. Technical problems? Email them to us with an exact description of the problem. Make sure to include:
  • Type of computer or mobile device your are using
  • Exact operating system and browser you are viewing the site on (TIP: You can easily determine your operating system here.)