Rachel Maddow says Fox 'changed' poll eligibility criteria on day it announced top 10 for GOP debate

There’s no love lost between MSNBC’s Rachel Maddow and Fox News. The liberal show host frequently chastises the conservative Fox News, and the network’s process of picking the top 10 candidates for the first Republican presidential debate gave Maddow another opportunity.

Maddow aimed her barb at the explanation Fox News gave for deciding who would be the last candidate to make the cut-off. Fox News gave Ohio Gov. John Kasich the coveted 10th slot at the expense of former Texas Gov. Rick Perry.

Maddow accused Fox News of unilaterally sounding the death knell for the lesser-ranked candidates seven months before the first Iowa voter would show up at the nation’s first caucus.

"If a candidate can't even make it to the debate stage, why would rational donors and volunteers continue giving money and time to what is apparently a lost cause," Maddow said on her Aug. 4, 2015, show.

A bit later, Maddow said, "Fox changed around their poll eligibility criteria on the day they announced who is in and out of their debate."

In particular, Maddow chastised Fox for not using an NBC/Wall Street Journal poll that was the fifth-most recent poll on record. Fox News skipped that poll and used a slightly older poll from Quinnipiac. Maddow said this shift cost Rick Perry a shot at the nomination.

"He didn't lose it, it was stolen from him,"  she said. "Not because of the vagaries of some poll but because of some cable news executives moving things around to effectively stack things against him."

Maddow compared what Fox News had said before and what it said after it made its picks. In this fact-check, we focus just on the question of whether Fox changed the rules of the game for the polls it would use.

Fox News became more nuanced with the polls

In May, when Fox News first announced the debates, it laid down several selection criteria for the candidates. Among them was this: "Must place in the top 10 of an average of the five most recent national polls, as recognized by FOX News leading up to August 4th at 5 PM/ET. Such polling must be conducted by major, nationally recognized organizations that use standard methodological techniques."

Fox News directed us to a Bloomberg article that added just one detail from Michael Clemente, executive vice president of news for Fox. Clemente said online and partisan polls were unacceptable.

But on Aug. 4, when Fox News unveiled who would make it to the main stage (a forum for the runners-up takes place earlier at 5 p.m.), it gave a far more detailed explanation of its selection rules than what it laid out in May:

• "They are the five most recent national polls from non-partisan, nationally-recognized organizations using standard methodological techniques. Recency was determined by actual interview dates (not release date)."  

• "They used live interviewers, random digit-dial sampling techniques and included both landlines and cellphones. (No online or automated (IVR) polls were used.)"

• "Their GOP primary vote question mirrored the ballot by reading all candidate names in random order and without honorifics."

• "Results among registered voters were used rather than national adult results when available. Furthermore, only results among either 1) self-identified Republican primary voters or 2) Republican and Republican-leaning voters were used."

• "The rank order was determined by a simple arithmetic average of publicly available results. Averages were rounded to the nearest tenth of a percentage point."

• "Each poll has a different margin of error, and averaging requires a distinct test of statistical significance. Given the over 2,400 interviews contained within the five polls, from a purely statistical perspective it is at least 90% likely that the tenth place Kasich is ahead the eleventh place Perry."

Featured Fact-check

Fox News Channel

stated on January 26, 2016 in a news release

Donald Trump’s decision to skip the Fox News/Google debate in Des Moines "is near unprecedented."

• "We did not include the highly-respected NBC/WSJ poll, which is the fifth most recent poll, only because it did not meet our criterion that the poll read the names of each Republican candidate in the vote question. We would note, however, even though their ballot question included Perry but did not name Kasich, the unaided "Kasich" response tied the aided "Perry" response. In short, their results are consistent with the results of the other polls in our review, and consistent with the resulting placement in the Fox News debate."

In May, Fox News took 45 words to describe how it would use polls to pick the candidates. In August, it used more than 260. Without a doubt, Fox moved from a broad to a detailed approach. We turned to political scientists to help us assess whether that shift is the same as Maddow’s claim that "Fox changed around their poll eligibility criteria."

A change of uncertain significance

The bottom line from the analysts we reached is that the original poll criteria were destined to be insufficient to meet the goal of giving Fox News a clear ranking of the top 10. During her show, Maddow interviewed Larry Sabato from the University of Virginia Center for Politics. We asked Sabato to assess her statement.

"We were never sure which polls could be included and which could not, beyond the requirements about live calls/interviewers and the like," Sabato told PunditFact. "You can call it ‘changing the rules’ if you want, but you could also say they reserved to themselves some flexibility, not knowing precisely what the polling community would serve up."

David P. Redlawsk, a political scientist at Rutgers University, said Fox could have done more to avoid this situation.

"They added criteria after they saw what the polls were," Redlawsk said. "To me this is different. Fox should have been more specific from the beginning."

By comparison, in May, CNN named the polls it would use for the debates it plans for September and provided a much more detailed description of how it would use those polling results.

That said, Fox News’ decision to use polls based on when they called voters and how they posed their questions passed muster with Robert Erikson at Columbia University.

"If Kasich was not on the NBC/WSJ ballot, the Fox decision makes sense," Erikson said. "Release data is less important than date in the field. It was my impression that even with NBC/WSJ included, Perry would lose out, only by an even more trivial amount."

Michael Traugott, a professor of political science at the University of Michigan, has taken a careful look at how news organizations use polls. He wrote a critique of CNN’s "poll of polls" after the 2008 election.

"The most important point is that the system is arbitrary, used to satisfy production values for television rather than being based on any substantive or political consideration," he told us. "The network is trying to use a statistical justification to cover a news judgment decision."

Traugott said that taking the average of candidate standing in polls is a bad idea on a number of grounds. Mainly, differences in how polling organizations chose their respondents and phrase their questions have a much greater impact than the statistical margin of error.

Traugott said that in large measure, Fox set itself up for this problem.

"If you are vague initially, when the time comes, you have to specify the details by which you acted," he said.

We contacted both MSNBC and Fox News. MSNBC said Maddow’s statement was correct based on Fox News adding the criteria that the GOP primary vote question mirror "the ballot by reading all candidate names in random order and without honorifics." Fox News declared that this made no difference.

Our ruling

Maddow said that "Fox changed around their poll eligibility criteria on the day they announced who is in and out of their debate." We found that the poll criteria were much more detailed in August (over 260 words long) than they were in May (about 45 words long).

The network could have been more precise early on, but Fox did leave some room for itself when it qualified the polls it planned to use for the debate back in May with "as recognized by Fox News."

Also, some experts said, Maddow exaggerated the impact of the approach Fox News ultimately used, and she implied that statistical certainty was achievable under any circumstances. 

Fox News moved from a broad approach to a more detailed one, and experts aren't convinced that it slighted Perry. With those missing caveats, we rate Maddow's claim Half True.