Houston mayoral poll doesn't tell us much of anything

At the end of last week, Rice University released the results of a polling it conducted on the Mayor's race for NPR's KUHF and TV station KHOU.

The poll, which interviewed 424 registered voters (margin of error of +/-4.76 percent) showed the following results:

48%: Undecided:
34%: Annise Parker
13%: Ben Hall:
5 %: other candidates

Based on these results, KHOU and Rice’s PR office both proclaimed respectively that a run-off election is likely, as Parker wasn’t close to the 50 percent mark necessary to gain a majority and win outright in the general election.

Here’s Rice’s subhead:

Despite lead, findings suggest a runoff election 

Here’s KHOU’s lede and follow-up grafs:

Annise Parker seems headed for a runoff in her campaign to keep her job, but she commands more than twice as many supporters as her leading challenger in a newly released poll commissioned by KHOU 11 News and KUHF – Houston Public Radio.

Still, just six weeks before Election Day, roughly half of all surveyed voters either didn’t know or wouldn’t say how they’re going to vote.

Parker leads the pack of candidates at 34 percent, with former city attorney Ben Hall at 14 percent. About 48 percent of voters are classified as undecided, indicating the incumbent mayor will have to fight to keep the post to which she was narrowly re-elected two years ago.

Ahem. (Patrick toggles controller to engage political scientist “beast mode”)

*HULK SMASH BAD HORSE-RACE REPORTING ON POLLING AAARGHHH*.

(Patrick jams large pins in Politico reporter Voodoo doll on desk, returns to human form. Takes deep breath, feels better)

Sorry about that. Rice’s PR office and KHOU way overplay these results. The poll itself has several very serious limitations that their reporting ignored.

Complaint 1: The poll is of registered voters

The sample size is OK, and I’ll assume the methodology to draw the sample is fine (Bob Stein knows what he’s doing and has a decent history of polling local issues within the constraints given him by his clients). But the poll talks to registered voters about a local race, where turnout is usually under 200,000 voters in a municipality with well more than a million registered voters (see the always-insightful Charles Kuffner on this point).  And we know from studying politics that those undecided voters generally aren’t thoughtful moderates agonizing over the issues. Instead, they often don’t have a clue as to what is going on and will probably sit the election out unless they can be reached by a good mobilizing effort.

So let’s assume for a second that none of the 48 percent “undecided” will vote and recalculate the numbers We roughly get:

65% Parker (34/52)
27% Hall (14/52)
8% Everyone else (4/52)

That would put Parker safely in the Mayor’s chair without a run-off (even with the larger margin of error created with the smaller sample size, see explanation below).  Of course, this still would leave us with a turnout rate of 52 percent of registered voters, which is far too high for even the most hotly contested local election. If we assume that the drop-off rate of supporters is roughly equal among candidates, then Parker still wins in a breeze. I certainly don’t endorse that assumption – turn-out operations are key in mobilizing voters and I don’t know enough judge the various campaigns’ abilities. (Though Hall hasn’t impressed me much in the little that I have seen.)

What the poll needed was a screening question or two that asked residents if they voted in the last municipal election, which would have sorted out likely voters from registered voters. It wouldn’t be perfect, but it I think it would have improved the results quite a bit. (I suspect that cost scuttled this idea, because it would have entailed another 1,000 phone calls or more.)

Complaint 2: The emphasis on subgroup reporting with small samples.

My second complaint about the coverage is its discussion of the candidates’ support. See Rice’s release:

Parker’s share of the vote by race and ethnicity is broad, according to the poll. She garnered 38 percent of the Anglo vote, 42 percent of the Hispanic vote and 24 percent of African-American vote. Hall, an African-American, has support from 29 percent of African-American voters. Parker garnered 27 percent of Anglo Republican voters’ support, compared with only 11 percent for Ben Hall.

Remember that margin of error for the total poll being 4.76 percent? Well, that’s based on the sample size of 423 respondents. When we’re talking about support among different racial/ethnic groups, we’re looking at the Anglo subgroup, the African American subgroup and the Latino subgroup. Each of these groups is probably between 100 and 200 people (except for Anglo Republicans, which is even smaller). Guess what? That means the margin of error goes up accordingly, roughly 7 percentage points for a sample size of 200 and 10 percentage points for a sample size of 100, which makes these subgroup analyses practically useless.^1,2

As usual, Kuffner is all over some of these points. He's kinder than I am to the poll's methodology, but his speculation is generally intelligent, appropriately cautious, and well-leavened with historical context.

The point is to be VERY WARY of extrapolating from a single poll, especially a small one conducted on a local race that doesn’t have a good likely voter screen.

1.  This assumes a confidence level of 95 percent
2. To pile on here, the Rice release also incorrectly states that Parker won narrowly against former City Attorney Gene Locke with 51 percent of the vote in 2011. She beat him in a runoff in 2009 with 52 percent of the vote. She avoided a runoff with 51 percent in 2011's general election against 5 relatively unknown candidates.