SRLC Straw Poll Numbers Don’t Add Up. Try Again, Wilson Research.
This afternoon, two and a half days after the last votes were cast in a paid-registrant-only straw poll of the Southern Republican Leadership Conference, Wilson Research Strategies publicly released their full tabulation of the straw poll results.
To get right to the point, Wilson Research Strategies’ full tabulation of the SRLC straw poll does not add up.
The following are the first-pass SRLC straw poll results released to and reported by the news media by Wilson Research Strategies shortly after the close of straw poll voting:
(If the primary election for president were held today, for whom would you vote?)
Newt Gingrich 18% (321)
Mike Huckabee 4% (80)
Gary Johnson 1% (3)
Sarah Palin 18% (330)
Ron Paul 24% (438)
Tim Pawlenty 3% (54)
Mike Pence 3% (58)
Mitt Romney 24% (439)
Rick Santorum 2% (41)
(Who would be your second choice in the Republican Primary Election for president?)
Newt Gingrich 20% (339)
Mike Huckabee 11% (178)
Gary Johnson 6% (104)
Sarah Palin 20% (332)
Ron Paul 6% (98)
Tim Pawlenty 7% (114)
Mike Pence 8% (141)
Mitt Romney 14% (242)
Rick Santorum 7% (125)
The full tabulation of SRLC results released this afternoon at once provides more detail and less detail. According to this tabulation, 1806 attendees of the SRLC completed the straw poll.
Wilson Research’s accounting of the first question looks like this:
You may notice three small differences. First, absolute numbers of responses for each category are not included; there are only percentages. While this is pesky, it’s not a problem since the initial release of results included those absolute numbers. Second, the percentage who responded either with “Don’t Know” (DK) or with a refusal to respond is included in this version of results, while it was not in the initial results. If there were 1806 respondents, then the 42 respondents not accounted for in the initial results would make up about 2%, rounded off — the DK/Refused category. Third, the “1%” share of Gary Johnson reported in the first report is properly changed to “<1%”; if Gary Johnson got three votes, then that’s only 0.17% of the 1806 votes, and that value doesn’t round up to 1%.
So far, these discrepancies are small potatoes and not consequential. But let’s continue on to
Wilson Research’s accounting of the second question:
The percentages sum to 101%, but this is not necessarily a flaw, because the percentages are rounded and when there’s more rounding up than rounding down it’s possible to legitimately get an extra percentage point in a sum. Again, in this version there is no count of the absolute number of votes; we can be glad that there was such a count provided in the first reporting of results. According to that first count (which includes the same percentage distribution as in the second reporting), there were only 1673 responses provided for the question about a second-tier presidential preference. The leaves 133 unclassified responses, 7.4% of the total number of respondents to the straw poll. In the full report today, just 2% of respondents are listed in the “DK[don’t know]/ Refused” category.
So where are the other 5%? There are a few possibilities I can think of:
1. The absolute count in the first report is wrong, but the percentage distribution in the second report is correct. In this case, a corrected count should be posted. Regardless, the count should have been included in the second report, along with an explanation of how 5% of responses lost in the first count were found in the second count.
2. The absolute count in the first report is correct, but the percentage of “DK/Refused” respondents in second report (which was missing altogether from the first report) is incorrect. This would be a reporting error which should be fixed.
3. 5% of respondents indicated a second choice for president who was not listed as an option on the official ballot. If this possibility is true, the percentage distribution in the second report is incorrect and misleading, inflating the responses for each candidate on the straw poll ballot from what the actual percentage distribution of responses would be in such a case:
Newt Gingrich: 18.8% (339)
Mike Huckabee: 9.9% (178)
Gary Johnson: 5.8% (104)
Sarah Palin: 18.4% (332)
Ron Paul: 5.4% (98)
Tim Pawlenty: 6.3% (114)
Mike Pence: 7.8% (141)
Mitt Romney: 13.4% (242)
Rick Santorum: 7.0% (125)
Other: approx 5% (??)
DK/Refused: approx. 2% (??)
The incorrect single-variable reporting of responses to the second straw poll question either leads to or is compounded by errors in the two-variable crosstabulation that follows. This is an untitled and unlabeled table:
The reference row, “% of Total,” reminds us of the (incorrect) percentage of respondents who picked a certain candidate as a second choice to be the Republican nominee.
This would be, if correctly realized, a really interesting table, since it would indicate the closeness of pairs of Republican candidates in the minds of the attendees of the Southern Republican Leadership Conference. That set is not representative of the Republican electorate, consisting of activists paying or on paid sponsorship to attend, overwhelminglycoming from a small set of states of the Deep South. But still, the results would have been interpretable to some extent. These results are not interpretable to any extent because even on its own terms the table is certainly incorrect.
What do the cells in the table mean? They are reported as percentages, but percentages of what? The unlabeled, untitled table does not say. The percentages can not refer to the rows, since the rows add up to varying amounts significantly more or less than 100%. The percentages must refer to the columns, since each column of percentages (not counting the reference row at the top) sums to 99%, 100%, or 101%.
But what do the columns mean, and what do the rows mean in contrast? There are two possibilities:
Possibility 1: columns represent respondents’ answer to question 2 (“Who would be your second choice in the Republican Primary election for President?”) and rows represent respondents’ answer to question 1 (“If the primary election for president were held today, for whom would you vote?”).
This possibility is the most intuitive one: after all, the total % of the second-choice vote each candidate got is placed at the top of each column. If this is the correct interpretation, then the percentages in each row below should be interpreted as the percentage of people who picked candidate X as a 2nd choice who picked candidate Y as a 1st choice. For instance, apparently 25% of those who picked Mike Huckabee as a 2nd choice picked Newt Gingrich as a 1st choice. Apparently 1% of those who picked Sarah Palin as a 2nd choice picked Gary Johnson as a 1st choice. Apparently 2% of those who picked Rick Santorum as a 2nd choice picked Gary Johnson as a 1st choice. Apparently 26% of those who picked Ron Paul as a 2nd choice picked Gary Johnson as a 1st choice. And apparently 33% of those who picked Gary Johnson as a 2nd choice also picked him as a 1st choice…
… that can’t be. Remember that according to Wilson Research Strategies, only three people voted for Gary Johnson as a first choice. Unless someone voted for two second choices (which would bust the straw poll methodology) you just can’t distribute three first-choice Gary Johnson voters across four cells.
So possibility #2 must be correct:
Possibility 2: rows represent respondents’ answer to question 2 (“Who would be your second choice in the Republican Primary election for President?”) and columns represent respondents’ answer to question 1 (“If the primary election for president were held today, for whom would you vote?”).
This is counterintuitive, since at the top of each column the percent a candidate got as a second-choice, not a first-choice, is listed. But as we’ve demonstrated, possibility #1 can’t be true. So possibility #2 it is.
But even if possibility #2 provides the correct interpretation of this mystery table, the problem of the mystery 7.4% returns. Remember from above that 7.4% of respondents are not accounted for in Wilson Research Strategies’ count of the second-choice vote, and that only 2% of that 7.4% is accounted for by the addition of a “DK/Refused” in the full tabulation provided this afternoon, a tabulation missing the vote counts. If approximately 5% of respondents to the second-choice question simply weren’t counted in the one-variable tally, then they also weren’t counted in this two-variable crosstabulation. If those 5% of respondents chose someone not listed here as a second-choice Republican, then the column cells shouldn’t add up to 100%, but should add up to some total averaging about 95% from column to column.
I’d love to be able to talk to you about what the responses to the SRLC straw poll mean. I really would. But missing respondents and missing labels and odd roundings add up to something I just can’t interpret with any confidence. Until corrections come along, I’m not sure you can do anything meaningful at all with the straw poll report other than tossing it in the waste paper basket.