Intercollegiate Studies Institute Civics Test Flunks Out
“Americans Don’t Know Civics,” flatly declares the USA Today headline. Agence France Presse and Washington Post columnist Kathleen Parker dutifully follow suit, passing on the press release of the “Intercollegiate Studies Institute.” Americans are expected to recoil, recoil! at the following fact points:
- More than 2/3 of those surveyed earned a failing grade on a 33-point “civic literacy” or “civic knowledge” quiz.
- “Americans across all economic, educational and political/social backgrounds are equally lacking.”
- “US elected officials scored abysmally.”
These news organizations do a disservice by failing to fully examine the nature of, methodology of and conclusions drawn from this ISI quiz. Let’s dig a bit deeper here.
An Economic Ideology Quiz
The newspaper articles reporting on the ISI quiz focus on its questions having to do with American history and national politics. That leaves out the significant portion of the quiz having to do with economic ideology, an ideology that the Intercollegiate Studies Institute (an advocacy organization not sponsored by a college or any consortium of colleges) may feel is central to American life but which is not grounded in the founding documents of this nation. 27% of the questions in the ISI quiz have centrally to do with economics, and a further 12% significantly touch on economics. Nearly 40% of the questions in this quiz test a respondent’s ability to answer the sort of economic questions the ISI has interest in and in the manner the ISI approves of.
The ISI poses a question about what it terms “threatening” behavior by Franklin Delano Roosevelt, another about business profits and two questions about the benefits of “free markets” or “free enterprise.” The ISI does not pose a question about the WPA, unemployment insurance, the origin of overtime pay, or the record of industrial workplace safety in the 1800s. The latter questions not asked by the ISI have much more to do with the experience of industrial and service workers in American economic systems. The former questions, the ones actually asked by the ISI, are concerned with economic theories of relevance to business executives. There are many more industrial and service workers in America than there are business executives. Why does the ISI ask questions relevant to business executives and not relevant to workers? What does that pattern have to do with the accomplishment of answers the ISI considers correct? These are questions worth pondering.
Especially when it comes to economics, the ISI has implemented a quiz that doesn’t test fact but rather ISI’s favored economic theory as a “bedrock conservative organization”, the members and leaders of which helped implement supply-side economics and deregulation in the Reagan administration and beyond. For example, the very premise of the ISI quiz question, “Free markets typically secure more economic prosperity than government’s centralized planning because:” is up for serious debate. The mortgage securities market screwed up when it was turned into a free market from a regulated one. The “correct” answer that ISI supplies — “the price system utilizes more local knowledge of means and ends” — may therefore not actually be correct. Indeed, that answer is a pretty good description of either corruption or of the kind of risk-taking by financiers that placed our entire economic system in jeopardy.
In another example of ISI’s ideologically-based “political correctness” test, some instrumentalists in political theory would argue that the correct answer to the ISI quiz question “Free enterprise or capitalism exists insofar as:” is actually what ISI calls an “incorrect” answer, “government implements policies that favor businesses over consumers,” asserting that capitalism cannot exist without state institutions to buffer and stabilize it. Others would dispute ISI’s “correct” answer to this question, “individual citizens create, exchange, and control goods and resources,” by noting the rise of corporations, not individuals, as significant loci of control and exchange in modern capitalist systems.
In a final example from the economic ideology section of the ISI quiz, “Which of the following fiscal policy combinations would a government most likely follow to stimulate economic activity when the economy is in a severe recession” is a matter for debate, with the answer “decreasing taxes and increasing spending” possibly wrong. During the fall campaign of 2008, Barack Obama won the presidency in part on the basis of promising to raise taxes again on the rich while spending more on human needs. As the ISI-allied magazine Human Events notes, FDR raised taxes and increased spending during the Great Depression. I am at a loss as to why the answer “increasing both taxes and spending” is deemed incorrect by ISI, since at least in these two instances (and most likely in more instances if we were to look at history in more detail) the plan was for government to follow that policy combination… unless by the phrase “would a government most likely follow” ISI means to say “should a government most likely follow in our opinion.”
Inaccuracy and Inconsistency in Other Quiz Answers
The lack of consistency and attention to factual detail in ISI’s quiz is not limited to the subject matter of economics. When ISI inserts culture war questions regarding abortion and religion into its quiz, it succeeds more in revealing its own polemical positions than in assessing quiztakers’ command of facts. ISI asks the question “Which of the following statements is true about abortion?” and codes the response “the National Organization for Women has lobbied for legal restrictions on it” as incorrect. Yet NOW is currently lobbying in favor of the Freedom of Choice Act (FOCA), which would impose legal restrictions on late-term abortions. It is therefore true that “the National Organization for Women has lobbied for legal restrictions on it”; the organization just hasn’t lobbied for as many legal restrictions as the questioners at ISI would prefer.
In another question more taking sides in a culture war than addressing matters of fact in law, ISI asks respondents to complete the phrase “The Bill of Rights explicitly prohibits” by selecting “establishing an official religion for the United States” (deemed correct) or “prayer in public school” (deemed incorrect) among other responses. Yet if ISI means by “explicitly” the literal mention of the phrase “official religion,” then the explicit prohibition of “establishing an official religion” is not present in the Bill of Rights. The Bill of Rights phrase “establishment of religion” has been interpreted by the Supreme Court to prohibit the establishment of an official government religion, making the answer true if “explicitly” refers to the Supreme Court interpretation of explicit text. By that standard, however, the answer “prayer in public school” is also correct, since the Supreme Court has interpreted the Bill of Rights’ text regarding “establishment of religion” to refer to proselytization in government schools and the phrase “free exercise thereof” to restrict government-established religious ceremonies such as prayer in schools. The ISI has been either sloppily or maliciously inaccurate in its quiz construction.
In a final example of an incorrectly-scored answer, the ISI quiz asks respondents to answer the question, “What is one power of the federal government?” The answer “Make treaties” is clearly true and scored as such. The answer “Maintain prisons” is scored as incorrect. Again, this is a reflection less of reality and more of the ideological wishes of the authors of the ISI quiz. Clearly, the federal government does have the power to maintain prisons, because it does so. The Supreme Court has made a number of rulings discussing in great detail the scope of federal power to administer prisons. As a matter of historical fact, the ISI quiz is again inaccurate.
It is most certainly not true, as USA Today maintains in its article, that this ISI “study expands the focus and concludes Americans across all economic, educational and political/social backgrounds are equally lacking.” ISI’s own results show that individuals with doctoral degrees score on average 37 percentage points higher on the ISI quiz (as ISI scores it) than individuals who have not attained a high school diploma. Those with a high school diploma score better than those without. Those with an undergraduate diploma do still better, and those with a master’s degree do still better. This is exactly the pattern we should expect to find if the American educational system imparts civic knowledge. The policy implication is not that we should tear down the current educational system, but rather that we should strengthen it.
Given the factual problems explored above regarding ISI’s scoring of its own quiz, it is not surprising that individuals with a doctoral degree should end up with a score of less than 100%. This may be due to informed disagreement. Quizzed PhDs may have a stronger command of historical fact than the authors of the ISI.
Methodological Questions on “Elected Officials”
The Intercollegiate Studies Institute tells a story of serendipity leading to its conclusions regarding America’s elected leaders:
The ISI civic Literacy survey was not designed to test the civic knowledge of elected officials, but it did discover evidence of an interesting pattern that may merit further exploration.
The ISI claims that from a sample generated by dialing random phone numbers, 164 respondents out of a total of 2,508 said “they have been elected to a government office.” The ISI then goes on to share its “finding” that “Elected Officials Score Lower than the General Public.”
According to the U.S. State Department:
There are over 500,000 elected officials in the United States. Of these, fewer than 8,500 are at the national and state level. The rest are local government officials—city council members, school board members, mayors, sheriffs, and an array of other individuals who serve in various capacities.
The only federally elected officials, of course, are the 100 Senators, 435 Representatives, 1 President and 1 Vice President of the United States. That makes for 537 federal elected officials.
The U.S. population this year is about 305,000,000. The probability that any one person in the United States is a federal elected official is 537/305,000,000, or 0.000176%. The probability that any one person in the United States is a federal or state elected official is about 8,500/305,000,000, or 0.00279%.
By chance, assuming the ISI is correct when it asserts that:
The telephone survey data can be taken to represent a probability sample of all individuals who reside in households with residential telephone service in the United States.
then the ISI should by chance have 0 federal elected officials and 0 state elected officials in its sample. If the ISI repeated its study a thousand times, it could be expected to quiz at least one federal elected official in five of those studies, and to quiz at least one state elected official in seventy of those studies. Clearly, we’re not talking about federal or state elected officials here in the conclusions. We’re talking about America’s county sheriffs, city clerks and municipal dogcatchers.
Remember, according to the U.S. State Department “there are over 500,000 elected officials in the United States.” Let’s be generous and say that by “over 500,000” the State Department is talking about a number that is nearly a million. Let’s say it’s 950,000 people who are elected officials in the United States, just for the sake of argument. 950,000/305,000,000 is about 0.31% of the American population. From a really random sample of 2,508, we should expect about 8 people to turn up who are elected officials in the United States.
Let’s be a bit more expansive and include everybody who has ever been an elected official in the United States. Let’s say there are 10 million of those people. That’s still just 3% of the American population, leading to an expected 75 such people turning up in ISI’s sample. But ISI reports 164 such people. How did ISI just happen to get so many elected officials? It’s unlikely, even if you include the idea of a larger number of the ever-elected. An explanation is in order, especially considering that ISI’s sample wasn’t actually representative overall; it had to weight its sample a fair bit to make it look representative.
But let’s move on and just ignore all that. Let’s take the ISI at its word. What is the margin of error going to be for a group of 164 people within a sample of 2,508? Well, here’s what the ISI says:
The margin of error for the sample of 2,508 adults is +/- 2.0 percentage points at the 95% level of confidence. When reporting on subgroups of adults (e.g., men, women, college graduates, etc.), the sampling error is higher.
That’s right, folks: the margin of error gets higher when you report on subgroups like men or women, who constitute 48.8% and 50.2% of the ISI sample respectively. It’s going to get quite a bit higher when you get down to a subgroup like ISI’s “elected officials,” who constitute just 6.5% of the overall sample. Assuming that the survey was well implemented (not a good assumption), the margin error for a subgroup of size 164 is about +/- 7.6 percentage points. If you get a difference between “elected officials” and everyone else that is smaller than 7.6 percentage points, then it is unreliable and should not be reported.
Of the 33 questions:
On 10 questions did “elected officials” on average do worse than everyone else by greater than 7.6 percentage points. (On only 5 of these questions did “elected officials” on average do worse than everyone else by greater than 8.6 percentage points.)
On 24 questions, the responses for “elected officials” were within the margin of error when compared to everyone else. (On 5 of these questions, the sampled “elected officials” actually did better than everyone else.)
If I were in the PR department for the Intercollegiate Studies Institute, I wouldn’t have confidence in that conclusion about “elected officials.” If that conclusion ended up on a press release (as it did), I would be keeping my head down and looking for another job.
The mainstream news articles you may have read last week about the study on civic knowledge were shallow and uncritical in their regurgitation of what the ISI wants you to conclude. But the ISI quiz used to generate those conclusions is riddled with inconsistency and outright error. It was used to advance an ideological agenda. Its findings about “elected officials” are dubious at best and should be questioned for multiple reasons. And you won’t hear that on the evening news.
P.S. If you really want to, you can take the quiz here. And in case you are wondering whether this article is a “sour grapes” reaction to a low score on my part, the answer is no. Figuring out pretty quickly what the ISI was looking for, I scored a 93%.