Evangelicals Behaving Badly with Statistics

American evangelicals, who profess to be committed to Truth, are among the worst abusers of simple descriptive statistics, which claim to represent the truth about reality, of any group I have ever seen. At stake in this misuse are evangelicals' own integrity, credibility with outsiders, and effectiveness in the world. It is an issue worth making a fuss over. And so I write.

By simple descriptive statistics I mean elementary ways of quantifying differences in the world, such as percentages and averages describing distributions and trends in populations. Now, I am not among those who believe that the ability to quantify is the true test of authoritative knowledge. I am not even a quantoid-geek type of sociologist. But I do think that statistics can often usefully represent what is going on in reality. They can help to get at what seems to be true about what is actual. So statistics are well worth doing well and getting right.

Of course statistics are generally well known for their misuse. Anyone can easily twist, misrepresent, and lie with statistics. And it takes a bit of basic knowledge for even well meaning people to avoid common statistical pitfalls. But none of that exempts evangelicals from having to use statistics responsibly. The problem is, they often do not.

Why do evangelicals recurrently abuse statistics? My observation is that they are usually trying desperately to attract attention and raise people's concern in order to mobilize resources and action for some cause. In a world awash in information and burdened by myriad problems, some evangelicals may justify the problematic misuse of statistics to get people to pay attention to what they think are good causes. But this is inexcusable. Such desperation, alarmism, and sloppiness reflect the worst, not the best, in evangelicalism.

Let me here give one specific illustration of the larger problem I am talking about, although examples could be multiplied many times. In a recent issue of American evangelicalism's flagship magazine, I found a glossy, four-page, centerfold advertisement for a national leadership summit, announcing this:

Wake Up Call
"Christianity in America Won't Survive Another Decade."
Unless We Do Something Now

Anyone who gives this claim a moment's thought should realize immediately that it is preposterous. Unless we act now, U.S. Christianity will be dead in ten years? Please. I hope somebody will let God know. Still, the quotation suggests that some authority has determined this, so let us explore further. Inside the ad we are greeted by "A Call to Arms," and are informed that, "This generation of teens is the largest in history—and current trends show that only 4 percent will be evangelical believers by the time they become adults. Compare this with 34 percent of adults today who are evangelicals. We are on the verge of a catastrophe." This is really incredible. But still, specific statistics are staring up from the page: 4 percent compared to 34 percent. Plus, the ad shows a lineup of evangelical heavyweights endorsing this summit, including Chuck Colson, Ted Haggard (before his fall), Jerry Falwell, Pat Robertson, Jack Hayford, and many more. Not only was the president of the National Association of Evangelicals (Haggard) backing this, but a recent official nae resolution on youth ministry is also reprinted in the ad, suggesting a general nae endorsement of this program.

It turns out not to be possible from the ad to determine who is quoted saying that "Christianity in America Won't Survive," but the statistic that only 4 percent of today's teens will grow up to be evangelicals does have a reference. The fine print shows that it comes from a 1997 youth ministry book about "the bridger generation" written by a professor at a significant evangelical seminary. Having read nearly every book on youth ministry published in the last 15 years, I realize that I own the very book. I take it off my shelf and find the source of the 4 percent statistic. It turns out, as reported on pages 165-66, that this professor had done an "informal survey" of 211 young people in three states over seven months in the mid-1990s. No information about the methodology in sampling these youth is given—most likely it was a convenience sample. Based on this evidence, the author writes, "The vast majority of bridgers today have not accepted Christ. In a recent and informal survey of 211 bridgers, only 4 percent responded that they were born-again Christians who had trusted in Christ alone for salvation." Older Americans are then said, based on another similar "informal survey," to have accepted Christ at higher rates. The conclusion: "According to present trends, we are about to lose eternally the second largest generation in America's history."

Let us ponder what is wrong with this picture. There is nothing wrong per se with an "informal survey" of a convenience sample of a few hundred people—we can often learn a lot of value from such exercises. However, it is impossible to make accurate inferences about a huge generation of youth at a national level from only 211 of even the best of samples, much less from an "informal," non-probability, most likely convenience sample.

Furthermore, even if the numbers were somehow nationally representative, they still tell us absolutely nothing about "present trends." Trends have to do with changes observed over time. The numbers presented in the book have to do with differences in generations measured at one point in time. Without knowing whether the observed cross-sectional differences reflect a cohort effect (what the author assumes) or a life course effect (a more plausible alternative), we are no more able to project the claim that we are on the verge of losing a generation eternally than we can predict that all babies today are going to grow up to wear diapers as adults because we see that they are wearing diapers now.

And even if the differences represent a cohort (not life course) effect that will remain stable as this young generation ages, it still makes no sense to think that "we are about to lose" an entire generation. About to? Like, real soon? In fact, it would take many decades for the projection to play itself out, even if all of the assumptions behind it were correct, which they are not. The claim to imminent demise is silly on elementary logical grounds. Thus, the book's author is guilty of making some unwarranted inferences about national representation and future events based on limited data and faulty logic. This is not a crime. But it has problematic consequences.

Namely, based on such erroneous conclusions, a national movement is now being organized to re-educate 20,000 youth pastors in 44 cities around the nation. Tickets for the three-hour event are $39 each, $99 for a church's entire pastoral staff. Pastors who attend this "high level briefing" in which "top voices" will "present the hard facts," the ad states, will be doing their part to avert the catastrophe of dwindling church attendance, increase in peer pressure, a growth in porn and violence on tv, and the decay of "our Christian nation." By implication, those who don't attend are enabling the catastrophe. How do we know that this national catastrophe is imminent, that this "crisis" "demands our response," and that "this generation is depending on our attendance"? Again, because a seminary ministry professor conducted an "informal survey" nine years ago with 211 young people in three states, selected by methods about which we know nothing, asked questions we know almost nothing about, and then made the logical error of drawing an unwarranted inference from a small and non-representative sample about trends and future faith conditions of entire generations.

Now, anyone familiar with my own work knows that I am deeply concerned about the lives and faith of American teenagers—I agree there is cause for concern and that many churches need to do a much better job working with youth than they currently do. But I find the misuse of statistics described above appalling. If this were an isolated incident, it might be excusable. But, having been a watcher of evangelicalism for many years, I know that this is not an aberrant case. Evangelical leaders and organizations routinely use descriptive statistics in sloppy, unwarranted, misrepresenting, and sometimes absolutely preposterous ways, usually to get attention and sound alarms, at least some of which are false alarms. The widespread influence of much-cited evangelical pollsters, who do not actually come entirely clean on their methods, does not help matters either. It seems that one of two situations pertains. Either statistically reckless evangelicals are somewhat aware that they are playing fast and loose with numbers. Or they are not, they simply do not know better. Either is unacceptable. In the first case, we are talking about intellectual dishonesty and the distortion of what is true, in the name of promoting truth. In the second, we are dealing with elementary ineptitude, gullibility, and irresponsibility conducted in full public view. This is what orthodox Christianity—assuming that is what evangelicalism is—has to offer the culture?

The point is not that all evangelical leaders need an M.A. in statistics, or that all evangelical organizations need to hire statisticians. Indeed, let me be entirely clear here. I am not finally pushing for the authority of experts. The last thing the world needs is more worship in the cult of expertise. I am actually arguing for the opposite: for ordinary evangelical leaders, pastors, and organizational staff-people to better exercise their God-given minds on pretty basic matters of percentages, averages, trends, and logical inferences so as to not say indefensible and embarrassing things in public.

It's not that hard. People simply need to ask themselves things like: Is it really plausible that Christianity will be dead one decade from now because today's young people appear to be less religious? Of course not. Anyone who could think that is clearly so gullible, so ill-informed about what reality is and how it works that they have no business offering, for example, "high level briefings" involving "top voices" about "what must be done to reverse the 4% trend" that doesn't exist. It's an embarrassment, a disgrace. It reflects the lowest of standards of operation and the feeblest of thinking. Non-evangelicals paying any attention to this have every right to ridicule and dismiss such ill-informed nonsense. And evangelical programs that miscalculate reality in such ways—however well meaning and enthusiastic they are—surely undermine their own long-term credibility and effectiveness.

But I suspect that to squarely address this question, the self-criticism and raising of standards needs to go deeper. The real question is not whether evangelicals can clean up their statistical act. The deeper question is whether American evangelicals can learn to live without the alarmism that is so comfortably familiar to them. Evangelicals, by my observation, thrive on fear of impending catastrophe, accelerating decay, apocalyptic crises that demand immediate action (and maybe money). All of that can be energizing and mobilizing. The problem is, it also often distorts, misrepresents, or falsifies what actually happens to be true about reality. And to sacrifice what is actually true for the sake of immediate attention and action is plain wrong. It should be redefined as a very un-evangelical thing to do.

Christian Smith is William R. Kenan, Jr. Professor of Sociology at the University of Notre Dame, author of Soul Searching: The Religious and Spiritual Lives of American Teenagers (Oxford Univ. Press), and principal investigator of the National Study of Youth and Religion, www.youthandreligion.org