3 Ways to Recognize Bad Stats
Often times, a statistic is like a piece of candy thrown at a parade—you really don't know if you should bite into it or not. We've all heard Mark Twain's famous quote about lies and statistics. There is a reason so many people have had skepticism toward stats. Too frequently, people repeat inaccurate, bad, or explicitly made-up numbers.
Still, I keep hearing statistics quoted at conferences and through blogs and social media that make me scratch my head in amazement. I'm not sure where some of these stats originate, and I'm the president of LifeWay Research.
So how can you really discern good stats from bad?
1. Be Wary of Statistics in Promotions
First, be skeptical of stats in promotional pieces. They are trying to convince you to consume an ideology or a product. Advertisers do not hold to the same standards as a researcher.
Statistics are effective at moving people toward action, but it can border on manipulation, depending on how you use or change the numbers and information. Using unverified statistics can be an example. Even if the action is good, it is not acceptable to use a statistic that is not valid.
For example, maybe you've heard of books that say the church is dying—and this book can fix it (Yet the numbers show that the church is not dying). Or, that 80% of church plants fail—and this conference can stop that (yet, about two-thirds of church plants are around in four years after starting).
You've probably heard someone say the stats show that pastors are miserable and just want to quit—and our ministry saves them (when the vast majority are happy and consider it a privilege to serve). Or that the Super Bowl is the biggest sex trafficking event in history—and you should support our ministry to stop it (when there is no law enforcement evidence that this is true).
All of these things are important: we need the church to thrive, church plants to be successful, pastors to be happy, and sex trafficking to end. But we don't need made up facts to evidence the situation and the need.
2. Be Wary of Stats that Cannot be Verified
Secondly, is there really a way to know the fact claimed? In other words, you should always ask "How do you know that?" when someone quotes a statistic.
For example, we have heard that 94% of evangelical youth drop out of church after high school never to return. The three words "never to return" should raise red flags. How could anyone know that?
Realistically, how could they keep track of those 94%? Did a researcher track them their whole lives or implant them with transponders and see if they come back?
We have also heard the statistic that 80% of pastors don't make it to retirement. How would someone know that? Some of this is just common sense.
If you can't think of a way to verify a statistic, it is probably not true. Unless someone spent millions of dollars on research conducted over decades, some of these stats could not be verified.
If it is not verifiable, you should not be quoting it.
3. Be Wary of Stats that do not Line up with Reality
Thirdly, be skeptical about statistics that do not line up with your reality. For example, one statistic seems to indicate that Christians and non-Christians divorce at the same rate. That hasn't been my experience. The truth is that those who practice their faith have a much lower rate of divorce.
I wrote a post called That Divorce Stat You Quoted Was Probably Wrong due to the confusion over this issue. The oft-quoted statistic comes from a Barna study. However, if you read what Barna reported, the research found those that say they're Christians divorce at the same rate as those that say they're not Christians. But the reality is that a much larger number of people claim to be Christians than really are, particularly when you look at beliefs and actions-- and that has to be factored in before we start saying, "Christians do this" (unless you really believe 75% of Americans are Christians).
So, stop getting up before your church and leading them to believe that they have a 50/50 chance of divorce. That just demoralizes them. Sociologist Bradley Wright, author of Christians are Hate-filled Hypocrites… and Other Lies You've Been Told, indicated that 60% of people who identify as Christians but rarely attend church have been divorced. But of those who attend church regularly, only 38% have been divorced. That is a huge difference.
Not to mention you can use your own common sense. I recently read the 50% of pastors will get divorced. Because this stat is all over the place, I'm going to write a full-length post on it in the future.
But just use some common sense for the moment and ask the question. Do we REALLY believe that half of all pastors get divorced? Does that REALLY make any sense to you? If the 50% divorce rate is a myth for all Americans (and it is), do we REALLY think that would be the case for pastors?
Of course not.
You could be wrong, but if something is way out there, pause, then ask, is the evidence stronger than my common sense? If it is, then change your opinion. If it's not, then require more clarity.
Be Skeptical when Needed and Always Discerning
Simply put, we need to be more skeptical when it comes to statistics. If a researcher cannot tell you their research methodology and the questions they asked, you should immediately disregard the statistic. Researchers have an ethical code that we supposedly follow—if they don't, stop listening to them, and never quote them.
Occasionally, researchers will come up with different conclusions to the same concept. It depends on the question posed. That's not the same thing.
I recently read some comments from David Kinnaman of Barna about Millennials and evangelism. He says that while most born-again Christians say they should share their faith, few do. At Lifeway Research, we've found the same thing.
However, David's report found that Millennials who are born again are the exception, and share their faith at a higher rate. Our data shows something different, something I'll discuss with David soon. My guess is that different samples and wording made the difference between the samples—but you should be able to tell by looking at the methodology!
But, in both cases, there is a reasonable methodology that led us to these different conclusions. As a reader, you should look at the methodology of study in order to know where the researcher is coming from, and to know the context for the statistics. You just need to be discerning.
You Don't Need a Degree in Stats
To discern if research is of good quality, it may help to understand some of the process. It's not that complicated and it will keep you out of trouble.
In any study, a researcher needs to ask, "Does this question ask what it needs to ask?" Every question is imperfect. There is only so much money available to research a topic therefore the number of questions is limited.
So a researcher is only going to be able to ask a limited number of questions (sometimes only one or two) about a topic. From that research, we're not going to be able to draw every conclusion—although some try to extrapolate it to ridiculous conclusions. But, every well-framed question tells us something, as long as it is done correctly.
So, How Do We Know?
What are some ways to tell if the stat you've heard is trustworthy? You can tell the quality of the research if it is:
- done on a good sample (a random set with a large enough sample, not a group at a conference, a website where people click to vote, etc)
- a good question (is it fair and not leading), and
- a good methodology (is it transparent, honest, and credible).
You can guard against believing and spreading bad statistics by determining if its source is a promotional piece, if it cannot possibly be verified scientifically, and if it does not line up with reality. Then, if it is quality research, quote away.
Anything else is just spreading misinformation. We can and should do better than that.