Can an evangelical research firm poll fairly on matters of religion?
Hemant Mehta, the “Friendly Atheist,” references our poll and compares it to a more recent one commissioned by the American Humanist Association (AHA). He said the AHA chose to conduct a new poll because they felt our results were “skewed.”
Actually, originally he indicated that the poll was useless since, of course Americans would say they wanted to keep the pledge. It appears that the AHA felt differently—so differently that they had to commission another poll, specifically to disprove the useless LifeWay Research poll.
The AHA recently released the results of a survey, asking Americans if the words "Under God" should be removed from the Pledge. They found that about a third of Americans say the phrase, "Under God" should be taken out.
By contrast, a 2013 poll from LifeWay Research found that only 8 percent of Americans support such a change.
Why such a big difference? The AHA's researchers says our poll was biased.
The full report on the survey points out that LifeWay Research is a “evangelical research firm.”
"As such, it appears to have an inherent bias in conducting research on religious attitudes," says the report.
Perhaps the good folks at the AHA missed this poll from 2013, which showed that less than 4 in 10 Americans believe that homosexuality is sin, and the number of Americans who think that homosexuality is sinful continues to drop.
That's not the outcome that most evangelicals want to hear. But it's what the survey said, so that's what we reported.
I've got a better explanation for why the LifeWay and AHA polls are different—we asked different questions.
In our research, and in asking the basic question of whether the phrase ‘under God’ should remain in the Pledge of Allegiance or not, LifeWay Research found only 8 percent of Americans believe it should be removed.
For their part, AHA inserted “a brief description of the history of the Pledge of Allegiance, including the information that ‘under God’ was only added as recently as 1954 in response to the Cold War and that some Americans feel that the Pledge should focus on unity rather than religion.”
Their press release explains:
The study, conducted in May of 2014, responded to a 2013 poll by Lifeway Research, which stated that only 8 percent of American adults felt that “under God” should be removed from the Pledge. Unlike the Lifeway Research poll, the survey done by The Seidewitz Group included a brief description of the history of the Pledge of Allegiance, including the information that “under God” was only added as recently as 1954 in response to the Cold War and that some Americans feel that the Pledge should focus on unity rather than religion.
It’s a bit difficult to tell exactly how they asked the question—be sure to read their actual release here.
This added information, not surprisingly, resulted in dramatic changes to the results. Instead of 8 percent, 34 percent of Americans agreed with changing the Pledge.
For his part of the discussion, Mehta, like most of his writings, is friendly. I think we would both agree that when you ask Americans if they want “under God” removed, they overwhelmingly do not.
When you add more context, information, “Cold War” references, and to “focus on unity,” however, the numbers shift. No surprise there. It is actually helpful in research to have more context and additional questions, as long as those questions do not sway the respondent.
In our attempt to find the most accurate results, we asked the simplest question. Of course, the AHA did not see it that way. They wanted a question that did more than just ask a simple question—they wanted more.
In their report, they state:
Two issues bring into question the Lifeway Research poll results. First, Lifeway is a self-described “evangelical research firm.” As such, it appears to have an inherent bias in conducting research on religious attitudes. Second, there is likely to be little public awareness of the history of the phrase “under God,” which was added to pledge in 1954. An understanding of this history could have a significant impact on support for inclusion of “under God” in the Pledge.
From the perspective of the AHA, LifeWay Research’s results are biased because we have a certain perspective on religion and the public has little awareness of the history of the Pledge.
I’m curious as to how AHA escapes their own assertion of bias since they have a specific opinion on religions as well and on what topics they feel an evangelical research firm should poll, but those are different blog posts for another day.
To assert bias and untrustworthiness is a serious charge to make against a research firm. The question is on what grounds does the AHA have to make such assertions.
Was it in the random sampling? Was it the secular polling contractors we use? Was our wording of the question in some way biased? Nope—it’s that we did not include the information that they thought people would need to answer correctly, and that we are not (as people of faith) to be trusted to ask faith-based questions.
When I spoke at the Washington Post building, during the Religion Newswriters Association meeting, I shared that research is done by Catholic, evangelical, and atheist groups—you just need to check their sample, methodology, and wording. If those are good, the results will be good.
That’s something that gets at bias—so let’s take a moment to do that.
That leads us to the questions asked by LifeWay Research and the AHA. While we asked the simple question, they inserted additional information, including the opinion of (I assume) the AHA and more, which admittedly impacted the answers given by the respondents.
I’m not going to make the same assertions about their research as they made about ours, but the wording of a question absolutely matters and will generate different results. That’s not surprising.
If a researcher asked, “Would you be in favor of removing from our Pledge of Allegiance the phrase ‘under God’ which references the Creator mentioned in our founding documents,” the numbers would go the other way from the AHA results. But, that’s not how we work.
Facts Are Our Friends
So far, ironically, the first major news coverage the AHA poll has received is from Glenn Beck’s The Blaze, which reported, in click-bait fashion, “Atheists Added Something to a Question About Removing ‘Under God’ From the Pledge That Yielded a Shocking Result."
I don’t really find the result shocking. In fact, perhaps surprising to the AHA, I find them helpful. Facts are our friends, so all research-based information is helpful.
Thanks to the AHA, we now know that when you tell people “under God” was added in the “Cold War” and point out that some value “unity more than religion,” people give a different answer. LifeWay Research found that when merely asked about its removal, the vast majority of Americans wanted the phrase to remain. Actually, that’s what every polling firm would find it they just asked the simple question.
Both of those things can be true. The more information we have the better. It’s good to know more, but it is also important to know that wording matters—particularly if someone shapes the wording to accomplish their desired results.
For example, asking (if Presidents could have a third term), “Do you plan to vote for President Obama?” will yield one result. Asking, “Do you vote for President Obama, who captured Osama bin Laden?” or ““Do you vote for President Obama, even though Iraq has spiraled into chaos?,” will almost certainly change the numbers.
All actually give helpful information; though we would not present the latter two alone as it they answered the question definitively.
You have to decide which is the best approach, whether questions are biased, and which is/are most trustworthy.