December 4, 2008Leadership, Research

A Response to David Allen's Critique of NAMB / LifeWay's Calvinism Research

Over the years, we have learned a few things about research in SBC life. Research tends to get people in our denomination excited. Many people quote it, most like it, and some despise it. People will quote and misquote statistics regularly to prove or disprove whatever matters to them. Simply put, we are an enthusiastic, passionate, and often imprecise people when it comes to church research.

As such, when we do research, it gets a lot of debate and discussion. We do not mind at all--and sometimes we read a comment and think, "Good point!" or "We should have thought of that." As such, we very infrequently respond to inappropriate uses or criticisms of our research.

However, I do see a pattern developing. It appears that when one of the faculty members at one of our seminaries disagrees with the results of our research, they write a rebuttal or a criticism. We actually don't mind a (good) rebuttal and questioning the wording of questions is normal and expected. However, it does seem that some of the faculty at Southwestern are making a habit of taking time away from their important tasks to critique our research. I thought it would be wise for me to take this opportunity to respond in what I hope is a gracious way.

Actually, our team debated whether to respond (and have not in the past). Since few people have read or commented on the critique, it is regrettable to bring attention to it. However, since these critiques seem to be a recurring (and public) pattern, it seems best to address it publicly.

On to the specifics...

Dr. David L. Allen of Southwestern Baptist Theological Seminary recently released a white paper entitled, "Calvinism: A Review" at In his white paper, Allen reviews the book, Calvinism: A Southern Baptist Dialogue of which I contributed the chapter, "Calvinism, Evangelism, and SBC Leadership." In my chapter, I present results from two recently collected data sources, the North American Mission Board's 2007 New Minister's Study and LifeWay Research's 2006 Calvinism Study.

In his review of the book, Allen presents a critical review of the data analysis done by the North American Mission Board and LifeWay Research. He raises a number of "weaknesses" surrounding the study. Actually, such comments are fine--no study is perfect and this study representing the NAMB and LifeWay surveys is far from flawless due to several factors I will list below. There are weaknesses to this study as in every study. And, we don't mind a good debate on the issues at hand.

However, this is the best, current study on the topic of Calvinism trends in the Southern Baptist Convention. Moreover, the data analysis is sound and accurate. And, in light of the pattern of such critiques coming from the same sister entity, my colleagues and I decided to respond.

The following response to Allen's review will take each of Dr. Allen's points one by one, providing further evidence and clarity for each issue raised by Allen.

First, let me say that I believe Dr. Allen's concerns were not offered in malice. They are occasionally helpful and (I believe) delivered with good intentions, but they don't alter the findings. In essence, Allen's points are more distracting than clarifying to the topic at hand.

First, Allen points to the data collection of the surveys. He states: "There is no statement whether any pastors who were in the first study might have been also in the second study, thus making direct comparison problematic." There are really two issues addressed in this sentence: the possibility of double counting or double sampling of the same individuals, and secondly the mistaken impression that the NAMB and LifeWay data sources are meant to be comparative.

It is certainly possible that some individuals were surveyed twice between the two surveys; however, it is unlikely. Although the LifeWay survey did not ask for the graduation year from an SBC seminary, we can make an estimate as to the probability of selecting a seminary graduate from 1998-2004 within LifeWay's pastor survey.

If we assume that of the estimate within the New Minister's Survey that 25% of newly graduated seminary students are now serving as senior or lead pastors of SBC congregations (527 lead SBC pastors out of 2,134 respondents), this would result in approximately 1,875 (25% X 7500 seminary graduates in 1998-2004) potential pastors of the 43,699 churches included in the sampling frame of the random sample conducted by LifeWay who would be eligible for duplicate invitations to both surveys. However, in reality, only 527 lead or senior pastors of SBC churches actually responded to the New Minister Survey, leaving about a 1% chance (527 pastors / 43699 churches) that a pastor surveyed in NAMB's New Minister's Survey also responded to LifeWay's survey. This is a level of duplication with which we are quite comfortable. We do not believe that any potential duplication would greatly alter the results.

Richie Stanley was part of the team that surveyed the recent graduates and he is now Team Leader at NAMB's Research Department. Richie Stanley has been a denominational researcher for 20 years. He has also worked as a statistician for the federal government, taught college statistics, and served as a minister of education. He holds a master of science in statistics (from the University of Kentucky) and a master of divinity with religious education (from SEBTS). Richie explained:

The idea that two surveys are better than one seems so logical to me. As you mentioned, two sources that point in the same direction are complementary even if not tied at the hip methodologically. And I think the small number of pastors who may have responded to both surveys would have no statistical impact. The reason I would try to avoid overlap in samples is to avoid annoying the invited respondents, more than concern about invalidating the results.

The second issue related to Allen's first point is the premise of data comparison. Allen alludes to this point in more detail:

Again, drawing conclusions from responses from two possibly different populations is problematic. Third, Stetzer concluded Calvinism is on the rise based on the percentages from the two groups that identified themselves as five-point Calvinists. Although this may be warranted from a common sense standpoint, direct comparisons from one group to another cannot be statistically interpreted in such a fashion.

If Allen considers my study to be a traditional comparative study (which is a technical term that we do not use at any point), Allen is correct that the methodology is problematic. We do a general comparison, but with the clear caveat that the samples cannot be compared as their samples and methodologies are different.

In other words, we make it clear that this is not a technical comparative study. We run separate analysis on each data set, demonstrating a common directionality to the results. We do not believe that you can make a direct comparison and we stated such in the chapter. Instead, I made a general comment about comparing the two studies and add, "However, it is difficult to make a direct comparison between the two studies since the 2007 NAMB New Minister's Study includes any seminary graduate between 1998 and 2004 who is serving at any level of church staff leadership. The LifeWay Calvinism study only looked at SBC senior or lead pastors." Here, I am stating the obvious difficulty in comparing the two studies since each have a different set of population parameters and methodologies.

Instead, this analysis seeks to identify common results in two different data sources, each with their own distinct sampling strategies. As in any scientific endeavor, repeated results denoting a common direction to results only strengthens theory. If our research had included only the LifeWay study (my current employer), the criticisms would be that only one study cannot detect any trends. However, in including the NAMB study (my former employer), the similarity in results in both studies only reinforces the results. Using multiple yet distinctly different data sources in demonstrating similar results is a common strategy in social science research and I believe the results are both more striking and responsible when we analyze both data sources.

Allen goes on to question the sample size for early birth cohorts included in the New Minister's Survey. Essentially, his argument is that since this is a sample of recent seminary graduates, older cohorts would not have adequate numbers to have representative estimates for trends in Calvinistic beliefs over time. At first glance, Allen makes a good point. In many reports of popular level research we do not include the population size for each cohort. That was the case in the New Minister's study. For Dr. Allen's and your reference, the sample sizes for each cohort by each study are listed below.

As expected the number of respondents for each cohort in the new minister's study are higher in later cohorts; however, the number of respondents for earlier cohorts in the LifeWay study are not much higher than the New Minister's Study. In our analysis, we did not present the results for the 1976 or later cohort in the LifeWay study as we believed the sample size (n=18) to be too small. It was our best judgment to include the earliest cohort (1945 or before) in the New Minister's Study (n=21). In statistical terms, any cell size below 30 can be problematic, using 20 as a cutoff was a judgment call. Nonetheless, even with the removal of this earliest cohort in the New Minister's Study, the trend of a higher proportion of 5-point Calvinists for each younger cohort persists.

Again from Richie Stanley:

The age cohorts are what they are. The fact from the NAMB seminary graduate study is that seminary grads became more open to Calvinism from 1998 to 2004. Our goal was to invite all the graduates to respond, without controlling for age. The age cohorts we ended up with were the result of who chose to respond. Again two surveys, two methodologies, complementary results.

Lastly, Allen speculates that we did not perform any statistical tests on our data. Dr. Allen writes, "Given the dissimilarity of the size of the groups, and since no actual statistical measurement of a statistical nature such as chi-square is used, one simply cannot draw and report statistical-sounding conclusions as Stetzer does here."

First, I would first like to make a sidebar comment as to how we conduct our statistical analysis at LifeWay Research. We are always cognizant of the fact that descriptive statistics (i.e. means, percentages, etc.) can be misrepresentative of actual characteristics within the population under study. Therefore, when comparing averages across groups, we are careful to run statistical tests behind the scenes in order to verify our findings. However, we also use restraint when presenting statistical results to lay audiences.

Although appropriate in some contexts, statistical language is more often than not a stumbling block for the majority of our readers. It is for this reason that we often do not publish detailed methodology and statistical tests and when we do we place them at the end of a chapter. Most of our readers are pastors interested in the results, not the statistical or methodological details. Thus, we often put the details at the end of the chapter. And, we must confess that it seems a bit odd to criticize the placement of the methodology in the chapter.

However, in response to Allen's remarks, I am happy to provide t-tests for the difference in means between Calvinists and non-Calvinists led congregations for annual baptism rates and weekly worship attendance.

Unfortunately, Allen's suggestion for "chi-squared tests" is, to use Allen's terminology, "statistical-sounding language," but makes little sense as chi-squared tests are typically reserved for categorical not continuous or numeric variables like we have for worship attendance and baptism rates. To remind you of my overall conclusions: 1) worship attendance is generally lower in Calvinist congregations than non-Calvinist led congregations, and 2) baptism rates are not that different between Calvinist and non-Calvinist congregations. The two tables below (worship attendance and baptism rates) list the survey, the means for each group (Calvinist vs. non-Calvinist), the t-test statistic, and the corresponding probability that the two means are statistically similar in the true population.

In terms of worship attendance, the difference in means for the LifeWay survey is not statistically significant (probability exceeds the conventional 0.05 alpha level); however, it is on the border of statistical significance in the NAMB survey. This was a difficult call to make given the high probability for the LifeWay survey, yet we decided there was at least partial evidence to present a difference in church size according to Calvinist leadership, especially given the common direction to the result.

As for the mean annual baptism rate, the difference in means for both surveys is not statistically significant. This means that little difference in baptism rates exist between Calvinist and Non-Calvinist led churches. As I stated in my report: "The difference between Calvinists and non-Calvinists within these surveys is small, always less than 1 percent. From this we can conclude that Calvinist-led and non-Calvinist-led churches evidence similar rates through their annual baptisms."

The data is not perfect, but it is the best we have at this point. Phillip Connor was formerly at NAMB where he did the data analysis on the NAMB study. He is now a Fellow at the Princeton Center for the Study of Religion. I asked him to review my response and he put it simply, "All data has its limitations. However, I believe this is good analysis that works within these limitations and presents sound and reliable results to the questions under review."

It is my hope that this response to Allen's review will be received in the spirit it is offered. Dr. Allen's concern about Calvinism is clearly known--one can just Google "David Allen and Calvinism" and it is quite clear. As stated in my introduction to this response, no study is perfect. However, this study is the best we have and the statistical analysis is sound.

Let me close with one exhortation. I recently spoke at the Washington Post at the invitation of the Religion Newswriters Association. They asked me to speak on how to discern religious research--identifying the good from the bad. One of my points was that all researchers have a bias--this researcher included. However, ethical researchers report what they find even when such findings disagree with our presuppositions.

To be honest, I am among those who are concerned about the lack of evangelistic passion, practical training, and leadership ability I see in some Calvinists graduating from seminary. Simply put, these are not the results I expected. However, in the world of research we report what the numbers are, not what we think they should be. We present the numbers and, yes, we can give our evaluation and opinion of the numbers. However, they are what they are. I can't change them to prove my point or make someone else happy. Facts are our friends, even when they show a different conclusion than we expected.

In closing, let me make the same exhortation I made at the "Building Bridges" conference where our conclusions were presented. Our results demonstrate that neither the Calvinist camp nor the non-Calvinist camp can claim superiority in baptism rates. And worse yet, an average annual baptism rate of 8-9 persons per 100 attendees is nothing to be proud about, regardless if you are a Calvinist or not. I think we can all agree that the world is in desperate need of the redemption accomplished by Christ on the cross. It is in need of the Savior. So let us be about the business of preaching Christ crucified, calling men, women and children to repent and believe in the gospel.

I will leave it at that. And, my hope is that all will read this in the spirit I offer it. Let's join hands and help all kinds of Southern Baptists to do their evangelism more faithfully.

Support our work. Subscribe to CT and get one year free.

More from The Exchange

Christianity Today
A Response to David Allen's Critique of NAMB / ...