Science and Asking

24 Feb Science and Asking

25 February 2025

By David Allen, Development for Conservation

Before I start in on this week’s topic, I want to ask for your help in gathering data for a future blog post. If you are willing to participate, here’s what I will need from you:

• Isolate the members and donors you have who made first gifts to your organization at some point – any point – during the calendar year 2020.
• Now add up everything those donors gave to your organization – as a group – since then (1/1/2020 through 12/31/2024).
• Send those two numbers to me – the number of donors and the total amount they have given. The email address is David (at) DevelopmentForConservation (dot) com.
• Any kind of narrative you might be willing to share from your experience recruiting members during COVID would be welcomed as well.

If you would like to understand more about what I am doing and how the data you provide will be used, see last week’s post here: How Much is a New Donor Worth?

Thank you! in advance.

***********************

The Nobel Prize (Economics) winner Daniel Kahneman tells an amusing anecdote in his book Thinking Fast and Slow about a study of new diagnoses of kidney cancer in the 3,141 counties of the U.S. “The counties with the lowest incidence of cancer are mostly rural, sparsely populated, and located in the traditionally Republican states in the Midwest, the South, and the West.”

What explains this? Clean living? Low air and water pollution, access to fresh food with no additives?

It makes perfect sense.

But then, where are the counties with the highest incidence of cancer? Also mostly rural, sparsely populated, and located in the traditionally Republican states in the Midwest, the South, and the West. Now probably due to what? Poverty, low access to good medical care, high fat diet, and too much alcohol and tobacco?

It makes perfect sense.

But both can’t be right.

The lesson is not about cancer research. The lesson is about small samples sizes. According to Kahneman, large samples are more precise than small samples, and small samples yield extreme results more often than large samples do.

The problem most of us have in testing our assumptions about what works and what doesn’t for fundraising in “our” small-sample-size service territories is that we can’t get to a sample size large enough to trust the precise results.

Does this imply that we should stop testing? Absolutely not. But neither should we be surprised when our results differ – even by a lot – with those from other small-sample-size organizations.

And it means that we need to pay attention to experiments that take advantage of larger sample sizes.

BTW, one answer to this is to aggregate our results, which is what I am doing with the new member study I started last week.

One such information source is NextAfter. NextAfter is a digital-first company that is actively testing all the time. And they make their research available online. The beauty of being able to do digital research is that the research window is so short. Experiments that might take weeks or even months to complete with mail-based marketing campaigns can take just days to a week using email. That’s also a problem, of course, because not all of the conclusions they draw are equally valid using mail. So keep an open mind and think critically.

One such experiment I believe does cross over is a recent one using ask strings versus a carousel of testimonials designed to evoke emotional responses. The non-profit was CaringBridge, with which you are probably familiar. The sample size was 79,000. Here’s how they summarized the experiment:

We believe that a testimony carousel for users on the homepage will achieve an increase in donor conversion rate because in comparison to the normal slideup, it helps incorporate and visualize the real impact a donor’s gift has on those who use the CaringBridge platform every day. In the control, users were presented with a slide-up that contained three giving options ($25, $50, and $100). In the treatment, users were presented with a similar slide-up onsite, but beside the copy was a rotating carousel of real images and stories from the CaringBridge blog about who is impacted by the platform.

So, which do YOU think performed better?

If you said the control, you are correct. That wasn’t surprising to me. What was surprising to me was the differential. Offering the three giving options increased the response rate by 62% and the money raised by 59%.

Here’s how they explained the results:

This outcome suggests that the rotating carousel of real images and stories from the CaringBridge blog may have been less effective or possibly distracting for users, leading to a lower conversion rate. This may also indicate that the effectiveness of the control is in using that right-hand side of the slideup in order to simplify the giving process rather than expanding on more reasons to give. Having a gift array that prefills on the donation page, is ultimately more impactful on the donor’s completion of their gift than the additional testimonies on the carousel.

I’ve been writing about this for years (See Ask for Something Specific!). But it’s nice to have my assertion backed up with current science, too.

I’m sure that some reader will come back with a story of how asking for something specific led to poorer results.

I’m blaming the small sample size.

It makes perfect sense!

Cheers and Have a Good Week!

-da

PS: Your comments on these posts are welcomed and warmly requested. If you have not posted a comment before, or if you are using a new email address, please know that there may be a delay in seeing your posted comment. That’s my SPAM defense at work. I approve all comments as soon as I am able during the day.

Photo by Lori courtesy of Pixabay