Marketing Hold-out Cells: Why You Need Them
Because you need a place to confine your marketing team when they’ve underperformed.
No, no. We’re not talking banishment to Castle Black. (Apologies. It’s been a long, rainy winter, so I’ve been watching GOT. Again.) We’re talking about a group of prospects that you DO NOT CONTACT. What?!? Choose not to advertise to a population? Why on earth?
Background: I’m an analytical, skeptical marketer by nature. I want to see the data, and I want proof that the advertising is working. I’m a proponent of the scientific approach: assume the null hypothesis, that our treatment group (those receiving ads) will NOT behave differently than the control group (those not seeing ads).
Many times this approach is unnecessary. In acquisition, for example, when speaking to a population that is unaware of your brand. Or in search, when you’re intercepting a potential customer as they’re seeking a solution, and otherwise wouldn’t have considered yours.
But, when marketing to a defined, known population, like your own customer base, a hold-out cell can be critical. Let me give you an example. A couple years ago, during the height of COVID, I was working with a well-known consumer brand in the social gathering space. They had pivoted (because who’s socializing during a pandemic, right?), and were desperately trying to get traction with video. They’d engaged an agency to run a Facebook campaign, and had just under $1M to invest in the effort.
Initial results were poor. Cost per acquisition was too high, unsustainable. The company began to focus on advertising to its own customer base (millions strong) because the reach was superior to email. Results got a little better. Then the agency advocated to expand the conversion window, originally 7 day click-through, to 30 day view-through. (Veeeeeeeery generous.) And lo and behold, results dramatically improved. The campaign was suddenly worth continuing. This was when the skeptic in me started to think, “Hmmmmm.”
And so I began lobbying for a hold-out cell. We were delivering customer lists to the agency for use in Facebook, why not just withhold a subset of the population? To their credit, the company was receptive. (Not all companies would be.) They held out a sample of tens of thousands of customers, and after a few weeks, analyzed the data. And guess what? The hold-out sample had the same conversion rate as the population receiving ads. The Facebook ads were not changing customer behavior; they were simply claiming credit for actions what would have happened anyway. The campaign was shut down, and several hundred thousand dollars were saved.
This is why you need a holdout. Because to truly understand your results, you must strive for the most accurate measurement possible. It can be difficult for us marketers to accept (we excel at taking credit), but if we are humble, if we seek the truth, a simple holdout can be indispensable.
