Can you trust the test that says your ads worked?

Last week I spoke about testing an ad campaign to see how many of the users the ad actually brought in.

I got a few DMs this week asking how it works, so this post will cover that.

I also wanna cover the point that when we show someone a test that says the ads worked, it's very much like marking our own homework. What we're doing now, and what's coming in our next release.

For the sceptics.

It's a fair point and good to look at closely.

So here's how the homework part works, and which parts to pay attention to.

Incrementality testing is about comparing people who saw the ad, with similar people who didn't, and the gap between them is what the ad caused.

Example:
Say 8 in every 50 people who saw the ad went on to use the product, and 3 in every 50 similar people who never saw it did too.

So the ad brought in 5 users, because the other 3 were coming anyway.

Everything rests on the accuracy of that second group, the comparison group, and there are two ways to build one.

The textbook way is before the campaign starts.

You take everyone you plan to target, pick some of them at random, and make sure they never see the ad. That's your holdout group.

Because the split is random, both groups end up alike, so it's a clean, fair test.

But it also costs you, because everyone you hold back is someone you don't advertise to, including some people the ad would have brought in.

Another problem is that most crypto channels can’t do this at all.

You can't hide a KOL's post from a random slice of their followers, or keep a quest away from half the people who find it.

What we do now

Right now, we do it the other way round, after the campaign.

First, we group wallets that belong to the same user, so no one is counted twice.

Then we build the comparison group from wallets that matched the same targeting, weren't already using the product, very similar to the people who saw the ad.

We take out anyone who saw the ad, along with all of their other wallets.

After that, both groups are measured over the same window.

The trade-off is that the two groups can’t be an exact match, so extra care has to go into picking the comparison group.

With a random holdout, you split one audience at random, so both portions are the same kind of people.

We don't split the audience, we find lookalikes instead, so you don't have to hold back any of your audience.

We match on what we can see: the same targeting, and how recently they've been active.

Because everything happens onchain, we can see a lot: what each wallet does, and when.

That gets the two groups close.

But anything we can’t see, we can’t match, and that’s why the matching is the part worth double checking.

What's Next

It'll never be perfect, though.

Lookalikes can still differ in ways we can't see, like why someone is active or how likely they were to convert anyway.

So we're building the textbook option into Specify: a random holdout set aside before the campaign starts. The difference is size.

Instead of a big holdout, we'll hold back a much smaller slice, around 10%.

That keeps the test clean while giving up far less of your audience.

The tests will also run in the app itself, so the results and the data behind them will be right there in your dashboard. (That's coming in our next release)

So which parts can you check?

There are two. The conversions are public and you can check them yourself. Then there's the comparison group. It isn't public, so you'll have to ask about it.

The conversions.

You don’t need an incrementality test to check them.

Every conversion for every campaign is in your dashboard, with the wallet and the transactions behind it, plus a link to a block explorer.

Pick a user, open the transactions, click the transaction link and you’ll see the swap or deposit it’s counted for.

The comparison group.

You can't check this one onchain, because only the people who built it know which wallets are in it.

Good questions to ask:

• How was the comparison group picked: at random before the campaign, or matched afterwards? Matched on what?
• Was it measured over the same period as the campaign?
• Were wallets belonging to the same person grouped together, on both sides?
• Can I see the data behind it?

If you're testing with us, just ask and we'll share ours.

Good answers tell you how much trust to put in that lift figure.