How to run a user survey that actually qualifies

The first proper product we built from scratch, we built without ever talking to a user.

None of us being marketers, we started where we were comfortable, with the technology. We found a solution we liked, shipped it, and then spent a long stretch releasing features into the dark and hoping one of them would land.

We did have a feedback form, but if I'm honest, we mostly used it for the nice replies that made us feel like we were onto something. The awkward feedback was easy enough to ignore and discount to outliers.

That product is Outposts, and the marketing was the least developed part of it by far. Some of that is the channel because a newsletter keeps your users at arm's length, and unless you go out of your way to reach them, you get very little back.

But the real reason is much simpler: we didn't talk to users because it never occurred to us that we had to. We assumed a good enough product would speak for itself. The good ol' "build it and they will come" philosophy.

The marketing suffered for it, because you can't market a product when you've never once heard how the people using it talk about it.

By the time we started the next thing, we'd learned that the slow and expensive way. The first thing we did, before we'd built anything at all, was sit down and talk to users.

It changes everything that comes after, and it's the cheapest move a team can make for replacing guesses with answers.

So when we came back to upgrading Outposts, we finally did the part we'd skipped.

We ran a survey. A real one, designed to show us something we didn't already know rather than to make us feel clever.

This is how we put it together, and the one finding that took apart an assumption we'd been running on for over two years!

Building a survey that disqualifies

Our first instinct was a form that basically said: do you think this is a good idea? would you use this? And what features do you want to see?

The problem is you get back exactly the answer you went looking for. People are polite, especially the ones who already signed up for your thing, and a form full of leading questions just returns your own biases. We weren't making that mistake again.

Like Google

We settled on a structure the way Google runs its opinion surveys. You open with broad, low-stakes questions and funnel toward the ones that actually qualify someone.

A lot more people fill it out, and you also get the subset you care about. The early warm people up and sort them, so the answers that matter most are at the end, and come in with a little more honesty.

Break it up

We also split it into segments instead of one long form, because one giant questionnaire either gets ignored, or gets half-answered by someone who loses interest.

The first segment just worked out what kind of web3 user someone is: how often they're active, whether they've got capital deployed, how many protocols they use. We hold most of that data ourselves already, but we wanted it in their words.

From there it moved into how they follow news, how informed they feel, and what they make of the product they already get from us.

The heavier asks we saved for the end. If they make it this far, they are users who want to answer, they're also most likely to answer honestly.

The whole thing was built to open the door to an ongoing relationship. The final questions offered a next step: try the new build, come on as a tester, and get on a call with us.

One mistake

We didn’t run it by a small group first to test the answers. That’s definitely recommended if you’re basing serious product decisions on the results.

It’s where you catch questions that read fine in your head, but land ambiguously on the page. It’s also where you find out who you’re really hearing from.

In our case, one set of responses came in a little unclear and left us guessing at the meaning. Not serious for us, but you can see how that could lead to bigger problems.

The finding that broke our assumption

From the beginning we've sold Outposts as a way to spend less time online. The early copy was touch-grass material: step back, let us handle the feed so you don't have to. A digital detox for people drowning in notifications and crypto news.

So we asked people how they felt about the time they spend online. They feel fine about it btw. Better than fine even. They're on X for hours a day and they're happy about it. The crowd we'd been promising to rescue from the timeline have no interest in being rescued.

That was the single result most at odds with everything we'd assumed. Our entire pitch rested on giving people their time back, and here the people themselves were telling us they didn't want it back.

I can't even argue with that because even though Outposts got me off Discord, which I'm thankful for, I'm also on X for several hours most days, and I'm not unhappy about that either.

So the promise is changing. The job was never really to get anyone offline. It's to make the hours they're already spending count for more, to put the thing they'd otherwise have missed in front of them. We'd never have known this without asking.

One caveat though because forty-odd responses and a few imperfectly worded questions are not gospel. We're not turning the strategy on its head off the back of one survey. But it was a clear enough signal to go and check for real. That’s what the follow-up calls are for, and if needed, even more surveys.

The part that actually mattered

The most useful thing the survey gave us was thirty-seven people who agreed to test and keep talking to us. Over a third were up for a call, most were happy to give more feedback. Only a handful said they get nothing out of the product, but still answered the survey!

That's the actual job of a survey, and it's amazing how many teams skip it.

Crypto has a bad habit of treating users as exit liquidity, something to acquire and then ignore while chasing the next batch.

A survey built to start a conversation is a small, nearly free way of doing the opposite, of treating the people already there as important and worth listening to. Which, it turns out, is also just good marketing.

We took the long road, but we got there in the end.

If you've ever read back your own glowing feedback form but had this nagging voice in the back of your mind, you now know what to do. Talk to your users.

The Quick-Reference Framework

We went into our last survey certain our users wanted a digital detox. One well-structured survey took that apart, and how it did is the playbook for running a survey that qualifies people instead of collecting opinions.

1. Borrow Google's funnel: vague → qualifying.

Opening broad then narrowing toward the questions that sorts people. They warm up on the soft questions and answer the sharp ones with more candour.

Watch out: a few of ours came out ambiguous or double-loaded, and we only caught it on the deep-dive. Maybe a pilot is for you.

2. Pilot before you send wide.

If a real product decision rides on the answers, or you're with one of the bigger project, test on a small group to make sure the answers are getting you the information you want. Revise if needed, then do the full send.

Watch out: Know who you're hearing from before you generalise. Some questions might need to be split to get clear data.

3. Segment instead of one big form.

We split by who the respondent is: web3 identity, capital deployed, protocols used, news habits, hours on X, how they feel about their time online.

Watch out: don't fish for the conclusion you want. Let the flow and questions deliver real data and accept it at face value.

4. Qualify first, follow up second.

Use the early answers to decide who's worth a deeper ask. Don't ask everyone everything.

Watch out: every extra required question costs you completion, so gate the deep asks behind the qualifier and only show them to the right people.

5. Treat the survey as the start of a relationship, not the end.

Offer a follow-up option, a call or a tester group, and route opt-ins straight into the next step.

Almost all of ours opted in as testers, which is what we wanted.

Video Breakdown