When you're putting any survey together for a research or white paper project, it's critical to remember that the topic will ultimately be what makes the project shine.
If the topic is the skeleton of your project — and we all know there's only so much you can do to hide wonky bones — then the survey itself will provide the vital organs.
So if you choose the wrong type of survey, or ask the wrong questions, then you'll have almost as big a problem as a wonky skeleton-topic.
There are two basic options that you have when it comes to survey structure: Qualitative and quantitative.
Deep or wide?
A qualitative survey is a "deep" survey — you identify a few select people to survey and then ask them in-depth questions in one-on-one interviews about your topic.
A quantitative survey is a "wide" survey — you deploy it via mail, email, phone or what-have-you to as many (relevant) respondents as possible and ask them multiple-choice or short-answer questions about your topic.
The perks of going deep
Qualitative surveys have plenty of advantages. If you want to gain a deep, rich understanding of a subject — especially one that few people have studied before — then there is rarely a better way to do it than through qualitative research.
Qualitative interviews or questionnaires are long, taking sometimes hours to complete. These types of interviews give you the opportunity to get inside the heads of your chosen subjects — and if those subjects are experts in some aspect of the business or have a unique perspective, they can be a content gold mine, too.
The drawbacks of deep surveys
Getting to sit down with an expert or interesting person and ask them a ton of deep questions sounds like fun, right? Well, here are some reasons why it might not be the best idea for your particular project.
- You can't pick just anybody. Selecting the appropriate respondents is crucial; your qualitative research will be only as good as the sum of your respondents. Choose wisely.
- It can take a long time. Booking an hour or two of time with the right people can take some time on its own.
- You probably won't have a lot of data. There will (hopefully) be some supporting research or data that you can use, but it's unlikely that you'll collect significant metrics from your qualitative research. This can be an issue come design time, so don't overlook it.
- You'll have to draw some conclusions. There aren't any algorithms that will look at your qualitative research and tell you what the findings should be. You'll need to figure that out on your own.
- Sometimes your findings can be mushy. It's wonderful when you find threads of continuity in your qualitative research that you know you can pursue in analysis; it's less wonderful when your interviews are all over the place. That's usually a sign you need to tighten up your topic, so there is a fix — but it's less than fun to realize that you need to go back to the drawing board after you've invested several days or even weeks already.
When qualitative trumps quantitative
There are some instances when only a qualitative survey will do.
One example that comes to mind is my own master's project — I conducted qualitative interviews with six journalists at a chain of alternative newsweekly papers who had written stories featuring anonymous sources in the past five years.
Perhaps you can see from that description why qualitative research was the best (probably only) option. There simply aren't that many people who fit that specific description, which I chose intentionally for research purposes, so it made a lot more sense to identify the few people who did fit the description and reach out to them instead of try to capture their responses in some kind of online questionnaire.
Another example: If the top names in your field are suddenly all focused on some new thing (let's say it's Snapchat), then does it make sense to ask everyone in your field why and how they use Snapchat — or would it make more sense to ask just those top names what they're doing and why?
The perks of going wide
There will be times when it makes a lot more sense to go wide than to go deep.
Let's say you're curious about the typical career path of people in your industry, for example. Or you want to know what kind of continuing education is common, or what news sources people in your industry generally use, or what hobbies or after-work activities are most popular.
In this case, it wouldn't make sense to ask just a few people these questions, however targeted those people are. You'd rather ask as many people as possible so that you can get a broad idea of what's "average" and what's unusual.
You'll also have data or metrics to help support any analysis you make (and perk up that final product, whatever it may be).
And just because it's wide does not mean it's shallow — you can mix in open-ended questions in with your multiple-choice to get a better idea of why your respondents feel the way they do and how they react.
You can also filter responses by (for example) years in the industry or salary range, which can help you identify how top performers differ from entry-level beginners, among other powerful parsing options.
What wide won't tell you
Wide can help you identify some "best practices" and top performers, but remember: it's telling you what's most popular, not what's most effective. To nail effectiveness, I'd suggest finding highly effective people and conducting qualitative interviews with them.
Wide also typically can't help you understand what's happening behind the scenes, why people are doing what they're doing or how they really feel about an issue. Nuance and gray areas — including emotions or ethical questions — can be very tough to explore using quantitative research.
When quantitative trumps qualitative
Anytime you want to figure out where "normal" is, you're best off using a quantitative survey.
- What's the normal salary range?
- What's the normal number of months/years spent in this position?
- How many hours do normal people spend on Facebook?
And anytime you want to show general comparisons between groups or industry subsections, a quantitative survey with its available filters can be a good way to do it.
Deciding what type of survey will support your project is the precursor to getting your questions together — one of my favorite parts of any survey project, and one I'll explore for both quantitative and qualitative surveys in the near future.