A Survey of Survey Design

Apr 30, 2014

There are many use cases for surveys in the startup world:

Founders want to survey early customers to understand their wants and needs.
Investors want to survey portfolio companies to learn how to be more helpful.
HR wants to survey employees to find out what’s going well and what’s not.
Engineers and PMs want to survey users to get product feedback.

Given the important role of surveys, it’s critical to design them as well as possible to increase response rates and maximize learning.

A few weeks ago, I decided it might be a good idea to survey my fund’s portfolio companies to understand where they excel, where they struggle, where they wish they had more help. Since founders are incredibly busy, I wanted to make sure the survey was designed well enough that it would be useful on first try. I read dozens blog posts, articles, and research papers on survey design, and this post is a summary of what I learned.

Before you start

Understand your goals for doing the survey and make sure the questions you ask will provide you with the data you need to accomplish your goals.
Realize that bad survey design can yield very misleading data. For example, when asked what they considered to be the most important thing for children to prepare them for life, over 60% of respondents chose the option “to think for themselves” when it was presented. However, when no options were presented, less than 5% of responders wrote an answer that fell into this category.

Question ordering

Ask simple questions first to build trust and rapport with the responder. If you start with tough questions, people will get frustrated and will be more likely to abandon the survey.
Group questions by topic, and organize the questions from more general to more specific within each topic.
Use filter questions to allow respondents to skip sections that do not apply to them.

Question phrasing

Options should be exhaustive and mutually exclusive.
Bad: “Do you think XYZ is good, quite good, or very good?”
Better: “Do you think XYZ is bad, neutral, or good?”
Use simple, unambiguous words and avoid jargon.
Bad: “How many occupants inhabit your household?”
Better: “How many people do you live with?”
Questions should be neutral, not leading.
Bad: “Agree/Disagree: XYZ is an incredibly bad product.”
Better: “How would you rate XYZ?”
Ask one thing at a time and avoid single or double negations.
Bad: “Does product XYZ have enough features to replace your existing payroll provider, or do you not have an existing payroll provider?”
Better: “Question 1 – Do you have an existing payroll provider? If the answer is ‘no’, skip the next question. Question 2 – Does product XYZ have enough features to replace your payroll provider?”

Categorical multiple choice questions

When asking people which of several categories they prefer, the order of options matters. People are biased toward the first satisfactory option when the options are presented visually (satisficing). On the other hand, they are biased toward the most recent option when options are presented orally (poor short-term memory). To compensate for these biases, it’s best to randomize the option order for different responders. For example, ask 1/6th of your responders if they prefer X, Y, or Z; ask another 1/6th if they prefer Y, X, or Z; and so on.
It’s tempting to include an “I don’t know” option, but while that encourages people who can’t pick a good answer to be honest, it also encourages people to be lazy. Adding “I don’t know” as an option generally doesn’t improve survey data quality.

Rating and ranking questions

Use 5-point scales for unipolar questions (“On a scale of 1 (not very happy) to 5 (very happy), how happy are you?”)
Use 7-point scales for bipolar questions (“On a scale of 1 (very unhappy) to 7 (very happy), how happy are you?”)
Common scales for different qualities can be found here.
People avoid extremes and stick to middle-ground values. Scales with 3 values are problematic because people will tend to pick the middle value.
Ask people to rate their attitudes on 5- (or 7-) point scales instead of asking them agree/disagree questions. People tend to acquiesce and will be biased toward agreement in questions where the options are binary (agree/disagree, true/false, yes/no, et cetera). For example, in one set of studies, people agreed with assertions 52% of the time and disagreed with the opposite assertions 42% of the time.
Rating questions can result in all answers being the same. For example, you might ask someone to rate ten movies and find that almost every movie receives 4 out of 5 points. Asking responders to rank options is more time-consuming but can provide more meaningful data when you want relative rankings.

Open-ended questions

Open-ended questions are great at collecting feedback, but they are hard to analyze automatically. Participants sometimes skip open-ended questions because they are more challenging and time-consuming to answer.
Open questions are effective for eliciting numerical responses, whereas categorical options might introduce biases. For instance, in one study, only 16% of individuals reported watching >2.5 hours of TV when that was the highest possible answer, but 40% of individuals reported watching >2.5 hours of TV when that option was broken into 5 sub-ranges. Asking an open-ended question like “How many hours of TV do you watch daily?” can remove this issue.

Miscellaneous tips

People are biased toward answers that make them look good, especially if surveys are administered by someone else. Self-administered surveys and anonymous surveys partially compensate for this bias.
Keep surveys to a reasonable length. As responders tire of answering questions, their tendencies to agree, to satisfice, and to skip questions all increase.

Coding VC

Discussion about this post