Be Sure You Aren’t Asking Unanswerable Survey Questions
How to make every question answerable to all participants
One of the most frequent mistakes that survey designers make is writing unanswerable questions.
A golden rule of survey design is: Every question should be answerable by all participants. The challenge here comes from the “all participants” part, which is where many survey writers fail.
This issue occurs in multiple choice questions when the answer choices provided do not cover all possible scenarios, and the respondent is not given an option to exit via a “trapdoor” (more on this later).
When drafting a multiple choice question, we generally give a list of answer choices based on our study objectives or even our own experiences.
A marketer at a peanut butter company who wants to know what kind of peanut butter a consumer bought recently might ask a question like this:
How would a respondent answer this question if they had only purchased Kroger private label brand peanut butter in the last 3 months? Could a respondent answer if they did not buy any peanut butter at all?
This question is unanswerable by all participants. Respondents hit a brick wall, figuratively speaking, when they are unable to answer a question. This is especially true if a question is a required or forced response.
If answering the question is required to move forward in the survey, the respondent is faced with a tough choice. They’ll do one of two things — both detrimental to survey quality.
- The respondent will quit or drop out of the survey. Don’t lose a respondent because of a poorly written question.
- The respondent may give a false response and pick an answer just to move on. This will lead to inaccurate results. On top of that, being forced to lie could impact the respondents attitude and undermine quality of responses going forward in the survey.
To avoid this, writers must think through all the possible ways respondents might want or need to answer a question. Offer several response options but also offer a ‘trap door’ option.
A ‘trap door’ is a quick escape from the question that allows the participant to advance past the question without having to choose any of the listed options. A common way to do this is by adding a “none” or “other” option.
The respondent who purchased Kroger private label and the person who did not buy any peanut butter can now answer accurately.
Making every question answerable for all participants means that there is an answer option that suits every person who is asked to answer the question. This will lead to better data, more engaged participants, and insightful results.
Take a moment to think like participants next time you write a survey. Make sure you are offering each respondent an answer or if not, at least a trapdoor.
If you found this article useful, hit the clap button and follow me for more. Here are some other articles you may enjoy:
