How the order of questions can affect accuracy
In design research, the type of questions you ask has a huge influence on the accuracy of the insights you gather. What many don’t realise however, is that the order of the tasks you get participants to complete, or the questions you ask them, can have a big influence on your results. Order effects can arise when completing one task or question influences the participant’s opinion of or response to another task or question, thus giving you unreliable and possibly misleading data.
For example, say you wanted to ask participants the two following questions:
Question one: Which of these words best describes your best friend: Strong, Clever, Brave, Passionate, Wise, Friendly
Question two: Describe yourself in four words
In this order, question one has told respondents the sort of words you expect them to use in question two – in this case, positive adjectives. So they are much more likely to use those sorts of words and to avoid negative words. If this is what you want, great. If it isn’t what you want, then you need to find a way to prevent order effects from damaging your data.
Ask open questions first
One useful rule of thumb that often helps reduce the impact of order effects is to ask open questions, where people are free to respond as they want, before asking closed questions, where they have to choose from a list of answers. The purpose of this rule is to stop the options given for closed questions influencing the responses given to the closed questions.
Try counterbalancing
Another way to try and counteract order effects is ‘counterbalancing,’ which means varying the order in which the various elements of the research are presented, so that any order effects caused by one element being first for one participant will be counteracted by another participant seeing it last. For example, if you wanted to counterbalance 3 questions, A, B and C, in a survey, you’d first work out all the possible combinations of these three questions:
- ABC
- ACB
- BAC
- BCA
- CAB
- CBA
Then, make sure an equal amount of people experienced each combination. So in this example participants would be equally spread across each of the six combinations. If you had 30 respondents, five would do A, B then C, five would do A, C then B, etc. While counterbalancing can be effective, it can be difficult and complex to implement.
Order effects can render the results of otherwise good research useless or confusing. A combination of experience, common sense and research design knowledge is required to order parts of research in a way that will get you the most useful insights.
If you would like advice on how to run a user research project, or you would like Nightingale to run research for you, contact us.
