Multiple Response Questions and Web Survey Software
Web page surveys are a visual medium. How you present questions in a browser window can affect the responses you get to those questions. There are three main ways you can present questions that ask each person to pick more than one choice: a drop-down list, a visible list with a series of checkboxes and a grid or matrix. The most capable web survey software lets you choose which of the three to use in any situation.
Web questionnaire designers sometimes use drop-down lists to save space when there are a large number of choices. This can work well if you’re asking a question like “Which other country have you most recently visited?” This question can offer something like 190 answer choices, and people can scroll down to find the right one. Everyone visiting a Web page knows how to do that. But if you want people to be able to select multiple answers, drop-down lists don’t work well. Many people, perhaps most people, don’t know that to select more than one item from a drop-down list they have to hold the control key while clicking on the second and higher choices. They forget to do that, they will only select the choice they click on. This problem makes using drop-down lists when people can select more than one answer a poor choice in most circumstances.
Your choice between the other two methods is less clear. If you have a question such as “Which of the following magazines do you read?” you can present a list of checkboxes or a grid in which each magazine is listed and columns are labeled Yes and No. If you force people to have to select a Yes or No for each magazine, you will get more Yeses than the number of checkboxes that would be checked. Research has not conclusively shown whether or not that difference makes the grids better.
On the positive side, forcing people to pick yes or no for each individual magazine makes them think more about those magazines, at least ideally. One danger is that some people will “straight line.” This is a problem in which some people taking surveys will just go down a grid and check answers, without giving much consideration to each item, as a way to save time. Another negative point about using a grid is that forcing people to indicate a yes or a no for every choice is much more time-consuming than just having them checked the choices that apply. Anything that makes a survey take longer is a negative. It may prompt some people to drop out. It may prompt others to get less thought to answers later in the survey.
One commonality in all three methods is that the order of the choices can affect the answers chosen. Some people tend to pick items near the start of a list and then move on. In the case of a grid in which people can’t move on without answering for each item, people tend to give more consideration for items near the top of the grid. In all these cases sophisticated web survey software should let you randomize the order in which the choices are presented. While doing so does not eliminate this problem, it distributes it equally among the different answer choices and so minimizes its effect.