Analyzing Comments with Survey Software
Questionnaires often have questions that ask for a comment about a subject. Researchers often aren’t happy including these kinds of questions in a project, because they are harder to analyze then other answers. But researchers also recognize that asking people to comment can often provide insights that they wouldn’t learn if they just offer people a list of answer choices.
For example, if you ask people why they gave a product a poor rating, you may get some answers you weren’t expecting, if you ask it in open-ended form, instead of just giving people a list of choices. Another common case in which you can get value in a text answer is if you give people a list of choices in response to a question and have “other” below the other choices. In this instance, you may find it useful to have space where people can tell you which other answers that would like to give.
How do you analyze these comments? Sophisticated survey software offers multiple options. One is to just give you a count of how many times each word appears in the answers. This is most useful with short answers, such as those clarifying what someone means by “other.” For example, if the question was “Which bank do you use most often?” a list of words given in response can show you how frequently banks that were not on your list are mentioned.
Another way to analyze the results is to use keyword coding. In this approach, you create a list of words or phrases you want to find in your answers, and your survey software can then tell you how many times each of these words or phrases appeared. Since coding puts the numeric codes into your data, you can then create cross tabs and other tables and charts illustrating how many times each word or phrase was used by different demographics.
Another kind of coding is inspection coding. This is the most laborious coding but can be the most useful. When using inspection coding, you have a person look at a comment above a code sheet and select one or more codes to describe the comment. This technique is particularly useful when you are looking for concepts not easily represented by individual words or phrases. Just as we keyword coding, you can use these codes to produce tables and charts.
One concern with inspection coding is if you have more than one person doing the coding, you need to make sure that they are doing the coding consistently, in other words, that all of your coders would give the same codes to the same comments. In these cases, you may wish to do some training.