Weighting Your Sample with Survey Software II
Our last blog discussed weighting. Weighting is a technique that lets some people count more than others when determining overall responses. The most common reason to do that is to make the percentages of certain kinds of people in your sample better match their percentages in the population your sample represents. For example, you might have been able to reach more older people in a telephone survey of than younger people, but the population you are studying has equal numbers in certain age groups. You can use weighting to make the older people each count a little less and the younger people count a little more, so that when you view overall answers, each age group contributes the desired percentage of the answers.
An occasional use of weighting based on demographic groups is to make the numbers in your sample reflect the numbers in the complete population your sample represents. For example, in a recent survey done by a municipality, they wanted the numbers of each group in the sample to look like the numbers of the total number of people in each group in the municipality.
There is another kind of weighting that the most sophisticated survey software allows. That is the ability to weight answers to some questions based on the answers people give to other questions. For example, if you are studying how corn farmers use fertilizer, you can ask the farmers how many acres of corn they plant, and then use that answer to wait other answers. This way the answers from a large farm could count more than the answers of a smaller farm. If the purpose of your survey is marketing, then your client might be well served by having the answers reflect the size of possible sales in this way. Many programs only let you weight based on demographics, if they that you weight at all. The most sophisticated survey software lets you do both kinds of weighting.
There are some caveats when doing weighting. The biggest concerns statistical significance testing. Significance tests help you learn how confident you can be that the answers you received in your sample closely reflect the answers you would have gotten, if you had been able to contact everyone in the population your sample is trying to represent. Significance testing depends on accurate sample sizes. If your weighting inflates the apparent number of the people in your sample, most significance testing would give you an inflated idea of how confident you can be.
In some cases it can be possible for sophisticated survey software to show the weighted numbers of different kinds of people on a report, but use the unweighted number of people in the calculations. Whether this is workable depends on the particular statistic being used. For example, capable survey software can create difference between proportions (or Z) tests using unweighted group sizes, even when displaying weighted numbers of people in each group.
In sum, you should always be cautious when using weighting, but in some cases it can be a very valuable tool.