When conducting surveys by email, polling on a website, or approaching people on the street, the main hurdle is knowing how many responses to collect.
The following formula can predict the necessary sample size based on your population, such as users, email addresses, potential customers.
- N = Population size, for example, the average number of users per month on your website, the size of your email list, or population in your city.
- z = Desired confidence level. Statisticians typically use 95 percent, which would put the z value at 1.96.
- e = Margin of error, which is usually 5 percent.
- p = Sample proportion. This is what you expect the results to be — the percentage of your population that you’ll sample. If you’re unsure, use 50 percent.
The formula can be difficult. But many online calculators can compute it for you, including one from SurveyMonkey.
Say you want to survey visitors to your ecommerce site. You can use the formula above to calculate the number of responses for the survey to be statistically valid. To do so, input your population size, which could be monthly unique visitors if you plan on running a survey throughout the month or weekly unique visitors if for a week. Then use the formula to calculate the number of people you need.
For example, 1,000 unique monthly visitors, a 95-percent confidence level, and a 5-percent margin of error would require a sample size of 278 responses.
The tricky part is knowing what percentage of your visitors would complete the survey. If your response rate is only 15 percent (which is typical for external surveys), to get 278 responses you would need to poll 1,850 people, which is more than your total monthly visitors!
If that is the case, you would either run the survey for all visitors for a few months or assume a lower confidence level. If you poll only a limited number of visitors, say 20 percent, your confidence level would be much lower, and your survey could be skewed by a specific group of people.
A gut feeling comes in handy here. If you notice that most respondents had a negative experience with your company, you know it is skewed. Alternatively, you may find that responses are coming from a certain demographic group that may not be representative of your core customers.
Demographics
Another way to establish survey size is by using demographic information. If you don’t have demographic info on your population, third-party companies can supply it based on email addresses or physical addresses.
Another way to establish survey size is based on demographic information.
Consider this example. Say you want to determine preferred colors by men and women. You surveyed your database randomly, emailing 1,000 people. But then you realize that 80 percent of respondents are women. Since the goal of the survey is to identify the color preferences of men and women, you would need to send an additional survey only to men to obtain an equal amount of responses for each gender.
To calculate how many additional emails to deploy, use the actual amount of responses from females (800 in this example) and the male response rate (say 10 percent for 200 responses). Therefore, you would need 600 more from males, which when divided by the 10-percent response rate would mean 6,000 more emails should be sent to the male population to balance the survey.
Confirming Survey Size
Assume you surveyed your audience and received responses. How do you know if the number of responses is sufficient to make a business decision? The answer is to calculate if conducting more surveys would change your mind.
For example, say you had 1,000 responses on question A with a split of 30 percent, 45 percent, and 25 percent for answers A, B, and C, respectively. You decide to poll 100 more people, and question A gets 100 percent of the answers. Thus the percentages would change to 36 percent, 41 percent, and 23 percent — not enough to change your decision, likely. In other words, experiment with the numbers by assigning 100 percent value to a certain answer and determine if it would, hypothetically, make a difference.
What’s the Reason?
Instead of sample size, focus on the reason for the survey. If the responses that you have received provide actionable results and they make business sense, chances are you have polled enough people.
Remember, third-party data can eliminate the need for surveys and collecting and analyzing the results. Qualitative surveys — those that ask open-ended questions — sometimes require just a few people to uncover a material problem. That can be more helpful than hearing from a bunch of happy customers.
