1. You cannot measure what has not happened.
Any question that allows the respondent to speculate about “what they would do” is invalid and cannot be trusted. Admittedly, this limits that scope of what can be learned from Public Opinion Surveys. In other words, you cannot measure what “would” work or “wouldn’t” work. You can only measure what “is” and “isn’t” working. When you ask a question about an experience that – by necessity – exists only in the imagination, the answers you receive will be, at best, imaginary. What a person will do when faced with an actual choice is quite different from what they thought they would do when that choice was purely theoretical. This is because a huge gap exists between who we think we are and who we actually are. We refer to this illusion as “self-image.”
2. Do not use focus groups.*
An interesting story appeared in the Oct 26, 2003, New York Times. In it, reporter Clive Thompson quotes Joey Reiman, a founding partner of the BrightHouse Institute: “…over years of producing marketing concepts for companies like Coca-Cola and Red Lobster, he [Reiman] has come to the conclusion that focus groups are ultimately less about gathering hard data and more about pretending to have concrete justifications for a hugely expensive ad campaign. ‘The sad fact is, people tell you what you want to hear, not what they really think,’ he says. ‘Sometimes there’s a focus-group bully, a loudmouth who’s so insistent about his opinion that it influences everyone else. This is not a science; it’s a circus.’”
The story goes on to say: “Advertising’s main tool, of course, has been the focus group, a classic technique of social science. Marketers in the United States spent more than $1 billion last year on focus groups, the results of which guided about $120 billion in advertising. But focus groups are plagued by a basic flaw of human psychology: people often do not know their own minds.”
3. Do not offer multiple-choice answers to your questions.
In other words, don’t offer a list of answers from which the respondent is allowed to choose. Not only is this technique invalid, it can lead to the worst type of self-delusion. Disastrous marketing decisions have often been the result of allowing survey participants to select their responses from a pick-list of possible answers.
4. Do not trust any survey of less than 384 qualified respondents.
The mathematics of public opinion are not linear, but exponential. Because of this, 1,500 qualified respondents can reveal the opinions of the entire population of North America with reasonable accuracy while 250 respondents will represent a universe of only 700 people with that same degree of accuracy! Strangely, by adding just 134 additional respondents (for a total of 384) you will have increased the size of your measurable universe to more than 1,000,000 people.
If your universe consists of just a few hundred people, you need to survey almost all of them in order to achieve any meaningful level of accuracy. As the size of your universe increases, the percentage of people needed to achieve the same accuracy decreases rapidly.
Larger population = Smaller percentage must be surveyed
Smaller population = Larger percentage must be surveyed
Error – This is the “plus or minus” percentage. It means that your results have an error potential of no more than the named percentage. (But the underlying math assumes that your question was valid AND that it was worded correctly.)
Confidence – Expressed as a percentage, “level of confidence” basically says that if you were to conduct the survey multiple times, how often would you get similar results.
95% confidence with an error of plus or minus 5% means that if you were to conduct the same survey 100 times, the results – 95 times out of 100 – would be within 5% of the first time you ran the survey.
If it’s worth conducting a survey to know the information, it’s worth asking a sample size that will yield a 95% level of confidence with an error potential of no more than 5%.
Bottom line: Never believe a local, regional, or industry survey based on a sample size of less than 384 people.
Most marketing surveys are invalid by either sample size or question phrasing.
Wizard of Ads, Inc. will write your Outcall Survey Questionnaire for a flat fee of $1,000. For more information contact Corrine@WizardofAds.com or call Corrine Taylor at 512-295-5700.
