Earlier this week I had a FB discussion with a fellow researcher about appropriate sample sizes (steekproefgroottes, muestras) in market research. It prompted me to write a post I always want to write each time that discussion comes up in Curacao.
As I write, I realize that the discussion might arise from the different goals of business research versus scientific research. In business (and other organizations) information is gathered to assist in decision-making. It has to make business sense. Scientific research often merely seeks to describe a condition. The findings are not necessarily used to aid in decision-making.
The research field allows for all sample sizes. Researcher just have to indicate the accuracy of the findings.
The larger the sample size, the more accurate the result, i.e. the more likely it is that the result of the study is closer to the reality. What is the error?
| Sample (rounded) | Error (rounded) |
| 2.500 | 2% |
| 1.000 | 3% |
| 600 | 4% |
| 400 | 5% |
| 100 | 10% |
This means that if you interview 400 people and 60% says they like your product, the reality is that between 55% and 65% like your product. If you interview 2.500 people, and 60% says they like your product, the real answer is likely to lie between 58% and 62%.
So, why would a market research not choose to work with the most accurate (large) sample size?
1. ROI reasons. Let’s say that a mass market research study with a sample size of 2.500 costs the same in Curacao, Holland and the USA, e.g. USD 10.000. All else equal, if the accurate answer on which you will base a decision yields you USD 1million extra profit in Holland, it will yield you USD 10.000 extra profit in Curacao – a BREAKEVEN -, just because our market is 100 times smaller (16 million vs 150.000). Why bother? What are the chances that you will sell this research proposal to your US boss, who is accustomed to a market where the same investment can yield 2.000 times more profit?
2. Maybe the ROI is good, but there is just no budget.
3. There are not enough qualified respondents in the market. There are 2000 babies born in Curacao every year. If a survey with 2% margin of error is desired, you need to interview all the new moms, and then some. How much it will cost in money and time to get to ALL of them? We were recently asked to interview 100 regular users of a minor cigarette brand in a certain age group. How difficult is that? How large is the risk that interviewers or respondents will fake interviews or come up with unqualified respondents?
4. There is not enough time. See point 3
5. Often a high level of accuracy is not necessary. For strategy and marketing purposes we are usually more interested in understanding and exploiting the top 3-4 alternatives, than in making an accurate top 20-ranking. We are interested in the top 3-4 reasons people buy a product, the top 3-4 media vehicles they use, the 3-4 advertising or product enhancing concepts that most appeal to them, the 3-4 closest competitors. All the other reasons, media vehicles, advertising or new product concepts and competitors become irrelevant. Why only the top 3-4? Because those are the ones we will duly consider in strategy development and/or marketing.
6. Borderline cases always mean that your work as a strategist is not done. If 52% like the product concept and 48% dislike, it is imperative to continue working on a better concept, regardless of the margin of error of the study. The risk of the product flopping in the market is simply much too great. What if further enhancement is not possible and a clear winner is not obvious? Well, if budget and time permit and the client wants to assume the risk of a product flopping, you can continue interviewing (making the sample size larger) until the desired accuracy level is reached. I.e. you can start with 400 and work your way up to a sample size of 1000 or more.
7. Segmentation is not very detailed. If the intention is to analyze and develop strategy for different segments, then you need enough respondents in each segment (subgroup) to properly analyze it (there is a formula for that). But, in small markets it is often not profitable to divide the market in many different segments, each with a different strategy. The most we might segment a small market by is 2 or 3 demographic and/or lifestyle variables. The best approach is to first decide which segments you want to study (because you can and intend to develop a separate strategy for them) and decide on minimum sample size later.
3. The answers might have been recorded, coded, processed or interpreted wrongly
4. Interviewers might have faked the interview. Unsupervised and without validation (i.e. call backs) that is a big risk. Our company asks the name and telephone number of respondents so we can call them back and check (validate). Either that or interviewers are duly supervised.
5. Certain segments (ages, gender, social economic class) might have been over- or underrepresented. Perhaps they could not be found (as young men often are), they were not willing to participate, or participated but systematically refused to answer certain questions. Personally I always participate in surveys but systematically refuse to answer questions about income, politics, religion, sex, etc. That’s one of the reasons we don’t include them in our surveys: never ask a question you would not want to answer.
Sample sizes of around 400 are the norm rather than the exception in the small markets of the Caribbean. The constant discussion about sample size inhibits the further development of the research industry. Researchers are afraid to propose small sample sizes, clients are afraid to accept them since they have been led to believe that small sample sizes provide useless answers. Yet, there is no money to pay for more. So, research is forgone, decisions are made on no data at all, while we all know that some data is better than no data at all.
No comments:
Post a Comment