Friday, March 12, 2010

9 issues to consider when deciding between Telephone or Door-to-door surveys in small markets?

The source for this post is the textbook Marketing Research by Aaker, Kumar and Day, supplemented by our own experience in the small markets of the Caribbean.

When trying to decide between a telephone or door-to-door (face-to-face) survey, these are some issues to consider:
  1. turn-around time - Because it is faster to call someone than to drive or walk to them, the turn around time for a telephone survey is shorter than for a face-to-face survey. In our experience, 400 telephone interviews can easily be done in less than 80 man-hours. Not so with face-to-face interviews
  2. length of the questionnaire - If the questionnaire is long or complex, it is often not desirable to conduct the interview over the phone. We have found that sometimes respondents are more willing to reveal certain personal information over the phone than in person and vice versa.
  3. distance to travel to respondents. If respondents live far apart, a telephone survey might be more cost-effective because of the cost and time of travel.
  4. level of supervision. It is easier (and more cost effective) to duly supervise interviewers from a centralized call area, than on the road. Mistakes are also easier caught and corrected, especially if data input happens at the same time, as we do at MarkStra.
  5. accessibility to respondents. Not all people have a telephone; interviewers can often not gain access to gated residential communities or apartments; interviewer safety may preclude visits to certain neighborhoods or during certain times of the day. Other groups are systematically inaccessible because of lifestyle or unwillingness to cooperate. Young men are often not home. When they are not, they are likely to be engaging in a group activity and cannot participate, even when called on a mobile phone. Workers in hospitality and health care often work shifts and may not be accessible during "regular" interviewing hours. In Curacao, we find that European Dutch people are systematically unavailable during dinner hours when interviewers are most likely to call or visit.
  6. product category. If the survey is about Ringtones, a phone is a pre-requisite. Hence a telephone survey is highly appropriate. If the survey is about a basic necessity (such as water or electricity), some respondents are likely to not have a phone, while their opinion is not only important, but possibly varies significantly from people with phones. If someone has an issue with the electricity company and has a phone, the road to a solution is probably much different than if he doesn't have a phone.
  7. sampling method used. Random sampling can be more effectively executed with a telephone survey. It is easier to call and call-back the chosen respondents, than to drive to their house with the risk of not finding them home or the risk of them not qualifying for the survey (f.i. because they are not a user of the product).
  8. the number of phones available at the call center
  9. budget- total cost is determined among others by the issues above
I started writing this post to explain the limitations of telephone surveys, but have realized that face-to-face interviews have at least as many limitations. Point 5, not being able to enter gated communities and apartments and interviewer safety issues are especially telling.

In the Caribbean, ROI and budget issues are especially important.Telephone surveys probably fit the budget better. Don't dismiss them as being less-than-ideal. Door-to-door surveys have at least as many limitations.

Survey sample sizes in small markets

Earlier this week I had a FB discussion with a fellow researcher about appropriate sample sizes (steekproefgroottes, muestras) in market research. It prompted me to write a post I always want to write each time that discussion comes up in Curacao.

As I write, I realize that the discussion might arise from the different goals of business research versus scientific research. In business (and other organizations) information is gathered to assist in decision-making. It has to make business sense. Scientific research often merely seeks to describe a condition. The findings are not necessarily used to aid in decision-making.

The research field allows for all sample sizes. Researcher just have to indicate the accuracy of the findings.

The larger the sample size, the more accurate the result, i.e. the more likely it is that the result of the study is closer to the reality. What is the error?

Sample (rounded)

Error (rounded)











This means that if you interview 400 people and 60% says they like your product, the reality is that between 55% and 65% like your product. If you interview 2.500 people, and 60% says they like your product, the real answer is likely to lie between 58% and 62%.

So, why would a market research not choose to work with the most accurate (large) sample size?

1. ROI reasons. Let’s say that a mass market research study with a sample size of 2.500 costs the same in Curacao, Holland and the USA, e.g. USD 10.000. All else equal, if the accurate answer on which you will base a decision yields you USD 1million extra profit in Holland, it will yield you USD 10.000 extra profit in Curacao – a BREAKEVEN -, just because our market is 100 times smaller (16 million vs 150.000). Why bother? What are the chances that you will sell this research proposal to your US boss, who is accustomed to a market where the same investment can yield 2.000 times more profit?

2. Maybe the ROI is good, but there is just no budget.

3. There are not enough qualified respondents in the market. There are 2000 babies born in Curacao every year. If a survey with 2% margin of error is desired, you need to interview all the new moms, and then some. How much it will cost in money and time to get to ALL of them? We were recently asked to interview 100 regular users of a minor cigarette brand in a certain age group. How difficult is that? How large is the risk that interviewers or respondents will fake interviews or come up with unqualified respondents?

4. There is not enough time. See point 3

5. Often a high level of accuracy is not necessary. For strategy and marketing purposes we are usually more interested in understanding and exploiting the top 3-4 alternatives, than in making an accurate top 20-ranking. We are interested in the top 3-4 reasons people buy a product, the top 3-4 media vehicles they use, the 3-4 advertising or product enhancing concepts that most appeal to them, the 3-4 closest competitors. All the other reasons, media vehicles, advertising or new product concepts and competitors become irrelevant. Why only the top 3-4? Because those are the ones we will duly consider in strategy development and/or marketing.

6. Borderline cases always mean that your work as a strategist is not done. If 52% like the product concept and 48% dislike, it is imperative to continue working on a better concept, regardless of the margin of error of the study. The risk of the product flopping in the market is simply much too great. What if further enhancement is not possible and a clear winner is not obvious? Well, if budget and time permit and the client wants to assume the risk of a product flopping, you can continue interviewing (making the sample size larger) until the desired accuracy level is reached. I.e. you can start with 400 and work your way up to a sample size of 1000 or more.

7. Segmentation is not very detailed. If the intention is to analyze and develop strategy for different segments, then you need enough respondents in each segment (subgroup) to properly analyze it (there is a formula for that). But, in small markets it is often not profitable to divide the market in many different segments, each with a different strategy. The most we might segment a small market by is 2 or 3 demographic and/or lifestyle variables. The best approach is to first decide which segments you want to study (because you can and intend to develop a separate strategy for them) and decide on minimum sample size later.

Why then have some studies with a sample size of 400 yielded vastly different results from the reality?

There are many reasons other than sample size, that could lead to the “wrong” answer. Following are some of the reasons listed by Aaker, Kumar and Day in Marketing Research (the quintessential marketing research textbook):

1. Interviewers might have phrased questions wrongly

2. Respondents might have intentionally or unintentionally given inaccurate answers

3. The answers might have been recorded, coded, processed or interpreted wronglyNumbered List

4. Interviewers might have faked the interview. Unsupervised and without validation (i.e. call backs) that is a big risk. Our company asks the name and telephone number of respondents so we can call them back and check (validate). Either that or interviewers are duly supervised.

5. Certain segments (ages, gender, social economic class) might have been over- or underrepresented. Perhaps they could not be found (as young men often are), they were not willing to participate, or participated but systematically refused to answer certain questions. Personally I always participate in surveys but systematically refuse to answer questions about income, politics, religion, sex, etc. That’s one of the reasons we don’t include them in our surveys: never ask a question you would not want to answer.

Sample sizes in the Caribbean

Sample sizes of around 400 are the norm rather than the exception in the small markets of the Caribbean. The constant discussion about sample size inhibits the further development of the research industry. Researchers are afraid to propose small sample sizes, clients are afraid to accept them since they have been led to believe that small sample sizes provide useless answers. Yet, there is no money to pay for more. So, research is forgone, decisions are made on no data at all, while we all know that some data is better than no data at all.

Further reading

Margin of error, Market Research World, Marketing Research by Aaker, Kumar, Day or google the topic.