At the recent AAAA account planners' conference (AdAge, 8/8/05), Malcolm Gladwell (Blink, Tipping Point author) threw down the gauntlet on decades of market research tradition with the assertion focus groups should be banned. This is akin to saying email should be banned because much of it is poorly written. Mr. Gladwell’s view of focus groups as random groups of people asked to make decisions on marketing in an artificial setting bears little resemblance to the reality of qualitative research today. After conducting hundreds of groups in the past two years, our conclusion is that it’s not the technique that is flawed, but how it’s used.
Focus groups are fast and easy to execute. The flexible format makes them ideal for exploring strategic issues (who is my target?) to tactical decisions (is a red arrow more readable than a copy block?). Before the M&M’s get thrown out with the candy dish, we think it’s worth trying to putting the focus back into focus groups. Here are some suggestions for making qualitative research more meaningful and less prone to misleading errors.
When appropriate, conduct groups online. A series of groups comprised of people from all over the country is automatically more representative than groups conducted in Denver, Chicago and LA. Our experience is that online groups are also better recruited and less susceptible to manipulation by individual clients or respondents. Here’s why:
Setting: Less artificial as respondents are in their own home, office or library rather than a sterile focus group facility. Observing from any Internet enabled location increases client participation.
Sample: Even hard to find populations can be identified and recruited. There are fewer ‘professional’ respondents since recruiting is not limited to those living near a facility. We have successfully recruited owners of $2000 exercise machines, users of flavoried rum, even concession stand volunteers.
Control. Loud-mouths have a harder time dominating in a forum where everyone speaks at the same time and at the same volume. Less time is spent controlling and more on listening.
Validity. People are more likely to admit to habits or opinions that they think others would disapprove of in the anonymity of cyberspace than they would face-to-face.
Add a quantitative component. A screener is merely a quantitative survey in sheep’s clothing. It can be treated as part of the overall research design in order to put qualitative findings in a larger context. Such “hybrid’ designs’ reduce overall costs by sharing the burden of recruiting across two studies. For example, in a recent study a survey was used to identify cross channel shopping habits for a large retailer, focus group respondents were selected based on their survey responses. The resulting samples were unusually homogeneous, allowing nuances in behavior and attitudes to be identified.
Make focus groups more observational. Groups where the moderator asks a series of yes no questions is nothing more than small sample quantitative survey. Effort and creativity are required to encourage respondents to talk to each other so their natural responses can be observed. Interactive collaborative tools -- such as trips to the web, projective techniques, survivor-type elimination games and whiteboard exercises -- make the experience fun and involving, as well as reduce artificiality.
Wednesday, August 24, 2005
Tuesday, May 31, 2005
Online Research in the U.S.: What's Taking So Long?
This just in: according to a new study (by Cambiar), the value of the U.S. online research market will grow from $1.2 Billion in 2003 to $4 billion by 2008, largely by introducing a large number of smaller users. Online usage is firmly established among larger corporations with 28% of their budgets devoted to online research. For smaller companies, those using online also devote 28% of their budget to it, but overall penetration of online techniques is lower. 51% of non-users surveyed saying they definitely or probably will start using this medium in the next three years. Hence the opportunity to grow the category lies in penetrating larger numbers of smaller companies.
No doubt this is an opportunity, but there is still plenty of room to grow among those larger companies. Twenty-eight percent is a shockingly low number, particularly when one considers how many advantages online represents over traditional methods. Data gathering by phone, mail, fax or intercept all have significant disadvantages in sampling and setting, not to mention cost and timing. What are the big companies waiting for?
This is where the real opportunity lies for the industry. Our experience is that the industry has been incredibly slow to embrace the Internet, slower than consumers, in fact. Issues of response rates and quality of sample are still the most widely mentioned causes for concern when considering online research.
It's different in the rest of the world. A parallel study among 50 research companies worldwide, representing c.$1.2bn in research revenue, and found commitment to heavy investment in online over the next three years: 55% intend to increase their scripting resources, 77% to invest in panel expansion and 55% to build new panels. 78% ‘already conduct online research and have invested heavily in order to compete in this market’ and 59% are able to manage entire studies using their own resources.
The U.S. needs to catch up with consumers and with the rest of the world and start thinking of online as THE way to do research, not just the NEW way to do research.
Comments?
No doubt this is an opportunity, but there is still plenty of room to grow among those larger companies. Twenty-eight percent is a shockingly low number, particularly when one considers how many advantages online represents over traditional methods. Data gathering by phone, mail, fax or intercept all have significant disadvantages in sampling and setting, not to mention cost and timing. What are the big companies waiting for?
This is where the real opportunity lies for the industry. Our experience is that the industry has been incredibly slow to embrace the Internet, slower than consumers, in fact. Issues of response rates and quality of sample are still the most widely mentioned causes for concern when considering online research.
It's different in the rest of the world. A parallel study among 50 research companies worldwide, representing c.$1.2bn in research revenue, and found commitment to heavy investment in online over the next three years: 55% intend to increase their scripting resources, 77% to invest in panel expansion and 55% to build new panels. 78% ‘already conduct online research and have invested heavily in order to compete in this market’ and 59% are able to manage entire studies using their own resources.
The U.S. needs to catch up with consumers and with the rest of the world and start thinking of online as THE way to do research, not just the NEW way to do research.
Comments?
Tuesday, May 17, 2005
The Case for Online Focus Groups
In his new book, Blink, Malcolm Gladwell threw down the gauntlet on decades of market research tradition with the assertion that focus groups should be banned. As intended, Gladwell’s challenge provoked controversy. A Reveries.com survey put the question to marketers: “Should focus groups be banned?” 500 marketers responded, the most the site has experienced; nearly two-thirds said “NO!” Only 10% agreed with Gladwell.
Based on the written comments, it is tempting to conclude focus groups are a research technique marketers love to hate. While we aren’t quite ready to ban them, ‘dislikes’ outnumbered ‘likes’ by a wide margin and the ‘dislikes’ were strongly negative.
Focus groups are described as ‘unscientific’, the results are considered untrustworthy, and even misleading due to manipulation by less than honest or domineering respondents, bad moderators, and agenda-driven clients. Small sample sizes, social desirability pressures, and sterile settings are all cited as reasons for questioning the validity of focus group findings. They are described as artificial settings where respondents say what they think moderators want to hear, where moderators say what they think clients want to hear and clients hear only what they want to hear. Add to that the inconvenience of travel to distant cities, bad food, and too many M&M’s, and one has to ask, ‘Who would use such a technique?”
Plenty of companies. There are over 400 focus group facilities and 131 professional moderators listed in the U.S. (the actual number is probably much higher). Focus groups are fast and easy to execute. The flexible format makes them ideal for exploratory work on a wide variety of issues from product design to positioning.
While no one is ready to throw out the M&M’s with the candy dish, it is time to look at what can be done to restore rigor and credibility to focus groups. Some steps are obvious – stop overuse, improve moderator training, and increase objectivity. Here’s a less obvious solution: move focus groups online.
At 75% of US adults, the online population is arguably more representative than the telephone population (or at least the population that will answer). That insight is prompting a revolution in survey techniques. According to Inside Research, in 2004, 24% of all U.S. research revenue came from online studies. While to date, most of that revenue is likely from surveys, there is also a growing movement toward using online technology for qualitative research as well. As one of the first companies to embrace online qualitative research, our experience is that they are more better recruited and less susceptible to domination by individual clients or respondents. Here’s why:
Setting: Less artificial as respondents are in their own home, office or library rather than a sterile focus group facility. Observing from any Internet enabled location increases client participation.
Sample: Geographically dispersed and conform to more precise recruiting specifications. Even hard to find populations can be identified and recruited. There are fewer ‘professional’ respondents since recruiting is not limited to those living near a facility.
Results: Loud-mouths have a harder time dominating in a forum where everyone speaks at the same time and at the same volume. Anonymity reduces social pressure.
“Hybrid’ designs’ where qualitative and quantitative techniques are ‘blended’ reduce overall costs by sharing the burden of recruiting across two studies. For example, in a recent study a survey was used to identify cross channel shopping habits major retailers. Focus group respondents were selected based on their survey responses. The resulting samples were unusually homogeneous, allowing nuances in behavior and attitudes to be identified. A similar was used by Whirlpool to study proposed designs for a long purchase cycle durable good. Two geographically dispersed groups meeting exacting specifications were recruited and sent prototypes of a new gas cook top knob. Interviews showed respondents to be very enthusiastic to the knob, but not to the overall design of the range.
Beyond addressing the shortcomings of traditional groups, online groups offer some unique advantages. Interactive collaborative tools -- such as trips to the web, projective techniques, elimination games and whiteboard exercises -- make the experience fun and involving. Consequently, a one hour group often yields as much content as two ‘in-person’ hours.
It’s time to put the focus back into groups. Online technology promises to be one way to do it.
Based on the written comments, it is tempting to conclude focus groups are a research technique marketers love to hate. While we aren’t quite ready to ban them, ‘dislikes’ outnumbered ‘likes’ by a wide margin and the ‘dislikes’ were strongly negative.
Focus groups are described as ‘unscientific’, the results are considered untrustworthy, and even misleading due to manipulation by less than honest or domineering respondents, bad moderators, and agenda-driven clients. Small sample sizes, social desirability pressures, and sterile settings are all cited as reasons for questioning the validity of focus group findings. They are described as artificial settings where respondents say what they think moderators want to hear, where moderators say what they think clients want to hear and clients hear only what they want to hear. Add to that the inconvenience of travel to distant cities, bad food, and too many M&M’s, and one has to ask, ‘Who would use such a technique?”
Plenty of companies. There are over 400 focus group facilities and 131 professional moderators listed in the U.S. (the actual number is probably much higher). Focus groups are fast and easy to execute. The flexible format makes them ideal for exploratory work on a wide variety of issues from product design to positioning.
While no one is ready to throw out the M&M’s with the candy dish, it is time to look at what can be done to restore rigor and credibility to focus groups. Some steps are obvious – stop overuse, improve moderator training, and increase objectivity. Here’s a less obvious solution: move focus groups online.
At 75% of US adults, the online population is arguably more representative than the telephone population (or at least the population that will answer). That insight is prompting a revolution in survey techniques. According to Inside Research, in 2004, 24% of all U.S. research revenue came from online studies. While to date, most of that revenue is likely from surveys, there is also a growing movement toward using online technology for qualitative research as well. As one of the first companies to embrace online qualitative research, our experience is that they are more better recruited and less susceptible to domination by individual clients or respondents. Here’s why:
Setting: Less artificial as respondents are in their own home, office or library rather than a sterile focus group facility. Observing from any Internet enabled location increases client participation.
Sample: Geographically dispersed and conform to more precise recruiting specifications. Even hard to find populations can be identified and recruited. There are fewer ‘professional’ respondents since recruiting is not limited to those living near a facility.
Results: Loud-mouths have a harder time dominating in a forum where everyone speaks at the same time and at the same volume. Anonymity reduces social pressure.
“Hybrid’ designs’ where qualitative and quantitative techniques are ‘blended’ reduce overall costs by sharing the burden of recruiting across two studies. For example, in a recent study a survey was used to identify cross channel shopping habits major retailers. Focus group respondents were selected based on their survey responses. The resulting samples were unusually homogeneous, allowing nuances in behavior and attitudes to be identified. A similar was used by Whirlpool to study proposed designs for a long purchase cycle durable good. Two geographically dispersed groups meeting exacting specifications were recruited and sent prototypes of a new gas cook top knob. Interviews showed respondents to be very enthusiastic to the knob, but not to the overall design of the range.
Beyond addressing the shortcomings of traditional groups, online groups offer some unique advantages. Interactive collaborative tools -- such as trips to the web, projective techniques, elimination games and whiteboard exercises -- make the experience fun and involving. Consequently, a one hour group often yields as much content as two ‘in-person’ hours.
It’s time to put the focus back into groups. Online technology promises to be one way to do it.
Monday, April 04, 2005
Different Research Methods Yield Markedly Different Results
Seeing is believing is especially true in market research. Ball State researchers have just provided solid evidence that observation research is superior to telephone or diary based methods for measuring consumer behavior.
Observation methods of measuring media usage show consumers significantly underreport their daily minutes of time by medium when questioned using diaries or phone survey.
Daily Minutes of Time Attributed To Media Via Each Method
Phone Diary Observation
Home Computer* 21 52 64
Online 29 57 78
Television 121 278 319
Books 18 17 36
Magazines 8 10 14
Radio 74 132 129
Newspapers 15 26 17
Source: Ball State University, Center for Media Design.
Not surprisingly, the least discrepancies were for media with the greatest social desirability (books, magazines) and therefore the least incentive to misrepresent the actual time spent. The implications for researchers are clear: beware! Even relatively innocuous questions such as how many minutes a day do you watch television should be regarded with suspicion and whenever possible, backed up by observation research.
We have seen evidence of social desirability repeatedly in our research. A recent project involved asking adults who in the household was responsible for financial decision-making. Both men and women answer in the affirmative. Further questioning reveal that men are more likely to THINK they are the decision-maker, while women actually handle more of the day-to-day financial tasks and decisions.
Observation methods of measuring media usage show consumers significantly underreport their daily minutes of time by medium when questioned using diaries or phone survey.
Daily Minutes of Time Attributed To Media Via Each Method
Phone Diary Observation
Home Computer* 21 52 64
Online 29 57 78
Television 121 278 319
Books 18 17 36
Magazines 8 10 14
Radio 74 132 129
Newspapers 15 26 17
Source: Ball State University, Center for Media Design.
Not surprisingly, the least discrepancies were for media with the greatest social desirability (books, magazines) and therefore the least incentive to misrepresent the actual time spent. The implications for researchers are clear: beware! Even relatively innocuous questions such as how many minutes a day do you watch television should be regarded with suspicion and whenever possible, backed up by observation research.
We have seen evidence of social desirability repeatedly in our research. A recent project involved asking adults who in the household was responsible for financial decision-making. Both men and women answer in the affirmative. Further questioning reveal that men are more likely to THINK they are the decision-maker, while women actually handle more of the day-to-day financial tasks and decisions.
Tuesday, January 18, 2005
Online Research Reveals Mysteries of ROI?
Why didn't we think of that? Media Post reported today (1.17.05) that BIGresearch simply ASKED 14,000 online consumers to tell them which media they find most influential for purchases across a variety of categories. Ask and you shall be told, apparently. "SIMM relies on real consumers to determine the value of each media for their purchasing decisions," said Dr. Joe Pilotta. "If consumers are not receptive to a media, blanket exposures cannot make it influential."
Pilotta claims most marketers have so far "relied upon abstract ROI and optimization models as surrogates for real consumers' media behavior." BIGresearch collaborated with renowned academics Don Schultz and Martin Block or Northwestern University's Medill School. Among other things, the system shows that "word of mouth" is more influential than advertising media for product categories such as electronics and home improvement products, while broadcast TV dominated a category like apparel products (see table below).
Media That Influencing Consumer Purchases
Electronics Apparel Grocery Home Improvement
Word Of Mouth 35.0% 26.1% 30.8% 26.4%
FSI 26.4% 27.6% 40.2% 21.7%
Broadcast TV 23.6% 19.7% 22.9% 19.0%
Internet 19.0% 11.4% 8.9% 7.9%
Email 16.7% 12.7% 8.7% 6.8%
Coupon 14.1% 16.7% 52.2% 11.2%
Source: BIGresearch Simultaneous Media Survey. Derived from interviews with more than 14,000 consumers, measuring their consumption of 28 different media, which was aligned with their retail store shopping behavior, product consumption and media influence.
Tuesday, December 14, 2004
Online panels are superior to telephone surveys. Those who are still doing surveys the old fashioned way are paying a high price, and not just in dollar terms. Telephone surveys have always been suspect from a representation perspective. Who really believed that they ever represented a 'random sample? Now with many respondents believing that do not call applies to polls as well as sales calls, 8% of the population completely disconnected from a landline, do not call lists now extending to cell phones, anyone who uses the phone for survey recruiting should beware.
That's not to say online panels are without their problems as well. I can speak from first hand experience that online panels are not all of equal quality. In general, I advise clients to stick withpanels that use email over Internet, have enough representation of young men and minorities to be useful and balance their samples mathematically to census figures. Beyond that some good indications of quality are response rates. Panels that are 'grabbing whoever they can and pelting them with questionnaires probably are going to have lower response rates. Another indicator is how respondents are compensated. You don't want professional respondents.
Finally ask whether the panel allow you to recruit qualitative research respondents from among survey respondents? This powerful technique is quickly becoming popular as a way to get extra insights about difficult to recruit for populations such as concept acceptors or super heavy users.
I am aware of just one panel that meets all of these criteria, BIGresearch. Unfortunately, their panel is not available for custom research unless you are already a subscriber to one of their syndicated panel products. If you decide to go with a self-serve panel such as those offered by Survey Savvy, Zoomerang, Harris or Greenfield, just be sure you know what you are getting. And (hint) be sure to check the back-end analytic software.
Carol
That's not to say online panels are without their problems as well. I can speak from first hand experience that online panels are not all of equal quality. In general, I advise clients to stick withpanels that use email over Internet, have enough representation of young men and minorities to be useful and balance their samples mathematically to census figures. Beyond that some good indications of quality are response rates. Panels that are 'grabbing whoever they can and pelting them with questionnaires probably are going to have lower response rates. Another indicator is how respondents are compensated. You don't want professional respondents.
Finally ask whether the panel allow you to recruit qualitative research respondents from among survey respondents? This powerful technique is quickly becoming popular as a way to get extra insights about difficult to recruit for populations such as concept acceptors or super heavy users.
I am aware of just one panel that meets all of these criteria, BIGresearch. Unfortunately, their panel is not available for custom research unless you are already a subscriber to one of their syndicated panel products. If you decide to go with a self-serve panel such as those offered by Survey Savvy, Zoomerang, Harris or Greenfield, just be sure you know what you are getting. And (hint) be sure to check the back-end analytic software.
Carol
Saturday, November 27, 2004
Closing of the Digital Divide
Years from now, 2004 will be seen as a watershed for the Internet. It will be known as the year that the demograhpic differences that defined the online world from the unconnected world literally vanished. Those of us who conduct market research online felt the change as it happened. In the old days of 2003, we recruited focus groups based on how long a respondent had been online, this no longer was important. Newbies and old hands responded the same. It seemed the internet population was more and more, like, well like us!
Now there is data to support this conclusion. According to Dr. Jeffrey Cole is Director of the USC Annenberg School Center for the Digital Future, the founder of the World Internet Project (reported first on iMedia) the differences between new and experienced Internet users have almost disappeared and Internet users look more like a subset of the US population as a whole.
In the past, the most experienced users connected over twice as long as the newcomers and were far more likely to be connected through a high-speed connection. Long-time users also connected from more places, both inside and outside the home. New users were much more likely to be looking at chat rooms, playing games and searching for entertainment information and searching for medical information. Experienced users spent much more time than novices buying online, doing work related to their jobs and looking at news online. Fours years ago the average new Internet users did not make an online purchase until they had been online between 18 and 24 months. Prospective shoppers four years ago also did not buy online because they feared the product would not be delivered or would be delivered damaged.
Sometime in late 2003 or early 2004 everything began to change. Now Although long-time users still connect longer, new users are only slightly more likely to be looking at chat rooms or playing games online, and they are Today, Internet users are no longer a subset of America as a wholejust about as likely to be looking at news, entertainment information or doing work related to their jobs. Today, new users buy online almost from the day they get connected. The 18- to 24-month lag period is gone. Both Internet users and non-users believe that prices are lower online and that the availability of products is greater.
The most likely cause four years ago for the vast differences in Internet use by experienced and new users was demographic differences. The earliest Internet users were much more likely to be white or Asian, highly educated, male and with higher incomes. They were also much more technologically inclined. Over the past four years, more and more of America has gone online, with the fastest growing groups being African-Americans and Latinos, females, lower income and those with less education. New users go online knowing what to expect from the Internet having, in many cases, been online before with a friend’s or relative’s connection. New users get down to business much faster than new users of several years ago, with less exploring and experimenting.
The implications - for polling, for research, for online commerce and more are enormous. But in the short-term, I am just happy to be able to answer the inevitable question, 'but is my audience online?'
Now there is data to support this conclusion. According to Dr. Jeffrey Cole is Director of the USC Annenberg School Center for the Digital Future, the founder of the World Internet Project (reported first on iMedia) the differences between new and experienced Internet users have almost disappeared and Internet users look more like a subset of the US population as a whole.
In the past, the most experienced users connected over twice as long as the newcomers and were far more likely to be connected through a high-speed connection. Long-time users also connected from more places, both inside and outside the home. New users were much more likely to be looking at chat rooms, playing games and searching for entertainment information and searching for medical information. Experienced users spent much more time than novices buying online, doing work related to their jobs and looking at news online. Fours years ago the average new Internet users did not make an online purchase until they had been online between 18 and 24 months. Prospective shoppers four years ago also did not buy online because they feared the product would not be delivered or would be delivered damaged.
Sometime in late 2003 or early 2004 everything began to change. Now Although long-time users still connect longer, new users are only slightly more likely to be looking at chat rooms or playing games online, and they are Today, Internet users are no longer a subset of America as a wholejust about as likely to be looking at news, entertainment information or doing work related to their jobs. Today, new users buy online almost from the day they get connected. The 18- to 24-month lag period is gone. Both Internet users and non-users believe that prices are lower online and that the availability of products is greater.
The most likely cause four years ago for the vast differences in Internet use by experienced and new users was demographic differences. The earliest Internet users were much more likely to be white or Asian, highly educated, male and with higher incomes. They were also much more technologically inclined. Over the past four years, more and more of America has gone online, with the fastest growing groups being African-Americans and Latinos, females, lower income and those with less education. New users go online knowing what to expect from the Internet having, in many cases, been online before with a friend’s or relative’s connection. New users get down to business much faster than new users of several years ago, with less exploring and experimenting.
The implications - for polling, for research, for online commerce and more are enormous. But in the short-term, I am just happy to be able to answer the inevitable question, 'but is my audience online?'
Digital World Goes Mainstream
The Best Data Wins
According to a Nov. 5 Wall Street Journal article, “Polls Taken Just Before Voting Hit Target On US Pres”, “elections are one of the best chances to measure polling results against reality.” That is precisely the reason as a market research professional I closely followed the polling results. My conclusions are 1) that online polling beats telephone polling and 2) sample size matters – a lot.
The most accurate predictions were all from online polls. Telephones, and the people who answer them, do not reflect the population at large as well as the Internet population. Within the online world, the polls that got it right had the biggest sample sizes. Only one poll got it exactly right – BIGresearch of Columbus, OH. BIGresearch’s October survey tapped over 8000 online respondents, balanced 14 ways against the Census population. The data said 51% Bush, 46% Kerry and 3% Nader among likely voters who are decided, a close match to the final actual result of 51% Bush, 48% Kerry and 1% Nader. (http://www.bigresearch.com/samples/BIG-president-final.pdf)
If the results of all the other polls had been aggregated to create a similar sample size, similar results would likely have emerged. Given this kind of validation, it is little wonder that corporate America is seeing a surge in large sample, online market research techniques. In times of uncertainty, the best data wins.
Carol
According to a Nov. 5 Wall Street Journal article, “Polls Taken Just Before Voting Hit Target On US Pres”, “elections are one of the best chances to measure polling results against reality.” That is precisely the reason as a market research professional I closely followed the polling results. My conclusions are 1) that online polling beats telephone polling and 2) sample size matters – a lot.
The most accurate predictions were all from online polls. Telephones, and the people who answer them, do not reflect the population at large as well as the Internet population. Within the online world, the polls that got it right had the biggest sample sizes. Only one poll got it exactly right – BIGresearch of Columbus, OH. BIGresearch’s October survey tapped over 8000 online respondents, balanced 14 ways against the Census population. The data said 51% Bush, 46% Kerry and 3% Nader among likely voters who are decided, a close match to the final actual result of 51% Bush, 48% Kerry and 1% Nader. (http://www.bigresearch.com/samples/BIG-president-final.pdf)
If the results of all the other polls had been aggregated to create a similar sample size, similar results would likely have emerged. Given this kind of validation, it is little wonder that corporate America is seeing a surge in large sample, online market research techniques. In times of uncertainty, the best data wins.
Carol
Subscribe to:
Posts (Atom)
