As a direct marketer, I’m of the opinion that focus groups are where reliable statistics go to die.
This is adequately summed up in the following example: your dog doesn’t sleep on the bed when you’re around, but as soon as you leave, that’s exactly where he heads. People behave differently when someone is watching them; it’s human nature, and focus groups are no different.
Focus groups are expensive, deficient, inaccurate, and evil. I love this position because it gets people all riled up and sparks debate on the benefits of qualitative versus quantitative feedback. When you get right down to it, it’s a matter of opinion; but, hopefully, I can persuade you to see the benefits of multivariate testing for digital experiences instead of using focus groups.
1. People behave differently when under a microscope. Ah yes, the Hawthorne Effect: The alteration of behavior by the subjects of a study because they are being observed. If you know that people are behaving differently when they are knowingly under a microscope, how much integrity does the data really have? How do you know people aren’t just there for the coffee and cookies because they have a lot of free time, and have zero intention of ever purchasing your product?
Also, if you’ve ever participated in a focus group, then you know that whoever is running the forum has a significant influence on the results, especially when there are creative people involved who already have an opinion on what direction they want to move forward with (think: Mr. Burns commenting “excellent” and tenting his fingers at the head of the room).
It’s human nature to look for those subtle signs and cater your reaction accordingly, which leads to tainted results. Online, you don’t necessarily know if anyone, or who, may be watching (actually, I just assume someone always is, but I work in that industry). As a result, my reactions are more native, less influenced, and therefore, more actionable from a marketing perspective.
2. Focus group audiences are not reflective of your actual audience, especially online. Take size out of the equation, as you simply can’t compare the impact or validity of the behavior of thousands of website visitors to however many focus group participants who:
a) Want to participate
b) Meet the selection criteria
c) Fit into conference room B that you booked for this purpose
A focus group audience is selected, and someone had to make a decision to arrive at that selection, hence an automatic bias. Additionally, it’s difficult to mimic the natural diversity of a digital audience.
On the flip side, a multivariate experiment encompasses all the actual traffic that’s arriving on your website via numerous channels and at various stages of the purchase cycle with different commitment levels. As an aggregate, those results are going to be a more accurate reflection of your true audience receptivity and behavior than anyone you pick off the street or segment from your customer database. And you don’t have to spend time educating them on why they’re there to begin with.
3. Results obtained through focus groups can’t be tied to an actual purchase. Focus group questions always seem to ride the gray line, leaving room for interpretation in the results and further impacting the validity of the data. Additionally, my experience with or exposure to your product or service does not mean I’ll necessarily buy it now or ever. Wouldn’t you rather know which elements or variations your prospective audience is responding to that actually drives them to purchase? Think of how much more useful those insights would be. Realistically, why do you care what 15 people who may never buy your product think of the creative direction?
4. Focus groups bleed money. Brand agencies love focus groups because they’re expensive and take a lot of “strategic brainpower” to plan and oversee (cough, billable hours). Plus they can spend even more hours sitting around pontificating on the sheer simplicity of the findings after the fact. Oh, to live in that world.
You can spend a lot of time and energy on focus groups only to end up marginally ahead (if at all) of where you were based on the assumptions going into it.
On the flip side, you can be up and running with a fairly simple split A/B or multivariate test experiment in a number of hours, and much more cost-effectively (some basic testing programs are even free). You can also test more assumptions over a shorter time period, depending on the amount of traffic you have driving to your site.
Forget about hand selecting your focus group. With the click of a few buttons, you can make all of your website traffic immediately part of your focus group segment. And you don’t even have to buy them lunch.
Real-world data is always better. Period.
Pippa Nutt is senior vice president, online & Canadian media, at Northern Lights Direct.
