All projects: Gel, Jobs, Gootodo, Games, Uncle Mark, Goovite, Blog, Bit Literacy
Focus groups on Broadway
The New York Times reported yesterday that focus groups have now arrived in the theater business. Focus groups are a popular research method; companies spend hundreds of millions of dollars a year on them, generally run by one moderator asking questions to a panel of strangers. Group dynamics among the respondents, and (often) leading questions from the moderator, make focus groups a poor choice for understanding the customer experience.
But now traditional market research is invading Broadway, tempting producers to take artistic direction from surveys, highly directed group interviews, and a worldview that gave us elevator music, Garfield movie sequels, and other masterpieces in blandness.
What really caught my eye in the article was the case study of one recent project. The research giant Nielsen (they of consumer TV ratings) analyzed the audience makeup of a performance of "Wicked," a popular Broadway musical. It's a good example of a poor research investment.
From Nielsen Brings a New Marketing Strategy to Broadway - New York Times:
[T]he arrival of a market research powerhouse like Nielsen is a long-awaited development. Which brings us back to "Wicked." At six performances in March, the Live Theatrical Events staff covered the 1,933-seat Gershwin Theater with questionnaires, getting about 6,100 back. That assumption about all the adolescent girls, as it turns out, was false. In a report that ran 30 pages, Mr. Craig said that "in line with typical theatergoers, audience members were slightly more apt to be 35 and older."
There's an error here. Can you spot it?
Unless they failed to mention some other key data, the conclusion is baseless, very possibly false. They covered roughly 12,000 seats with questionnaires, and about 50% came back, a good response rate. After analyzing the data, the consultants found the unexpected: instead of being mostly adolescents, "audience members were slightly more apt to be 35 and older." Please tell me this isn't the level of data analysis theater producers are paying for. Obviously, the data could be affected by self-selection on the part of the attendees. Adolescents may have ignored the surveys on their chairs, and thirty-somethings may have filled in the surveys at a much higher rate; so even if adolescents filled most of the seats, the surveys still could have come back with a majority of thirty-something responses.
There's an easier way to get the data on audience makeup: watch who comes in. It's not like the people are hard to count - they're sitting in the theater for two hours! Companies often resist direct customer observation - it's decidedly low-tech, for one thing - yet here it would have offered a more accurate data sample, quicker and cheaper.
Also note that the article seems to praise the analyst for creating "a report that ran 30 pages." Is 30 pages good in itself, just because of the "plop factor"? The bigger the sound when the report hits the desk, the more impressive it is? Of course not. Executives investing in research should be looking for understanding, not reams of data.
Thankfully, there was a good counterpoint - or voice of reason, I'd call it - included in the article.
Jeffrey Seller, a producer of "Rent" and "Avenue Q," said his one experience with focus groups a few years ago had been enough to convince him that market research is, on the whole, worthless. A focus group indicated that "Rent," six months after its New York opening, had no brand recognition in Chicago - an indication that was proved wrong when the show sold half a million dollars in tickets the first day they were on sale there. ... "'Chorus Line,' 'Annie,' 'Hair,' 'Les Misérables,' 'Rent,'" Mr. Seller added. "Were any of those shows built by focus groups?"
I agree that Broadway Shows should not come out of focus group research. Although after moderating several there might be a plot line for a comedy. Let's remember there are many research tools out there. Let's hope/pray that the reseacher and client both know which one to use at the appropriate time. (My presonal history: surveys, focus groups, usability and listener labs.)
While any methodology can be flawed, I am not sure it is fair to condemn focus groups en masse. The "focus group" is often a catch phrase for qualitative methodology which includes the customer observation (called ethnographic research) you recommend. Ethnographic research is usually conducted while a consumer is using a product/service "in context". Your recommendation, while I like its straightforward simplicity, “There's an easier way to get the data on audience makeup: watch who comes in." could also be terrifically flawed.
How many adolescent teens at say, 12-15, can you readily identify as a teen-age girl? As you are already aware, more and more, teenagers are defying categorization. Remember, there are almost 2000 people coming through the door at curtain call. I am not sure that “counting adolescents on-site” is a reliable methodology either! Professional researchers and their client have presumably (we can only hope!) had discussions with regard to an appropriate mix of methodologies to deploy to arrive at the data the client is seeking. Furthermore, rarely is just one research protocol used exclusively.
Since this Good Experience newsletter deals with market research, which is what my company does, I feel compelled to respond.
I agree that AC Nielsen's methodology when surveying Gershwin Theater attendees was flawed. It suffered from what we call "response bias" (i.e., the people who responded to the questionnaire are probably not representative of the population being assessed.) However, the quote included from Mr. Seller, "market research is, on the whole, worthless" I believe is a bad message to put in front of readers.
A hundred year old, $12B a year industry must add some value to its customers, or surely it wouldn't still be in existence. Research done right can add a great deal of insight to business decisions, even in the performing arts industry.
This reminds me of a wonderful NPR piece from several years ago, in which a market research company surveyed the pop-music listening public and determined the ideal song style, topic, and length. They fed this information to a few songwriters, who created "surefire" hits and also, hilariously, songs certain to flop. Needless to say, the "hits" were bland and uninspired, while one of the flops, an operatic ballad about cuticles, or something like that, smacked of genius.
Market research produces me-too stuff. In consumer products, another flavor of toothpaste isn't the worst thing in the world, I suppose, but in the arts, following the mob means giving them more of what they already had last month.
Market research in the arts can determine schedules, ticket sale strategies, advertising choices, even casting. But content? I might as well stay home and watch reruns of shows that at least were original once.
I work in politics where we don't do focus groups as such. We do polling and analyze voting patterns from data gathered from previous election cycles. But that's as close as we get because focus groups are a time consumptive process (96 days left until the general election - not sure why I think I have the time to respond to this, much less think about focus groups). So we do a lot of on-the-spot profilng based upon instinct and experience. So I understand the inclination towards eyeballing it.
But as we know in politics, it's an uncertain science. Especially these days with age. My mother was fond of saying, when she turned 40 and people were astonished at her ageless face, "this is what 40 looks like." Gender is frequently a fluid line. And race. And economic status - no way. What about the millionaire next door who got his money when his mother won the lottery and died suddenly under mysterious circumstances?
Why not do a focus group for marketing purposes if you have the time & money? Certainly, it shouldn't drive content. The purpose of art is not necessarily to entertain. And it's not like you're going to be able to tell an artist what to do anyway. But Circe de Soleil (Ka) in Las Vegas has performers who double as ushers, making one's experience a remarkable organic whole. Make the focus group part of the experience by using people like that who ask questions in interesting ways.
Might even be able to make the experience better while you're learning more about the people enjoying it.
I am so disturbed by the tone of this article. Condemning research and focus groups in particular on the basis of the faulty research planning and methodology of a couple of ineffective analysts and blinded clients is just irresponsible. And, quite frankly, it's disrespectful to the thousands of knowledgeable researchers who read your newsletter. Could the article have been more one-sided?!
I am a professionally trained focus group moderator who also does quantitative research. I make it a point to avoid leading questions -- to the point that I've had to argue with my clients on occasion. Many of us are ethical and take great pains to ensure data and process integrity.
Research is helpful to ALL disciplines when planned and completed with integrity, focus and accuracy.
While the tone of the article is a bit harsh it does bring to light two things:
1. Misapplied data is a bad thing.
2. Creative brilliance does not result from researching the masses but from the vision of an individual or small group.
As we have heard from research professionals already on this thread, well done research can compliment any endeavor but misapplying the results of that research can lead to bad, if not horrible, conclusions.
But truly innovative ideas, products, songs, Broadway shows, etc. tend to occupy an “idea space” that is sparsely populated and can be misunderstood or difficult for the general population to understand. For example, how could a focus group predict the success of The Blueman Group? How do you explain it to them? Can you capture the essence of the performance without actually seeing it? Creative people find themselves forging ahead in spite of pressure from people/institutions to conform and might say things like “market research is…worthless” because during the creative process they are successful in the face of pressure to change or modify their vision.
(my background: Technology Product Manager. I have used focus groups and other market research tools but am not a professional researcher.)
Typically, people who dislike market/ing research have one good story about how it once failed them, and they tend to generalize about why they don't need it or why it's without merit (e.g., Avenue Q example above).
However, those people often fail to point out about all the times when market research was valuable, when it does provide useful insights, and times when it will be useful. Why? Simply put, out of ignorance: a kind of bounded rationality dilemma--nobody can know everything, so why to people always try to convince others that they do?
Mark takes issue with focus groups. I've heard pretty much the same argument for surveys and other quantitative techniques. I often have heard from developers and executives alike that they know exactly what the market wants.
But, let's keep this in perspective: I'm pretty sure that somebody out there has a problem with "listening labs." So, why not try to keep an open mind and learn about something more than what is only thought to be known or well-understood?
Focus groups are one only form of research that yeilds consumer insights. In this instance, clarifying business objectives is paramount. If the business objective were to increase theatre attendance, did the participant sample represent the target market? Hence, to identify the variables that prevent would-be theatre goers from attending would be most effective in making strategic changes increase sales. Variables worth examining include price, location, topic, and competitive entertainment options at the same price point. Nonethless, as noted by other experts in this forum, focus groups bring a clear value to the research process. Especially uncovering hidden consumer insights not revealed in quantative data.
[My Focus Group Moderator]