Tips on Moderating Listening Labs

Dec 7, 2004

At the heart of good customer research is a non-directed conversation which puts the customer in charge; so I wrote in October 2003 in Four Words to Improve User Research. At the end of the column I promised to write more about our research method, "listening labs" - but with the exception of some sidelong glances (Sep 22, 2004 and Aug 24, 2004) I haven't really dug in until now.

Today I cover one major part of running a listening lab: moderating. There are other important activities - working "behind the glass," where the observers sit, for example - but moderating the customer interaction is crucial.

The best way to learn moderating skills is to practice, in person, with real live customers. Still, I've put together some tips I've accumulated over the years in the hopes that they help, too. (Another opportunity for learning is our new LabFest event, which we'll run in three U.S. cities in 2005. Let me know if you want hear more - mark at goodexperience.)

Moderation is a lot harder than it looks. The moderator's job in listening labs is to facilitate the interaction, but not to lead it. A good moderator will appear, to most observers, to be doing almost nothing, while the customer clicks around and talks. The key to good listening lab moderation is to stay quiet, except in those moments where intervention is necessary. Such passivity can be especially hard for experienced moderators of traditional focus groups or usability tests, which require the moderator to take charge of the test and shepherd customers through their tasks and questions.

In listening labs, the moderator must accomplish all the goals of the lab - getting all the data and showing the customer experience to the client - without explicitly leading the process. Instead, the moderator must allow the customer to lead (within some boundaries) and still accomplish those goals.

Here are some tips to keep in mind about listening lab moderation:

- Don't start task definition until you build rapport with the customer.

As I wrote in "Four Words to Improve User Research" [1], moderators should not write out specific tasks before going into the labs. How can a moderator presume to know where, how, and why customers will use the site, without asking them in person first?

But don't misunderstand: listening lab sessions do require for users to complete tasks. It's just that the task definition takes place in the initial interview at the beginning of the lab session. This interview must proceed at a relaxed pace, allowing the customer's context to emerge.

This is one of the toughest things for a new moderator to master: not ending the interview too soon. The initial interview is intended to give the customer some time to get used to the moderator, to understand that the interaction isn't a "test," and to relax and get into a mindset of using the site in a natural manner (i.e. in the same way they would use the site from work or from home). If the moderator starts task definition too soon, the customer might be nervous, stiff, or otherwise unprepared for the open-ended interaction that follows.

- As the customer uses the site or product, don't ask leading questions or lead the customer in any way.

This is perhaps the most important rule of lab moderation, and the most difficult for new moderators to adhere to. It's simply too easy for the moderator to insert himself or herself into the lab with a comment or question ("It's not very good, is it?") or an instruction when the customer gets uncomfortable ("Just click there and you'll move to the next page"). Unfortunately, task-based usability tests can often include such leading questions, ignoring any context the customer may bring to the table, and so tainting the customer's feedback.

- Act only on the lead of the customer.

Unless the customer gives something for the moderator to respond to, the moderator must not insert any instruction into the session - except to move to a totally different step in the session. Acting only on the customer's lead is especially important because it shows the client observing the sessions that the data is coming straight from the customer, unmodified by the moderator. There is thus no possible reason, political or otherwise, for the client to doubt the honesty and accuracy of what the customer states.

- Ask the customer to repeat.

It's always OK for the moderator to ask the customer to repeat what they said. Often customers will say what they really think by muttering something under their breath - often a negative comment that they're sensitive about voicing too loudly, or not realizing the importance of what they're saying. The moderator can then, without showing too much interest, ask the customer to repeat what they said. This is partly for the moderator's benefit in taking accurate notes; it's at least as much for the benefit of the observers, who must not be deprived of the opportunity to hear a customer's honest feedback.

- Ask the customer to clarify.

Sometimes the customer will give feedback that makes sense at that moment, but won't translate well to the observation room or the discussion after the session (let alone back at the company headquarters after lab day). It's always OK to ask the customer to clarify what they meant - since this is not leading the customer to say or do anything that they didn't originate himself or herself.

- Avoid opinion-based questions.

Remember that listening labs are based on what customers do, not on what they say they feel (or what they say they do, for that matter). Therefore it's important to avoid focus group-style questions that ask customers to describe their emotional reactions to some graphical element, a "brand", or some intangible aspect of the experience. Labs must show the customer using the site in as natural a manner as possible, consistent with how they use the site outside the lab setting. Don't get us wrong; emotions are important - but generally they can be obtained by observation during the lab session, without asking the customer to break out of the natural scenario and describe their emotional state.

- Avoid conditional or theoretical questions.

Sometimes it's tempting to ask the customer what they would do if this function were available, or if that section looked different, or if if if... These tend not to be very helpful, since (once again) the lab is intended to spotlight customers' real-world actions, not their theoretical actions or reactions. The exception here is if the question clarifies something the customer has already said - i.e. if it follows the customer's lead.

- Keep the customer comfortable, but don't always avoid the uncomfortable moment.

This is a balancing act that takes moderators some time to learn well. On the one hand, customers must be at ease during the lab, in order to use the site in the way that they normally use it outside the listening lab. However, moderators are often tempted to help customers when they struggle. The moderator must do nothing and allow the customer to struggle; this is for the benefit of the client observing behind the glass. The client must be allowed to see the customer struggling. If the customer says that they give up, has tried all other options, and would normally leave the site, then the moderator should honor the customer's statement and move on to another task or site (or, if there is nothing else to accomplish in the session, end the session).

Of course, if the customer is showing visible anxiety or embarrassment, the moderator shouldn't prolong the frustration. Just allow the honest moment to occur, and then move on.

- Keep the customer in "use mode", and avoid "critique mode."

While the customer is naturally progressing through their task, it's tempting for the moderator to ask the customer to step outside the task and critique the site. "If you could redesign it, where would you put this button?" Don't do this. Customers should remain in "use mode," using the site as they would outside the lab session. Asking customers to critique the site and offer solutions breaks their natural use of the site and puts the lab session at risk.

- Feel free to ask the clients' pre-written questions at the end of the session.

Occasionally a stakeholder within the client organization will insist that each customer get asked a few survey questions. The time to defer to the client's request is at the end of the session, when the customer has finished giving their more useful feedback. After asking the final questions, the moderator can dismiss the user and walk back to the observation room for the discussion that follows each session.

---

• IN: Column Archive
• #