There’s a common practice in conversation design, where a design team will role play a conversation with the intention of identifying common conversational patterns, and distilling that down into an ideal ‘flow’ or ‘happy path’. You’d then use this as your first-pass design and iterate from there. Is this practice still relevant today? Honestly, I don’t really think so.
Many of you may have done this exercise. Heck, I’ve even taught this to people back in the day; as many conversation design training courses still do; and I’ve used it countless times myself when scoping and designing a conversational user interface or chatbot.
Over the years, though, I’ve found it less worthwhile for a number of reasons, which I’ll get into here, as well as tell you a few things you can do instead to properly scope out the conversations to be had
What is role play and why is it used?
If you don’t know, role play is where two designers would sit on a chair each, facing away from each other, and simulate a conversation around a specific use case. An observer would take notes to document the pathways the conversations take, as well as any notable events, both good and bad. Ideally, you’d have had a discussion beforehand to at least get together a broad scope for the conversation: these are the needs the user has, the goals they’re trying to complete, the content or capabilities the system would have access to, and so on.
The first few times you do this, it’s awkward. Everything is all over the place. However, with practice, it gets a little better. By the end of a few rounds of role play, so the theory goes, you’d have your happy path: the ideal journey through a conversation. From there, you can start prototyping and iterating your design.
The issues with role play
- It’s not for everyone. Some people don’t like the pressure of performing in front of their colleagues. This is amplified if they don’t know the subject matter. The exercise is supposed to be fun, and it can be, but after running countless workshops with this exercise, I can say that some people just don’t like it.
- It’s done with the wrong people. These exercises are often conducted by the design team, with the design team. A bunch of designers sitting in a room theorising over what an interaction should be like is like sailing without a compass, blindfolded. As an absolute minimum, you need subject matter experts in the room that know the use case. They’re the only ones who can often finish a conversation. Otherwise, you have an isolated group of people with no knowledge over the use case stumbling through assumptions.
- It’s not realistic. The main downside of them all is that it’s just not realistic. I had a conversation recently with Liz Stokoe,which helped me unpack some of the activities we practice at VUX, and added further validation to the notion that’s been swirling around my subconscious for years now: role play isn’t realistic and isn’t as effective as other methods.
Liz was kind enough to share two studies that prove this out.
In this study of police training it found that, in simulated exercises, people exhibited behaviour that they didn’t in realistic settings, namely, being a bit ‘over the top’ and including superfluous detail. Not conducive to documenting real conversation pathways.
And, in this mystery shopper study, it found that people placing ‘fake’ calls spoke entirely differently, using different language. They had shorter call durations and saw different call outcomes. If you were to tranlate this example in to conversation design role play, you could draw the following conclusions:
- People speak differently in simulated environments. This means that, when you role play, you aren’t gathering good enough data to build a NLU model on because people aren’t using the same language as they would if they had the genuine need. A single prompt or response phrased differently can lead to the conversation going in a totally different direction. This is the risk you run with role play.
- Shorter call lengths. This shows that pieces of the conversation are missing in role played conversations (or superfluous detail added, as in the police study). You’re either going to end up with details of the conversation missing, or you’re going to have create more work for yourself by adding intents and flows that aren’t likely to actually occur in reality.
- Different outcomes. When you’re role playing, you don’t have the genuine need that a real user has, so your eye isn’t on the prize. You’re not focused on the goal of the conversation, you’re focusing on playing the part. I’ve seen this play out in reality countless times where a role played conversation finishes, and I ask the person playing the customer whether they achieved what they set out to, and the response is often “Oh, I don’t know actually (laughs)”. Without a real user with a real goal, you can often miss the point of the conversation.
These two studies might not be directly related to AI design, but they are related to human behaviour in simulation situations, and I’ve seen these conclusions play out in reality.
So what else can you do instead? I’ll cover that in the next post: 4 ways to scope the design of a conversational AI use case.
You can listen to the full podcast with Liz Stokoe on Apple Podcasts, Spotify, YouTube or wherever you get your podcasts.
