UX Research for Conversational AI Experiences: The Definitive Guide

User experience (UX) research is the process of uncovering user wants, needs, pain points, motivations, and behaviors. The goal is to create data-driven, user-centered experiences that meet user needs and expectations. It should ideally be conducted from conception, continuously throughout the design process, and after launching. 

UX research is crucial for conversational AI experiences because of the layered complexity in terms of technology, devices, channels, integrations, and more within the experience and added human elements such as conversation, voice, and tone. 

When you add human elements like conversation, users expect the experience to resemble human-to-human interaction. If it doesn’t meet user expectations, it can cause more harm than good for your business. UX research, particularly user research and usability testing, will help you meet those expectations.

• User research is more exploratory in nature and typically happens at the beginning of the design process. It can help determine what problems a conversational AI experience will solve, when conversational AI is the right solution to use for the problem, how people want to interact with the experience, and define the scope of the solution.
• Usability testing happens when you’ve built something like a prototype or new feature and want to test it with users. The goal is to determine if the experience works, if it’s meeting user expectations, and to identify any issues, gaps, and opportunities to improve the experience. Usability testing should happen continuously throughout the design and development process and after launching.

UX research is often one of the first things to get cut because of budget and time, but don’t let anyone use that as an excuse to cut it out of the process completely! Doing it will save time and money in the long run. Even with minimal time and no budget or a low budget, there are still ways to gather valuable data and insights quickly and effectively. 

I’ve created a 6-step approach along with templates and tips to help you conduct UX research for conversational AI experiences. The 6 steps are: 

1. Align with your team on research or testing plan
2. Prepare for user interviews (if applicable)
3. Recruit participants (if applicable)
4. Conduct research or testing
5. Synthesize your findings
6. Present findings to stakeholders
   

Step 1: Align with your team on a research or testing plan

Creating a research plan can help you in many ways such as: 

• Getting approval to conduct research or testing.
• Improving cross-functional communication.
• Giving more visibility to stakeholders and anyone new that comes onto the project.
• Clarifying and shaping conversational AI strategy, design, and implementation.

Ideally you’ll want to collaborate with your team to help you fill out the plan. If you can’t, fill out the plan as best you can and then present it to the rest of the team to make sure you’re aligned on the details. The plan should include research goals, scope, timeline, budget, methodologies, recruitment details, and any next steps.

Goals

Include what you’re exploring or trying to uncover with the user research or usability testing. Try to tie these goals to business goals if you can.

Be as specific as you can with the goals. If anything is unclear, it will hurt the accuracy of your results and insights.    

Scope

The scope establishes clear boundaries for the depth of your research. For example, if your goal is to identify usability issues with your bot (your goal should be much more specific, but using this to demonstrate a point), you need to figure out exactly what that means to you and what success looks like for that goal. Questions to help you define that scope could be:

• Do you want to test specific flows or intents within the bot? If so, which ones?
• Could the wording and language affect usability issues? Is that something that should be tested? 
• Does discoverability of the bot fall into that scope? If users can’t discover it, they can’t use it. 
• Does anything else fall into the scope of usability issues that should be covered by the testing?
• If your bot lives in multiple places, do you want to test it in all places? On a website? On mobile? In an app?
• Is there anything the scope shouldn’t cover?  

These are the types of things you need to think about when coming up with the scope. It’s a lot easier to define when your goals are clear and specific.

Timeline

Timelines tend to be tight for UX research, but make sure to give realistic time estimates and leave enough time for planning, recruiting, preparing, conducting the research, summarizing and synthesizing the research, and presenting the findings. These are all a crucial part of the process and it takes more time than people realize to do them well. 

Budget 

Determine how much money is allotted. Once you determine the budget, plan which methodologies and the number of participants you can manage within your budget. 

If you don’t have any budget, you can still conduct research and testing using quite a few different methodologies such as user interviews and surveys, but you may not be able to get as many participants or use methodologies that take more time and effort, like diary studies.

Methodologies

The methodologies you use will depend on your budget, timeline, and where you are in the design process. More traditional options include:

• Competitive analyses 
• Surveys
• User interviews
• Card sorting
• A/B testing
• Diary studies
• Focus groups
• Customer feedback review (Anywhere you can find real customer reviews, whether that’s through a feedback form in a chatbot, a review in an app store, social media, or anywhere else)  

Options specific to conversational AI include: 

• Conversation review, meaning going through actual customer conversations to identify patterns, issues, gaps, and opportunities. Customer service call transcripts are also incredibly useful here.
• Wizard-of-Oz testing

I won’t go into more detail on each of the methodologies here because there are a ton of resources out there already, but the ones I’ve personally used the most are competitive analyses, surveys, user interviews, conversation review, and customer feedback review. Those can be all done with no or low budget and a tight timeline.

Recruitment details:

If you’re working with participants, decide: 

• How many participants are you going to need for each method?
• Are you using a recruitment platform, tool, or service? 
• If so, which ones are you going to use for each method? 
• How much do you estimate spending on recruitment for each method and does that align with the budget?  
• How long will sessions be?
• Will sessions be in person, virtual, or a combination?
• Do participants need to download anything for the session, such as Zoom or another platform? Do they need a strong internet connection? Anything else?
• On your end, are you going to have multiple people helping with the research? If so, define roles and responsibilities.
• Do you need to book a room or need anything else for a successful session?

You may not need to recruit participants if you’re only doing a competitive analysis or conversation review, so you can skip this if the methodologies you use don’t require participants.

Templates

User research plan 

Google Slides | PowerPoint | Figma

Usability testing plan 

Google Slides | PowerPoint | Figma

Note: If you’re part of a bigger team, you’ll most likely need to get approval from representatives that have influence on the decision-making process for the goal, budget, timeline, methodology, scope, and recruitment methods. The makeup of the group will be different depending on what titles and roles your organization uses and what you’re researching, but the types of people that are typically critical for these types of decisions are project or product managers, product owners or architects, conversation designers, conversation analysts, UX researchers, and product designers. For smaller teams, you might have 2-3 people covering all those roles. 

Sometimes marketing, executives, or others in the C-suite need to be involved. If that’s the case, loop them in. Optional people that should have visibility but may not be needed to make decisions are product architects, data and analytics, data scientists, developers, engineers, and QA.  

Step 2: Prepare for user interviews (if applicable)

If you’re doing user interviews, you need to prepare questions or tasks to make sure you’re getting the insights you need out of the participants. These questions and tasks should be based on your goals and scope. 

Below are a few conversational AI-specific things you may want to consider to help you come up with the right questions to ask. 

For exploratory research to define the scope of a conversational AI experience or feature:

• What are people doing before, during, and after interacting with the experience? 
  • Are they at home surfing on the web? 
    • If so, a chatbot could be a good option.
  • Are they multitasking? In a car? Cooking? 
    • If so, you may want to think about using voice.
  • Are people on the go and using different devices and modalities while interacting with the experience? 
    • If so, a multimodal experience makes sense here. 

• What is their state of mind when interacting with the experience?
  • Are they trying to get something like medical results? 
    • If so, their state of mind will be different than when they are using it to meditate, for example.
  • Are they trying to complete a time-sensitive task? 
    • This can cause stress and anxiety for people, so it’s important to understand how people might react in this situation and what you might be able to do to make it easier for them.
  • Are they in a car? 
    • Timing for notifications and responses are crucial, especially when people are driving or in other situations where distractions can potentially harm other people.
  • Find out as much as you can about how they might be feeling before, during, and after the experience. 

• How are people interacting with the experience?
  • Are they using a phone? The web? Smart speaker? Car? Watch? Glasses? Any other types of devices?
  • Can the devices and modalities change throughout the experience? 

• When do people interact with the experience?
  • Are they using it first thing in the morning? Late at night? Multiple times throughout the day?  

• What is the motivation for engaging with the experience? 
  • Are they looking to complete a specific task? Or is it more for fun?
  • Are they using it to complete a critical task? 

• Where are all possible locations people could be interacting with this experience? 
  • Are they at home? On the go? In the car? Walking? Hiking?
  • Is there going to be background noise or people talking in the background? 
    • If so, voice may not be the best option.
  • Is this experience going to be used in different countries? Different states? 
    • This is important for localization. 
  • Will people have access to the Internet?
    • This is important for accessibility and inclusivity. If not, you’ll have to think about ways to solve for internet access issues. 

• Are there any barriers to using this experience? If so, what would give people enough value or confidence to overcome those barriers and use the experience? 
  • E.g. trust, privacy, safety, accessibility, location 

• What ethical considerations might there be?
  • E.g. sustainability, privacy and security, safety, inclusivity, accessibility, transparency

For a visual representation of a bot, you’ll want to discover what user expectations are in terms of design, gender, physical features (if applicable), persona, and character. 

If you’re exploring voice, questions to consider include: 

• Should the voice conform to expectations or try to break traditional stereotypes? Is it going to potentially hurt business if it does break traditional stereotypes?
• What are user expectations of the sound of the voice? Gender? Character? Tone? Ethnicity? Pitch? Age? 
• Should there be a gender or should it be genderless?
• Are people expecting a certain ethnicity? Accent? 
• How can you make the voice as diverse and inclusive as possible?
• Should you use a synthetic or human voice?
• Do people want the option to choose what type of voice they can use?

For usability testing, you can test anything in an experience such as flows or intents or task completion. You can even test how people say things to a bot like asking for a piece of information or requesting something.

If you’re using voice, you can test different types of voices to determine if people resonate with one more than others. If your experience is multimodal, meaning you use a combination of modalities such as touch, visuals, and voice, you can ask users to interact with the experience in different situations to make sure you’re using the right modalities at the right time. You can also test voice and tone to make sure it’s coming across the way you want it to or language to see how different words affect user reactions and behaviors. 

This is by no means a comprehensive list, but it should be plenty to start with. It costs a lot of time and money to add to or rebuild an experience later, so asking the right questions and testing the right things from the start will help you make the most of the interviews and get the insights you need.

Step 3: Recruit participants (if applicable)

Your participant pool will depend on your methodology, budget, and timeline. If you’re using a quantitative method such as a survey, try to recruit as many people as you can. If you’re using more qualitative methods like user interviews or focus groups, you can gain valuable insights from interviewing even 5 people. If you have the time and budget to do more, go for it. If you can’t get that many, try for at least a minimum of 3.

If you have little or no budget, you can use friends, family, or coworkers to start. Just be mindful of company policies and NDAs and clear it with the right people first. If you end up going with friends, family, or coworkers:

• Do your best to recruit people in your target audience, but you can still gain valuable insights from people that aren’t. 
• Be careful of bias, particularly with people you know.
• Be as objective as you can with the questions and follow-up questions.
• Don’t let personal connections get in the way of the research. 

If you do have a budget, you can use sites to recruit people like: 

• Optimal Workshop
• Lyssna (formerly UsabilityHub)
• Maze
• dscout
• UserTesting
• User Interviews
• Great Question

Note: Think about inclusivity and accessibility and do your best to interview a diverse group of people within your target audience. Vary ages, ethnicities, locations, jobs, life experiences, or other factors as much as you can. Diversity of thoughts, abilities, and perspectives are much more valuable than quantity. 

Step 4: Conduct UX research

Here you’ll conduct the research using the methodologies you’ve chosen. I’ve included more
information, tips, and templates for the competitive analysis and user interview methodologies.

Competitive analysis

The goal of a competitive analysis is to evaluate where you are in relation to your competitors and find gaps and opportunities you can take advantage of that your competitors aren’t fulfilling. The level of detail for a competitive analysis depends on what you’re researching.

If you’re researching other products, services, or platforms, you’ll want to do a comprehensive analysis to see where you are in the market and what you can do to stand out. Do these every so often to re-evaluate where you stand and potentially find new gaps and opportunities. 

If you’re adding or changing a feature and want to research what competitors are doing, you can grab a few screenshots and make a few notes. It doesn’t have to be very extensive.  

I’ve found SWOT (strengths, weaknesses, opportunities, threats) analyses to be helpful here. If you can, do one internally for yourself as well as your competitors.

• Strengths: What’s successful? 
• Weaknesses: What can be improved? 
• Opportunities: Where can you stand out from your competitors?
• Threats: What barriers do you face?

SWOT analysis templates

Google Slides | PowerPoint | FigJam

Surveys

• Try to recruit as many people as you can within your target audience.
• Do your best to run your survey by a few people before you send it out. Many times I’ve sent out a survey and realized there either there were multiple questions that were too similar or I forgot to add a question that I really wanted to get insights on. 
• Keep the questions specific to what you’re researching, but broad so you get as many insights as possible.

The final tip I have is to document your findings in one place. This will make it easier to organize and synthesize the results.

User interviews

I won’t get too detailed about how to conduct user interviews since there are a bunch of resources out there already, but I’ve created a sample interview script template for both user research and usability testing with some tips from my own experience: 

Sample interview script template

For documenting and storing the actual interviews, Lookback might be worth checking out. 

Step 5: Synthesize your findings

Once you finish conducting research, gather the results together in one place. This makes it easier to find and document themes and patterns. Create categories from the groupings and organize them however you want. Those categories become your high-level insights that you can share out with stakeholders.

It’s also really helpful to document direct user quotes to use when you present to stakeholders. They can be very impactful and can help sway opinions when making decisions.

For user research, one framework you can use is Jobs to Be Done (JTBD). This framework focuses on jobs, pains, and gains and aims to get to the root of why someone wants to use a product or service. Below is an example of how I used it to conduct research for a meditation app:

There are plenty of other methods, but I’ve found this one has been the best at helping me translate user jobs directly into features. 

For usability testing, one option for synthesizing results is a rainbow spreadsheet. In this spreadsheet, you add in participant details, assign a color to each of the participants, add in the insights from the research, and then map the participants to each of the insights with a mark in the color they were assigned. 

Then you can add up how many participants matched each insight, prioritize accordingly, and come up with possible solutions and next steps. There’s also a tab for metrics in case you want to document any.

I use the following categories for the rainbow spreadsheet: 

• Errors
• Suggestions
• Observations
• Positive quotes
• Negative quotes

I’ve included an example below. In the example, there’s a column called an Error Rating, but Priority might be a more positive way to reframe it. It’s basically a way to triage and prioritize the most important insights you want to focus on. 

Rainbow spreadsheet template

It really doesn’t matter what you use as long as it works for you and your team.  

Step 6: Present to stakeholders

After you’re done collecting your insights, you’ll most likely need to translate your research findings and present them in an easily understandable format to other stakeholders. A meeting is probably the best way to do this so you can walk everyone through the research and answer any questions.

Most of the time stakeholders don’t care enough or have the time to comb through the details, so as hard as it may be, keep it brief and high-level. Try to pull out the most impactful information and then share links to the research or any other documentation in case they want to dive in deeper.

Include anyone involved with the strategy and design phases and anyone else that needs visibility. You can have a larger group of people in this meeting because it’s a share out. Leave time for questions after you present. I would also recommend giving people time to look through and process the research before starting to strategize.

Templates

User research

Google Slides | PowerPoint | Figma

Usability testing

Google Slides | PowerPoint | Figma

Presentation tips: 

• Present information in simple, bite-size chunks. 
• Try not to add too much text on slides. No one will be able to process it. 
• Use visuals when you can along with minimal text.
• Pick out only the most important information to highlight. 
• Quotes are always impactful.
• Be aware of who you’re presenting to and tailor the presentation to them

Final thoughts

This guide is only one approach based on my own experience, but I hope it gives you a starting
point for what to think about, what questions to ask, and the types of methodologies that can
help you get the best insights for your experience. Please feel free to reach out if you have any
comments, questions, or feedback!