top of page
Search

AI Role-Play for Training: How to Build a Practice Conversation That Talks Back

Writer: Odin Training
Odin Training
6 days ago
5 min read

Anyone who has run scenario training knows the role-player problem. You need someone to play the driver, the complainant or the difficult employee, and it is usually the same three people, who are usually needed somewhere else. Online learning never solved this. It went around it, with branching scenarios where learners click a choice instead of producing a response.

That constraint is loosening. AI role-play lets a learner speak with a character who answers back, then receive a debrief. This post covers what the evidence says, how to design a role-play character that is worth talking to, where AI scoring goes wrong, and what it costs. It draws on a voice-based traffic stop I had AI build and test over one week.

You can try the scenario here.

What AI Role-Play Actually Changes

The practical shift is that a conversation can now be practised alone, repeatedly, on demand. A vendor guide to AI role-play for L&D leaders reports figures such as 90% of learners finding it less stressful than live role-play, but those numbers are self-reported by the vendor with no methodology given, so treat them as marketing rather than evidence.

Independent evidence is thinner but pointing the same way. A case study of 171 learners using AI digital role-play for leadership conversations, published in the International Journal of Advanced Corporate Learning, measured engagement, behavioural performance and satisfaction, and the authors describe the approach as effective for soft skills where practice opportunities have historically been scarce. Those are the authors' own conclusions from an abstract, so read the full paper before citing a result.

Law enforcement is paying attention too. Fort Worth police are testing an AI simulator developed at UT Arlington that plays subjects at different difficulty levels and scores each officer statement as positive or negative. The officer involved says that without a human role-player, training is easier to run often.

Design the Character Before You Touch Any Tool

The traffic stop I built works because of its design, not its technology. The driver, Mark, is on a rural Alberta highway, clocked at 128 km/h in a 100 zone. He has a realistic reason for being there, and he is hiding things. His insurance card expired six days ago. There is an open can in the console and a duffel bag on the back seat. He will explain any of it if asked well, and none of it otherwise. His behaviour also shifts with the learner's tone: respectful officers get an apologetic driver, sarcastic ones get short answers.

That structure transfers to any hard conversation. Here is a prompt you can run in any AI tool without building anything:

"I am designing a role-play for [audience] practising [skill]. Create a character the learner will speak with. Give the character a realistic reason for being in this situation, two pieces of information they will only reveal if the learner asks a good open question, one thing that is visibly off that the learner should notice, and how the character's behaviour changes if the learner is respectful versus dismissive. Then write five criteria an observer could use to score the conversation."

Try it for a complaint intake, a supervisor's performance conversation or a next-of-kin notification rehearsal. The output is a role-player brief you can hand to a person tomorrow. If it works with a person, it will work with an AI.

What Went Wrong, and Why It Matters

Most of my build problems were visible. A police car parked facing the wrong way is obvious. A laptop that folded itself in half is obvious. The character problems were more instructive. Early versions of Mark volunteered his expired insurance before anyone asked, which defeats a scenario about asking. The fix was an explicit section in his instructions headed "Mark does not volunteer information," with examples of correct and incorrect responses.

The failure that matters came from the AI coach, Sergeant Whitford. In one test run she told me Mark had mentioned being up since 4:30 that morning. He never said it. The detail was in her background notes, and she presented it as something that happened in the conversation. She also scored me five out of six when the transcript showed I had earned six.

An AI coach can sound completely confident while describing evidence that does not exist. Research on generated feedback supports the concern. A 2024 study on the faithfulness of LLM-generated feedback found hallucinated content in roughly 27% of sentences from a fine-tuned model and 23.5% from a prompted ChatGPT-4 system. Those are older systems on a different task, so the exact rates will not match yours, but the lesson holds.

For a law enforcement audience the parallel is uncomfortable. Training spends enormous time on the difference between what was observed, what was said and what was inferred, because reports, courts and complaint investigators examine that difference. A coach that blurs it teaches the opposite, with the authority of a sergeant.

The fix is a rule at the top of the coach's instructions: evidence comes only from the conversation. Missed clues are framed as what the learner would have found if they had asked. Even so, have a person compare a sample of transcripts against the scores before any cohort uses it.

What It Costs and Where It Falls Short

My demo was cheap. I started on the ElevenLabs Starter plan at $6 US a month, then moved to the Creator plan at $22 US a month before launch so the demo would not run out mid-conversation. Each full run, stop plus debrief, takes about three and a half minutes and uses roughly 900 to 1,600 credits. Hosting was free. Plan prices change, so check current pricing before you budget.

A public demo and a cohort are different arithmetic. Twelve supervisors doing three practice runs each is 36 conversations a month. Price it per learner before you promise it.

There are real limits. Conversations are recorded and stored by a US company, so a government or police client should run a privacy review, and real incident details should never go into a character's instructions. It needs a live connection and a microphone, which some agency networks block. It can be launched from an LMS but is not a drop-in SCORM package. And it is practice, not certification. Mark is convincing for five minutes, but he is no substitute for a live scenario with physical stressors.

Where to Start

Start with the design. Pick one conversation your learners find hard, write the hidden information and scoring criteria using the prompt above, and run it with a person first. You will learn what a good transcript looks like before you ask an AI to judge one. Then add the technology, and keep a human checking the scores until the pattern has earned trust.

Want to Build These Skills With Your Team?

I offer private 4-hour virtual workshops designed specifically for training developers. We work through your department's actual projects using multiple AI platforms, so participants leave with practical skills and working materials, not just theory.

Format: Private virtual sessions for up to 20 participants

Investment: $2,000 USD / $2,500 CDN per workshop

Sources

A Note on AI Use

This post was researched and drafted with AI assistance, then reviewed and edited for accuracy and voice. All practical recommendations reflect my own instructional design experience.

 
 
 

Comments

bottom of page