Building future-ready talent pipelines means seeing how leaders actually behave. This paper shows how AI-based role-plays make that visible, reliably and at scale.
Assessment centres rely on trained role-players and assessors to see how leaders behave in difficult conversations. That approach is reliable but hard to scale, and it varies with the person playing the role. This pilot asked whether an AI-based role-play could do the same job across a whole cohort of first-line managers.
We went in with three questions:
| Question | What a "yes" would look like | Answer from the data |
|---|---|---|
1. Can we observe key behaviours clearly? |
Behaviours such as ownership, questioning and use of evidence show up as distinct, countable moments in the transcripts | Yes. Ownership split the cohort 46% / 54%, and questioning fell from 25% to 2% of messages |
2. Does the AI perform well and consistently? |
The character stays in role, replies at conversational speed and reacts to what each manager says | Yes. Zero persona breaks in 1,353 messages, 7-second median reply, and sentiment that tracks the manager's approach |
3. Do participants engage effectively? |
Managers stay long enough and write enough for their behaviour to be judged, without being made to | Yes. 96% held a real conversation, for a median of 23 minutes and 458 words |
An AI character ran 112 unsupervised feedback conversations with first-line managers over 52 days without once breaking role, and the transcripts show clear, measurable differences in how managers handled them.
The last row is the point. An assessor panel sees a handful of managers a day. Here, a behaviour that matters (owning a mistake) was observed across an entire cohort, in the managers' own words.
First-line managers at an IT services company each held one written conversation with an AI character: a team member disputing an "average" annual rating after receiving no critical feedback all year. The scenario is built around a failure on the manager's side, so it tests ownership, listening and resolution under mild pressure.
The deployment ran from 28 July to 17 September 2026. It was optional, unsupervised and available at any hour. The dataset holds 112 sessions and 2,706 messages, split evenly between manager and character, with zero structural defects (no missing, duplicated or out-of-order messages).
A behavioural measure is only as good as the conditions it is taken under. On every check that needs no human validation, the AI held those conditions steady.
Same character, every time. Zero breaks of persona in 1,353 AI messages, confirmed by a keyword screen and a hand-checked contradiction test. Every manager met the same person in the same emotional starting state: no rater fatigue, no order effects, no rapport bias.
Fast enough to feel like conversation. Median reply 7 seconds; 90% inside 12 seconds; only 14 of 1,353 replies over 30 seconds.
Available when managers were. Sessions ran on 43 of 52 days; one in eight ran at the weekend and 14% started between midnight and 6am.
Deep enough to produce evidence. Managers wrote 0.8 words for every word the character wrote, so the transcript captures their behaviour, not a script.
Only 4 of 112 sessions ended before three exchanges, and 29% ran past half an hour. People chose to continue rather than endured it.
The transcripts surfaced three observable behaviours, each measured across all 112 managers rather than a sampled few.
54% of managers did not acknowledge a failure of their own, in a scenario where the failure (feedback withheld all year) was objectively theirs.
The 46% who did used phrasing such as "I should have shared this earlier" or "I take responsibility for that". This measures accountability, not politeness: 83% of managers said thank you.
Managers open by asking, then switch to defending and fixing. Concrete examples peak at 44% of messages in the second fifth, mostly after the rating had already been defended.
Her negative sentiment rises early and falls to its lowest at the close, a shape four separate model measures agree on. Her sense of feeling unheard peaks at 19% of messages in the second fifth and drops to 4% by the end. In 53 sessions her sentiment shows a clear, lasting upturn at a median of exchange 7; in 15 it turns down, at a median of exchange 4. Conversations that go wrong, go wrong early.
Managers also reuse more of her vocabulary as the conversation goes on (10% overlap in the opening fifth, 20% in the fourth), a supporting signal of active listening.
AI role-play turns live interaction into evidence at cohort scale. Three properties make that evidence usable:
Every participant faces an identical stimulus, which human role-players cannot guarantee across a cohort.
Behaviours such as ownership, questioning and use of evidence appear as countable moments in a full transcript, traceable to the exact exchange.
Because the turning point of each conversation can be located, feedback can point to the specific exchange where it changed, and what the manager said there.
What this data does not yet show.
Behaviour markers here come from keyword and language-model analysis, not trained assessor ratings. Agreement with human assessors, inter-rater reliability and links to on-the-job performance have not yet been tested. Each participant took part once, so behaviour change cannot be measured.
info@thinktalentindia.com
+91-9910955257
+91-8828158509
