Targeted Advisory case study

Same Behaviour, Different Ratings: Blame the Boundary

14live sessions in 5 weeks, for supervisors and for staff
360+supervisors and staff who rated the same scenarios
8scenarios, each with an agreed and a contested reading
51/44supervisors’ split at the hardest boundary, Good or Strong

The situation

The question behind the question.

This organisation had moved to a new performance framework in which half of every rating rests on behaviour. Each role type has one culture goal, rated on 5 levels, and strong results cannot lift the rating of someone whose behaviour falls short, so a manager’s reading of the scale can cap a strong performer. HR wanted managers who apply it consistently and fairly, and staff who understand how they are assessed. But a scale written in words is read in as many ways as there are readers.

As it was asked

“Train our managers and staff on culture goals.”

As we reframed it

Find where the scale splits its readers, let managers discover their own disagreements before ratings count, and show staff how they are rated and what they can do about it.

How we worked

How the work unfolded.

01

Find where the scale splits

We started from the scale, not from slides: where would 2 fair managers read the same behaviour differently? With HR’s talent partners we wrote 8 scenarios. Each had an anchor reading most people should agree on, contested readings near the boundaries between levels, and an expected vote split written down in advance, so the room’s result could be read against it. Real cases, contract consequences and conduct issues stayed out, so people could disagree safely.

02

2 tracks for 2 jobs

Then 2 tracks for 2 jobs. Supervisors calibrated: rate a scenario alone, compare in pairs, see the room’s vote, through lenses on bias, culture and neurodiversity, the last shaped with the inclusion team: am I scoring what they did, or how it looked to me? Each left with 1 evidence statement drafted and 1 personal commitment. Staff learned how the rating works from a facilitator independent of HR, with silent reflection, questions by chat and a demo of where evidence is recorded.

03

Run, then redesign

Then live, online sessions of 90 minutes rather than the 2 hours requested, because attention fades before the second hour, deliberately not recorded and grouped by sector where possible. A dry run with the talent partners came first. After the first real session we redesigned overnight: scenario first, framework second, because people engaged with a case before a definition. 14 sessions in 5 weeks reached more than 360 people in one window.

04

Read the votes

Then the votes, read as data. “Subjectivity” was the first word for both groups, from 42% of staff and 33% of supervisors, but consistency worried 13% of supervisors and almost no staff: one fear, seen from both sides of the desk. On clear cases, 2 in 3 agreed. At the line between good and strong, supervisors split 51 to 44 on the same behaviour: that boundary, not the definitions, needed calibrating. The questions raised became job aids, guides and FAQs.

The instrument

Rate, compare, reveal

Rate alone. Each manager rates the same scenario privately, before hearing anyone else.

Compare in pairs. 2 managers explain their ratings to each other, evidence first.

Reveal the room. The vote appears; the split shows where the scale needs a shared reading.

Consistency is not taught. It is discovered, one disagreement at a time.

What moved

What changed, and what didn't.

360+supervisors and staff rated the same scenarios on the same scale, in 14 live sessions
Evidence Decision taken Adopted in practice Measured impact
What moved

360+ supervisors and staff worked through the same scenarios on the same scale. HR now uses what the sessions produced: job aids for both tracks, the scenario library to calibrate ratings, guides on the culture goal and its evidence, and FAQs built from the questions people actually asked.

What didn't move

The engagement closed with 14 of the 20 planned sessions delivered: the other 6, and a session for senior leaders, were not run. The only data were the votes in the room: no check yet of how culture-goal ratings spread at year-end. Sessions ran in English only, live only, with no train-the-trainer for HR.

What we'd say to a peer

A rating scale written in words is read in as many ways as there are managers. Staff will call it subjective; managers will call it inconsistent. Before the cycle, let managers rate the same cases privately and see where they split: the hardest boundary, not the definitions, is what to train. And explain the rating to staff through someone who does not do the rating.

Let's talk Read more cases
Method note:

Own approach in 4 steps; 8-scenario library with anchor and contested readings; private rating, pair comparison and live vote; bias, culture and neurodiversity lenses; job aids, guides and FAQs.

WeAreHuman@Work

One piece of evidence on sustainable work, every Sunday. Free, and one click to leave.

Get the weekly edition