---
title: 5 Instrument today: behaviourally anchored questions and rubrics
---

# 5 Instrument today: behaviourally anchored questions and rubrics

## 5.1 The assessment item and what makes it work

The current instrument is the assessment item. It consists of three connected parts: a competency description, a question and a scoring rubric. The description defines which ability the item intends to measure and provides the shared reference for the candidate's reading of the task and the reviewer's reading of the answer; it works when it names one clear ability, in behavioural terms, with one main action verb, and matches what the question asks and what the rubric evaluates. The question prompts the candidate to produce evidence – actions, decisions, reasoning or outcomes – rather than opinions; it works when it is clear, focuses on one competency, asks for observable behaviour and can be interpreted by candidates from different backgrounds. The rubric gives the evaluator the criteria for interpreting the answer; it works when its levels describe observable features of responses, are ordered and consistent, and differ in ways that are clear and interpretable \[20\].

Six quality properties follow from the relationships between these artefacts: construct clarity (description → question and rubric), task accessibility (question → candidate), evidence relevance (question → answer), criteria precision (rubric → reviewer), judgement consistency (rubric → judgement) and candidate separability (answer → judgement) \[20, 21\]. They are the checklist against which generated items are reviewed, and they reappear in Part III as the properties any new evidence type has to satisfy before it is scored.

![Figure 7 – The assessment item as a chain from competency description to reviewer judgement, with the quality property each link carries.](../media/image8.png)

*Figure 7 – The assessment item as a chain from competency description to reviewer judgement, with the quality property each link carries.*

## 5.2 Generating assessment content

Items are produced by the Generation Service. Its tasks are grounded: a task carries the prompt, the configuration and the ontology grounding that make its output reproducible, in contrast to the four raw model endpoints (generate, complete, embed, list models) which carry none of it and exist for trying a prompt. The tasks are generate_vacancy_text, generate_vacancy_question, generate_scoring_rubric and generate_interview_questions with its scoring rubric, and for the first three a feedback counterpart that judges an existing text instead of producing one.

Configuration lives in a config store per tenant – prompt, model, thresholds – and the service reports which configuration answered in response headers (X-Config-Tenant, X-Config-Source), not in the body \[22\]. The AI lifecycle framework describes the generation functionality's value precisely: not generic text generation, but translating an abstract competency into an assessment item that produces observable evidence and can be evaluated consistently.

Its core risk is that an item may look fluent while failing to measure the intended competency; generated content is therefore evaluated with separate LLM-as-judge prompts for language clarity, alignment, cognitive load, faithfulness, usefulness, tone, observability and unsupported-claim risk, runs are captured in MLflow against the tested endpoint and dataset, and nothing is published automatically – the reviewer approves every item before candidate use \[21\].

![Figure 8 – The Generation Service: brief, configuration and ontology ground a task; every writing task has a judging counterpart.](../media/image9.png)

*Figure 8 – The Generation Service: brief, configuration and ontology ground a task; every writing task has a judging counterpart.*

## 5.3 Scoring answers: the Candidate Evaluation pipeline

Answers are scored by the Candidate Evaluation Service. Because a language model takes several seconds per criterion and an application may submit dozens of candidates at once, the service is asynchronous: a submission returns a job id, becomes one run with one case per answer, and the caller polls or receives a webhook when the last case finishes. The worker claims a case, the ledger marks it processing atomically, and the worker reports back; the database is the source of truth, so a worker restarting mid-evaluation loses the case, not the run \[19\].

Inside the worker one case passes through three explicit stages: an evaluator (the LLM judge) judges each rubric criterion and extracts structured features – rubric score, completeness, specificity, impact, informativeness – a scoring model turns those features into score probabilities on the five-point scale, and a summariser explains the result and cites the evidence in the answer. A final validation stage checks that the result still agrees with the pipeline state that produced it, so an explanation cannot drift away from the scores it explains.

Which components run, with which settings, is configuration rather than code \[19\].

Behind the scoring model sits a calibration layer. Historical reviewer scores are processed through a Multi-Facet Rasch model, which converts discrete reviewer scores into calibrated estimates with certainty bands and corrects for reviewer strictness, question difficulty, competency differences and rubric variation; these calibrated targets, not raw reviewer scores, are the training and validation reference of the score model. The documented main limitation is score compression: the model produces less variation than human reviewers and underuses the ends of the scale – acceptable for reviewer support, not for autonomous decisions, and exactly what the Box 3 reviewers noticed as "not sharp enough" \[1, 21\].

![Figure 9 – How a single evaluation is produced: evaluator, scoring model, summariser, validation – asynchronous, configuration-driven, Rasch-calibrated.](../media/image10.png)

*Figure 9 – How a single evaluation is produced: evaluator, scoring model, summariser, validation – asynchronous, configuration-driven, Rasch-calibrated.*

## 5.4 What the pilots show

Two sets of pilot results describe how the instrument behaves. In the DNB pilots – the DNB Traineeship with 631 candidates and 9,500 review scores, and the DNB Data & Tech Traineeship with 282 candidates and 4,230 review scores, five questions each – AI–human agreement was substantial (Cohen's κ = 0.667 and 0.695) and higher than human–human agreement (κ = 0.531 and 0.545). The more precise reading is not that the AI is superior but that it is more consistent, while human reviewers vary among themselves: the reviewer distribution plots show some reviewers favouring score 4, others 2 and 3, score 5 used unevenly.

The confusion matrix shows disagreement concentrated within one score point, with the AI somewhat conservative at the maximum and relatively generous around 4 \[14\]. In the Box 3 pilot the picture is consistent: the AI's average level is 0.05 above the reviewers', the candidate ranking correlates at 0.88, top-10 overlap is 7 of 10, but the AI compresses – 69.7 % of its scores on level 3 against 54.5 % for reviewers – and when it misses it is almost always too high (above every reviewer on 4.0 % of answers, below every reviewer on 0.5 %).

A within-candidate spread ratio of 0.79 shows that the instrument, scored answer by answer, makes candidates look more uniform than reviewers judge them \[1\].

These results fix the baseline the extension has to respect. The question-and-rubric instrument is reliable, explainable and well-calibrated against reviewers in the central range; its weaknesses are compression at the ends of the scale, the absence of a cross-answer view of the candidate, and the effort it asks of reviewers – precisely the three points the reviewers raised.