---
title: 6 Quality assurance today
---

# 6 Quality assurance today

## 6.1 The Bias Monitoring Framework

The Bias Monitoring Framework does not treat bias as only an AI risk; it monitors the full assessment workflow – question design, rubric quality, human judgement, AI-supported scoring, reviewer consistency, group outcomes and representativeness – in three scopes. Scope 1 asks whether AI and human reviewers score candidates differently in a systematic way and whether reviewers are consistent with each other, using AI–human score differences, Cohen's kappa, pass-rate differences by threshold and reviewer-level distributions; agreement is treated as a reliability signal, not a fairness conclusion.

Scope 2 asks whether candidate groups receive comparable outcomes overall and across questions, competencies, reviewers and scoring source, using score and pass-rate parity, effect sizes (A12, Cliff's delta) and factor-level breakdowns. Scope 3 asks whether observed differences change who progresses when selection thresholds are applied, using pass rates per threshold, group composition among top scorers and the balance point – the most selective cut-off at which representation still matches the applicant pool.

The interpretation logic is layered: an AI–human difference without group impact indicates a calibration issue; a parity difference without threshold impact calls for monitoring; a parity difference that also changes pass rates at a relevant threshold is the high-priority case \[14\].

![Figure 10 – The three scopes of the Bias Monitoring Framework and the interpretation logic that connects them.](../media/image11.png)

*Figure 10 – The three scopes of the Bias Monitoring Framework and the interpretation logic that connects them.*

Applied to the DNB pilots the framework found near gender parity in Pilot A (A12 = 0.485, Cliff's delta = −0.030; pass rates at threshold ≥ 4 of 59.5 % for women and 56.0 % for men) and a small female advantage in Pilot B (A12 = 0.555, Cliff's delta = 0.110; 61.6 % versus 52.3 % at threshold ≥ 4), with no evidence of systematic disadvantage against female candidates; in Pilot A the balance point lay at the top 18 % – female representation stayed at its 31.5 % baseline up to a shortlist of about 113 of 631 candidates \[14\]. The framework is what the extension's bias measurement builds on: the same three scopes, applied per requirement and per scoring rule rather than per question, with the biographical scorer as an additional scoring source.

## 6.2 The AI lifecycle management framework

The AI lifecycle management framework governs validation and release of the AI components. It documents the system at five levels – system, functionality, model, run and evidence, governance and release – so that a business reviewer can understand the system without reading model runs and a technical reviewer can trace any claim to its evidence. Each reviewer-facing functionality is documented as a functional contract: purpose, inputs, outputs, the point of human validation, the components involved and the evidence required before use.

Validation is proportional to risk and specific to role: a scoring component requires evidence on score behaviour, calibration, reviewer agreement and fairness; a search component on relevance, coverage, freshness and permission safety; a generation component on faithfulness, alignment, usefulness and unsupported claims. Evidence is tracked in MLflow with datasets split into training, validation and test, and release conditions name the validated roles, rubrics, languages, evidence types and populations outside which a functionality is not valid. Human authority is a design principle: AI components support evaluation, search and content creation; reviewers inspect, edit, override or reject every output \[21\].

For the extension this framework is not optional context but the form in which the new components will be documented: the biographical scorer, the work-probe criteria scoring and the coverage matching each get a functional contract, a model card and release conditions – and the six properties of section 3.3 are the method-level counterpart of the evidence the framework asks for.