ObjectivEye · AI & Data team · Methodology

Assessment redesign:methodology and flow

Evidence-based skills profiles, blind biographical scoring and skills-based matching on a shared taxonomy

Concept report for review · current methodology and planned extension · Sprint 2, deliverables 5.1–5.3 · 18 October 2026

Marc Lenz, Head of AI · Status: concept for review. In the second sprint week this concept is turned into requirements, a project plan and an actionable follow-up sequence per work stream.

The mechanism this report proposes: CV and vacancy are extracted into graphs, projected onto the shared skills graph and matched there, requirement by requirement.

#Contents

Part I Motivation

• 1 Summary
• 2 Motivation
  • 2.1 What the reviewers told us, and what the data confirms
  • 2.2 What the customer cases ask for
  • 2.3 What the redesign has to deliver
• 3 Methodological requirements
  • 3.1 What the selection research says about validity and bias
  • 3.2 What follows for ObjectivEye
  • 3.3 What every method has to document before use
  • 3.4 Scope of this concept

Part II Current system

• 4 The platform today
  • 4.1 Services and the gateway
  • 4.2 The Ontology Store
  • 4.3 Data: what the platform reads, stores and measures
• 5 Instrument today: behaviourally anchored questions and rubrics
  • 5.1 The assessment item and what makes it work
  • 5.2 Generating assessment content
  • 5.3 Scoring answers: the Candidate Evaluation pipeline
  • 5.4 What the pilots show
• 6 Quality assurance today
  • 6.1 The Bias Monitoring Framework
  • 6.2 The AI lifecycle management framework

Part III · A Planned extension – biographical data and generalisation

• 7 The extended assessment model
  • 7.0 Introduction: what this extension answers
  • 7.1 What is kept and what is added
  • 7.2 Three formats, one methodological bridge
  • 7.3 The scorecard as translation layer
• 8 Instrument: behaviourally anchored questions in the extended model
• 9 Instrument: biodata – evaluating biographical information for job fit
  • 9.1 A CV score is not a CV score
  • 9.2 What the biodata literature says about keying
  • 9.3 From questionnaire items to standardised CV records
  • 9.4 The Skills Passport as classification
  • 9.5 Skills-based matching on the taxonomy graph
  • 9.6 Bias: what remains after blinding, and why the graph helps
• 10 Instrument: work samples as written work probes
• 11 Engineering storyline
  • 11.1 Who builds what
  • 11.2 The shared ontology
  • 11.3 Employer flow
  • 11.4 Candidate flow
  • 11.5 Reviewer flow

Part III · B Planned extension – roadmap and plan forward

• 12 Follow-up: one proof of concept in three sprints
• 13 Open questions for the follow-up
  • 13.1 A – A valid training and evaluation corpus for CV ingestion
  • 13.2 B – Mitigating bias in practice
  • 13.3 C – Legal implications and GDPR constraints
  • 13.4 D – Validating validity and bias-proneness
• 14 References

View Markdown source

Part I

Motivation

Why the assessment model is extended now: what reviewers and customers asked for, what the selection research allows, and what every method must document before it is used.

View Markdown source

#1 Summary

ObjectivEye assesses candidates today with written items that are scored against behaviourally anchored rubrics. That method gives focused and defensible evidence, but two things have become clear in the current pilots:

1. Reviewers want a fuller and more consistent picture of a candidate and find it slow to judge answer after answer;

2. Several customer cases – talent pools, internal mobility, re-allocation – need candidate evidence that can be interpreted beyond the single application it was collected for.

This report proposes how to extend the assessment model accordingly, and it does so with one constraint in mind throughout: every extension must be at least as valid and at most as biased as what it replaces, and both properties must be measurable.

The proposal is staged as an extension of the current methodology, not as its replacement. Everything the platform does today – generating questions and rubrics from a vacancy, scoring answers against behaviourally anchored rubrics with an explained, calibrated pipeline, monitoring bias across three scopes – keeps working unchanged; the extension puts a requirements profile in front of it, adds two evidence types beside it and a reusable profile and matching around it. Part II of this report therefore describes the current methodology and system first, instrument by instrument, so that Part III can state precisely what is kept, what is added and where the two meet.The proposal has three parts.

• First, the evidence model is broadened: structured education, experience and biographical evidence and short written work probes are added next to the behavioural items, and all of them are turned into the same kind of record – an evidence claim against a requirement, evaluated with a scoring guide that is fixed before any candidate is seen.

• Second, biographical information is handled as a scored method in its own right: the CV is normalised into blind, classified records in the style of a Skills Passport, mapped to the shared taxonomy, and scored with a rationally keyed guide over taxonomy paths; empirical calibration is prepared for later.

• Third, matching between a candidate and a position becomes path-finding on the taxonomy graph: coverage per requirement, with the path as the reason, never a single similarity number or a total score.

The methodological case rests on the selection literature. Sackett, Zhang, Berry and Lievens [3, 4] rank structured interviews, job-knowledge tests, empirically keyed biodata and work samples as the strongest job-specific predictors and show that composites without cognitive ability lose almost no validity while reducing group differences. The biodata literature [6–8] supports starting with narrowly specified rational scales and calibrating weights once outcome data exists. And our own work on skills-based matching [9] shows why the taxonomy matters for fairness: vector representations of occupations carry gender segregation into any matching model, and only a structure in which a match is a path lets that bias be located and changed rather than merely observed.

The engineering storyline (section 11) describes what the platform and the AI & Data team build for each of the three flows – employer, candidate, reviewer – and specifies the shared ontology that connects them. The follow-up (section 12) unifies three existing capabilities into one proof of concept over three sprints. Section 13 raises the four questions that decide whether the biographical part can reach production: the training corpus for CV ingestion, bias mitigation in practice, the legal and GDPR constraints on learning from candidate data, and how validity and bias-proneness will be validated.

Figure 1 – Why, what and how – the summary cards of the concept presentation.

View Markdown source

#2 Motivation

#2.1 What the reviewers told us, and what the data confirms

The pilot with the Belastingdienst produced two reviewer questionnaires, one for the Box 3 vacancy and one for an earlier vacancy. Eight reviewers answered. The answers were linked to the evaluation data of the Box 3 vacancy – 198 answers from 33 candidates – so that each piece of feedback could be tested against what the scores actually show [1]. The questionnaire itself is very negative: on the question whether they would recommend human-plus-AI assessment to other selection committees, both Box 3 reviewers answered 0 out of 10 and the earlier vacancy's reviewers ranged from 0 to 7.

The feedback groups into four topics. The most consistent one is efficiency: seven of eight reviewers say the process is not more efficient, because judging each answer separately takes too much time. This cannot be checked in the evaluation data, since review time per answer is not logged; it is a process finding, not a scoring finding. The second topic is consistency: reviewers notice that contradictions between a candidate's answers are not weighed when every answer is scored on its own, and they want one picture of the candidate. The data confirms the mechanism behind this: a candidate's levels vary less across their six answers under the AI than under the reviewers – a spread ratio of 0.79 – so the AI makes candidates look more uniform than reviewers judge them.

The third topic is the CV: reviewers say that answers alone are not enough and that candidates refer to a CV the reviewer cannot see. Only two of 198 answers mention a CV explicitly, so the data gives weak evidence here, but the competency with the lowest reviewer average is the one that asks for qualifications and years of experience – material a CV would normally supply. The fourth topic is scoring: the AI should be stricter and sharper and look at what was actually answered. This is partly supported.

The overall bias is small (the AI's average level is 0.05 above the reviewers' and the candidate ranking correlates at 0.88), but the AI compresses the scale: 69.7 % of its scores sit on level 3 against 54.5 % of the reviewers' scores, and it uses the ends of the scale on 6.6 % of answers where reviewers use them on 16.9 %.

Figure 2 – Reviewer feedback by topic next to what the Box 3 data confirms.

The analysis draws one conclusion that this concept takes as its starting point: better scores alone will not change the feedback. What reviewers ask for is a compact and consistent overview of the whole candidate with less effort – a product question before it is a scoring question [1].

#2.2 What the customer cases ask for

The objectives and priorities overview [2] defines four product objectives – Dynamic Profile, Decision Support, Dashboard and Talent Pool – and ties their acceptance conditions to concrete customer cases. Read together, the cases ask for a change in the unit of assessment. IND needs decision quality under pressure: criterion-level evidence per candidate, compared on one screen. NVWA needs explainable selection and talent continuity: evidence that stays usable after the vacancy is closed. TNO wants evidence before CV review, which presupposes a structured requirements profile and evidence collected against it rather than a document to read.

SZW needs skills visibility and assessor consistency, and the TalentLead cases need skills mobility and talent reuse – a candidate's evidence matched against another requirements profile without a new assessment. None of these is served by a score per vacancy and nothing else. All of them are served by a normalised evidence profile per candidate that can be interpreted against several requirements profiles.

The ontology note [16] states the same two motivations from the system side: recommendations, talent pooling and matching across positions and organisations need a reusable candidate profile, and complementary evidence types let organisations balance assessment depth, candidate effort and assessor time against application volume.

#2.3 What the redesign has to deliver

Four outcomes follow. Flexible matching: reusable evidence that can be compared with several requirements profiles instead of a re-assessment per vacancy. A broader picture: background, specific examples and demonstrated work next to the behavioural items. Proportionate effort: assessment depth that follows the position and the recruitment stage, and the rule that nothing a candidate already recorded is asked again. Traceable assessment: every conclusion cites its requirement, the scoring guide used and the original evidence. The rest of this report is organised around these four outcomes and the constraints under which they must be reached.

Figure 3 – The two needs and the four intended outcomes.

View Markdown source

#3 Methodological requirements

#3.1 What the selection research says about validity and bias

The meta-analytic picture of personnel selection was revised in 2022. Sackett, Zhang, Berry and Lievens [3] showed that the corrections for range restriction used since Schmidt and Hunter (1998) had systematically overstated the validity of most predictors, and they re-estimated operational validities with more defensible corrections. The order of predictors changed as a result. Structured interviews now have the highest mean validity (ρ = .42), followed by job-knowledge tests (.40), empirically keyed biodata (.38) and work samples (.33). Cognitive ability tests fell from .51 to .31, integrity tests to .31, and the predictors that selection practice relies on most casually – unstructured interviews (.19), years of job experience (.07) – stay at the bottom. Rationally keyed biodata sits at .22. The same paper reports Black–White standardised mean differences: .23 for structured interviews, .33 for both forms of biodata, .54 for job-knowledge tests, .67 for work samples and .79 for cognitive ability.

In their follow-up focal article [4] the authors draw the design consequences. They favour job-specific predictors – those that sample job behaviour rather than infer a general construct – over measures of general psychological constructs.

They argue that the long-standing validity–diversity dilemma has largely dissolved: because the validity of cognitive ability has fallen, composites with and without it have nearly identical mean validity, and dropping ability from a six-predictor composite costs .05 of validity under the new estimates instead of .20 under the old ones; equally valid composites with smaller group differences can therefore be built.

They also warn against reading the ranking as a mandate: the validity of structured interviews is "really .42, plus or minus .24" across its credibility interval, work samples are less applicable where the relevant skills are learned after hiring, and a risk-averse employer should compare predictors on the lower end of their credibility interval as well as on the mean. Local validity studies retain their evidentiary value.

Table 1 – Corrected operational validities and Black–White standardised mean differences from Table 3 in Sackett et al. (2022) [3], with the consequence for ObjectivEye.

#3.2 What follows for ObjectivEye

A method, in this concept, is a defined combination of evidence collection and a scoring guide. That definition matters because validity belongs to the procedure in use, not to a format or a scale. Three cautions follow directly [16]. The categories in Table 1 are research categories, and their estimates do not transfer automatically to a particular digital or AI-assisted implementation. Behaviourally anchored rating scales are a scoring approach within a method, not an independently validated method; the same holds in the other direction – normalised CV information is not automatically validated biodata, and a written assessment item is not automatically a structured interview. And predictive validity, assessor agreement and bias are three different properties that must be evaluated separately: agreement between the AI and a reviewer supports a scoring-consistency claim but does not establish prediction, and a group-score difference is not by itself a test of measurement bias.

With those cautions the development priorities are as follows. Near term: structured education, experience and biodata scorecards, because they broaden the evidence model while building on the existing evaluation structure and give the reusable evidence that recommendations and talent pooling need; and written work-probe evaluation – an essay, a project plan, another deliverable – scored with task-specific structured rubrics that can reuse the existing scoring mechanism.

Continued and integrated: the existing assessment items with behaviourally anchored rubrics, as one component of the broader model. Later: conversational structured assessment as a digital adaptation of the structured interview; it could be more engaging and more resistant to externally generated answers, but adaptive questioning, consistent scoring and validation need substantial development.

Unstructured interviews, holistic CV judgements, credentials or years of experience alone, and general personality or cognitive tests are lower priorities: they provide less directly traceable task evidence or require additional justification of relevance, validity and bias [16].

#3.3 What every method has to document before use

Without sufficient support a method remains experimental, and its scoring guide must not imply a validation it does not have. Before a method is used, six properties are documented – for the method from the research, for the implementation from local evidence. The specification names the evidence elicitation, the item or task content, the scoring guide, the score format and the target requirements, together with language, model version and allowed tools. The research support names the relevant studies, their populations and performance criteria and the limits of transfer to the proposed method. Predictive validity states why scores should relate to the intended performance criterion and how this is evaluated locally.

Bias and fairness covers content relevance, group outcomes, accessibility and differential prediction, examined separately. Scoring consistency covers agreement between qualified assessors and the consistency of automated or AI-assisted scoring against the guide. Efficiency covers candidate completion effort, assessor effort, dropout and the information gained per stage. For the three formats of this concept the status differs: the question items have an item bank and rubrics and a measured AI–reviewer agreement; biographical scoring has its fields and normalisation rules to specify and its extraction accuracy to measure; work probes have a task bank to define and their consistency still to measure.

Translated or AI-assisted adaptations need explicit evaluation in every case.

Figure 4 – The six properties every method documents, with the status per format.

#3.4 Scope of this concept

In scope is the digital application process: evidence capture and semi-automated assessment; the existing behavioural items retained, with education, experience and biodata evidence and written work probes added; one requirements profile per position and one reusable evidence profile per candidate; requirement-level scorecards as the translation layer.

Out of scope are live interviews and in-person observation, conversational structured assessment as a later research and development option, a validated matching algorithm – enabled by the shared model but not specified here – and any certification of the current implementation or a validated assessment manual.

The concept assumes no new overall scoring formula, validity threshold or automatic rejection rule, no new ontology classes beyond proposed fields and relationship names that are for review, and no delivery date, owner or sprint commitment; access and candidate permission for reuse across organisations remain explicit design decisions. Customer feedback is design input, not a quantified research finding, and the screens in this report are illustrative.

View Markdown source

Part II

Current system

What exists today and how well it works: the services, the question-and-rubric instrument with its generation and scoring pipeline, and the two frameworks that keep it honest. Everything in this part is kept by the extension.

View Markdown source

#4 The platform today

At a glance

Table 2 – The current system at a glance; sections 4–6 describe each row.

#4.1 Services and the gateway

ObjectivEye is a skills-based, explainable hiring platform. It reads vacancies and candidate material, resolves what they say against a shared ontology, and generates and scores against that ontology rather than against free text – which is what makes a result something that can be accounted for afterwards [17]. Technically it is a handful of services, each with its own API, behind one service gateway, and one document store holding every document the platform has read or written. The Ontology Store holds the General Skills Taxonomy – ESCO, ISCO and the Competentiegids Rijk – with semantic search over it and the resolution of free text into the concepts it names.

The Generation Service writes what a hiring process needs written: a vacancy from a brief, questions for a role, a rubric to score answers against, and for every writing task a counterpart that judges an existing text. The Candidate Evaluation Service scores a candidate against a vacancy asynchronously and returns the reasoning alongside the score; its worker runs the pipelines. The Document Store keeps categorised documents, their Markdown and parsed structure and the templates learned from them; the Search Worker provides semantic search over the in-repo index. Every service is reached through one gateway address and a path prefix; a service moving is a routing change rather than an announcement.

The Ontology Store is the one exception: it runs in a different Azure environment and the application calls it directly [17, 18].

Figure 5 – The platform today: caller, service gateway and the six services it routes to (drawn from the gateway documentation, dev environment).

#4.2 The Ontology Store

Everything the platform generates or scores resolves its language against the Ontology Store first. That is what makes a downstream result explainable: a score that points at a concept with a stable URI can be accounted for and compared across candidates, where a score pointing at a phrase that happened to match cannot [18].

The store answers two kinds of question – what concept does this text mean, and what does this concept connect to – and it decouples the platform from any single framework: ESCO, CompetentNL and internal terminology are normalised into one canonical shape through adapters, so internal workflows never learn which source a skill came from.

Search is semantic, not lexical: the query is embedded (Azure OpenAI ada-002) and compared by cosine distance against vectors computed from each record's label and description in PostgreSQL with pgvector; asking for "strategisch beleid" returns "organisatorisch beleid" and "strategische planning implementeren" although neither label contains the query.

The General Skills Framework behind it models occupations, competencies, tasks, knowledge, outcomes and behavioural indicators with their relationships, and around it sit skills profiles per role, behaviour-anchored rubrics per criterion, an OWL/SHACL concept model and crosswalks between frameworks.

Three operations matter for the extension: extraction (/v1/extract-skills for a vacancy, /extract-skills/cv for a CV, /extract-skills/profile for a profile in which each skill carries a proficiency, a year last used and the evidence for it), matching (/v1/match-skills, returning three buckets – shared, asked for but missing, brought but not asked for – with an overall score) and the neighbourhood of an entity (/v1/entities/{uri}/neighbours). The service is judged on precision@k against a labelled query set, p95 latency and ingestion success [18].

Figure 6 – The General Skills Framework and its relationships, as modelled in the Ontology Store.

#4.3 Data: what the platform reads, stores and measures

Three kinds of data flow through the current methodology. Content data: vacancies, the templates they are generated from, questions, rubrics and this documentation, all held in the Document Store and searchable by meaning. Assessment data: candidate answers, the AI's judgement per rubric criterion with its reasoning, the final score, and the human reviewer's scores – one run per application, one case per answer, persisted in the evaluation ledger (PostgreSQL) which is the source of truth for polling; a configuration is addressed by the hash of the model that produced it, so a past result stays reproducible [19].

Monitoring data: the Bias Monitoring Framework reads candidate scores, human reviewer scores, AI-generated scores, candidate group variables where legally available, question and competency identifiers, pseudonymous reviewer identifiers, selection thresholds and progression outcomes [14]; the Box 3 analysis adds per-criterion scores from a re-run, the experiment bundle with configs and model outputs, and the Forms exports of the reviewer questionnaires [1]. Personal data stays in the platform database and in named SharePoint folders; the Data Registry – the controlled home for data sets with personal data – is being prepared and the integration of the training pipeline with it moved to Sprint 3.

This is the data the extension starts from: it is the training and validation base of the current scorer, and it is what any new scorer will be compared against.

View Markdown source

#5 Instrument today: behaviourally anchored questions and rubrics

#5.1 The assessment item and what makes it work

The current instrument is the assessment item. It consists of three connected parts: a competency description, a question and a scoring rubric. The description defines which ability the item intends to measure and provides the shared reference for the candidate's reading of the task and the reviewer's reading of the answer; it works when it names one clear ability, in behavioural terms, with one main action verb, and matches what the question asks and what the rubric evaluates. The question prompts the candidate to produce evidence – actions, decisions, reasoning or outcomes – rather than opinions; it works when it is clear, focuses on one competency, asks for observable behaviour and can be interpreted by candidates from different backgrounds. The rubric gives the evaluator the criteria for interpreting the answer; it works when its levels describe observable features of responses, are ordered and consistent, and differ in ways that are clear and interpretable [20].

Six quality properties follow from the relationships between these artefacts: construct clarity (description → question and rubric), task accessibility (question → candidate), evidence relevance (question → answer), criteria precision (rubric → reviewer), judgement consistency (rubric → judgement) and candidate separability (answer → judgement) [20, 21]. They are the checklist against which generated items are reviewed, and they reappear in Part III as the properties any new evidence type has to satisfy before it is scored.

Figure 7 – The assessment item as a chain from competency description to reviewer judgement, with the quality property each link carries.

#5.2 Generating assessment content

Items are produced by the Generation Service. Its tasks are grounded: a task carries the prompt, the configuration and the ontology grounding that make its output reproducible, in contrast to the four raw model endpoints (generate, complete, embed, list models) which carry none of it and exist for trying a prompt. The tasks are generate_vacancy_text, generate_vacancy_question, generate_scoring_rubric and generate_interview_questions with its scoring rubric, and for the first three a feedback counterpart that judges an existing text instead of producing one.

Configuration lives in a config store per tenant – prompt, model, thresholds – and the service reports which configuration answered in response headers (X-Config-Tenant, X-Config-Source), not in the body [22]. The AI lifecycle framework describes the generation functionality's value precisely: not generic text generation, but translating an abstract competency into an assessment item that produces observable evidence and can be evaluated consistently.

Its core risk is that an item may look fluent while failing to measure the intended competency; generated content is therefore evaluated with separate LLM-as-judge prompts for language clarity, alignment, cognitive load, faithfulness, usefulness, tone, observability and unsupported-claim risk, runs are captured in MLflow against the tested endpoint and dataset, and nothing is published automatically – the reviewer approves every item before candidate use [21].

Figure 8 – The Generation Service: brief, configuration and ontology ground a task; every writing task has a judging counterpart.

#5.3 Scoring answers: the Candidate Evaluation pipeline

Answers are scored by the Candidate Evaluation Service. Because a language model takes several seconds per criterion and an application may submit dozens of candidates at once, the service is asynchronous: a submission returns a job id, becomes one run with one case per answer, and the caller polls or receives a webhook when the last case finishes. The worker claims a case, the ledger marks it processing atomically, and the worker reports back; the database is the source of truth, so a worker restarting mid-evaluation loses the case, not the run [19].

Inside the worker one case passes through three explicit stages: an evaluator (the LLM judge) judges each rubric criterion and extracts structured features – rubric score, completeness, specificity, impact, informativeness – a scoring model turns those features into score probabilities on the five-point scale, and a summariser explains the result and cites the evidence in the answer. A final validation stage checks that the result still agrees with the pipeline state that produced it, so an explanation cannot drift away from the scores it explains.

Which components run, with which settings, is configuration rather than code [19].

Behind the scoring model sits a calibration layer. Historical reviewer scores are processed through a Multi-Facet Rasch model, which converts discrete reviewer scores into calibrated estimates with certainty bands and corrects for reviewer strictness, question difficulty, competency differences and rubric variation; these calibrated targets, not raw reviewer scores, are the training and validation reference of the score model. The documented main limitation is score compression: the model produces less variation than human reviewers and underuses the ends of the scale – acceptable for reviewer support, not for autonomous decisions, and exactly what the Box 3 reviewers noticed as "not sharp enough" [1, 21].

Figure 9 – How a single evaluation is produced: evaluator, scoring model, summariser, validation – asynchronous, configuration-driven, Rasch-calibrated.

#5.4 What the pilots show

Two sets of pilot results describe how the instrument behaves. In the DNB pilots – the DNB Traineeship with 631 candidates and 9,500 review scores, and the DNB Data & Tech Traineeship with 282 candidates and 4,230 review scores, five questions each – AI–human agreement was substantial (Cohen's κ = 0.667 and 0.695) and higher than human–human agreement (κ = 0.531 and 0.545). The more precise reading is not that the AI is superior but that it is more consistent, while human reviewers vary among themselves: the reviewer distribution plots show some reviewers favouring score 4, others 2 and 3, score 5 used unevenly.

The confusion matrix shows disagreement concentrated within one score point, with the AI somewhat conservative at the maximum and relatively generous around 4 [14]. In the Box 3 pilot the picture is consistent: the AI's average level is 0.05 above the reviewers', the candidate ranking correlates at 0.88, top-10 overlap is 7 of 10, but the AI compresses – 69.7 % of its scores on level 3 against 54.5 % for reviewers – and when it misses it is almost always too high (above every reviewer on 4.0 % of answers, below every reviewer on 0.5 %).

A within-candidate spread ratio of 0.79 shows that the instrument, scored answer by answer, makes candidates look more uniform than reviewers judge them [1].

These results fix the baseline the extension has to respect. The question-and-rubric instrument is reliable, explainable and well-calibrated against reviewers in the central range; its weaknesses are compression at the ends of the scale, the absence of a cross-answer view of the candidate, and the effort it asks of reviewers – precisely the three points the reviewers raised.

View Markdown source

#6 Quality assurance today

#6.1 The Bias Monitoring Framework

The Bias Monitoring Framework does not treat bias as only an AI risk; it monitors the full assessment workflow – question design, rubric quality, human judgement, AI-supported scoring, reviewer consistency, group outcomes and representativeness – in three scopes. Scope 1 asks whether AI and human reviewers score candidates differently in a systematic way and whether reviewers are consistent with each other, using AI–human score differences, Cohen's kappa, pass-rate differences by threshold and reviewer-level distributions; agreement is treated as a reliability signal, not a fairness conclusion.

Scope 2 asks whether candidate groups receive comparable outcomes overall and across questions, competencies, reviewers and scoring source, using score and pass-rate parity, effect sizes (A12, Cliff's delta) and factor-level breakdowns. Scope 3 asks whether observed differences change who progresses when selection thresholds are applied, using pass rates per threshold, group composition among top scorers and the balance point – the most selective cut-off at which representation still matches the applicant pool.

The interpretation logic is layered: an AI–human difference without group impact indicates a calibration issue; a parity difference without threshold impact calls for monitoring; a parity difference that also changes pass rates at a relevant threshold is the high-priority case [14].

Figure 10 – The three scopes of the Bias Monitoring Framework and the interpretation logic that connects them.

Applied to the DNB pilots the framework found near gender parity in Pilot A (A12 = 0.485, Cliff's delta = −0.030; pass rates at threshold ≥ 4 of 59.5 % for women and 56.0 % for men) and a small female advantage in Pilot B (A12 = 0.555, Cliff's delta = 0.110; 61.6 % versus 52.3 % at threshold ≥ 4), with no evidence of systematic disadvantage against female candidates; in Pilot A the balance point lay at the top 18 % – female representation stayed at its 31.5 % baseline up to a shortlist of about 113 of 631 candidates [14]. The framework is what the extension's bias measurement builds on: the same three scopes, applied per requirement and per scoring rule rather than per question, with the biographical scorer as an additional scoring source.

#6.2 The AI lifecycle management framework

The AI lifecycle management framework governs validation and release of the AI components. It documents the system at five levels – system, functionality, model, run and evidence, governance and release – so that a business reviewer can understand the system without reading model runs and a technical reviewer can trace any claim to its evidence. Each reviewer-facing functionality is documented as a functional contract: purpose, inputs, outputs, the point of human validation, the components involved and the evidence required before use.

Validation is proportional to risk and specific to role: a scoring component requires evidence on score behaviour, calibration, reviewer agreement and fairness; a search component on relevance, coverage, freshness and permission safety; a generation component on faithfulness, alignment, usefulness and unsupported claims. Evidence is tracked in MLflow with datasets split into training, validation and test, and release conditions name the validated roles, rubrics, languages, evidence types and populations outside which a functionality is not valid. Human authority is a design principle: AI components support evaluation, search and content creation; reviewers inspect, edit, override or reject every output [21].

For the extension this framework is not optional context but the form in which the new components will be documented: the biographical scorer, the work-probe criteria scoring and the coverage matching each get a functional contract, a model card and release conditions – and the six properties of section 3.3 are the method-level counterpart of the evidence the framework asks for.

View Markdown source

Part III · A

Planned extension

Biographical data and generalisation

The methodological extension: one evidence model for three instruments, biographical data as a scored method in its own right, matching on the taxonomy graph, and the engineering storyline that puts it into the existing services.

View Markdown source

#7 The extended assessment model

#7.0 Introduction: what this extension answers

ObjectivEye assesses candidates with written questions. For each competency a position needs, the candidate answers one question, and the answer is scored against a behaviourally anchored rubric – a scale whose levels describe what a weaker and a stronger answer look like. An AI scores first and explains why; a human reviewer confirms or corrects. This works and is validated, but it has limits that the people using it have told us about. Reviewers find it slow, because they judge one answer at a time and never see the candidate as a whole. They miss the CV: a candidate's education and experience are not part of the assessment, although they would normally be the first thing a recruiter looks at. They want sharper scores. And our customers want to reuse what they learn about a candidate – for another vacancy, for an internal move, for a talent pool – which a set of answers to one vacancy's questions does not allow.

The extension answers this with one main addition and the structure around it. The main addition is biographical data: a candidate's education, experience, certificates and achievements become a structured, anonymised profile – a Skills Passport, filled quickly from the CV and corrected by the candidate – and that profile is scored against the requirements of a position, in the same way an answer is scored against a rubric. The structure around it is a requirements profile per position, so that there is something explicit to score against; a third evidence type, short work probes, for the cases where demonstrated work is the better evidence; and a scorecard that shows, per requirement, what the biographical profile, the questions and the probes each say – side by side, never added up – so that the reviewer decides per requirement with all evidence in view.

Three questions therefore define this part. How is biographical data scored against a requirements profile in a way that is valid, blind and explainable? Chapter 9 answers it: standardised records, a rule-based scoring guide over the skills taxonomy, and a measurable approach to the bias that biographical data brings. How do biographical scores sit alongside the behaviourally anchored rubric scales?

Chapters 7 and 8 answer it: the questions stay exactly as they are, they are asked only for requirements the profile does not already cover, and both are expressed in the same evidence model so that they can be read together. And how are the sources combined in the decision? Chapter 7.3 and the reviewer flow in chapter 11 answer it: they are not combined into a total – each source keeps its own score and its own certainty, the scorecard places them next to each other per requirement, and the reviewer records a judgement per requirement.

Everything that exists today is kept; the extension puts a requirements profile in front of it, biographical data and work probes beside it, and the scorecard and matching after it.

#7.1 What is kept and what is added

The extension keeps every component of Part II and changes where it sits in the flow. Today a vacancy produces questions and rubrics, candidates answer, the pipeline scores and the reviewer confirms. In the extended model a vacancy first produces a confirmed requirements profile; per requirement the employer decides which evidence counts and under which scoring guide; the candidate's Skills Passport covers what it can, questions and work probes are asked only for the rest; and the scorecard shows the three sources side by side per requirement. The Generation Service, the Candidate Evaluation pipeline with its Rasch calibration, the Ontology Store and the Bias Monitoring Framework stay as they are; what is added are the requirements profile and rule sets, the passport records and biographical scorer, work-probe criteria, the scorecard and coverage matching, and certainty and explanations per level.

Figure 11 – From the current methodology to the extended model: what is kept and what is added.

At a glance

Table 3 – What the extension keeps and what it adds.

#7.2 Three formats, one methodological bridge

The three formats differ only in how evidence is elicited. From the CV and biographical data the candidate provides an upload; education, experience and achievements are extracted, normalised and confirmed by the candidate, become education, experience and demonstrated-task records linked to taxonomy concepts, and are scored with requirement-linked guides for background – relevance, level, recency – rather than for credentials or years. From the behaviourally anchored questions the candidate provides a written answer to one situation per requirement; it becomes an answer record with the situation, the reported behaviour and the outcome, and is scored with the requirement's anchored rubric, AI-assisted and reviewer-confirmed. From a work sample the candidate provides a short task output – a project plan, a case response, a written piece; it becomes a demonstrated-task record with the submission as source and is scored against task-specific criteria derived from the requirement.

Everything after elicitation is shared, and that is the bridge. Every format produces the same record type, an evidence claim: what the candidate claims or shows, for which requirement, from which source. Every claim is evaluated with a predefined, requirement-linked scoring guide whose levels and anchors are fixed before any candidate is seen. All scored claims form one evidence profile per candidate, which the scorecard reads per requirement; evidence that is missing stays visible as missing. Evaluation, storage and traceability are therefore identical across formats, and the ontology in section 11.2 holds the concepts they share.

Figure 12 – Three formats, one methodological bridge (concept presentation, slide 12).

#7.3 The scorecard as translation layer

Candidate evidence and position-specific judgements remain distinct records. The same experience record, answer or work probe may support several assessments while its relevance and interpretation differ between positions. The scorecard connects what the evidence supports to what a position requires through the shared task, competency and knowledge concepts [16]. For a project-coordinator position the chain runs as follows: a position requirement – prepare project plans with clear milestones and dependencies – is met by candidate evidence consisting of an experience record, a reported project example and a submitted sample plan; from these follow evidence claims (the candidate reports having planned projects; the work probe shows planning performance); separate scoring guides evaluate the reported contribution and the quality of the plan; and the scorecard entry for the requirement cites both sources, explains their contribution and states the remaining uncertainty.

Figure 13 – From position requirement to scorecard entry, and the three rules that keep the chain honest.

Three rules keep the chain honest. Nothing is silently merged: extracted associations, candidate statements, verification and assessment conclusions stay separate records with provenance – source, extraction activity, evaluator, version. Scores are not portable: a record can support several requirements or positions, but its evaluation may change with the target, the context or the guide, and a score under one rubric is not a score under another. And there is no universal total score: missing evidence is distinguishable from low performance, records from the same source are not double counted, and any aggregation needs an explicit, reviewed interpretation. Education or occupational experience may support a conclusion, but it does not establish demonstrated capability; that remains the role of the behavioural items and the work probes.

The three instruments side by side

Table 4 – The three instruments of the extended model. Figures from Sackett et al. (2022) [3].

View Markdown source

#8 Instrument: behaviourally anchored questions in the extended model

The question-and-rubric instrument is retained as it is and becomes one of three evidence formats. What changes is its place, not its mechanics. First, an item is now generated for a requirement in the confirmed requirements profile rather than for a competency chosen on its own: the Generation Service receives the requirement, its taxonomy concept and the organisation's template, and the item's description, question and rubric are checked for the same six properties as today – construct clarity, task accessibility, evidence relevance, criteria precision, judgement consistency, separability [20, 21].

Second, an item is asked only where the package needs it: when the passport already covers a requirement with high certainty, or when a work probe is the better evidence, the employer's package leaves the question out; this is how the instrument's main cost – reviewer and candidate effort per answer – is reduced without touching its quality. Third, the answer becomes an evidence claim in the shared model: a candidate answer record, scored by the existing pipeline into a level on the requirement's scale, with the judge's criterion judgements as the explanation and the Rasch-calibrated certainty band as the certainty input.

Fourth, the item's result is read per requirement next to the other sources and is never summed with them; the cross-answer view the reviewers asked for comes from the scorecard, not from a change to the item.

Two findings from the pilots turn into requirements on the instrument in the extended model. Compression at the ends of the scale [1, 21] is addressed where it arises – in the score model's calibration and in the relevance check of the evaluator – and it is monitored per requirement with the existing Scope 1 metrics; the extension does not hide it behind other sources. And the measured length and relevance effects [1] – answers at the 150-word limit, off-topic but well-built answers – stay part of the item's validation, because a work probe or a passport record next to the answer does not make the answer's own score more valid. In short: the instrument keeps its validated scope, its documentation and its release conditions, and gains a defined place in a package that no longer uses it for everything.

View Markdown source

#9 Biodata: evaluating biographical information for job fit

Biodata is the new instrument this sprint researched most deeply, and the one on which the Blind CV review and the knock-out criteria depend. This chapter argues in six steps that biographical information can be a valid and fair source of evidence – but only under a specific method – and specifies that method.

#9.1 A CV score is not a CV score

How valid and how biased the scoring of biographical data is depends entirely on the method, not on whether names and photographs have been removed. Three methods are in play. Naive CV screening rates the document as a whole – titles, employers, years, education, presentation – without a fixed framework or fixed weights and with the candidate's identity in view. There is no validity estimate for the whole, and its ingredients are weak: years of job experience predict performance at ρ = .07, years of education and the training-and-experience point method at .10 and .11 respectively [3].

Its bias is the largest of the three: years alone carry a group difference of d = .49 [3], and the correspondence audits since Bertrand and Mullainathan [10] show fewer call-backs for minority and female names on otherwise identical CVs. Its reasoning cannot be reconstructed. Rationally keyed biodata scores standardised entries with rules that follow from the requirement – which tasks, skills and knowledge count, how duration, recency and responsibility are weighed – fixed before any candidate is seen.

Its validity is moderate (ρ = .22) but job-relevant by construction, so it adds information next to the question scores; its group difference is d = .33, and because the score needs neither name nor age nor employer, those can be removed and proxies excluded explicitly; every level cites a rule, a requirement and the entry. Empirically keyed biodata uses the same entries with weights estimated from the relation between past candidates' biodata and a criterion, cross-validated.

It is the most valid of the three (ρ = .38) once enough outcome data exists per role family, carries the same d = .33, but can learn who was hired rather than who performed, and its weights are statable without being logically justified.

Figure 14 – Naive screening, rationally keyed and empirically keyed biodata compared on validity, bias and explainability.

The consequence is the position this report takes. A naive, rule-of-thumb score on the CV, even anonymised, has poor validity, and that noise would leak into the overall assessment. The minimum standard is a standardised format, rationally scored, and ideally bias-mitigated. The remaining bias potential is real and higher than for the anonymous answers – but, unlike the answers, biographical evidence is what the customers actually asked for, and section 9.6 shows that in a taxonomy-based system that potential can be measured and reduced.

#9.2 What the biodata literature says about keying

Biodata instruments have been used in selection for decades, traditionally as standardised questionnaires whose responses are converted into scores either by expert judgement (rational keying), by their statistical relation with an outcome (empirical keying) or by a combination of both (hybrid keying); factor-analytic scaling is a scale-construction technique rather than a separate source of justification [5]. Three studies shape the path this concept proposes.

Stokes and Searcy [6] built biodata scales for mechanical-equipment franchise owners and compared broad rational scales, narrowly specified rational scales, internally derived scales and an empirical item key against sales performance and supervisory ratings. Their abstract reports higher validities and less faking with empirical keys overall – and, decisive for us, that the specific rational scales were as predictive as the empirical item key. The lesson is not that rational scoring equals empirical scoring in general, but that rational scoring works when it is tied to specific, clearly defined constructs rather than to a loose combination of education, titles and years.

Cucina, Caputo, Thibodeaux and Maclane [7] compared rational, empirical and hybrid scoring on 139 items of the US Office of Personnel Management's Individual Achievement Record. Rational keying yielded the lowest validities in their data; hybrid methods performed best at the smallest development samples, empirical keying at moderate sizes and stepwise regression weighting at the largest. Their vertical-percentage example makes the empirical logic concrete: a response chosen by 18.2 % of high performers and 9.5 % of low performers receives a weight of +8.7 – it scores because it discriminates, not because it sounds desirable.

Speer, Siver and Christiansen [8] addressed the main criticism of empirical keys, their opacity and poor generalisation, and showed across two field studies that the similarity between development and application context affects how empirical keys relate to external measures; empirical associations therefore have to be read against the underlying construct and the job context.

Figure 15 – The three families of biodata scoring and the chosen path: rational now, hybrid next, empirical later.

Taken together the three studies justify a staged path. Rational keying first: it needs no outcome data, it is transparent, and it is defensible when constructs are narrow and requirement-linked – which is exactly what the taxonomy provides. Hybrid calibration next: once reliable performance observations exist, the weights of the same constructs are estimated statistically while the construct structure, and with it the explanation, is kept. Empirical keying later, where volume and outcome quality allow it. Two guardrails hold throughout [5]. The evidence record stays separate from the score, so a change in scoring method never requires re-extracting or rewriting a record. And calibration uses reviewer judgements and, where available, job performance – never hiring decisions or recruiter preferences, which would reproduce existing bias rather than predict performance.

#9.3 From questionnaire items to standardised CV records

A classic biodata item measures one construct with an ordered response scale: how many projects the candidate has coordinated independently in the last five years (none, one, two to three, four or more); the highest level of project responsibility in the most recent role (participated, parts, independently, several complex projects); how often the candidate coordinated with external suppliers; how many non-required trainings they completed. The scale value is the rational score.

ObjectivEye's operational requirement is time efficiency [5]: a second questionnaire next to the assessment is not acceptable. The same constructs can, however, be filled from standardised CV records. Experience records that carry the task "coordinate projects" with responsibility "independent" and a duration give the coordination level; the highest responsibility attribute across records gives the responsibility level; a certificate record gives the learning construct. What the CV cannot show – frequency, scale – is asked as a single confirmation question rather than as a form.

The CV thus becomes the quick way to fill a biodata instrument whose items the candidate never has to see.

Figure 16 – Typical biodata questionnaire items, and the same constructs filled from the Skills Passport.

#9.4 The Skills Passport as classification

TNO's Skills Passport research provides the classification for those records. The passport is positioned as a digital tool for recording competences that gives workers an overview for career development and helps employers with recruitment and selection, as an alternative to relying on occupations and formal qualifications alone [11]. In TNO's 2022 stakeholder study nearly 60 % of employers considered it a good idea and 82 % of those would use it; workers named privacy and self-determination – control over sharing, correcting and deleting their data – as preconditions, and ownership and governance were explicitly left open [11].

The 2025 programme in the metal and electrical engineering sector tests three technical concepts, insists on interoperability between systems and on a range of validation from soft (online tests) to hard (demonstrated skills), and notes that AI used on the data must be explainable and fair [12]; the Metalektro pilot with seven companies reported in July 2026 [13].

The methodological note [5] summarises the expected contents as formal qualifications, professional experience, skills and competencies, practical achievements, validated proficiency and development interests, and cites TNO's stakeholder research for the weight employers give to them (93 % qualifications, 92 % work experience, 91 % a skills list, 84 % certificates).

Figure 17 – What a Skills Passport contains (TNO) and a schematic record set with validation status per record.

Two features of the passport carry over into this concept. First, the record types themselves: the standardised profile classifies biographical information into those six types, each record linked to taxonomy concepts. Second, the distinction between self-reported and validated evidence. A skill listed on a CV, two years in a role that uses it, a portfolio that applies it, a standardised assessment of it and independently verified work establish different things and must not receive fixed scores by source alone; a certificate proves completion, not proficiency. Every record therefore carries a validation status – self-reported, confirmed by the candidate, verified, assessed – and the scoring guide decides what a status is worth for a given requirement. Ownership follows TNO's model: the passport belongs to the candidate, who controls what is shared.

Operationally, a CV becomes a set of blind, classified passport records in four steps. Extraction splits the document into entries – experience, education, certificate – and free text into task statements. Mapping links each entry to taxonomy concepts: occupation, tasks, skills, knowledge, education level (ESCO, ISCED). Standardisation computes the same fields for every candidate – duration in years, recency, responsibility level, sector. Confirmation shows the candidate only the standardised entries to correct and complete; nothing is inferred silently. Name, age, address, photograph, gender, employer names and exact dates are removed before anything is scored, and every entry keeps its source span in the CV so that provenance is available without the document. The records are reused for the next requirements profile; the CV is never read again.

Figure 18 – From the uploaded CV to standardised, blind entries in four steps.

#9.5 Skills-based matching on the taxonomy graph

Once requirements and records are expressed in the same concepts, matching no longer needs documents or titles. Occupations, tasks, skills and knowledge are nodes of one graph – ESCO plus the ObjectivEye task layer. A requirements profile and a biographical profile are both sets of nodes with weights. The links of the graph define relatedness: an occupation has essential and optional skills, a skill has broader and narrower skills, a task uses skills, an occupation has typical tasks. A path between a requirement node and a candidate node is a reason, and the match is coverage per requirement: a required node is covered directly, via an allowed path with a distance penalty, or not at all. There is no single similarity number.

Figure 19 – Skills-based matching as path-finding on the taxonomy graph, applied to keyed biodata.

Applied to keyed biodata this gives the rational key a precise form. Every standardised entry becomes a set of nodes with duration, recency and level attached. The key is a weight per link type – same task above narrower skill, narrower skill above essential skill of the occupation, that above occupation group – scaled by duration and recency, fixed in advance and readable as rules. Where the taxonomy has no path between a requirement and an entry, nothing is scored: a missing link is a taxonomy task, not a candidate penalty, and the absence shows as missing evidence rather than as a low level. When empirical or hybrid calibration becomes possible, it re-estimates the same link-type weights; the structure, and with it the explanation, stays.

Figure 20 – The whole flow in one view for a fictitious project-coordinator case: blind CV → CV graph → skills graph ← requirements graph ← vacancy, with the match read off per requirement.

Reading the figure: the requirement "plan projects with milestones" points at the task node plan project schedules; the candidate's experience record as project officer contains the same task, five years, own projects, confirmed – a direct match and level 4 under the requirement's rule L4. "Apply public procurement rules" points at the knowledge node public procurement, which no candidate record reaches: no evidence, shown as missing, and because the requirement is a knock-out, a decision for the reviewer rather than a low score. The same picture holds for a Skills Passport that arrives already classified: its records already carry taxonomy concepts and a validation status, so it enters where the CV graph does.

#9.6 Bias: what remains after blinding, and why the graph helps

Blinding closes one door. Five others stay open in biographical scoring, and each has to be named with its control and its measure. The first is the structure of the taxonomy itself.

In Adhikari, Vethman, Vos, Lenz, Cocu, Tolios and Veenman [9] language-model representations of occupational skills – bag of words, word2vec and BERT – reproduce the gender segregation of the labour market, and across all representations and distance metrics tested that segregation carries through to which occupations a profile is matched to; the authors recommend evaluating matching models on matching performance and gender-segregation risk together and making the bias explicit.

The control here is to score through tasks and skills rather than occupation similarity, to let occupation nodes widen the search without scoring, and to down-weight or disable link types that carry segregated structure; the measure is the group gap per link type and per requirement and the balance point from the Bias Monitoring Framework [14]. The second door is the proxies inside the entries: years, employer prestige, credential level, career gaps and part-time periods correlate with age, gender and background.

The rational key scores task and responsibility rather than years or credentials alone, gaps and part-time are not fields, recency is capped; the measure is the pass-rate difference per entry attribute and the sensitivity of levels to years and credentials. The third is extraction and language: CV style, non-native phrasing and sector vocabulary change what an extractor finds, and some groups document more than others. The candidate's confirmation and completion step is the control, together with extraction evaluated per language; the measures are extraction recall by language and sector and the correction rate per group.

The fourth is historical outcome data, which enters with empirical or hybrid keying: a key can learn who was hired or rated well rather than who performed. Calibration therefore uses reviewer judgements and performance criteria only, with a group check before every weight update and weights that stay statable rules; the measure is the group gap before and after each calibration and cross-validated validity per group. The fifth is reviewer anchoring: a visible CV score steers the judgement of the answers and the reverse.

CV, question and probe scores stay separate and side by side, the order of reveal is a workflow setting, identity reveal is controlled; the measure is agreement with and without the CV score visible, one of the indicators the Box 3 analysis already lists [1].

Figure 21 – Five bias risks that remain after blinding, with control and measure for each.

What distinguishes a taxonomy-based system is not that these risks are smaller but that they are locatable. In a vector model a group gap can only be observed. Here every CV score is a set of paths – requirement node, link type, candidate node, weight – and a group difference in scores decomposes into the same parts: which requirement, which link type, which entry type produces it. The gap can be measured where it arises, on blind entries, so that the cause is a rule or a node and never a name. It can be corrected at that element – a link type down-weighted, a proxy node excluded, a missing transferable link added – with each change a readable rule that is re-measured afterwards. And because transferable tasks and skills count across occupations, more relevant evidence is used while every rule stays visible, which is how validity can rise and bias-proneness fall at the same time. That is the case for embedding keyed biodata in the graph rather than in a model.

View Markdown source

#10 Instrument: work samples as written work probes

The third format is the work sample, implemented as a written work probe: a short, representative task whose output can be completed digitally – a project plan, a case response, a memo, a piece of analysis – scored against task-specific criteria derived from the requirement.

The research basis is the work-sample category in Sackett et al. [3]: a corrected operational validity of .33, down from .54 in the older estimates, which still places it among the stronger job-specific predictors, and a Black–White standardised mean difference of .67 – the largest of the three formats in this concept, and the reason the probe needs the closest fairness check of the three.

Sackett et al. [4] add two cautions that shape its use: work samples are less applicable to entry-level hiring where the relevant skills are learned after hire, and the credibility interval around the mean matters as much as the mean. A probe is therefore not a default; it is chosen per requirement where demonstrated performance is the evidence that matters and where the task can be representative of the work.

Methodologically the probe is the clearest case of the method definition in section 3.2: evidence collection is the task, the scoring guide is the criteria sheet, and validity belongs to that pair in use.

Specification names the task, the allowed tools and time, the language and the criteria with their levels; research support cites the work-sample literature and its limits of transfer to a written, unsupervised format; predictive validity is evaluated locally as for the other formats; bias and fairness cover access to tools and time, language, and the group difference the literature reports; scoring consistency is agreement between qualified assessors and between the AI criteria scoring and reviewers; efficiency weighs the higher candidate effort against the information gained – a probe is used only where it adds information the passport and the questions do not give.

The existing evaluation pipeline is reused: the evaluator judges each criterion of the sheet, the scoring model produces a level, the summariser cites the submission, and the submission itself stays attached as evidence rather than being rewritten. What has to be built new is the task bank per role family with its criteria sheets, the candidate-facing delivery with time and tool rules, and the validation that a probe's criteria scoring behaves like the question scoring does today. In the extended model the probe level sits beside the passport and question levels on the scorecard; a requirement such as "plan projects with milestones" can carry all three, and the reviewer sees where they agree and where they do not.

Figure 22 – Why a taxonomy-based system makes bias measurable and tunable.

View Markdown source

#11 Engineering storyline

This section is written for the two teams that build the concept. It fixes who owns what, specifies the shared ontology both teams program against, and then walks through the employer, candidate and reviewer flows screen by screen, stating for each step what the front-end shows and what the back-end does.

#11.1 Who builds what

The split holds for every flow. The platform team builds screens, workflows, records and access: the employer, candidate and reviewer front-ends, the staging engine and the position lifecycle, the storage of Skills Passports and positions with versioning and consent, identity kept apart from records, the reveal policy and the audit log. It integrates the AI services through the gateway and renders the proposals, levels, certainty values and explanations it receives; it never computes a score itself.

The AI & Data team builds extraction and taxonomy – vacancy to requirements profile, CV to entries and task phrases, mapping to the Ontology Store, autocomplete and suggestions – scoring – rule sets per requirement, the biographical scorer over taxonomy paths, Candidate Evaluation for questions and probes, certainty and explanations – and quality: validation, bias measurement per rule and requirement, agreement metrics, with the internal insights service feeding the fairness panel.

The contract between them is simple to state: the platform sends documents and choices, the AI services return proposals, levels, certainty and explanations with provenance, and every number a reviewer sees can be traced back through the gateway.

Mapped onto the services of Part II: the Ontology Store gains the requirements-profile extraction (deliverable 1.2), the passport record mapping and the link-type traversal behind coverage; its existing /extract-skills/cv and /extract-skills/profile operations are the starting point of the quick-fill step, and /match-skills with its three buckets – shared, missing, not asked for – is the ancestor of coverage per requirement. The Generation Service produces questions and probes per requirement and proposes default rule sets from templates. The Candidate Evaluation pipeline scores questions and probes unchanged and gains the biographical scorer as a sibling stage that reads passport records and rule sets instead of answers and rubrics. The insights service computes the Bias Monitoring metrics per requirement and rule. The platform application (Next.js front-end, NestJS back-end behind the API gateway, OE-655) owns every screen and record described below.

Figure 23 – Who builds what: platform team and AI & Data team, and the contract between them.

#11.2 The shared ontology

The generalised evidence-based assessment ontology [16] is the data model both teams program against. It connects what work requires, what evidence a candidate provides and how that evidence is evaluated, in one graph with three namespaces. ESCO identifiers are reused where the meaning matches; ObjectivEye and customer concepts live in their own namespaces with explicit relationships to the shared vocabulary; the model is expressed in RDF/OWL with SHACL validation, PROV-O provenance and SKOS mappings, and it is to be integrated into the Ontology Store and followed across candidate profiles, requirements, evidence collection, generation, evaluation and matching, so that the same canonical concepts connect requirements, evidence and results throughout the system.

Namespace 1 – Work, tasks, skills and competencies

Core relationship: a task requires skills, competencies and knowledge applied in a context; its performance is expressed through behaviour and produces an outcome; occupations generalise recurring work patterns and occupation groups organise them.

Namespace 2 – Organisation, position and requirements

Namespace 3 – Candidate profiles, assessment evidence and scoring guides

Work-probe submissions remain evidence – no new class is needed for them. Extracted requirements stay drafts until the employer confirms them. Evidence and evaluations connect to Namespaces 1 and 2 through the references in the right-hand columns; that is what lets the scorecard translate between a candidate's evidence and a position's requirements without copying either.

Figure 24 – The three namespaces as presented to the team (concept presentation, slide 13).

#Flows

Defintion flow: -> inital info -> assessment config -> design -> publish

Application flow: -> visit -> create skills-profile -> answer questions -> complete and submit

Assessment flows -> review -> score -> select

#11.3 Employer flow

The employer flow follows one rule: the AI proposes, the employer decides, the platform stores. The platform team owns the position screens – create position, requirements editor with tags and taxonomy search, scoring-rules editor, package and staging screen, publish – the position record in which the confirmed requirements profile, the rule sets and the package are stored versioned together and frozen on publish, and the workflow settings for staging, reveal and knock-out handling, which are enforced server-side. The AI & Data team owns the requirements extraction (vacancy spans to requirements to taxonomy concepts with provenance, in the Ontology Store, deliverable 1.2), the default rule sets per requirement derived from the taxonomy and templates with validation that proxies are off, and the package generation in which the Generation Service produces questions and probes only for requirements that need them.

Figure 25 – Employer flow: create position from vacancy, scoring rules per requirement, assessment package and staging, review screen.

Step 1, create position from vacancy. The employer pastes or uploads the vacancy text and sees a proposed requirements profile: each requirement with its taxonomy concept and a must / should / knock-out tag, which can be edited, and a search over the skills list to add requirements. Confirming the profile stores it versioned; from then on it is the reference for scoring, package and review. Behind the screen the extraction service turns vacancy spans into requirements and links each to ESCO or task concepts with provenance to the span.

Step 2, scoring rules per requirement – the counterpart of defining questionnaire items today.

For a requirement the employer switches evidence sources on or off (passport, question, work probe; interview is listed but off), reads and edits the anchors of the biographical scoring guide – level 4 for the same task with at least three years, own responsibility and recent; level 3 for the same task under three years or a related task via a shared skill; level 2 for a related occupation only or the same task long ago in a supporting role; level 1 for the same occupation group without shared task or skill; no path, no evidence – and chooses the attributes that count: task match, duration, recency and responsibility are on, years alone, employer name and education level are off by default.

A template per organisation or role family pre-fills these; changed anchors are flagged for review; the rules can be applied to all requirements at once. In the back-end scoring guides are configuration, not code: a versioned rule set per requirement with link-type weights, attribute rules and knock-out logic, fixed before the first candidate is scored.

Step 3, package and staging. A matrix shows which requirement is measured with which evidence; the employer chooses the staging strategy – all at once, sequential for everyone, or staged by invitation with the passport first and questions for a shortlist – and sees the effort the candidate will be shown. Publishing composes the package from the rule sets (questions and probes only where needed), stores the staging strategy as a setting on the position and freezes profile, rules and package together.

#11.4 Candidate flow

The candidate flow follows the rule that the platform owns the passport and the AI services fill and map it, and the promise that nothing a candidate recorded is asked again. The platform team owns the passport screens – record types with add forms, CV upload and proposal review, edit forms with pickers, the status overview, apply and reuse – the passport record itself (typed records with taxonomy references, source span, validation status and versions; identity stored apart; consent for reuse across positions) and the assessment delivery that shows per position what is covered and runs the staged package for the uncovered requirements only. The AI & Data team owns the CV extraction and mapping (the existing extraction producing entries and task phrases, a mapper to occupation, task, skill and knowledge nodes with a confidence per proposal), the taxonomy services for autocomplete and suggestions and the normalisation of duration, recency and level, and the coverage computation of the matching service.

Figure 26 – Candidate flow: start a passport, quick fill from the CV, adjust an entry, confirm and apply.

Step 1, start a passport. The candidate sees five record types – education, experience, certificates, skills and tasks, achievements – each with an add form and an empty state that explains what a record is, and the shortcut to fill the passport from a CV. Step 2, quick fill from the CV. After one upload, proposals appear per record type, each to accept, edit or remove; suggested skills are marked as suggestions; nothing is saved until it is accepted. The extraction produces the entries and task phrases, the mapper links them to taxonomy nodes and keeps the source span; proposals are drafts with a confidence flag, and accepting turns a draft into a record with the status confirmed. Step 3, adjust and complete.

The edit form offers the role from the occupation list, the tasks as chips with suggestions from the Ontology Store, period and responsibility as structured fields, and a visibility switch for the employer name that is off for blind review; the source span and the status are shown. Normalisation computes duration, recency and level; the edited record keeps its provenance. Step 4, confirm, apply, reuse. The overview lists the records with their statuses; for a position it shows the coverage – three of four requirements in the example – and what the package still asks, so that the candidate starts an assessment that contains only the missing evidence, or reuses the passport for another position.

Coverage is computed by the matching service from passport and requirements profile through the taxonomy; the staging engine builds the package from the uncovered requirements; reuse across positions is consent-based, and records are versioned when they change.

#11.5 Reviewer flow

The reviewer flow follows the rule that the AI scores each source and the platform keeps the sources apart. Three scorers produce the levels: the biographical scorer applies the rational key over taxonomy paths, Candidate Evaluation scores the anchored questions, and criteria scoring handles the work probes; each has its own guide.

For each level the AI side also returns a certainty – derived from the number of independent sources, the validation status of the record, the agreement between sources and the length of the taxonomy path – and an explanation built from the stored path, rule, anchor and source span, not from free text. Aggregation happens per source; combination happens only per requirement, and only for the reviewer's eyes.

The platform owns the review screens, the review record (judgements, comments and reveal events per requirement, with the rule-set and profile versions that produced the levels) and the no-total rule: it never sums levels, sorts and filters per requirement only, and treats knock-outs as filters rather than ranks.

Figure 27 – Candidate profile view: overview, skills profile per requirement with the three source levels and their certainty, evidence detail per source.

The candidate profile view is structured from overview to detail. At the top, coverage, the knock-out status, the number and status of records and the agreement between sources – and, deliberately, "total score: none by design". In the middle, the skills profile as a picture: for each requirement the passport, question and probe levels as separate bars against the requirement's threshold band, with the certainty indicator beside them. At the bottom, the evidence per source, opened one at a time; for the biographical source every level with its taxonomy path, the record, the rule that produced it and the source span; questions and probes collapsed with their aggregates until opened. Certainty is about support, not quality: three independent sources that agree on a confirmed record with a direct path give high certainty; a single self-reported source gives low certainty; missing evidence lowers certainty, never the level.

Figure 28 – Comparison view across candidates: per requirement, source by source, with filters and decision chips – no total column.

The comparison view shows all candidates of a stage in one table, per requirement the three source levels with certainty, with filters for knock-out met, coverage and certainty, sorting per requirement only, and a decision chip per candidate. It draws on the same scorecard entries, aggregated per source across candidates. Group-gap monitoring per requirement column runs in the insights service and flags a column, not a candidate. Identity stays hidden throughout; reveal is a logged action governed by the position's workflow setting.

View Markdown source

Part III · B

Roadmap and plan forward

How the extension is built and proven: one proof of concept from existing capabilities over three sprints, and the four questions that decide whether the biographical part reaches production.

View Markdown source

#12 Follow-up: one proof of concept in three sprints

The concept is not built from scratch. Three capabilities that exist or are in preparation are unified into one initial proof of concept. The requirements profile extraction is Sprint 2 deliverable 1.2 and extends the Ontology Store; the existing CV extraction becomes the CV-assisted background step; and the assessment concept of this report – evidence profile, scorecard, methods, staging – is the model the two extractions are mapped to.

In the second week of Sprint 2 the concept is separated into work streams by capability and, per stream, the requirements (inputs, outputs and contracts between extraction, Ontology Store and scorecard), the goals (which reviewer feedback and customer-case outcomes the first version addresses) and the validation strategy (how the prototype is evaluated in Sprint 4: vacancies, populations, comparison against the current flow, bias indicators) are defined.

Sprint 3 builds the unified proof of concept, one flow from vacancy to evidence profile. Sprint 4 evaluates the prototype on real vacancies against the current flow.

Figure 29 – One proof of concept from three existing capabilities; the goal of the second sprint week.

Figure 30 – Three sprints – define, build, evaluate – with the work streams by capability (concept presentation, slide 39).

Table 5 – The three-sprint plan by work stream.

Seven work streams carry the plan: work streams and requirements (define); the Ontology Store extension from requirements profiles to the evidence profile (define, build); the requirements profile extraction mapped to the unified profile; the CV extraction as background step; the scorecard and evidence profile; the unified proof of concept (build); and the evaluation of the prototype (evaluate). Owners per stream are assigned in the sprint planning.

View Markdown source

#13 Open questions for the follow-up

Four questions decide whether the biographical part can go into production. They are stated with the options and considerations we see today; none of them is decided here, and the third is explicitly a request to the legal review rather than a legal opinion.

#13.1 A – A valid training and evaluation corpus for CV ingestion

Every biographical level rests on the ingestion step – extract, map, standardise – and its quality can only be claimed against a corpus that is representative, labelled and legally usable. What has to be labelled per CV are the span-level entries (experience, education, certificate, task phrases), their taxonomy concepts (occupation, task, skill, knowledge, ISCED level) and the normalised attributes (duration, recency, responsibility); agreement between two annotators per field is the first validity measure of the corpus itself. Four sources are candidates, and they differ in realism and in what they may be used for. Pilot CVs already in the platform are realistic but are personal data collected for a different purpose (see C).

Candidate-confirmed passport records from the new flow produce labels as a by-product of the confirmation step and are the most defensible source once consent covers it. Synthetic CVs generated from the taxonomy give coverage of rare occupations, languages and formats, which is useful for recall testing but not for a validity claim. Public or licensed corpora, such as ESCO-annotated job and CV datasets, need a check of their coverage of Dutch public-sector vocabulary. Whatever the mix, the corpus has to be stratified by language, sector, seniority, CV style and the groups we monitor, because the extraction-recall measure of section 9.6 cannot be computed per group otherwise. The decisions needed are which source may be used for which purpose (evaluation only versus tuning), the target size per stratum, and who annotates.

The proposal for the work stream is to start with evaluation-only use of confirmed passport records and a synthetic recall set, and to tune only once C is answered.

#13.2 B – Mitigating bias in practice

The design-time controls are in the concept: blind entries; a rational key over tasks and skills rather than occupation similarity; proxies off by default; no total; knock-outs that filter instead of rank; separate scores to limit anchoring. What practice adds is measurement and intervention. The internal insights service computes, on every scored population, the group gap, the pass-rate difference and the balance point from the Bias Monitoring Framework [14] per requirement and per link type, the extraction recall and correction rate per group, and agreement with and without the CV score visible. The intervention repertoire is deliberately small and readable: down-weight or disable a link type, exclude a node, add a transferable link to the taxonomy, change an anchor – each a rule change with an audit entry and a re-measurement.

For the learned components, the extractor and the mapper, constrained or debiased representations are evaluated on performance and segregation risk together, as [9] recommends. What the concept does not yet give is thresholds (which gap triggers which action), a review cadence per customer and a sign-off for rule changes; the proposal is to set thresholds after the first vacancy in Sprint 4 rather than before, and to have rule changes reviewed by the AI team together with the customer's assessor and logged with their measured effect. One limit has to be stated plainly: group variables are only available where a customer may legally collect them.

Where they are not, the system can still measure how much proxy attributes influence levels, but it cannot measure group gaps, and the fairness panel must say so rather than show a reassuring blank.

#13.3 C – Legal implications and GDPR constraints

This subsection lists the questions a legal review has to answer; it is not legal advice, and the AI team does not own these decisions. They are raised now because their answers shape the training corpus and the bias measurement.

Purpose and legal basis. Candidate data was collected to assess an application. Using it to train or tune extraction and scoring models is a further purpose that needs its own basis and a compatibility assessment (GDPR Art. 5(1)(b), 6(4)). The data processing agreements with the pilot customers have to be read for whether model improvement is covered; if it is not, the options are consent of the candidates, a documented legitimate-interest assessment, or an amendment of the agreement per customer. Evaluation-only use on a customer's own vacancy is easier to justify than cross-customer tuning.

Special categories and group variables. Bias measurement needs group variables, some of which are special categories (Art. 9) or sensitive in practice. Collecting them solely for fairness monitoring, separated from the records, aggregated and with minimum group sizes is the usual pattern, but it has to be in the processing agreement and the privacy notice, and some customers will set their own conditions – SZW's written privacy requirements and FG clearance [2] are an example. Automated decision-making and transparency.

The design keeps a human judgement per requirement and no automatic rejection; knock-outs used as filters come close to Art. 22 and require the reviewer step to remain a real one. Candidates have to be told that a CV is standardised and scored, how, and how to correct it – the confirmation step is also the Art. 13, 15 and 16 mechanism. AI Act.

Recruitment and selection systems are high-risk under Annex III; the documentation already prepared for ISO/IEC 42001 and Article 10 data governance has to cover the biographical scorer – data sets, representativeness, bias examination, logging, human oversight – and the six properties of section 3.3 map onto it. Retention and reuse. A passport reused for another position or organisation is a new purpose, and applicant-data retention periods conflict with a long-lived passport unless the candidate owns it and consents, which is TNO's ownership model and should be the product's.

The concrete asks for the legal work stream are five: read the pilot agreements for model-improvement and fairness-monitoring purposes; decide per corpus source whether evaluation and tuning are permitted; draft the candidate-facing text for the quick-fill and confirmation steps; define the collection and separation of group variables; and confirm the retention and consent model for passport reuse.

#13.4 D – Validating validity and bias-proneness

Validation proceeds from the ingestion to the claim. Extraction validity is precision and recall of entries, concepts and attributes against the annotated corpus of A, per language and sector, with the candidate correction rate in production as the running proxy. Scoring consistency is the agreement between the biographical scorer and qualified reviewers who see only the standardised entries, with agreement between reviewers as the ceiling; the Box 3 indicator set [1] is extended to biographical levels. Added information asks whether passport levels add information next to question and probe levels – incremental agreement with the final human judgement, not correlation with hiring.

Predictive validity is only possible with performance criteria after hiring; the customer cases with a baseline, IND and TalentLead 1 [2], are the candidates for a first local study, and until then the validity claim stays at the method level of section 3.1 and is labelled as such – which is also what [4] recommends when local studies are feasible. Bias-proneness is measured as group gaps and balance points per requirement and link type, the sensitivity of levels to proxy attributes, and a comparison of the rationally keyed score with a naive baseline on the same blind entries; that difference is the measured value of the method.

The protocol is the Sprint 4 evaluation on at least one new vacancy with the current flow as comparison, with populations, outcomes and separate tests defined in the validation strategy of the second sprint week, and with results flowing into the internal insights service so that they are not a one-off report.

View Markdown source

#14 References

[1] Vass, B. (2026). Box 3 reviewer feedback, linked to the evaluation data. ObjectivEye internal analysis, 16 October 2026.

[2] ObjectivEye (2026). Objectives and priorities – main objectives, customer cases and priorities, AI backend priorities, capability mapping. Internal document.

[3] Sackett, P. R., Zhang, C., Berry, C. M. & Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068. https://gwern.net/doc/statistics/meta-analysis/2021-sackett.pdf

[4] Sackett, P. R., Zhang, C., Berry, C. M. & Lievens, F. (2023). Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors. Industrial and Organizational Psychology, 16(3), 283–300. https://www.cambridge.org/core/journals/industrial-and-organizational-psychology

[5] ObjectivEye (2026). Biographical data and skills-based matching – from traditional scoring methods to evidence-based skills profiles. Internal methodological note.

[6] Stokes, G. S. & Searcy, C. A. (1999). Specification of scales in biodata form development: Rational vs. empirical and global vs. specific. International Journal of Selection and Assessment, 7(2), 72–85. https://doi.org/10.1111/1468-2389.00108

[7] Cucina, J. M., Caputo, P. M., Thibodeaux, H. F. & Maclane, C. N. (2012). Unlocking the key to biodata scoring: A comparison of empirical, rational, and hybrid approaches at different sample sizes. Personnel Psychology, 65(2), 385–428. https://doi.org/10.1111/j.1744-6570.2012.01244.x

[8] Speer, A. B., Siver, S. R. & Christiansen, N. D. (2020). Applying theory to the black box: A model for empirically scoring biodata. International Journal of Selection and Assessment, 28(1), 68–84. https://doi.org/10.1111/ijsa.12271

[9] Adhikari, A., Vethman, S., Vos, D., Lenz, M., Cocu, I., Tolios, I. & Veenman, C. J. (2024). Gender mobility in the labor market with skills-based matching models. AI and Ethics, 4(1), 163–167. https://link.springer.com/article/10.1007/s43681-023-00410-5

[10] Bertrand, M. & Mullainathan, S. (2004). Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American Economic Review, 94(4), 991–1013.

[11] TNO (2022). Het skillspaspoort: mogelijke oplossing voor 'vrij reizen' over de arbeidsmarkt. https://www.tno.nl/nl/newsroom/2022/01/skillspaspoort-oplossing-arbeidsmarkt/

[12] TNO (2025). Grootste experiment met skillspaspoorten. https://www.tno.nl/nl/newsroom/insights/2025/02/grootste-experiment-skillspaspoorten/

[13] TNO (2026). Pilot skillspaspoort in de Metalektro. https://www.tno.nl/nl/newsroom/papers/pilot-skillspaspoort-metalektro/

[14] ObjectivEye (2026). Bias Monitoring Framework – AI–human and reviewer consistency, group fairness and parity, selection-threshold impact. Internal document, May 2026.

[15] ObjectivEye (2026). ObjectivEye_Assessment_Concept.pptx, concept presentation (39 slides); ObjectivEye_Sprint2_Update_19Oct.pptx, sprint update; illustrations ObjectivEye_CV_to_match_flow, _flow_employer, _flow_candidate, _review_candidate_profile, _review_compare (PNG/SVG).

[16] Lenz, M. (2026). Generalised evidence-based assessment ontology – motivation, research basis, development priorities, Layers 1–3 and system-wide adoption. ObjectivEye internal concept note.

[17] ObjectivEye (2026). The ObjectivEye platform – services, gateway routing and upstreams (platform documentation, dev environment, revision 0345e03).

[18] ObjectivEye (2026). Ontology Service – General Skills Taxonomy, semantic search, extraction and matching (service documentation).

[19] ObjectivEye (2026). Candidate Evaluation Service – asynchronous scoring, pipeline stages, operations (service documentation).

[20] ObjectivEye (2026). Assessment items & when they work well – description, question, rubric; core quality checks (internal note).

[21] Lenz, M. (2026). AI lifecycle management framework – milestone achievement report, May 2026 (UNIIQ).

[22] ObjectivEye (2026). Generation Service – tasks, feedback counterparts, configuration (service documentation).

Sources [3], [4], [9], [11], [12] and [13] were re-read for this report; [14], [17]–[22] are the internal documents supplied for Part II; [6] and [7] were checked through their published abstracts (Stokes & Searcy: higher validities and less faking with empirical keys overall, specific rational scales as predictive as the empirical item key; Cucina et al.: rational keying lowest, hybrid best at small samples). [8] is cited through the methodological note [5]. All figures in the illustrations are fictitious.

View Markdown source