--- title: 3 Methodological requirements --- # 3 Methodological requirements ## 3.1 What the selection research says about validity and bias The meta-analytic picture of personnel selection was revised in 2022. Sackett, Zhang, Berry and Lievens \[3\] showed that the corrections for range restriction used since Schmidt and Hunter (1998) had systematically overstated the validity of most predictors, and they re-estimated operational validities with more defensible corrections. The order of predictors changed as a result. Structured interviews now have the highest mean validity (ρ = .42), followed by job-knowledge tests (.40), empirically keyed biodata (.38) and work samples (.33). Cognitive ability tests fell from .51 to .31, integrity tests to .31, and the predictors that selection practice relies on most casually – unstructured interviews (.19), years of job experience (.07) – stay at the bottom. Rationally keyed biodata sits at .22. The same paper reports Black–White standardised mean differences: .23 for structured interviews, .33 for both forms of biodata, .54 for job-knowledge tests, .67 for work samples and .79 for cognitive ability. In their follow-up focal article \[4\] the authors draw the design consequences. They favour job-specific predictors – those that sample job behaviour rather than infer a general construct – over measures of general psychological constructs. They argue that the long-standing validity–diversity dilemma has largely dissolved: because the validity of cognitive ability has fallen, composites with and without it have nearly identical mean validity, and dropping ability from a six-predictor composite costs .05 of validity under the new estimates instead of .20 under the old ones; equally valid composites with smaller group differences can therefore be built. They also warn against reading the ranking as a mandate: the validity of structured interviews is "really .42, plus or minus .24" across its credibility interval, work samples are less applicable where the relevant skills are learned after hiring, and a risk-averse employer should compare predictors on the lower end of their credibility interval as well as on the mean. Local validity studies retain their evidentiary value. | **Method** | **ρ (2022)** | **B–W d** | **What it means for a digital, semi-automated flow** | |---------------------------------------|--------------|-----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Structured interviews | .42 | .23 | Highest validity, wide credibility interval; live interviews out of scope – written, behaviourally anchored items are the digital relative and need their own validation. | | Job-knowledge tests | .40 | .54 | Feasible digitally where specific knowledge is a justified requirement; moderate group difference; later. | | Empirically keyed biodata | .38 | .33 | Strong once outcome data exists; the target of the calibration path in section 9.2. | | Work samples | .33 | .67 | Written work probes; the largest group difference of our formats and therefore the closest fairness check. | | Cognitive ability | .31 | .79 | Not pursued: smallest contribution to a composite relative to its adverse impact. | | Rationally keyed biodata | .22 | .33 | Our starting point for CV scoring; needs no outcome data, is explainable, and must be validated locally. | | Unstructured interviews · years alone | .19 · .07 | .32 · .49 | What naive CV screening amounts to; avoided by design. | *Table 1 – Corrected operational validities and Black–White standardised mean differences from Table 3 in Sackett et al. (2022) \[3\], with the consequence for ObjectivEye.* ## 3.2 What follows for ObjectivEye A method, in this concept, is a defined combination of evidence collection and a scoring guide. That definition matters because validity belongs to the procedure in use, not to a format or a scale. Three cautions follow directly \[16\]. The categories in Table 1 are research categories, and their estimates do not transfer automatically to a particular digital or AI-assisted implementation. Behaviourally anchored rating scales are a scoring approach within a method, not an independently validated method; the same holds in the other direction – normalised CV information is not automatically validated biodata, and a written assessment item is not automatically a structured interview. And predictive validity, assessor agreement and bias are three different properties that must be evaluated separately: agreement between the AI and a reviewer supports a scoring-consistency claim but does not establish prediction, and a group-score difference is not by itself a test of measurement bias. With those cautions the development priorities are as follows. Near term: structured education, experience and biodata scorecards, because they broaden the evidence model while building on the existing evaluation structure and give the reusable evidence that recommendations and talent pooling need; and written work-probe evaluation – an essay, a project plan, another deliverable – scored with task-specific structured rubrics that can reuse the existing scoring mechanism. Continued and integrated: the existing assessment items with behaviourally anchored rubrics, as one component of the broader model. Later: conversational structured assessment as a digital adaptation of the structured interview; it could be more engaging and more resistant to externally generated answers, but adaptive questioning, consistent scoring and validation need substantial development. Unstructured interviews, holistic CV judgements, credentials or years of experience alone, and general personality or cognitive tests are lower priorities: they provide less directly traceable task evidence or require additional justification of relevance, validity and bias \[16\]. ## 3.3 What every method has to document before use Without sufficient support a method remains experimental, and its scoring guide must not imply a validation it does not have. Before a method is used, six properties are documented – for the method from the research, for the implementation from local evidence. The specification names the evidence elicitation, the item or task content, the scoring guide, the score format and the target requirements, together with language, model version and allowed tools. The research support names the relevant studies, their populations and performance criteria and the limits of transfer to the proposed method. Predictive validity states why scores should relate to the intended performance criterion and how this is evaluated locally. Bias and fairness covers content relevance, group outcomes, accessibility and differential prediction, examined separately. Scoring consistency covers agreement between qualified assessors and the consistency of automated or AI-assisted scoring against the guide. Efficiency covers candidate completion effort, assessor effort, dropout and the information gained per stage. For the three formats of this concept the status differs: the question items have an item bank and rubrics and a measured AI–reviewer agreement; biographical scoring has its fields and normalisation rules to specify and its extraction accuracy to measure; work probes have a task bank to define and their consistency still to measure. Translated or AI-assisted adaptations need explicit evaluation in every case. ![Figure 4 – The six properties every method documents, with the status per format.](../media/image5.png) *Figure 4 – The six properties every method documents, with the status per format.* ## 3.4 Scope of this concept In scope is the digital application process: evidence capture and semi-automated assessment; the existing behavioural items retained, with education, experience and biodata evidence and written work probes added; one requirements profile per position and one reusable evidence profile per candidate; requirement-level scorecards as the translation layer. Out of scope are live interviews and in-person observation, conversational structured assessment as a later research and development option, a validated matching algorithm – enabled by the shared model but not specified here – and any certification of the current implementation or a validated assessment manual. The concept assumes no new overall scoring formula, validity threshold or automatic rejection rule, no new ontology classes beyond proposed fields and relationship names that are for review, and no delivery date, owner or sprint commitment; access and candidate permission for reuse across organisations remain explicit design decisions. Customer feedback is design input, not a quantified research finding, and the screens in this report are illustrative.