Influence of pairing in examiner leniency and stringency ('hawk-dove effect') in part II of the European Diploma of Anaesthesiology and Intensive Care: A cohort study.
Options
BORIS DOI
Publisher DOI
PubMed ID
39194037
Description
Background
The European Diploma of Anaesthesiology and Intensive Care (EDAIC) Part II examination is a supranational examination for anaesthesiologists.Objectives
We explore the impact of examiner pairing on leniency and stringency, commonly referred to as the 'hawk-dove effect'. We investigate the potential variations in grading approaches, resulting from different examiner pairs and their implications for candidate performance.Design
Retrospective cohort, observational design.Setting
EDAIC Part II examination data from 2021 to 2023.Participants
Three hundred and twenty-five examiners across 122 EDAIC Part II examination sessions.Interventions
We analysed the influence of examiner leniency and examiner pairing on candidate performance in the EDAIC Part II using many-facet Rasch modelling.Main Outcome Measures
The study's main outcome measure was determining a leniency score among the examiner population. The study also aimed to assess how examiner pairing influenced candidate performance, as measured by their scores in the examination.Results
During the study period, the number of examiners who participated in 2021, 2022 and 2023 were 253, 242 and 247, respectively. The median sessions attended were 7.0 (3 to 10). The examination data revealed a mean leniency score of 0 [95% confidence interval (CI) -0.046 to 0.046], with the standard deviation being one-third that of the candidates' ability scores. There were 1424 different pairs of examiners, with most pairs (97%) having only a one-point difference in marking. The mean leniency score for the pair of examiners was -0.053 (95% CI -0.069 to -0.037).Conclusion
The variations in grading approaches associated with different pairings emphasise the potential for the 'hawk-dove effect' to influence candidate performance and outcomes. Understanding these variations can guide curriculum development, examiner training and coupling, ensuring a balanced and equitable assessment process.Trial Registration
None.
The European Diploma of Anaesthesiology and Intensive Care (EDAIC) Part II examination is a supranational examination for anaesthesiologists.Objectives
We explore the impact of examiner pairing on leniency and stringency, commonly referred to as the 'hawk-dove effect'. We investigate the potential variations in grading approaches, resulting from different examiner pairs and their implications for candidate performance.Design
Retrospective cohort, observational design.Setting
EDAIC Part II examination data from 2021 to 2023.Participants
Three hundred and twenty-five examiners across 122 EDAIC Part II examination sessions.Interventions
We analysed the influence of examiner leniency and examiner pairing on candidate performance in the EDAIC Part II using many-facet Rasch modelling.Main Outcome Measures
The study's main outcome measure was determining a leniency score among the examiner population. The study also aimed to assess how examiner pairing influenced candidate performance, as measured by their scores in the examination.Results
During the study period, the number of examiners who participated in 2021, 2022 and 2023 were 253, 242 and 247, respectively. The median sessions attended were 7.0 (3 to 10). The examination data revealed a mean leniency score of 0 [95% confidence interval (CI) -0.046 to 0.046], with the standard deviation being one-third that of the candidates' ability scores. There were 1424 different pairs of examiners, with most pairs (97%) having only a one-point difference in marking. The mean leniency score for the pair of examiners was -0.053 (95% CI -0.069 to -0.037).Conclusion
The variations in grading approaches associated with different pairings emphasise the potential for the 'hawk-dove effect' to influence candidate performance and outcomes. Understanding these variations can guide curriculum development, examiner training and coupling, ensuring a balanced and equitable assessment process.Trial Registration
None.
Date of Publication
2024-12
Publication Type
Article
Language(s)
en
Contributor(s)
Sciberras, Stephen | |
Klimek, Markus | |
Ateleanu, Bazil | |
Scipioni, Hugues | |
Di Loreto, Rodolphe |
Additional Credits
Institut für Medizinische Lehre, Assessment und Evaluation, Forschung / Evaluation
Series
European Journal of Anaesthesiology
Publisher
Lippincott, Williams & Wilkins
ISSN
0265-0215
Access(Rights)
open.access