Generalisability analyses of the College of Intensive Care Medicine Second Part examination: Theory meets practice

We thank Bihari et al. for the endorsement of our findings¹ and welcome the opportunity to clarify the points they raised.² We believe in transparency and fairness in high-stakes examinations and are grateful to the College of Intensive Care Medicine (CICM) for supporting our external analysis to complement their internal reviews. We hope to encourage open discussion of the assessment process to further advance reliable and valid credentialing of intensive care specialists for independent practice. We have structured our response to parallel comments by Bihari et al.

We agree that the coefficients presented in our article are indices of dependability (Φ) rather than the relative coefficient (Eρ²). This was a deliberate choice, as candidates were judged against an absolute standard, and as a consequence, variation in station difficulty and examiner stringency affects the reproducibility of scores and pass/fail decisions. In contrast, coefficient alpha is a relative coefficient and will always be at least as large as dependability coefficients estimated from the same dataset. It is generally inappropriate to use coefficient alpha as an index of precision for Objective Structure Clinical Examinations (OSCE) because of the multiple sources of measurement error present, particularly when multiple circuits are used in test administration.[3], [4], [5] Although the benchmark of >0.8 is seldom achieved in OSCE-type examinations, even when coefficient alpha is (inappropriately) used as an index, we consider >0.8 an appropriate standard for high-stakes decisions.

We agree that it would be ideal to use indices of classification consistency and classification accuracy; however, these are difficult to estimate with the structure of the CICM Second Part examination. As a step in this direction, we report the (absolute) standard error of measurement (SEM) for various test lengths below (Table 1). SEMs quantify the expected variation in scores across administrations of similar (randomly parallel) examinations, and they are related closely to indices of decision consistency. The SEMs in the table are large for the test length used, particularly given the high failure rate, indicating that a substantial proportion of pass/fail results would change if candidates were retested with different sets of stations and examiners.

Table 1.

Re-reported D-study with projected generalisability coefficients including the (absolute) standard error of measurement (SEM). Bold values identify the current assessment length.

Total number of stations	Viva only G-coefficient	SEM	Viva stations + 2 Hot Cases G-coefficient	SEM
2	0.28	0.91	0.26	0.96
4	0.44	0.65	0.42	0.68
6	0.54	0.53	0.52	0.55
8	0.61	0.46	0.59	0.48
10	0.66	0.41	0.64	0.43
12	0.70	0.37	0.68	0.39
14	0.73	0.35	0.72	0.36
16	0.76	0.32	0.74	0.34
18	0.78	0.30	0.76	0.32
20	0.80	0.29	0.78	0.30

SEM, standard error of measurement.

Secondly, we agree that attributing 1.4% of the variance to examiner stringency (Hawk–Dove effects) underestimates the true examiner effect. As noted in our article,¹ examiners were permitted to discuss and amend scores prior to submission and only these final scores were recorded in the CICM examination database. After this analysis was completed in 2025, we recommended that the CICM record prediscussion scores to enable more accurate analysis of variation in examiner stringency for calibration and quality assurance. We hope this has been integrated into the CICM examination procedure.

Thirdly, we calculated indices of uncertainty; however, we did not report them in the article due to manuscript length and statistical complexity. We calculated an (absolute) SEM to estimate the likely variation in scores if a candidate was tested repeatedly with different stations and examiners. Table 1 re-reports our findings with the addition of SEMs as a function of test length. Examiners marked Viva stations on a 0 to 10 scale, with a mean score and standard deviation of 5.6 and 1.47, respectively,¹ so we have used this scaling in Table 1 to ease interpretation. SEMs can be used to form confidence intervals around a candidate's score. For example, for a candidate who scored the overall mean score of 5.6, a 95% confidence interval ( ± 2 SEMs) would range from 4.68 (fail) to 6.52 (clear pass).

Finally, our justification for conducting an analysis considering the two Hot Cases as Viva stations is, in part, related to feedback from the Australian Council for Education Research. When this research was conducted, the CICM combined the Hot Case and Viva scores into a single Oral Examination score, with a pass requiring 50%. This meant it was technically possible to fail the Hot Case or Viva component, but pass the examination overall due to compensatory marking. To address this, the Australian Council for Education Research recommended separating the Hot Case and Viva components, with independent pass requirements to ensure construct coherence and defensible decision making. Our benchmarking of the exam design at the time (combining the Hot Case and Viva scores) showed only modest increases in reproducibility of (composite) scores when aggregated, as expected. Given the reliability coefficients of 0.42 and 0.61 for the Hot Cases and Viva respectively, both components would require improvement to support independent pass/fail decisions.

We agree that a stratified composite model is statistically preferable. However, we are unaware of any statistical packages that can accommodate this complexity given the nested design of the examination. Additionally, heterogeneity of Viva stations (clinical, radiology, procedure and communication stations) is commonly ignored in statistical modelling the reproducibility of scores and pass/fail decisions. The Viva could be viewed as a test administration framework that permits diverse assessment formats to be combined, allowing flexibility in the skills that are assessed.³ Given the Viva and Hot Case scores were combined numerically for overall pass/fail decisions, treating the Hot Cases as additional stations provided a reasonable approximation for evaluating the existing system. Weighting the less reliable Hot Case component more heavily would reduce the composite reliability.

We acknowledge that, since the period covered by our dataset, improvements to the Second Part Examination have been implemented. Ongoing scientific evaluation of assessment processes is essential for transparent, fair and defensible decision making. Continued internal and external evaluations will support the CICM's commitment to high-quality assessment and the future of intensive care medicine in Australia and New Zealand.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

CRediT authorship contribution statement

Kenneth R. Hoffman Writing Original Draft, Writing Review and Editing, Project Administration. David Swanson Writing Original Draft, Writing Review and Editing. Stuart Lane Writing Review and Editing. Christopher P. Nickson Writing Review and Editing. Anna T. Ryan Writing Review and Editing.

Conflict of interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Assoc. Prof. Kenneth Hoffman Fellow of the CICM Workplace Based Assessments subcommittee 2025 CICM First Part Examiner 2026. Prof. Anna Ryan: No conflicts of interest. Prof. Stuart Lane Fellow of the CICM CICM Second Part examiner 2011–2023 CICM Second Part examination committee 2019–2023 Chair of the CICM Second Part examination panel 2020–2023 CICM First Part examiner 2012–2019. Assoc. Prof. Chris Nickson Fellow of the CICM CICM First Part examiner 2017-present CICM First Part examination committee 2019-present CICM Supervisor of Training 2018-present. Prof. David Swanson No conflicts of interest.