Skip to main content
. 2020 Apr 28;9:8. Originally published 2020 Jan 9. [Version 2] doi: 10.12688/f1000research.21046.2

Figure 1. Experimental design to assess inter-rater reliability of the Unified Career Outcomes Taxonomy 2017.

Figure 1.

In Experiment 1, six coders comprising four experienced coders (blue) and two inexperienced coders (orange) classified 572 PhD alumni records from three different institutions using UCOT Exp1 and the Draft Guidance Document. After measurements of inter-rater reliability were determined, the working group convened to refine the career taxonomy definitions and guidance materials, generating experimental version 2 (UCOT Exp2) and the Revised Guidance Document. The reliability of UCOT Exp2 and the Revised Guidance Document were then assessed using a new data set of 219 PhD alumni records. In Experiment 2, coding was performed by five experienced coders ( blue) and three new inexperienced coders ( orange). After classification of the records, inter-rater reliability was measured. The group convened again to review the results and introduce further clarifications to the Revised Guidance Document, therein creating the Final Guidance Document.