Skip to main content
PLOS One logoLink to PLOS One
. 2024 Nov 8;19(11):e0313476. doi: 10.1371/journal.pone.0313476

Consistency between inter-institutional panels using a three-level Angoff-standard setting in licensure tests of foreign-trained dentists in Sweden: A cohort study

Jesper Dalum 1,*, Liselotte Paulsson 2, Nikolaos Christidis 1, Mikael Andersson Franko 3, Klas Karlgren 4,5, Charlotte Leanderson 6, Gunilla Sandborgh-Englund 1
Editor: Andrej M Kielbassa7
PMCID: PMC11548739  PMID: 39514553

Abstract

Licensure exams play a crucial role in ensuring the competence of individuals entering a profession, thereby safeguarding the public and maintaining the quality and integrity of the profession. In Sweden, dentists educated outside the European Union seeking to practise dentistry must undergoa re-certification process. The re-certification process includes a theoretical examination where pass marks are set using a three-level Angoff method. This study aimed to determine the consistency of the Angoff ratings using independent panels at two Swedish universities. Two cohorts of panellists were included in the study: one reference and one external. The reference panel was responsible for rating the upcoming theoretical examinations in the proficiency test, which were used to set the pass mark. The external panel, recruited from a dental school at a university in another region in Sweden, provided ratings after the examinations. Three examinations during 2019–2020 were included in this study (267 items in total). There was a strong correlation (ρ ≥ 0.70, p < .001) between the ratings of the two independent panels, with no significant differences in item ratings across the full exams, dental disciplines, and professional qualifications analysed. This suggests that the three-level Angoff method reliably produces similar standards for assessing the competence of the minimally qualified dentist across different institutions. The expectations of the minimally qualified but still acceptable dentist were comparable between the two independent panels across the three theoretical examinations explored. The alignment between the panels indicates valid, reliable standards across institutions, despite the independent syllabi of the two study programmes. However, while there is an alignment, differences in ratings remain. Consequently, involving multiple institutions in future standard-setting processes could help ensure that the standards reflect a broader range of educational practices, supporting the credibility of licensure examinations.

Introduction

Licensure has a crucial function in ensuring the competence of individuals entering a profession and safeguarding the public, consequently maintaining quality and integrity within the profession. Dentists educated outside of the European Union who seek to practise dentistry in Sweden can undergo a re-certification process for licensure [1, 2], which involves two examinations: a theoretical exam and a clinical skills assessment. The examinations are administered by the Department of Dental Medicine at Karolinska Institutet. After passing these examinations, a six-month clinical training period follows [1, 2]. The examinations used in the re-certification process are competency-based, and the blueprint and standard setting are aligned with the standards of a dentist in Sweden for licensure, i.e., the learning outcomes of a Swedish university degree in dental surgery [2, 3].

In competency-based proficiency tests, it is important to establish reliable and valid measures aligned with the expected standards [4, 5]. There are several methods to address the challenge of setting a defensible pass-mark based on the difficulty level of the test items [4], such as the Angoff [6], Nedelsky [7], and Ebel [8] methods. The Angoff method is commonly used to set the pass markon theoretical examinations [9–12]. In Angoff, a panel of subject matter experts are used to rate each individual question or task. The panel ratings represent what is expected of the minimally qualified but still acceptable participant [12] and is used to distinguish those who have the knowledge or skills required [13, 14]. The original Angoff method does come with challenges, such as the need for several subject matter experts, the difficulty for them to define the minimally qualified participant [15], and the difficulty to estimate how many of these participants would answer a specific task or question correctly [16]. To simplify the method, two major modifications have emerged: a two-level method, where panellists decide whether the minimally qualified but still acceptable individual would answer a test item correctly ‘YES’ or ‘NO’ [17], and a three-level version introducing a ‘MAYBE’ option [18, 19]. In the theoretical examination included in the proficiency tests of non-EU dentists, a simplified version of the three-level Angoff method is used for setting the pass mark. Training and consensus have been redacted, which serves to balance a manageable workload.

The Angoff-panel for the theoretical examination is currently recruited from a single university and concerns how the credibility and reliability of the pass mark on these nationwide examinations may arise. Whilst all Swedish dental programmes adhere to the intended learning outcomes of the Swedish Degree of Master of Science in Dental Surgery, which is required for a Swedish degree in dental surgery [3], they design their programme syllabi to achieve these outcomes independently. Thus, the question arises if the re-certification Ang off ratings and expectations of the minimally qualified are consistently aligned across all Swedish dentistry programmes, something this study aimed to determine as reflected by the ratings from two universities using the three-level Angoff method. The null hypothesis was that there would be no significant difference in the consistency between the two independent panels.

Materials and methods

Two cohorts of panellists were included in the study, a reference and an external panel. The reference panel was used to consecutively rate the regular theoretical examinations of the proficiency test ratings used to set the pass mark of the exam. As part of the study, after the examinations, an external panel was recruited from a dental school at a university in another region in Sweden. This panel rated the same examinations as the reference panel. Both panels rated each question in each examination using a three-level Angoff method. Three examinations from the period 2019–2020 were included in this study (total n = 267 items).

The consistency of the item ratings was analysed for each full test, all items in total, and those items aligned to each dental discipline and learning outcome in the examinations. This was used to assess the method’s reasonableness and explore its credibility at the item level [13, 20]. Additionally, the correlation between item difficulty and the Angoff panellists’ mean item rating was analysed to explore whether participants reached the set Angoff standard within each dental discipline. Fig 1 presents a flowchart illustrating panel recruitment, number of panellists, age, sex, final pass-mark per exam, and item difficulty analysis for each examination, contributing to the consistency analyses of the inter-panel and panel/item difficulty.

Fig 1. Study design flowchart.

Fig 1

Flowchart illustrating panel recruitment, number of panellists, age, sex, final pass-mark per exam, and item difficulty analysis for each examination, contributing to the analysis of inter-panel and panel / item difficulty consistency.

Panel recruitment

It has been debated which kind of subject matter experts to use in Angoff panels [21, 22]. In this study we identified a subject matter expert as a dentist educated around 10 years ago with a mix of current clinical experience and a background of teaching at the programme of dentistry, and the reference panellists were recruited consecutively before each examination from the Department of Dental Medicine at Karolinska Institutet during the period 2019-08-01 to 2020-08-19. The panels comprised clinical teachers in the dentistry programme, with the number of panellists in each test ranging from 10 to 12, while the subject matter experts were a mix of recurring and novice Angoff raters across the three examinations included in the study.

The external panel was recruited from the Faculty of Odontology at Malmö University, Sweden. Recruitment targeted clinical teachers in the dentistry programme and took place between 2021-05-15 and 2021-12-31 following the completion of the three theoretical examinations. The participants were given written information about the study in addition to an online information session. A total of ten subject matter experts were recruited, and those panellists who accepted the invitation were given both written instructions and all test items to review. Notably, external panel members did not have access to the reference panel ratings, and all were novices to the Angoff methodology.

The simplified three-level Angoff item rating process

Test items in the theoretical test are primarily single best answer questions combined with very short essay and drag and drop question types. The panellists individually rated each question ‘YES’‘NO’ or ‘MAYBE’. There was no training- or panel meetings for consensus. The three exams included between 85 and 91 questions.

Variables

The theoretical proficiency test is a criterion-based examination. To systematically ensure adequate sampling of the professional qualifications, test items are created from a blueprint including the intended learning outcomes in the Swedish Degree of Master of Science in Dental Surgery [3]. These learning outcomes, here termed professional qualifications, establish a standard for the profession and constitute the foundation for the competency expected from minimally qualified dentists passing the exam. The blueprint from which the test items were created also includes the representation of dental disciplines, determined by the dimensioning within a Swedish syllabus of the study programme in dentistry. The alignment of professional qualifications and dental disciplines between and within the re-certification examinations has been presented in a previous publication [1]. The analyses in this paper cover a total of ten professional qualifications [3] and eight dental disciplines, namely cariology, oral radiology, prosthodontics, periodontology, endodontology, oral medicine, and paediatric dentistry. The dental disciplines presented in this study were selected from the major dental disciplines in the syllabus, as minor dental disciplines (2–3 questions on each exam) did notenable adequate analyses. The variable presented as Angoff rating is the calculated mean rating of each question from each panel. The Angoff ratings per professional qualification and dental discipline are the mean rating calculated from each question from the respective qualification or discipline, where the item difficulty is the calculated mean scores of the participants, whereas the item difficulty per professional qualification and dental discipline is the mean item difficulty calculated from the respective qualification or discipline.

The three options in the three-level Angoff are assigned weights as follows: ’YES’ is weighted as 1, ’NO’ is weighted as 0, and ’MAYBE’ is weighted as .3. Total scores were calculated by summing the number of ‘YES,’ ‘NO’ and ‘MAYBE’ answers multiplied by their respective weights.

Statistical analysis

Descriptive statistics were used to describe the number of items in each exam and demographic variables of the two panels. Nonparametric methods like medians, Spearman’s rho, Mann-Whitney’s test, and ordinal regression were used throughout since the method of calculating total scores implied that the normal distribution would be a poor model. The consistency of the two panel item ratings was analysed by Spearman’s correlation coefficient (ρ). Spearman’s ρ was also used to analyse the consistency between the panel ratings and the participant group scores. The consistency between panel ratings, and the panel’s rating versus participant group score, was analysed within the total items in each exam, both for dental discipline and professional qualification. A p-value ≤0.05 was regarded as statistically significant. To explore differences between panel ratings, i.e., which panel rated the highest score, the absolute difference was calculated from the median of each panel’s final rating. To explore differences between panel ratings and the participant group score, i.e., if the panels overestimated or underestimated the performance of participants, the absolute difference was calculated from the median of each panel’s final rating and the participants’ group score. Furthermore, the significance of median differences were determined using Mann-Whitney’s test, and regression models were used to visualise the relationship between the panel ratings and participant scores. Ordinal regression of the panellists’ responses (‘YES’‘NO’ or ‘MAYBE’) was moreover used to explore the impact of age and sex of the subject matter experts, where age was categorised and analysed in both groups (over or under the median age of 36 years) and three groups (29–33 years, 34–39 years, and 40–67 years). All analyses were conducted using R version 4.3.2 (R Foundation for Statistical Computing, Vienna, Austria).

Ethical considerations

Ethical approval for this study was granted by the Swedish Ethical Review Authority (Protocol identification number: 2019–06028, April 2nd, 2020). Data collected from the theoretical examination (scores) and reference panellists were gathered as part of regular examination, and consent was not collected, as the collected data do not contain sensitive information and were delivered anonymised for data curation and analysis. Informed consent was obtained from all participants in the external Angoff panel, as their ratings are not part of regular examination procedure. Finally, rater data from the external panel were anonymised upon collection, and a random ID was provided to all external panellists for rating submissions. The study is reported according to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement guidelines [23].

Results

In Fig 1 the number of items, number of participating subject matter experts, age, sex, and final pass mark per test and panel are presented. The two panels consisted primarily of women, which reflects the current sex-distribution of dental teachers in both universities. The age of the reference panel was slightly higher compared to the external panel. Analyses of the potential effects of age and sex showed no statistically significant difference between groups.

Inter-panel consistency

The ratings of the two panels of experts from different universities evaluated the difficulty of the items similarly across all exams, dental subjects, and professional qualifications (ρ≥ 0.70, p < .001). Although the reference panel set slightly higher pass marks overall (73% to 75%) compared to the external panel (72% to 74%), the difference in how the two panels rated individual exam questions was not significant. This consistency suggests that the standards for judging the exams were aligned between the two panels, despite being from different institutions. The average item correlation, the correlation per dental discipline and professional qualification, and the absolute difference between panels by exam are presented in Table 1.

Table 1. Average item correlations and absolute item differences between reference and external panel by exam, divided into dental disciplines, professional qualifications, and total items (n = 267).

Variable Inter-panel item correlation Inter-panel item difference
Exam 1 Exam 2 Exam 3 All items Exam 1 Exam 2 Exam 3 All items
(n = 91) (n = 85) (n = 91) (n = 267) (n = 91) (n = 85) (n = 91) (n = 267)
p p-value p p-value p p-value p p-value Dif. p-value Dif. p-value Dif. p-value Dif. p-value
Dental discipline
 Cariology .96 < .001 .79 < .001 .81 < .001 .87 < .001 .028 1 .050 .254 -.010 .940 .038 .603
 Endodontology .92 < .001 N/A .86 < .001 .94 < .001 .053 .596 -.027 .789 -.100 .466 .001 .485
 Oral radiology .87 < .001 .93 < .001 .79 < .001 .90 < .001 .063 .247 .010 .742 -.047 .271 .015 1
 Orofacial medicine .78 < .001 .79 < .001 .81 < .001 .80 < .001 .039 .631 .080 .176 .074 .364 .011 .574
 Periodontology .81 < .001 .87 < .001 .87 < .001 .87 < .001 .041 .344 .077 .448 -.038 .570 -.013 .852
 Paediatric .91 < .001 .93 < .001 .94 < .001 .93 < .001 -.021 .406 -.048 1 -.017 .759 .011 .916
 Dentistry
 Prosthodontics .83 < .001 .88 < .001 .83 < .001 .88 < .001 .031 .732 -.026 .947 .100 .172 .077 .410
Professional qualification
 PQ 1 .90 < .001 .89 < .001 .84 < .001 .88 < .001 .030 .912 .058 .771 .003 .912 .040 .519
 PQ 2 .91 < .001 .83 < .001 .69 < .001 .88 < .001 .015 .971 .013 .539 .017 .821 .031 .983
 PQ 3 .89 < .001 .89 < .001 .88 < .001 .89 < .001 .050 .850 -.011 .821 .024 .879 .057 .603
 PQ 6 .87 < .001 .91 < .001 .86 < .001 .88 < .001 .101 .353 .047 .582 -.007 .481 .041 .466
 PQ 7 N/A .73 < .001 .90 < .001 .89 < .001 -.040 .820 .007 .842 -.067 .446 -.048 .568
 PQ 8 N/A N/A N/A .90 < .001 .000 .399 -.100 .409 .300 .148 .034 .767
 PQ 9 .93 < .001 .93 < .001 N/A .89 < .001 .071 .520 -.028 .146 .033 .850 .004 .627
 PQ 11 .87 < .001 .84 < .001 .83 < .001 .88 < .001 .105 .280 .015 1. -.029 .650 .032 .633
 PQ 13 .89 < .001 .91 < .001 .86 < .001 .88 < .001 -.006 .631 -.008 .771 -.031 .597 -.004 .949
 PQ 14 .81 < .001 .86 < .001 .92 < .001 .91 < .001 -.006 .880 -.049 .895 -.097 .677 -.034 .663
Total .89 < .001 .89 < .001 .83 < .001 0.88 < .001 .057 .912 0.064 .771 -.002 .912 .032 .755

Statistically significant results in bold. Inter-panel correlations (Spearman rho (p)): ≥0.70 Very strong relationship, 0.40–0.69 Strong relationship, 0.30–0.39 Moderate relationship, 0.20–0.29 Weak relationship, 0.01–0.19 No or negligible relationship. Dif: Inter-panel absolute item difference (median absolute difference). Positive absolute difference indicates a higher Angoff score for the reference panel and negative absolute item difference indicates a higher Angoff score for the external panel. N/A = Professional qualifications with less than eight items were excluded. A full list of professional qualifications explored can be found in the supporting information.

Consistency of panel rating and item difficulty

There was a strong relationship between how the panels rated the exam items and how the participants performed in the dental disciplines of Cariology, Endodontology, Oral Radiology, Periodontology, and Paediatric Dentistry (ρ 0.40–0.69, p≤0.001). Prosthodontics showed the weakest correlation between panel ratings and participant scores(Reference = 0.148, External = 0.059). Orofacial Medicine had a moderate correlation(Reference = 0.306, External = 0.399), with the external panel rating slightly lower than participant performance(External = -0.022).Table 2 shows the correlation between the Angoff ratings and the item difficulty within each exam and dental discipline.

Table 2. Average item correlations and absolute item differences between panels and item difficulty in items per dental discipline and total items (n = 267).

Variable Average Item correlation Absolute Item Difference
Reference panel External panel Reference panel External panel
(n = 267) (n = 267) (n = 267) (n = 267)
ρ p-value ρ p-value Dif. p-value Dif. p-value
Dental discipline
 Cariology .623 < .001 .555 < .001 .091 .001 .121 .002
 Endodontology .709 < .001 .584 .001 .211 < .001 .171 < .001
 Oral radiology .471 < .001 .714 < .001 .218 < .001 .172 < .001
 Orofacial medicine .306 .074 .399 .018 .114 .15 -.022 .991
 Periodontology .640 < .001 .783 < .001 .127 .077 .120 .144
 Paediatric Dentistry .614 < .001 .601 < .001 .153 < .001 .208 < .001
 Prosthodontics .148 .428 .059 .752 .108 .027 .031 .231
Total .493 < .001 .448 < .001 .108 < .001 .120 < .001

Statistically significant results in bold. Inter-panel correlations (Spearman rho (p)): ≥0.70 Very strong relationship, 0.40–0.69 Strong relationship, 0.30–0.39 Moderate relationship, 0.20–0.29 Weak relationship, 0.01–0.19 No or negligible relationship. Dif: Inter-panel absolute item difference (median absolute difference). Positive absolute difference indicates a higher Angoff score for the reference panel and negative absolute item difference indicates a higher Angoff score for the external panel. N/A = Professional qualifications with less than eight items were excluded. A full list of professional qualifications explored can be found in the supporting information.

Discussion

The study investigated the three-level Angoff item ratings of two independent panels across three theoretical tests. The results showed strong to very strong item correlation with no significant absolute difference between the two panels’item ratings of the three exams, item ratings within the dental disciplines or item ratings within the professional qualifications. As dental schools in Sweden independently establish their syllabi, the results were not certain beforehand. The results strongly suggest that even with the panels being recruited from independent dental schools, there is alignment in the expectations regarding the standards of a minimally qualified yet still acceptable dentist. Thus, the null hypothesis was not contradicted.

The theoretical examinations are a standard set concerning what is expected from a Swedish dentist, i.e., the professional qualifications and curricula for a Swedish degree in dental surgery [1–3]. In some specific dental disciplines, e.g., Prosthodontics and Orofacial Medicine, the relationship between the Angoff ratings and the item difficulty was moderate to negligible. Generally, the participants’ median score did not reach the Angoff median set for either of the two panels. The results are in line with our previous exploration of the theoretical examinations, i.e., participants who pass the theoretical examination still can show knowledge gaps in single dental disciplines [1]. Dental education and licensure processes vary across countries, and efforts are made to harmonise dental education in the EU and worldwide [24–27]. Consequently, certain items, in a Swedish context, may be anticipated by panellists as answered correctly by the minimally qualified test taker, whilst in other educational systems these items may not be part of the standard syllabus or are regarded as specialist competence. It is important to emphasise that the purpose of the theoretical examinations is to differentiate participants that have the knowledge and skills that meet the standards of a Swedish degree in Dental Surgery.

In the present study 10–12panellists were used, a decision partly grounded in evidence-based practice since the recommended panel size typically falls within the range of five to fifteen members [21, 28]. This choice was also influenced by practical considerations, given the challenges associated with recruiting enough subject matter experts from a single institution for each theoretical examination. Whilst previous studies have suggested that panellist discussions can reduce variability and improve intra-rater agreement by presenting test data [29, 30], such measures would require substantial resources, making them impractical for regular examinations. The approach used in this study allowed for the inclusion of new item raters, easily integrated into the process with written instructions alone. The recruitment and standard-setting strategies used in our study could effectively expand the pool of available subject matter experts without necessitating the extensive resources typically associated with the original Angoff method and its modifications. Our findings align with earlier research indicating that individual ratings without consensus save time [31]. However, it is important to acknowledge the trade-offs involved in skipping consensus meetings. The method may not alter pass rates compared to consensus methods [31], but it limits opportunities for in-depth discussions on item relevance and difficulty. Moreover, the absence of consensus meetings may reduce the support available to new panellists, potentially affecting confidence in raters and the consistency of ratings. To address these concerns, future studies could explore the development of supplementary training materials or online platforms that allow for some level of interaction without the full resource demands of traditional consensus meetings. Involving external panellists, from all national universities with a dental programme, is therefore recommended to mitigate the potential influence of specific practices or biases from individual institutions. This approach may also enhance the credibility of the process by ensuring that the expectations for the minimally qualified but acceptable dentist are consistent at a national level, thereby maintaining accuracy of assessments in evaluating the necessary competencies for dental practice. Whilst the modified approach to recruitment and standard setting in this study offers a practical solution to resource constraints, it is essential to consider the potential impacts on the quality of the standard-setting process. Balancing the need for efficiency with the importance of examination validity will be crucial in further refining this method.

Limitations

The results suggest that even with the panels being recruited from independent dental schools, there is alignment in the expectations regarding the standards of a minimally qualified yet still acceptable dentist. However, the analysis of consistency is one of many ways to explore how the two panels rated the test items. It is important to clarify that the analyses of panellists’ ratings do not explore if the panellists agree with each other, as agreement was not the primary outcome of this study. Our main purpose was to compare the two panels final ratings, and even though we did not explore agreement, the average results indicate that the panellists’ combined ratings ordered the items similarly. Furthermore, the average absolute item differences between panels also indicate that panellists rated items similarly.

A weakness of the study is that item relevance is not explored. The Angoff ratings do not provide specific insight into item relevance, i.e., an item could be rated as “YES” but still be irrelevant for evaluating the competence of a dental practitioner. One way to further explore item relevance is to have the subject matter expert also rate the relevance of the items, while another method suggested for rating relevance, alongside difficulty, is the Ebel approach [8].

This study indicated consistency between panels rating the same exams using the three-level Angoff, and a high inter-rater reliability in the reference panel was also shown in a previous study [1]. The findings suggest that the three-level Angoff method can be used effectively in contexts where time and resources are limited. However, there are concerns about reduced variability when using only three levels in rating, where the original Angoff, which uses 1–100 for ratings, may yield a more reliable outcome [17]. How panel members conceptualise a minimally qualified candidate may also be affected by experience and training in the Angoff methodology [15, 32]. The panellists were instructed to regard whether the least qualified but still acceptable candidate will answer the item correctly, potentially offering panellists a clearer and more understandable scenario for item rating [33], especially for new raters. To ensure that the compromise inherent to the three-level Angoff is reliable, it is important to evaluate the effect of consensus meetings held to discuss deviant items (where panellists disagree to a larger extent) and help introduce new panellists to the method. Thus, additional studies are needed to fully understand the pros and the cons of the three-level Angoff method.

To further investigate the validity of item ratings from a national perspective, we plan to invite other Swedish study programmes in dentistry to participate in the regular Angoff rating before upcoming theoretical exams. These ratings will be analysed as part of quality assurance.

Conclusion

The expectations of the minimally qualified but still acceptable dentist were comparable between the two independent panels across the three theoretical examinations here explored. The alignment between the panels indicates valid, reliable standards across institutions, despite the independent syllabi of the two study programmes. However, while there is an alignment, differences in ratings remain. Consequently, involving multiple institutions in future standard-setting processes could help ensure that the standards reflect a broader range of educational practices, supporting credibility of licensure examinations.

Supporting information

S1 Checklist

(PDF)

pone.0313476.s001.pdf (173KB, pdf)

Acknowledgments

We thank the candidates who took part in the tests and the Angoff panellists. Ida Hed Myrberg and Henrike Häbel contributed to the process of data curation.

Data Availability

All data files are available from zenodo.org DOI https://doi.org/10.5281/zenodo.13951858.

Funding Statement

The author(s) received no specific funding for this work.

References

  • 1.Dalum J, Christidis N, Myrberg IH, Karlgren K, Leanderson C, Englund GS. Are we passing the acceptable? Standard setting of theoretical proficiency tests for foreign-trained dentists. Eur J Dent Educ. 2023;27(3):640–9. Epub 2022/08/31. doi: 10.1111/eje.12851 . [DOI] [PubMed] [Google Scholar]
  • 2.Swedish National Board of Health and Welfare. HSLF-FS 2017:80 The National Board of Health and Welfare’s regulations and general advice on licensing for professions within healthcare for education from third countries (In Swedish). 2017. p. 2. [cited 2024 Aug 25]. https://www.socialstyrelsen.se/globalassets/sharepointdokument/artikelkatalog/foreskrifter-och-allmanna-rad/2017-12-42.pdf.
  • 3.Swedish Council of Higher Education. Higher Education Act 1993:100, Annex 2, System of Qualifications. 1993. [cited 2024 Aug 25]. https://www.uhr.se/en/start/laws-and-regulations /Laws-and-regulations/The-Higher-Education-Ordinance/Annex 2/#MSc_DentalSurg.
  • 4.American Educational Research A, American Psychological A, National Council on Measurement in E, Joint Committee on Standards for E, Psychological T. Standards for educational and psychological testing. Washington: American Educational Research Association; 2014.
  • 5.Al-Sowygh ZH, Sukotjo C. Foreign-trained dentists in advanced education in prosthodontics programs in the United States: demographics, perspectives on current training, and future goals. J Prosthodont. 2011;20(2):161–5. Epub 2010/11/13. doi: 10.1111/j.1532-849X.2010.00658.x . [DOI] [PubMed] [Google Scholar]
  • 6.Angoff W. Scales, norms and equivalent scores. In: RL T, editor. Educational Measurement, Second Edition. Washington DC: American Council on Education; 1971. p. 508–600. [Google Scholar]
  • 7.Nedelsky L. Absolute Grading Standards for Objective Tests. Educ Psychol Meas. 1954;14(1):3–19. doi: 10.1177/001316445401400101 [DOI] [Google Scholar]
  • 8.Ebel RL, Frisbie DA. Essentials of educational measurement. Prentice Hall. 1972.
  • 9.Jelovsek JE, Walters MD, Korn A, Klingele C, Zite N, Ridgeway B, et al. Establishing cutoff scores on assessments of surgical skills to determine surgical competence. Am J Obstet Gynecol. 2010;203(1):81 e1–6. Epub 2010/04/27. doi: 10.1016/j.ajog.2010.01.073 . [DOI] [PubMed] [Google Scholar]
  • 10.McKinley DW, Norcini JJ. How to set standards on performance-based examinations: AMEE Guide No. 85. Med Teach. 2014;36(2):97–110. Epub 2013/11/22. doi: 10.3109/0142159X.2013.853119 . [DOI] [PubMed] [Google Scholar]
  • 11.Norcini J, Anderson MB, Bollela V, Burch V, Costa MJ, Duvivier R, et al. 2018 Consensus framework for good assessment. Med Teach. 2018;40(11):1102–9. Epub 2018/10/10. doi: 10.1080/0142159X.2018.1500016 . [DOI] [PubMed] [Google Scholar]
  • 12.Norcini JJ. Setting standards on educational tests. Med Educ. 2003;37(5):464–9. Epub 2003/04/24. doi: 10.1046/j.1365-2923.2003.01495.x . [DOI] [PubMed] [Google Scholar]
  • 13.Kane M. Validating the Performance Standards Associated With Passing Scores. Rev Educ Res. 1994;64(3):425–61. doi: 10.3102/00346543064003425 [DOI] [Google Scholar]
  • 14.Cusimano MD. Standard setting in medical education. Acad Med. 1996;71(10 Suppl):S112–20. Epub 1996/10/01. doi: 10.1097/00001888-199610000-00062 . [DOI] [PubMed] [Google Scholar]
  • 15.Boursicot K. Setting Standards in a Professional Higher Education Course: Defining the Concept of the Minimally Competent Student in Performance‐Based Assessment at the Level of Graduation from Medical School. High Educ Q. 2006;60(1):74–90. doi: 10.1111/j.1468-2273.2006.00308.x [DOI] [Google Scholar]
  • 16.Chinn RN, Hertz NR. Alternative Approaches to Standard Setting for Licensing and Certification Examinations. Appl Meas Educ. 2002;15(1):1–14. doi: 10.1207/s15324818ame1501_01 [DOI] [Google Scholar]
  • 17.Kim J, Yang JS. How to improve reliability of cut-off scores in dental competency exam: A comparison of rating methods in standard setting. Eur J Dent Educ. 2020;24(4):734–40. Epub 2020/07/01. doi: 10.1111/eje.12563 . [DOI] [PubMed] [Google Scholar]
  • 18.Jalili M, Hejri SM, Norcini JJ. Comparison of two methods of standard setting: the performance of the three-level Angoff method. Med Educ. 2011;45(12):1199–208. Epub 2011/11/30. doi: 10.1111/j.1365-2923.2011.04073.x . [DOI] [PubMed] [Google Scholar]
  • 19.Yudkowsky R, Downing SM, Popescu M. Setting standards for performance tests: a pilot study of a three-level Angoff method. Acad Med. 2008;83(10 Suppl):S13–6. Epub 2008/10/10. doi: 10.1097/ACM.0b013e318183c683 . [DOI] [PubMed] [Google Scholar]
  • 20.Hambleton RK, Jaeger RM, Plake BS, Mills C. Setting Performance Standards on Complex Educational Assessments. Appl Psychol Meas. 2000;24(4):355–66. doi: 10.1177/01466210022031804 [DOI] [Google Scholar]
  • 21.Verhoeven BH, van der Steeg AF, Scherpbier AJ, Muijtjens AM, Verwijnen GM, van der Vleuten CP. Reliability and credibility of an angoff standard setting procedure in progress testing using recent graduates as judges. Med Educ. 1999;33(11):832–7. Epub 1999/12/03. doi: 10.1046/j.1365-2923.1999.00487.x . [DOI] [PubMed] [Google Scholar]
  • 22.Verhoeven BH, Verwijnen GM, Muijtjens AM, Scherpbier AJ, van der Vleuten CP. Panel expertise for an Angoff standard setting procedure in progress testing: item writers compared to recently graduated students. Med Educ. 2002;36(9):860–7. Epub 2002/10/02. doi: 10.1046/j.1365-2923.2002.01301.x . [DOI] [PubMed] [Google Scholar]
  • 23.von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP, et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med. 2007;4(10):e296. Epub 2007/10/19. doi: 10.1371/journal.pmed.0040296 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Dixon J, Manzanares-Cespedes C, Davies J, Vital S, Gerber G, Paganelli C, et al. O-HEALTH-EDU: A scoping review on the reporting of oral health professional education in Europe. Eur J Dent Educ. 2021;25(1):56–77. Epub 2020/08/21. doi: 10.1111/eje.12577 . [DOI] [PubMed] [Google Scholar]
  • 25.Wang YH, Zhao Q, Tan Z. Current differences in dental education between Chinese and Western models. Eur J Dent Educ. 2017;21(4):e43–e9. Epub 2016/06/25. doi: 10.1111/eje.12216 . [DOI] [PubMed] [Google Scholar]
  • 26.De Vries J, Murtomaa H, Butler M, Cherrett H, Ferrillo P, Ferro MB, et al. The global network on dental education: a new vision for IFDEA. Eur J Dent Educ. 2008;12 Suppl 1(s1):167–75. Epub 2008/04/12. doi: 10.1111/j.1600-0579.2007.00498.x . [DOI] [PubMed] [Google Scholar]
  • 27.Field JC, Cowpe JG, Walmsley AD. The Graduating European Dentist: A New Undergraduate Curriculum Framework. Eur J Dent Educ. 2017;21 Suppl 1(S1):2–10. Epub 2017/12/06. doi: 10.1111/eje.12307 . [DOI] [PubMed] [Google Scholar]
  • 28.Hurtz GM, Hertz NR. How Many Raters Should be Used for Establishing Cutoff Scores with the Angoff Method? a Generalizability Theory Study. Educ Psychol Meas. 1999;59(6):885–97. doi: 10.1177/00131649921970233 [DOI] [Google Scholar]
  • 29.Taube KT. The incorporation of empirical item difficulty data into the Angoff standard-setting procedure. Eval Health Prof. 1997;20(4):479–98. Epub 1997/11/03. doi: 10.1177/016327879702000406 . [DOI] [PubMed] [Google Scholar]
  • 30.Hurtz GM, Auerbach MA. A Meta-Analysis of the Effects of Modifications to the Angoff Method on Cutoff Scores and Judgment Consensus. Educ Psychol Meas. 2003;63(4):584–601. doi: 10.1177/0013164403251284 [DOI] [Google Scholar]
  • 31.Senthong V, Chindaprasirt J, Sawanyawisuth K, Aekphachaisawat N, Chaowattanapanit S, Limpawattana P, et al. Group versus modified individual standard-setting on multiple-choice questions with the Angoff method for fourth-year medical students in the internal medicine clerkship. Adv Med Educ Pract. 2013;4(default):195–200. Epub 2013/10/09. doi: 10.2147/AMEP.S46705 . [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Verheggen MM, Muijtjens AM, Van Os J, Schuwirth LW. Is an Angoff standard an indication of minimal competence of examinees or of judges? Adv Health Sci Educ Theory Pract. 2008;13(2):203–11. Epub 2006/10/18. doi: 10.1007/s10459-006-9035-1 . [DOI] [PubMed] [Google Scholar]
  • 33.Puryer J, O’Sullivan D. An introduction to standard setting methods in dentistry. Br Dent J. 2015;219(7):355–8. Epub 2015/10/10. doi: 10.1038/sj.bdj.2015.755 . [DOI] [PubMed] [Google Scholar]

Decision Letter 0

Andrej M Kielbassa

1 Aug 2024

PONE-D-24-15178Consistency between inter-institutional panels using a three-level Angoff - standard setting in licensure tests of foreign-trained dentists in SwedenPLOS ONE

Dear Dr. Dalum,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please note that one of our external Referees has recommended rejection of your submitted draft ("The reasons for that (rejection) are manifold and could be obtained from the comments. A lack of readability and clarity are the first to be mentioned. Moreover the scientific impact of this study is very low."), while our second Reviewer obviously seem satisfied (at least to some major extent). Therefore, I have double-checked your paper (see comments forwarded as R #1). You surely will agree that with reference to the numerous shortcomings having been identified with your draft, your submitted manuscript is not considered ready to proceed. Please take the opportunity to carefully revise your paper, and stick to each and every comment, to speed up this review process. Remember that Plos One's policy is to go for one round of revisions only, to avoid any re-re-review, so ensure re-submitting a high-quality re-submission. 

Please submit your revised manuscript by Sep 15 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Andrej M Kielbassa

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at 

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and 

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. Please note that in order to use the direct billing option the corresponding author must be affiliated with the chosen institute. Please either amend your manuscript to change the affiliation or corresponding author, or email us at plosone@plos.org with a request to remove this option.

3. When completing the data availability statement of the submission form, you indicated that you will make your data available on acceptance. We strongly recommend all authors decide on a data sharing plan before acceptance, as the process can be lengthy and hold up publication timelines. Please note that, though access restrictions are acceptable now, your entire data will need to be made freely accessible if your manuscript is accepted for publication. This policy applies to all data except where public deposition would breach compliance with the protocol approved by your research ethics board. If you are unable to adhere to our open data policy, please kindly revise your statement to explain your reasoning and we will seek the editor's input on an exemption. Please be assured that, once you have provided your new statement, the assessment of your exemption will not hold up the peer review process.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

Reviewer #3: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Abstract

- Please revise, and stick to Journal style.

- No subheadings mandatory. Please consult some recently published Plos One papers.

- Please provide exact P values here. Remember that terms like "very strong relationship" and "scores were overall lower" would seem too vague, and do not provide sound information.

- With your revision, please stick to the allowed word maximum, to provide as much information as possible.

Intro

- "To address this, the present study aims to investigate (...)." This study has been finished hasn't it? Please switch to past tense.

- No subheadings with this section. -

- Please elaborate bot aims and objectives with this section. Do not provide a literature review here.

- A sound and valid null hypothesis is missing at the end of this section.

Meths

- "(...) by two separate panels; The reference panel was (...)." Please revise for typos.

- Details of statistical program missing.

Results

- With you full text, please provide exact results. See "weakest relationship" or "moderate relationship" - you surely will agree that this would seem hard to understand.

Disc

- Stick to H0 when starting this section.

- This section would seem perfectible. There would be many more aspects to discuss. Please revise carefully.

Concl

- With your revision, please exclusively stick to your aims.

Refs

- This list must be adopted to Journal style.

In total, this submission would seem interesting, is considered easily intelligible, and should be worth following after revision, depending on the latter's quality. Please stick to the comments provided above, and revise carefully.

Reviewer #2: Title page:

- It would be helpful to potential readers if the title of the publication already indicated the type of study that was involved in the publication.

- Please note the formal requirements of PLoS One regarding the font size and font type of your headings, please see "Abstract".

- Please use either American or British English in your manuscript, but do not mix the two.

Introduction:

- "To address this, the present study aims to investigate the consistency of three-level Angoff ratings of the proficiency test between two independent panels of raters from two universities." - "To address this, the aim of the present study was to investigate..."

- Please provide a sound and valid null hypothesis at the end of your Introduction section, which should be discussed at the beginning of the Discussion section.

Materials and methods:

- With the material used (i.e., the statistical software), please use general names with your text, followed by (brand name; manufacturer, city, ST[ate - abbreviated, if US], country) in parentheses. Stick to semicolon.

- Which test did you use to determine the (missing) normal distribution?

- The significance level α is missing. I guess it was 0.05?

- The ethics committee vote date is missing.

Discussion:

- The mentioning of statements regarding the null hypothesis at the beginning of this section is missing. Remember that the null hypothesis can be rejected or not rejected.

References:

- Please follow the author guidelines for ALL your References and provide all necessary data such as the correct abbreviated journal name, volume, issue, page numbers etc. Please carefully revise your data.

- Please make sure the title of each reference you access online is complete, correct, and includes access date and link.

- Please provide the correct and complete names of authors of websites used, e.g. References 2-4.

Reviewer #3: The authors present valuable insight into the Swedish process of re-certifying dentists trained outside the EU. Two expert panels assessed three different types of examinations using a modified Angoff method for setting exam pass-marks. The results suggest an overall high correlation between both (expert) groups in the majority of items although the syllabus is variable from one university to another. It was concluded that simplification of the Angoff method is appropriate, less time consuming and resource-saving compared to the originally described procedure.

The paper is well organized and concise. However, information is provided in a cumbersome style, which is hard to read and confusing at times. To me the paper needs extensive linguistic revision especially in the Materials and Methods and Results sections. There is a lack of clarity, consistency and readability in particular.

• For the above mentioned reasons the abstract needs to be rewritten. The results are almost incomprehensible. Who are the participants in this study? I assume you refer to the external panel but this is nowhere mentioned and confuses the reader to the maximum.

Why do you consider the modified Angoff method to be “promising” compared to the original one when you have not even investigated both methods?

• In the Results section, the paragraph starting on page 6 l151-page 7 l158 is incomprehensible at all and probably needs visualisation. (For you who are dealing with these closely related technical terms on a daily basis this all might be self-explanatory which is obviously not the case for your readers).

• Description of the study’s limitations is by far the best paragraph in the paper which arises the question whether an overall “similarity” (since agreement was not assessed in the first place) between to different panels assessing pass-marks of three exams is worth to be reported in a high-standard international publication?

• The closing sentence again refers to the traditional Angoff method which again has not been investigated in this study.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: PD Dr. Hans Juergen Jochen Michael Wicht

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2024 Nov 8;19(11):e0313476. doi: 10.1371/journal.pone.0313476.r002

Author response to Decision Letter 0


9 Sep 2024

PONE-D-24-15178

Consistency between inter-institutional panels using a three-level Angoff - standard setting in licensure tests of foreign-trained dentists in Sweden

We would like to thank the editor and reviewers for taking the time to review the manuscript, and considering the manuscript for publication. We hope that the changes made to the manuscript is up to you expectations.

Responses to comments, in detail, can be found in the document Response to reviewers. Below is a summarised version of the comments to specific reviewer and editor comments.

Reviewer 1

We have revised the abstract to conform with the journal's style. We have used the recently published article in formatting our manuscript. The exact p-values are now provided in the abstract.

The Introduction has been revised to reflect that the study is finished. We have removed the subheadings from the Introduction to align with the journal's style requirements. The null hypothesis has been explicitly stated at the end of the Introduction. Additionally, aims and objectives have been elaborated.

We have revised the text to correct any typographical errors. We have included detailed information about the statistical software used in the analyses, including the version, manufacturer, and location. The exact results from statistical analyses have been provided throughout the text.

We have revised the Discussion section to focus on the null hypothesis and expanded the discussion to cover additional relevant aspects.

The Conclusion has been revised to strictly adhere to the study's aims.

References have been reformatted to comply with the journal's guidelines.

--------------------------------

Reviewer 2

Thank you for your suggestion. We have revised the title to reflect the nature of the study more clearly. We have reviewed and adjusted the font size and type of all headings throughout the manuscript to comply with PLoS One formatting guidelines. We have also revised the text to British English.

We have revised the text in the introduction. The null hypothesis has been explicitly stated at the end of the Introduction. Additionally, aims and objectives have been elaborated. We have revised the Discussion section to focus on the null hypothesis and expanded the discussion to cover additional relevant aspects.

We have revised the Material and Methods section. We now include the brand, manufacturer, and location details for the statistical software used, as suggested. No formal test of possible normal distribution was made. Instead, we concluded that the normal distribution would be a poor model based on the way total scores were calculated. This has been clarified in the manuscript. We have revised the text to include the significance level. The date of the ethics committee approval has been included in the manuscript under Ethical consideration.

The Discussion begins with a reference to the null hypothesis. See revisions in first section of Discussion.

All references have been revised to follow the journal’s style, including online access information where relevant

-------------------------

Reviewer 3

The abstract and Methods section has been rewritten for clarity.

We have clarified why the modified Angoff method is considered promising, despite not directly comparing it to the original method. See Discussion and limitations section. We have also revised the conclusions regarding the reference to the original Angoff method.

We have revised the results section to provide clarity, adding an explanatory text in the beginning of each paragraph in the results section.

The Discussion section has been revised to ensure the study’s contribution is clear, in the context of the study’s limitations.

The Conclusion has been revised to remove ambiguous references to the traditional Angoff method and focuses solely on the findings.

Attachment

Submitted filename: Response to reviewers.docx

pone.0313476.s002.docx (29.7KB, docx)

Decision Letter 1

Andrej M Kielbassa

15 Sep 2024

PONE-D-24-15178R1Consistency between inter-institutional panels using a three-level Angoff - standard setting in licensure tests of foreign-trained dentists in Sweden: A cross-sectional studyPLOS ONE

Dear Dr. Dalum,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. Please note that one of the previous external reviewers rejecting your manuscript has declined to re-review, without double-checking your revised draft. The second reviewer (recommending revisions with the first round of reviews) has rejected your revised version, mainly due to formal aspects. No doubt, the latter have to be followed, and you will find the respective comments given below. Please note that our referees have raised several points that do jeopardize the outlook for publication of your manuscript, so please remember that these concerns must be fully addressed, and re-review of the manuscript will be necessary and extensive. Additionally, I have double checked your draft, and have commented on the revised version (for details, please see below). I am confident that you will stick to those aspects, and you will re-submitted an improved version of your manuscript clarifying the missing aspects. Please note that usually only one revision would seem acceptable, and repeated re-review is discouraged.

Please submit your revised manuscript by Oct 30 2024 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

  • A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

  • A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

  • An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Andrej M Kielbassa

Academic Editor

PLOS ONE

Additional Editor Comments:

- Unfortunately, the Co-Authors have not provided a point-by-point response to the previous Reviewers’ comments. Please go back to the recent recommendations, and clarify how you have responded to the recently forwarded recommendations.

- „Angoff - standard“ – please double check for correct spacebar use.

- „The reference panel ratings originated from the panel used to consecutively set the pass mark for regular examinations. The external panel was recruited, as part of the study, from a dental school at a university in another region in Sweden after the examinations.“ Please note that this would seem hard to understand; it would seem that you have compared regular students from two different universities. In particular when reflecting on your thoughts given with your Abstract section („This study aimed to determine the consistency of ratings by independent panels at two Swedish universities in re-certification exams for non-EU trained dentists“), this should be clarified. Please clearly define your reference and external groups.

- The Co-Authors state that „A cross-sectional design was used to analyze rating consistency of two separate panels.“ Please note that a cross-sectional study design is a type of observational study design. First, this would call for pre-registration (as with all observational studies, see https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2952011/ ). Second, cross sectional studies should be reported according to STROBE (see https://www.equator-network.org/wp-content/uploads/2015/10/STROBE_checklist_v4_cross-sectional.pdf), along with a dedicated information given with your manuscript. You surely will agree that both aspects would seem missing with your revised and re-submitted draft.

- „The panellists in the reference panel were recruited consecutively before each examination during 2019-08-01 to 2020-08-19 (...).“ With reference to the thoughts given above, this would be a group of people who share a common characteristic or experience within a defined period (e.g., have been examined. This usually would be a cohort. Consequently, this would be a cohort study comparing two subgroup cohorts (reference and external). Please double check, and give reasons why you have called this study a cross sectional study.

- „The two panels consisted primarily of women.“ Please provide reasons.

- With your Conclusions, please stick exclusively to your aims (these were „This study aimed to determine the consistency of ratings by independent panels (...)“, or „This study aimed to determine whether the expectations of minimally qualified dentists are aligned, as reflected by the ratings from two universities using the three-level Angoff method“). Do not simply repeat your results here. Do not speculate. Instead, provide a reasonable and generalizable extension of your outcome.

- „Additional studies are needed to fully understand the pros and the cons of the three-level Angoff method.“ This is not considered a conclusion from your study. Please, these aspects might be discussed with your Discussion section; please delete these thoughts from your conclusions.

- References still have not been adapted to Journal style (please see previous comments, and compare your responses: „References have been reformatted to comply with the journal's guidelines“, and „All references have been revised to follow the journal’s style“). With your revisions, you surely want to make up this leeway. Please revise for English titles, and provide doi numbers.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: No

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The Co-Authors have satisfyingly revised their previous draft. Unfortunately, doi numbers still would seem missing with most of the references, but this should be perfectible in cooperation with the typesetter.

Reviewer #2: Dear Authors,

Abstract:

- No subheadings are required in the Abstract. Please note the comment from Reviewer #1 in the last review.

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0310004

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0228249

- Please put a space before and after a mathematical operator (e.g., minus, plus, greater than, less than) in the whole manuscript as well as before and after the hyphen.

- Please note the comment from Reviewer #1 regarding the mention of exact p values.

- Shouldn't the order of the p and Spearman's Rho values be the other way around (significance/correlation) according to your explanation? - "There was a statistically significant and strong correlation (ρ ≥ 0.70, p ≤ 0.05) between ..." - "There was a statistically significant and strong correlation (p ≤ 0.05, ρ ≥ 0.70) between..."

- Please report exact p values (e.g., p = .015, p < .XYZ), unless p is < .001. In accordance with the APA style, p values smaller than p < .001 are written as p < .001 (https://www.graphpad.com/support/faq/how-to-report-p-values-in-journals).

- "There was a statistically significant and strong correlation (ρ ≥ 0.70, p ≤ 0.05) between the ratings of the two

independent panels, with no significant differences ..." - Here, the exact p value is missing as well.

- Please re-check your punctuation skills in English.

Introduction:

- "The null hypothesis was that there will be no significant difference..." - "The null hypothesis was that there would be no significant difference..."

Materials and methods

- Please put a space before and after a mathematical operator (e.g., minus, plus, greater than, less than) in the whole manuscript as well as before and after the hyphen.

- Please consistently omit the zero before the decimal point for the p values and the correlation coefficient because the statistic cannot be greater than 1 (APA style), and please do not change the notation within the manuscript.

Results

- Please put a space before and after a mathematical operator (e.g., minus, plus, greater than, less than) in the whole manuscript as well as before and after the hyphen.

- Please report exact p values (e.g., p = .015, p < .XYZ), unless p is < .001. In accordance with the APA style, p values smaller than p < .001 are written as p < .001 (https://www.graphpad.com/support/faq/how-to-report-p-values-in-journals).

- Please consistently omit the zero before the decimal point for the p values and the correlation coefficient because the statistic cannot be greater than 1 (APA style), and please do not change the notation within the manuscript.

Conclusion

This chapter is formulated in very general terms.

Refs

- Please revise this part with regard to up-to-dateness. For example, complete information is available on PubMed for Reference 1. https://pubmed.ncbi.nlm.nih.gov/36039793

I assume that it is the task of the authors to ensure that a revised version is up-to-date and free of errors.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2024 Nov 8;19(11):e0313476. doi: 10.1371/journal.pone.0313476.r004

Author response to Decision Letter 1


18 Oct 2024

PONE-D-24-15178R1

Consistency between inter-institutional panels using a three-level Angoff - standard setting in licensure tests of foreign-trained dentists in Sweden: A cross-sectional study

Revised title: Consistency between inter-institutional panels using a three-level Angoff-standard setting in licensure tests of foreign-trained dentists in Sweden: A cohort study

PLOS ONE

------------------------------------------------------------------------------------------------

Dear Editor and reviewers,

Thank you for taking the time to review and to improve upon our manuscript. In the following text you will find our point-by-point response to the comments. The comments are also uploaded as a word document. We have also included an updated point-by-point response for the 1st revision in attached files.

The manuscript has been formatted according to reference article: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0228249. The manuscript has also been sent for language check at a third party.

------------------------------------------------------------------------------------------------

Additional Editor Comments:

- Unfortunately, the Co-Authors have not provided a point-by-point response to the previous Reviewers’ comments. Please go back to the recent recommendations, and clarify how you have responded to the recently forwarded recommendations.

Reply: Point-by-point response to the first review is provided in the uploaded document PONE Response to reviewers 1st revision - point-by-point.

- „Angoff - standard“ – please double check for correct spacebar use.

Reply: Thank you, this is corrected.

- „The reference panel ratings originated from the panel used to consecutively set the pass mark for regular examinations. The external panel was recruited, as part of the study, from a dental school at a university in another region in Sweden after the examinations.“ Please note that this would seem hard to understand; it would seem that you have compared regular students from two different universities. In particular when reflecting on your thoughts given with your Abstract section („This study aimed to determine the consistency of ratings by independent panels at two Swedish universities in re-certification exams for non-EU trained dentists“), this should be clarified. Please clearly define your reference and external groups.

Reply: The abstract has been rephrased to clarify the design of the study.

- The Co-Authors state that „A cross-sectional design was used to analyze rating consistency of two separate panels.“ Please note that a cross-sectional study design is a type of observational study design. First, this would call for pre-registration (as with all observational studies, see https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2952011/ ). Second, cross sectional studies should be reported according to STROBE (see https://www.equator-network.org/wp-content/uploads/2015/10/STROBE_checklist_v4_cross-sectional.pdf), along with a dedicated information given with your manuscript. You surely will agree that both aspects would seem missing with your revised and re-submitted draft.

Reply: Yes, we agree. We acknowledge that cross-sectional studies are observational by nature and agree that pre-registration of medical studies and adherence to STROBE guidelines are important. This study was not pre-registered. A STROBE Statement has been added at the end of ethical consideration, and a document how the manuscript adheres to the STROBE guidelines is also added to the supplementary material.

- „The panellists in the reference panel were recruited consecutively before each examination during 2019-08-01 to 2020-08-19 (...).“ With reference to the thoughts given above, this would be a group of people who share a common characteristic or experience within a defined period (e.g., have been examined. This usually would be a cohort. Consequently, this would be a cohort study comparing two subgroup cohorts (reference and external). Please double check, and give reasons why you have called this study a cross sectional study.

Reply: Thank you for pointing this out, we have adjusted the text accordingly in the title and in the first sentence of the methods section.

- „The two panels consisted primarily of women.“ Please provide reasons.

Reply: Thank you for your comment. The absolute majority (at least 75%) of the dental teachers are women in both dental schools involved. This is reflected by the sex distribution of the panels. We have added text “The two panels consisted primarily of women, which reflects the current sex-distribution of dental teachers in both universities.” In the beginning of the Results. Page 7, Row number 165-166.

- With your Conclusions, please stick exclusively to your aims (these were „This study aimed to determine the consistency of ratings by independent panels (...)“, or „This study aimed to determine whether the expectations of minimally qualified dentists are aligned, as reflected by the ratings from two universities using the three-level Angoff method“). Do not simply repeat your results here. Do not speculate. Instead, provide a reasonable and generalizable extension of your outcome.

Reply: Thank you, the Conclusions have been revised accordingly.

- „Additional studies are needed to fully understand the pros and the cons of the three-level Angoff method.“ This is not considered a conclusion from your study. Please, these aspects might be discussed with your Discussion section; please delete these thoughts from your conclusions.

Reply: Thank you for your comment, the statement has been removed from the conclusion section.

- References still have not been adapted to Journal style (please see previous comments, and compare your responses: „References have been reformatted to comply with the journal's guidelines“, and „All references have been revised to follow the journal’s style“). With your revisions, you surely want to make up this leeway. Please revise for English titles, and provide doi numbers.

Reply: Thank you for your comment, the references are now formatted correctly

Reviewer #1: The Co-Authors have satisfyingly revised their previous draft. Unfortunately, doi numbers still would seem missing with most of the references, but this should be perfectible in cooperation with the typesetter.

Reply: Thank you for your comment, the references are now formatted correctly

Reviewer #2: Dear Authors,

Abstract:

- No subheadings are required in the Abstract. Please note the comment from Reviewer #1 in the last review.

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0310004

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0228249

Reply: Thank you for your comment. Subheadings have been deleted in the abstract

- Please put a space before and after a mathematical operator (e.g., minus, plus, greater than, less than) in the whole manuscript as well as before and after the hyphen.

Reply: Thank you for your observation, this has been corrected in the text.

- Please note the comment from Reviewer #1 regarding the mention of exact p values.

Reply: Yes, we have corrected this by adding exact p-values in the Tables.

- Shouldn't the order of the p and Spearman's Rho values be the other way around (significance/correlation) according to your explanation? - "There was a statistically significant and strong correlation (ρ ≥ 0.70, p ≤ 0.05) between ..." - "There was a statistically significant and strong correlation (p ≤ 0.05, ρ ≥ 0.70) between..."

Reply: Thank you for noticing, we have corrected these sentences.

- Please report exact p values (e.g., p = .015, p < .XYZ), unless p is < .001. In accordance with the APA style, p values smaller than p < .001 are written as p < .001 (https://www.graphpad.com/support/faq/how-to-report-p-values-in-journals).

Reply Yes, we have corrected this by formatting the text and adding exact p-values in the Tables.

- "There was a statistically significant and strong correlation (ρ ≥ 0.70, p ≤ 0.05) between the ratings of the two

independent panels, with no significant differences ..." - Here, the exact p value is missing as well.

Reply: Thank you for observing this. In running text, we refer to overall results. The p-values have been corrected. Exact p-values have been added in the revised Tables for reference.

- Please re-check your punctuation skills in English.

Reply: We have re-checked the punctuation, and hope this is OK now

Introduction:

- "The null hypothesis was that there will be no significant difference..." - "The null hypothesis was that there would be no significant difference..."

Reply Thank you, the text has been revised according to your comment.

Materials and methods

- Please put a space before and after a mathematical operator (e.g., minus, plus, greater than, less than) in the whole manuscript as well as before and after the hyphen.

Reply: We have checked the manuscript and have hopefully corrected these mistakes.

- Please consistently omit the zero before the decimal point for the p values and the correlation coefficient because the statistic cannot be greater than 1 (APA style), and please do not change the notation within the manuscript.

Reply: We have checked the manuscript and have hopefully corrected these mistakes.

Results

- Please put a space before and after a mathematical operator (e.g., minus, plus, greater than, less than) in the whole manuscript as well as before and after the hyphen.

Reply: We have checked the manuscript and have hopefully corrected these mistakes

- Please report exact p values (e.g., p = .015, p < .XYZ), unless p is < .001. In accordance with the APA style, p values smaller than p < .001 are written as p < .001 (https://www.graphpad.com/support/faq/how-to-report-p-values-in-journals).

Reply: We have checked the manuscript and have hopefully corrected these mistakes

- Please consistently omit the zero before the decimal point for the p values and the correlation coefficient because the statistic cannot be greater than 1 (APA style), and please do not change the notation within the manuscript.

Reply: We have checked the manuscript and have hopefully corrected these mistakes

Conclusion

This chapter is formulated in very general terms.

Reply: The conclusion section has been rewritten in order to provide a reasonable and generalizable extension.

Refs

- Please revise this part with regard to up-to-dateness. For example, complete information is available on PubMed for Reference 1. https://pubmed.ncbi.nlm.nih.gov/36039793

I assume that it is the task of the authors to ensure that a revised version is up-to-date and free of errors.

Reply: We’re sorry about these mistakes, and the reference list has been updated accordingly.

Attachment

Submitted filename: PONE Response to reviewers 2nd revision.docx

pone.0313476.s003.docx (28.4KB, docx)

Decision Letter 2

Andrej M Kielbassa

25 Oct 2024

Consistency between inter-institutional panels using a three-level Angoff - standard setting in licensure tests of foreign-trained dentists in Sweden: A cohort study

PONE-D-24-15178R2

Dear Dr. Dalum,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Please not that one of our external reviewers has noted that still some minor issues would call for improvements; these should be perfectible together with the typesetter, so please pay special attention to the proofs.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. If you have any questions relating to publication charges, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Prof. Dr. Dr. h. c. Andrej M Kielbassa

Academic Editor

PLOS ONE

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: (No Response)

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: With the help of the Reviewers, the Co-Authors have re-submitted a revised draft now, and the latter would seem ready to proceed.

Reviewer #2: Dear Authors,

your reference section unfortunately still contains errors and information that should not be listed, and unfortunately you have again not fully followed the Plos One author guidelines, i.e., “Epub” should not be listed and is even partially (according to PubMed) incorrect (see Ref. 17) and the page numbers should both be listed in full. It is your second revision.

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

**********

Acceptance letter

Andrej M Kielbassa

30 Oct 2024

PONE-D-24-15178R2

PLOS ONE

Dear Dr. Dalum,

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team.

At this stage, our production department will prepare your paper for publication. This includes ensuring the following:

* All references, tables, and figures are properly cited

* All relevant supporting information is included in the manuscript submission,

* There are no issues that prevent the paper from being properly typeset

If revisions are needed, the production department will contact you directly to resolve them. If no revisions are needed, you will receive an email when the publication date has been set. At this time, we do not offer pre-publication proofs to authors during production of the accepted work. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few weeks to review your paper and let you know the next and final steps.

Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

If we can help with anything else, please email us at customercare@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Prof. Dr. med. dent. Dr. h. c. Andrej M Kielbassa

Academic Editor

PLOS ONE

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Checklist

    (PDF)

    pone.0313476.s001.pdf (173KB, pdf)
    Attachment

    Submitted filename: Response to reviewers.docx

    pone.0313476.s002.docx (29.7KB, docx)
    Attachment

    Submitted filename: PONE Response to reviewers 2nd revision.docx

    pone.0313476.s003.docx (28.4KB, docx)

    Data Availability Statement

    All data files are available from zenodo.org DOI https://doi.org/10.5281/zenodo.13951858.


    Articles from PLOS ONE are provided here courtesy of PLOS

    RESOURCES