Abstract
Objectives: The aim of this study was to establish the face and content validity of the Caries Assessment Spectrum and Treatment (CAST) index. This hierarchical epidemiological index consists of 10 codes covering the spectrum of carious lesion progression and describing conditions ranging from the absence of carious lesions to the presence of sealants and restorations, the presence of lesions in enamel and dentine, and the presence of advanced stages of carious lesion in pulpal tissue and tissue surrounding the tooth. Methods: Using the RAND modified e-Delphi consensus method, a set of 17 statements related to the content, including the codes and descriptions used, and suitability of the CAST index were scored on a scale of 1–9 by 15 senior epidemiologists from 15 countries over three rounds of assessment. Agreement of ≥ 75% was required to indicate consensus on a statement. Results: Consensus was reached on 14 statements in the first round, none in the second and on the remaining three statements in the third round. To obtain feedback on the initial validation of the index and to test its external validity, 41 epidemiologists from 24 countries were requested to assess the validated statements. Minor changes to 10 of the 17 statements’ content and descriptions were suggested; this necessitated the resubmission of the modified CAST index to the original 15 epidemiologists. Consensus of ≥ 80% was reached on all 10 statements regarding codes and their descriptions. Conclusions: After a total of four rounds with the panel members and one round with the feedback group, the CAST index was approved for face and content validity. External validity was obtained. The participating epidemiologists found the RAND modified e-Delphi consensus method to be a suitable instrument for reaching consensus.
Key words: Delphi technique, dental caries, DMF index, validation studies, caries assessment spectrum and treatment index, epidemiology, caries assessment spectrum and treatment, international caries detection and assessment system
INTRODUCTION
Dental caries is a serious public health problem, which, in an individual, may cause pain, suffering and a decrease in that person’s quality of life. Despite large reductions in the prevalence and severity of dental caries in many countries during recent decades, the disease remains prevalent1., 2., 3.. The United Nations recognises oral diseases, including dental caries, as representing a major health burden4.
A variety of tools for the assessment of dental caries have been presented in the literature. The most widely known and used is the decayed, missing and filled teeth (DMFT) index, recommended by the World Health Organization5. Although the DMFT index has advantages, these are outweighed by its weaknesses in relation to current thinking in caries epidemiology because it does not account for the consequences of untreated cavitated dentine lesions and do not record carious lesions in enamel. Subsequently, the index provides an underestimation of the prevalence and severity of caries1.
In an attempt to overcome difficulties experienced with the DMFT index and to combine other caries assessment indices, a new visual and tactile dental caries detection system was developed for international use; this is designated the International Caries Detection and Assessment System (ICDAS)6., 7.. This index was meant to be used in clinical practice and for education, research and epidemiological purposes. The ICDAS Coordinating Committee has since made a few changes to the original index and has named the latest version of the index ICDAS II8.
However, the ICDAS II two-digit coding system is complicated for use in caries epidemiological research. The system encumbers data analysis and makes it difficult to present results in a meaningful and easy-to-read manner9., 10., 11.. Moreover, application of the system requires the use of an air compressor for drying each tooth surface, which necessitates additional financial resources and a source of electricity. The latter makes it difficult to consider ICDAS II an index for global use in epidemiological surveys. Furthermore, it is time-consuming to use11., 12..
A special index for assessing the very advanced stages of the carious process was reported recently. This is termed the pulpal involvement ulceration fistula abscess (pufa/PUFA) index13. The pufa/PUFA index provides information about the presence of oral conditions resulting from untreated dentine cavities. However, subsequent to its application in schoolchildren, it has been suggested that code ‘u’ should be excluded from the PUFA index and codes ‘f’ and ‘a’ combined14.
The use of both the ICDAS II and PUFA indices in an epidemiological survey in Brazil revealed operational and reporting difficulties11., 14.. Notwithstanding these difficulties, the use of these two independent indices allowed the assessment of carious lesions in enamel and of the consequences of untreated cavitated dentine lesions. The use of these indices avoided the underestimation of caries prevalence that tends to occur when the DMFT index is used, but the prevalence of caries was considered to have been overestimated as a result of the use of ICDAS II11.
In view of the respective advantages and disadvantages of the ICDAS II, pufa and DMFT indices, a new index was proposed. The rationale and content of this index, termed the Caries Assessment Spectrum and Treatment (CAST), have been reported15. The CAST index describes, in a hierarchical way, the complete range of carious conditions, from the absence of carious lesions, to the presence of caries protection (sealant) and caries treatment (restoration), lesions in enamel and dentine, lesions penetrating the pulp and tissue surrounding the tooth (abscess/fistulae), and loss of teeth. The hierarchical approach implies that a high CAST score is considered to represent more severe condition than a low CAST score. The index was developed to be used solely in epidemiological surveys globally. Assessment is performed visually and the use of compressed air is not required15.
As with any new index, the CAST index needs to be validated and its reliability assured. Validating an index consists of determining its face, content and construct validity. Face validity refers to the presentation of the instrument and its suitability to assess the domain of content. Content validity refers to the extent to which a measurement reflects the domain it is intended to measure, and construct validity refers to the extent to which the object of measurement is actually measured.
The aim of the present study was to assess the CAST index for face and content validity.
MATERIALS AND METHODS
Assessment instrument
The present validation study used the RAND modified e-Delphi method16. The Delphi method, a former version, is an interactive consensus method that combines scientific evidence with the opinions and experience of experts. The expert panel explores information through a number of interactive rounds of assessment, in which members are able to rate their agreement with a set of statements and to make comments17.
The first stage in the process of assessing the face and content validity of the CAST index involved the creation of a website on which to host panel rounds. The website used in the current investigation contained a scientific publication, information related to the CAST index and its characteristics15, information about the RAND modified e-Delphi method and a manual on how to use the website. Face and content validity were assessed through a set of statements related to the CAST index that referred to the following domains: (i) the name of the index; (ii) the presentation and order of the index’s codes and descriptions; (iii) the wording of the codes and descriptions, and (iv) the applicability of the index. Four members of the CAST index development group prepared the statements.
Assessment procedure
Panel process
The statements were assessed independently by a panel composed of 15 experienced epidemiologists from five continents. Establishing an expert panel requires the identifying of people from different backgrounds who are well informed and interested in the subject to be assessed17., 18.. Panel members rated each statement on a scale of 1–9, on which 1 = total disagreement and 9 = total agreement with the statement. The nine assessment codes were clustered as 1–3, 4–6 and 7–9, indicating, respectively, ‘disagreement’, ‘not in agreement/doubt’ and ‘agreement’ with the statement. A box for inserting remarks about each statement was provided. A statement was considered valid if ≥ 75% of the panel members rated it using a code within the cluster indicating ‘agreement’. Statements that failed to reach the 75% consensus mark were modified according to the remarks made by panel members and discussion by the authors. The modified statements were then re-presented to the panel members for reassessment. In total, three rounds were necessary to reach consensus on all 17 statements.
Feedback process
The consensus statements were then presented to a feedback group to determine external validity. Based on suggestions by the panel members and a PubMed search for recently published epidemiological surveys, a total of 45 epidemiologists were approached, of whom 41 agreed to participate. The method of assessing the statements described for the panel was applied in the feedback group.
Subsequent to the feedback group’s assessment of the statements, a few changes in the wording of some descriptions were made. Ten slightly modified statements were then presented to the panel members for final assessment.
The members of the panel and the feedback group were also requested to provide information on their gender, age and number of years of experience in conducting epidemiological studies. Members were also asked for their opinions on the suitability of the RAND modified e-Delphi method to assess a diagnostic instrument for face and content validity.
Ethical aspects
This study was approved by the ethics board of the University of Brasilia Medical School (reference CEP-FM 001/2011). Panel members were requested to complete an informed consent form before logging onto the website as a requirement for joining the study. This research was conducted in full accordance with the World Medical Association’s Declaration of Helsinki.
Data analysis
Percentages of agreement were calculated after each round of judgement.
RESULTS
Disposition of members of the panel and feedback group
Data on age group, gender and number of years of experience in performing epidemiological surveys for both members of the panel (n = 15) and the feedback group (n = 41) are presented in Table 1. Only two members of the validation panel had only 3–5 years of experience in epidemiological surveys.
Table 1.
Gender, age group and years of experience in epidemiological surveys of members of the panel and feedback group
| Characteristic | Panel, n | Feedback group, n |
|---|---|---|
| Gender | ||
| Male | 7 | 26 |
| Female | 8 | 15 |
| Age, years | ||
| 31–45 | 4 | 13 |
| 46–55 | 6 | 15 |
| ≥ 56 | 5 | 13 |
| Experience in epidemiology, years | ||
| 1–9 | 3 | 12 |
| 10–25 | 7 | 16 |
| ≥ 26 | 5 | 13 |
Panel process: rounds 1–3
Panel members represented 15 countries: Australia, Brazil, Canada, Chile, China, Finland, Kuwait, Mexico, the Netherlands, Nigeria, South Africa, Tanzania, Thailand, Turkey and the UK. Response rates in the first, second and third rounds were 100%, 93.3% and 100%, respectively.
In the first round, a set of 16 statements was presented; consensus was reached on 14 statements (Table 2). The failure to achieve consensus on two statements related to the wording of Code 2 ‘Restored’ (60.0% agreed) and Code 9 ‘Does not match with any other category’ (69.2% agreed).
Table 2.
Percentage of agreement on statements by members of the expert panel over the four rounds of assessments and final scores
| Agreement, % | |||||
|---|---|---|---|---|---|
| Item | Round 1 | Round 2 | Round 3 | Round 4 | Final score |
| Codes and description for ‘Sound’ | 76.9 | NA | 86.6 | 100 | 100 |
| Codes and description for ‘Sealed’ | NA | NA | 80.0 | 86.6 | 86.6 |
| Codes and description for ‘Restored’ | 60.0 | 66.6 | 80.0 | 86.6 | 86.6 |
| Codes and description for ‘Enamel’ | 76.9 | NA | 78.5 | 80.0 | 80.0 |
| Codes and description for ‘Dentine’– code 4 | 92.3 | NA | NA | 86.6 | 86.6 |
| Codes and description for ‘Dentine’– code 5 | 92.3 | NA | NA | 100 | 100 |
| Codes and description for ‘Pulpal involvement’ | 92.3 | NA | NA | 93.3 | 93.3 |
| Codes and description for ‘Abscess/fistula’ | 100 | NA | NA | 93.3 | 93.3 |
| Codes and description for ‘Lost’ | 84.6 | NA | NA | 93.3 | 93.3 |
| Codes and description for ‘Does not correspond to any of the other descriptions’ | 69.2 | 66.6 | 86.6 | 93.3 | 93.3 |
| Suitability of the word ‘spectrum’ | 92.3 | NA | NA | NA | 92.3 |
| Appropriateness of the index name ‘CAST’ | 76.9 | NA | NA | NA | 76.9 |
| Sequence of codes | 84.6 | NA | NA | NA | 84.6 |
| Suitability of index codes | 76.9 | NA | NA | NA | 76.9 |
| International implementation of the CAST index | 100 | NA | NA | NA | 100 |
| Straightforwardness of the CAST index | 76.9 | NA | NA | NA | 76.9 |
| Appropriateness of using one score to assess enamel carious lesions | 76.9 | NA | NA | NA | 76.9 |
NA, not applicable.
In the second round, one panel member was unable to participate for personal reasons. There was no increase in the number of statements on which consensus had been reached (Table 2). However, remarks made by members of the panel were considered valuable and resulted in changes in some descriptions.
The third round involved the assessment of five statements in which index codes and descriptions had been modified. The word ‘Sealant’ (Code 1) was added as a characteristic, upon recommendation by panel members. Consensus was reached on all statements, including three additional statements: ‘Sealed’ (85.7% agreed); Restored (80.0% agreed), and ‘Does not match with any other category’ (92.8% agreed). At the end of the third round, consensus had been reached on all 17 statements presented to members of the panel (Table 2).
Feedback process
The 41 members of the feedback group originated from 24 countries: Australia, Belgium, Brazil, Chile, China, Czech Republic, Egypt, Finland, Germany, Indonesia, Ireland, Italy, Kenya, Kuwait, the Netherlands, Nigeria, South Africa, Spain, Sweden, Switzerland, Tanzania, Thailand, Turkey and the USA.
A set of the original 17 statements plus three additional statements relating to the structure and applicability of the CAST index record form were rated by 95.1% of the feedback group as an external validity procedure. Consensus was reached on 14 statements. The six statements for which consensus was not reached were related to the index name (57.1% agreed), the suitability of the word ‘spectrum’ (71.4% agreed), the wording of the descriptions for ‘Sealed’ (68.2% agreed), ‘Restored’ (60.9% agreed), ‘Enamel’ (65.8% agreed) and ‘Does not match with any other category’ (64.1% agreed). Remarks made by members of the feedback group and the absence of consensus on six statements led to a few changes in the wording of the descriptions of the index codes.
Panel process: final round
As the wording of codes and descriptions used in the CAST index had been altered, the 15 members of the panel were invited to participate in a final rating of 10 statements, each related to one of the codes or descriptions used in the CAST index. Consensus was reached on all 10 statements with agreement of ≥ 80% (Table 2). Remarks made by panel members indicated support for the changes that had been made in the descriptions of the index codes. After five rounds of assessment, the CAST index had been successfully assessed for face and content validity. The changes to the codes and descriptions of the CAST index made during the validation exercise are presented in Table 3. The final validated CAST index is presented in Table 4.
Table 3.
Changes in codes and descriptions from the start of the process, when the index was termed ‘CAS’, to Round 3 and the final validation of the Caries Assessment Spectrum and Treatment (CAST) index
| CAST index versions | |||||
|---|---|---|---|---|---|
| Started as CAS | After Round 3 | Final version | |||
| Code | Description | Code | Description | Code | Description |
| 0 Sound | There should be no evidence of dental caries | 0 Sound | No visible evidence of a distinct carious lesion is present | 0 Sound | No visible evidence of a distinct carious lesion is present |
| 1 Enamel | Distinct visual change in enamel. A clear carious opacity (white or brown in colour) is visible under dry or wet circumstances, including localised enamel breakdown without clinical visual signs of dentinal involvement | 1 Sealed | Pits and fissures have been at least partially sealed with a sealant material | 1 Sealed | Pits and/or fissures are at least partially covered with a sealant material |
| 2 Dentine | Internal caries-related discolouration in dentine. The lesion appears as shadows of discoloured dentine visible through an apparently intact enamel which may or may not exhibit localised breakdown | 2 Restored | A cavity has been restored with an (in)direct restorative material currently without a dentine carious lesion and no fistula/abscess present | 2 Restored | A cavity is restored with an (in)direct restorative material |
| 3 Dentine | Distinct cavity with or without visible dentine. No (expected) pulpal involvement. Cavitation in opaque or discoloured enamel exposing the dentine beneath, no (expected) pulpal involvement | 3 Enamel | Distinct visual change in enamel. A clear carious related discolouration (white or brown in colour) is visible, with or without localised enamel breakdown without clinical signs of dentinal involvement | 3 Enamel | Distinct visual change in enamel only. A clear caries-related discolouration is visible, with or without localised enamel breakdown |
| 4 Pulp | Pulpal involvement. The opening of the pulp chamber is visible or the coronal tooth structures have been destroyed by the caries process and only root fragments are left | 4 Dentine | Internal caries-related discolouration in dentine. The lesion appears as shadows of discoloured dentine through enamel which may or may not exhibit a visible localised breakdown | 4 Dentine | Internal caries-related discolouration in dentine. The discoloured dentine is visible through enamel which may or may not exhibit a visible localised breakdown of enamel |
| 5 Pulp | Abscess/fistula. A pus-containing swelling or a pus-releasing sinus tract related to a tooth with pulpal involvement due to dental caries is present | 5 Dentine | Distinct cavitation into dentine. No (expected) pulpal involvement | 5 Dentine | Distinct cavitation into dentine. The pulp chamber is intact |
| 6 Missing | Tooth has been extracted because of dental caries | 6 Pulp | Involvement of the pulp chamber. Distinct cavitation reaching the pulp chamber or only root fragments are present | 6 Pulp | Involvement of the pulp chamber. Distinct cavitation reaching the pulp chamber or only root fragments are present |
| 7 Filled | A tooth cavity has been restored with a longlasting restorative material without a dentine carious lesion associated with it and no fistula/abscess present | 7 Abscess/fistula | A pus-containing swelling or a pus-releasing sinus tract related to a tooth with pulpal involvement due to caries is present | 7 Abscess/fistula | A pus-containing swelling or a-pus-releasing sinus tract related to a tooth with pulpal involvement |
| 9 | Unable to diagnose | 8 Lost | The tooth has been removed because of dental caries | 8 Lost | The tooth has been removed because of dental caries |
| 9 | Does not match to any of the other descriptions | 9 | Does not correspond to any of the other descriptions | ||
Table 4.
The codes and descriptions of the hierarchical Caries Assessment Spectrum and Treatment (CAST) index as approved for face and content validity
| Characteristic | Code | Description |
|---|---|---|
| Sound | 0 | No visible evidence of a distinct carious lesion is present |
| Sealant | 1 | Pits and/or fissures are at least partially covered with a sealant material |
| Restoration | 2 | A cavity is restored with an (in)direct restorative material |
| Enamel | 3 | Distinct visual change in enamel only. A clear caries-related discolouration is visible, with or without localised enamel breakdown |
| Dentine | 4 | Internal caries-related discolouration in dentine. The discoloured dentine is visible through enamel which may or may not exhibit a visible localised breakdown of enamel |
| 5 | Distinct cavitation into dentine. The pulp chamber is intact | |
| Pulp | 6 | Involvement of the pulp chamber. Distinct cavitation reaching the pulp chamber or only root fragments are present |
| Abscess/fistula | 7 | A pus-containing swelling or a pus-releasing sinus tract related to a tooth with pulpal involvement |
| Lost | 8 | The tooth has been removed because of dental caries |
| Other | 9 | Does not correspond to any of the other descriptions |
Suitability of the RAND modified e-Delphi method
The RAND modified e-Delphi method was considered by 78.0% of the feedback group to be a suitable procedure for the online assessment of a diagnostic instrument (the CAST index) for face and content validity.
DISCUSSION
The RAND modified e-Delphi method was used to assess the CAST index for face and content validity. A literature search of PubMed using the MeSH (medical subject headings) terms ‘Delphi technique’ and ‘dental caries’ brought up six papers describing the application of the method to obtain consensus. Topics covered included a review of the Delphi technique19, establishing priorities in oral health care20, designing clinical protocols21, developing educational systems22, validating criteria for a treatment decision23 and evaluating the effectiveness of dental care plans24. The method has not previously been used to validate an epidemiological index, as in the present study. However, the Delphi method has been used to assess face and content validity of diagnostic instruments in the medical25 and nursing17 fields and its use in the seeking of consensus in the health care field is increasing18 as a result of its flexibility26. The RAND modified e-Delphi is an electronic version of the Delphi method which eases data entry and enhances communication among panel members in comparison with the traditional Delphi method.
The suitability of the RAND modified e-Delphi method for establishing the face and content validity of a diagnostic and detection instrument, such as the CAST index, depends on its facility to allow for the gathering of opinions from experts in different parts of the world with diverse backgrounds in a process which, in combination with the appropriate scientific information, leads to interactive rounds of discussion and results in very reliable outcomes18. Another advantage of this method refers to the fact that participation is anonymous, which allows each panel member to present his or her ideas individually and to respond to statements in an unbiased manner. Members of the feedback group appear to have been aware of these advantages as the majority considered the RAND modified e-Delphi method suitable for establishing the face and content validity of a diagnostic instrument.
Usually, a panel consists of 10–15 participants, but consensus sessions involving over 100 people have been carried out26. The number of participants depends on the homogeneity of the sample of experts, the manageability of the process and whether verification is internal or external. In the present study, the panel comprised 15 senior epidemiologists and, as larger numbers of experts are required for assessing external validity26, the feedback group was estimated to require around 45 members.
Although the epidemiologists could be contacted online, the validation process took a long time. Some epidemiologists responded swiftly, but others did not for acceptable reasons. This, together with the different holiday periods in effect in the northern and southern hemispheres and despite the emailed reminders, contributed to a data collection period of 10 months. This long investment in time allowed the CAST index to be assessed by many epidemiologists from all corners of the world. It ensured the index’s external validity. The fact that native English-speaking epidemiologists were among the members greatly facilitated the clarity of words, meaning and grammar of descriptions. The participation of senior epidemiologists with vast knowledge of the subject ensured that the index was judged properly.
Notwithstanding the validation of the index by the 15 epidemiologists, the decision to ask a feedback group to assess the validated index for clarity and understanding turned out to be fruitful. Comments from these epidemiologists, who came from different cultural and social backgrounds, necessitated some adjustment to the officially validated index. It showed that the methodology used was able to fine-tune the codes and descriptions to produce an even more well-balanced index.
The CAST index covers the complete spectrum of the carious process, including in sealed and restored teeth, and is set up to describe a hierarchy of severity. It has the potential to become an important tool for epidemiological caries assessment. The index was rated as being straightforward, easy to apply and suitable for use internationally. The validation acceptance scores for codes and their descriptions were very high. The index has been assessed as demonstrating face and content validity with a high level of agreement, particularly for its index codes and their descriptions. External validity was obtained. Further development of the CAST index involves the assessment of its construct validity and reliability. Both activities are underway.
Acknowledgements
We are very grateful to the members of the panel and feedback groups for their participation and enthusiasm during the implementation of this study. We extend our appreciation to Fintan Planting, for designing and operating the website. We thank CAPES (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior) and Radboud University Nijmegen for supporting this study financially.
Conflicts of interest
None declared.
REFERENCES
- 1.Burt BA, Baelum V, Fejerskov O. In: Dental Caries – The Disease and its Clinical Management. 2nd ed. Fejerskov O, Kidd E, editors. Blackwell Munksgaard; Frederiksberg: 2008. The epidemiology of dental caries; pp. 123–146. [Google Scholar]
- 2.Piovesan C, Mendes FM, Ferreira FV, et al. Socioeconomic inequalities in the distribution of dental caries in Brazilian preschool children. J Public Health Dent. 2010;70:319–326. doi: 10.1111/j.1752-7325.2010.00191.x. [DOI] [PubMed] [Google Scholar]
- 3.Huew R, Waterhouse PJ, Moynihan PJ, et al. Prevalence and severity of dental caries in Libyan schoolchildren. Int Dent J. 2011;61:217–223. doi: 10.1111/j.1875-595X.2011.00060.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.United Nations General Assembly. Political declaration of the high-level meeting of the General Assembly on the Prevention and Control of Non-communicable Diseases. 66th Session. 2011. Available from: http://http://www.un.org/ga/search/view_doc.asp?Symbol=A/66/L.1. Accessed 12 September 2012
- 5.World Health Organization . 4th ed. WHO; Geneva: 1997. Oral Health Surveys: Basic Methods. [Google Scholar]
- 6.Pitts N. ICDAS – an international system for caries detection and assessment being developed to facilitate caries epidemiology, research and appropriate clinical management. Community Dent Health. 2004;21:193–198. [PubMed] [Google Scholar]
- 7.Ismail AI, Sohn W, Tellez M, et al. The International Caries Detection and Assessment System (ICDAS): an integrated system for measuring dental caries. Community Dent Oral Epidemiol. 2007;35:170–178. doi: 10.1111/j.1600-0528.2007.00347.x. [DOI] [PubMed] [Google Scholar]
- 8.Pitts N. Karger; Basel: 2009. Detection, Assessment, Diagnosis and Monitoring of Caries; pp. 1–41. [DOI] [PubMed] [Google Scholar]
- 9.Agustsdottir H, Gudmundsdottir H, Eggertsson H, et al. Caries prevalence of permanent teeth: a national survey of children in Iceland using ICDAS. Community Dent Oral Epidemiol. 2010;38:299–309. doi: 10.1111/j.1600-0528.2010.00538.x. [DOI] [PubMed] [Google Scholar]
- 10.Cadavid AS, Lince CM, Jaramillo MC. Dental caries in the primary dentition of a Colombian population according to the ICDAS criteria. Braz Oral Res. 2010;24:211–216. doi: 10.1590/s1806-83242010000200014. [DOI] [PubMed] [Google Scholar]
- 11.de Amorim RG, Figueiredo MJ, Leal SC, et al. Caries experience in a child population in a deprived area of Brazil, using ICDAS II. Clin Oral Invest. 2012;16:513–520. doi: 10.1007/s00784-011-0528-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Braga MM, Oliveira LB, Bonini GA, et al. Feasibility of the International Caries Detection and Assessment System (ICDAS-II) in epidemiological surveys and comparability with standard World Health Organization criteria. Caries Res. 2009;43:245–249. doi: 10.1159/000217855. [DOI] [PubMed] [Google Scholar]
- 13.Monse B, Heirich-Weltzien R, Benzian H, et al. PUFA – an index of clinical consequences of untreated dental caries. Community Dent Oral Epidemiol. 2010;38:77–82. doi: 10.1111/j.1600-0528.2009.00514.x. [DOI] [PubMed] [Google Scholar]
- 14.Figueiredo MJ, de Amorim RG, Leal SC, et al. Prevalence and severity of clinical consequences of untreated dentine carious lesions in children from a deprived area of Brazil. Caries Res. 2011;45:435–442. doi: 10.1159/000330531. [DOI] [PubMed] [Google Scholar]
- 15.Frencken JE, de Amorim RG, Faber J, et al. The Caries Assessment Spectrum and Treatment (CAST) index: rationale and development. Int Dent J. 2011;61:117–123. doi: 10.1111/j.1875-595X.2011.00022.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Dalkey NC. RAND Corporation; Santa Monica, CA: 1969. The Delphi Method: An Experimental Study of Group Opinion. [Google Scholar]
- 17.Chang AM, Gardner GE, Duffield C, et al. A Delphi study to validate an advanced practice nursing tool. J Adv Nurs. 2010;66:2320–2330. doi: 10.1111/j.1365-2648.2010.05367.x. [DOI] [PubMed] [Google Scholar]
- 18.Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud. 2001;38:195–200. doi: 10.1016/s0020-7489(00)00044-4. [DOI] [PubMed] [Google Scholar]
- 19.Woolgrove J. Delphi definitions. Br Dent J. 2006;201:186–187. doi: 10.1038/sj.bdj.4813903. [DOI] [PubMed] [Google Scholar]
- 20.Kuthy RA, Siegal MD, Wulf CA. Establishing maternal and child health data collection priorities for state and local oral health programmes. J Public Health Dent. 1997;57:197–205. doi: 10.1111/j.1752-7325.1997.tb02976.x. [DOI] [PubMed] [Google Scholar]
- 21.Paterson FM, Paterson RC, Watts A, et al. Initial stages in the development of valid criteria for the replacement of amalgam restorations. J Dent. 1995;23:137–143. doi: 10.1016/0300-5712(95)93570-r. [DOI] [PubMed] [Google Scholar]
- 22.Mettes TG, van der Sanden WJ, van Eeten-Kruiskamp L, et al. Routine oral examination: clinical vignettes, a promising tool for continuing professional development? J Dent. 2010;38:377–386. doi: 10.1016/j.jdent.2010.01.004. [DOI] [PubMed] [Google Scholar]
- 23.Milsom K, Tickle M, Jenner A, et al. The identification of agreed criteria for referral following the dental inspection of children in the school setting. Br Dent J. 1999;186:37–40. doi: 10.1038/sj.bdj.4800011. [DOI] [PubMed] [Google Scholar]
- 24.Bader JD, Shugars DA, White BA, et al. Development of effectiveness of care and use of services measures for dental care plans. J Public Health Dent. 1999;59:142–149. doi: 10.1111/j.1752-7325.1999.tb03263.x. [DOI] [PubMed] [Google Scholar]
- 25.Vakil N. Consensus guidelines: method or madness? Am J Gastroenterol. 2011;106:225–227. doi: 10.1038/ajg.2010.504. [DOI] [PubMed] [Google Scholar]
- 26.Skulmoski GJ, Hartman FT, Krahn J. The Delphi method for graduate research. J Inform Technol Educ. 2007;6:1–21. [Google Scholar]