Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Jun 5.
Published in final edited form as: Caries Res. 2019 Jun 5;53(6):650–658. doi: 10.1159/000499700

Developing and Testing Electronic Health Record-Derived Caries Indices

Joel M White 1,2,*, Elizabeth A Mertz 1,2,3, Joanna M Mullins 4, Joshua B Even 4, Trey Guy 4, Elena Blaga 4, Aubri M Kottek 2,3, Shwetha V Kumar 5, Suhasini Bangar 5, Ram Vaderhobli 1, Ryan Brandon 4, William Santo 1,2, Larry Jenson 1, Stuart A Gansky 1,2,3,6
PMCID: PMC6864222  NIHMSID: NIHMS1022066  PMID: 31167186

Abstract

Caries indices, the basis of epidemiologic caries measures, are not easily obtained in clinical settings. This study’s objective was to design, test, and validate an automated program (Valid Electronic Health Record Dental Caries Indices Calculator Tool: VERDICT) to calculate caries indices from an electronic health record (EHR). Synthetic use case scenarios and actual patient cases of primary, mixed, and permanent dentition, including decayed, missing, and filled teeth (DMFT/dmft) and tooth surfaces (DMFS/dmfs) were entered into the EHR. VERDICT measures were compared to a previously validated clinical electronic data capture (EDC) system and statistical program to calculate caries indices. Four university clinician-researchers abstracted EHR caries exam data for 45 synthetic use cases into the EDC and post-processed with SAS software creating a gold standard to compare the VERDICT-derived caries indices. Then two senior researchers abstracted EHR caries exam data and calculated caries indices for 24 patients, allowing further comparisons to VERDICT indices. Agreement statistics were computed among abstractors, and discrepancies were resolved by consensus. Agreement statistics between the two final phase abstractors and the VERDICT measures showed extremely high concordance: Lin’s concordance coefficients (LCCs)>0.99 for dmfs, dmft, DS, ds, DT, dt, ms, mt, FS, fs, FT, and ft; LCCs>0.95 for DMFS and DMFT; and LCCs of 0.92–0.93 for MS and MT. Caries indices, essential to developing primary health outcome measures for research, can be reliably derived from an EHR using VERDICT. Using these indices will enable population oral health management approaches and inform quality improvement efforts.

Keywords: Dental informatics/bioinformatics, Electronic dental records, Outcomes research, Epidemiology, Caries detection/diagnosis/prevention

Introduction

Electronic health records (EHRs) provide dental researchers the ability to extract vast amounts of patient data quickly, accurately and economically, thus enhancing and expanding all forms of scientific discovery. Perhaps the most useful oral health measure is the patient’s caries experience, expressed as a caries index [Bratthall, 2000; Ditmyer et al., 2011; Gruebbel, 1944; Klein et al., 1938; Marthaler, 2004; Monse et al., 2010; van Dommelen and Schuller, 2016]. Caries indices measure the severity of dental caries among populations and individual patients and can be used for prevalence estimates; they allow for longitudinally evaluating interventions for preventing and treating disease and can be used for incidence estimates. Traditional research methods of defining, measuring and comparing caries indices of populations have enabled successful epidemiologic comparisons. However, these methods have known limitations including standardization of measurements, high costs associated with labor-intensive data acquisition, difficult participant enrollment, examiner and abstractor calibration, slow results production, and often limitations in research designs that result in less generalizability [Kahn et al., 2007; Song et al., 2013].

EHRs seem ideal to overcome such limitations; yet using EHR data for research has specific challenges. These include the non-universal EHR implementation and utilization by practices and providers, different and incompatible EHR systems commonly used in practices, non-standardized data input interfaces, non-searchable data elements, missing data fields, lack of diagnostic codes, incomplete clinical information, and data governance and security issues [de Lusignan et al., 2006; Gilbert et al., 2011; Kahn et al., 2007; Kristianson et al., 2009; Liu et al., 2013; Mansson et al., 2004; Schleyer et al., 2007; Sung et al., 2003; Terry et al., 2010]. Overall, the predominant problem with current EHR systems is that they are designed and developed to focus on clinical care and administrative tasks rather than research. Thus, despite EHRs’ potential digital benefits, researchers needing to employ traditional research methods can be challenged using EHR data. In past caries indices research examiners and abstractors had to either infer if a filled or missing tooth was due to caries or simply not differentiate between them, thus reducing the validity of caries studies [National Health and Nutrition Examination Survey, 2016; World Health Organization, 2013].

Several recent developments have addressed the challenges to caries indices research. First, EHRs are increasingly common in dental practices. While only 11% of dental offices used computers in 1984, 78% did in 2013 [Schleyer et al., 2013]. Standardized dental diagnostic terminology (SNOMED, SNODENT, and the interface terminology within the EHR, SNODDS) now allow clinicians to input diagnosis data associated with dental procedures, thereby recording reasons for filled and missing teeth [Kalenderian et al., 2011; White et al., 2011]. Second, standardized diagnostic terminology within the EHR allows for more accurate caries index scores by designating the reasons for conditions found and recorded. Having expanded conditions and specific diagnoses allows for analyzing decayed surfaces (permanent and primary) and lesion depth, including enamel lesions (initial caries), shallow dentin lesions (moderate caries), and deep dentin lesions (extensive caries). Third, many dental schools, dental maintenance organizations, and dental accountable care organizations use the same EHR software. Data from these sources are more standardized, accessible, and extensive as an enormous repository for analysis [Walji et al., 2014]. Lastly, clinical trials research utilizing EHR data is potentially more cost-effective than using paper-based data collection [Pavlovic et al., 2009]. Even with good, standardized decayed, missing and filled data, carefully chosen with an eye for caries research efforts, EHR systems have generally lacked automated data abstraction algorithms that allow provider inputs in the EHR to be quickly converted into usable research formats. This is especially true of caries indices where no automated abstraction and indexing procedures to derive caries indices currently exists. This paper reports on the design, development and validation of an algorithm (denoted as Valid Electronic Health Record Dental Caries Indices Calculator Tool: VERDICT) that automatically abstracts EHR data on decayed, missing due to decay, and filled teeth due to decay at the tooth and tooth surface levels and then calculates traditional caries indices for primary and permanent dentition.

Materials and Methods

This study was conducted with Institutional Review Board approval to evaluate a program in the Willamette Dental Group (WDG), which serves over 400,000 patients in over 50 offices throughout Oregon, Washington and Idaho. WDG utilizes the same EHR as most United States and Canadian dental schools and several large practices, axiUm (Exan Corporation, Coquitlam, British Columbia, Canada). The EHR contains standardized clinical interfaces for charting conditions and findings of decay as well as required diagnostic terminology for all planned and completed procedures. The software also captures diagnosis with procedures so reasons for treatment are recorded, i.e., restorations or extractions due to decay. All WDG offices successfully implemented the EHR by 2013 and all care provider teams receive standardized training and calibration for data entry procedures to standardize patient care [Mertz et al., 2017]. All providers are required to utilize the EHR and are evaluated by their consistent and accurate adherence to the EHR through process and appropriateness of care measures. Adherence rates throughout WDG offices exceed 98% for process of care measures, including charting and treatment planning [Mertz et al., 2017].

For this study, caries was defined as any cavitated caries lesion that penetrated into dentin in accordance with WHO protocols [World Health Organization, 2013]. An algorithm was developed to translate existing WDG odontogram and treatment history data into caries indices, including indices of decayed (cavitated dentin caries, D, d), missing (M, m) and filled (F, f) designations of permanent (DMF) and deciduous (dmf) tooth surfaces (S, s) and teeth (T, t). The research team worked iteratively with programmers providing specifications and expertise to develop and configure the algorithm (VERDICT) to determine caries indices using EHR data. Appendix Table 1 lists the 36 caries indices produced from automated scripts that abstract data from each patient’s digital odontogram and treatment history.

The algorithm, based on the data schema framework and EHR structure, consists of four essential parts:

  1. EHR data – The actual charted findings of decay, restorations, dentition, missing teeth, and planned procedures due to decay are collected by providers during patient care. Importantly, no additional data collection were required; the system utilizes EHR data already recorded and available.

  2. Configuration – Configuration is the mapping process that allows the provider-charted entry to be interpreted by the algorithm to accurately determine decayed, missing, and filled designations and calculate the caries indices. The mapping includes designating the procedures and materials that represent restorations and the conditions and diagnoses that indicate decay. The algorithm was developed based on the data schema, framework, and EHR structure.

  3. Automated resolution process – When decayed surfaces are restored or teeth are extracted/exfoliated, charted decay conditions are updated (i.e., changed to be no longer decayed). For clinical care, a manual EHR process “resolves” a decay finding after restoration or extraction. When this manual process is not properly completed, an automated process each day resolves any decay conditions not manually resolved.

  4. Calculation – Using EHR data (1 and 3) and the configuration (2), the algorithm computes caries indices starting with the patient’s first EHR-recorded visit and all subsequent visits. No additional data are stored, as indices are calculated in real-time for any new visits since the last calculation.

Two-phase testing of VERDICT

Initial testing of the VERDICT algorithm entailed creating 45 synthetic use cases that represented synthetic patients with a wide range of possible oral conditions including positive and negative controls. Data included synthetic use cases with all teeth present, all teeth missing, all teeth decayed, no teeth with decay, all tooth surfaces filled, no tooth surfaces filled, and many more scenarios (see Appendix Table 2 and Figures). Cases also balanced primary, permanent, and mixed dentitions. Use cases were entered into a special research area of the EHR.

Four university clinician-researchers served as independent abstractors based on their extensive experience in utilizing dental EHRs and determining decayed, missing, and filled teeth from EHR odontograms and treatment histories. Abstractors were trained to use a validated electronic data capture program, the CAries Research INstrument (CARIN) software package [Regents of the University of California, 2009; Warren et al., 2015]. Modeled on the NHANES two-pass system, CARIN allows dental researchers to enter dental examination data in real time or by post-examination abstraction from paper or electronic odontograms via a graphical user interface and converts the data into a usable electronic form for epidemiologic and clinical research. Instructions were given about navigating the EHR, reading the odontogram and treatment history, and abstracting data to enter into CARIN. Each clinician-researcher independently abstracted EHR data (Figure 1a) from the 45 synthetic use cases into the CARIN software (Figure 1b). Sixteen of the most common caries indices or their components (Appendix Table 1) were calculated for each test case using SAS software (SAS Institute, Cary, NC). Disagreements among abstractor results were resolved by consensus. Each abstractor also completed six repeat patient scenarios randomly selected to estimate intra-rater reliability. These consensus caries indices became the gold standard for evaluating the validity of the VERDICT algorithm in this phase of testing.

Figure 1.

Figure 1.

Figure 1.

Open in a new tab

Example Synthetic Scenario Odontogram in axiUm (a) and CARIN (b)

VERDICT was executed on all 45 use cases for 16 indices. These were then compared to the gold standard-derived indices for each scenario using Lin’s concordance correlation (LCC) for person-level indices. The first phase of algorithm testing showed generally good concordance. Discrepancies were identified and further algorithm refinement was conducted to address missing teeth and surfaces due to decay (versus any other reason), utilize diagnostic codes (to better identify restorations and extractions due to caries), resolve conditions when there was a subsequent restoration of a tooth, add counts of decayed surfaces planned for restoration or extraction due to caries, address stainless steel crowns due to decay, and (optionally) include incisal surfaces. After algorithm refinement, testing was conducted on a purposive sample of patient data.

The second phase of testing utilized 24 actual WDG patient records. This patient sample was purposive; patients were chosen to best test the VERDICT algorithm with various caries experiences, ages, dentitions, and insurance plans, as well as new and recall patients. A general goal of patient selection was to identify a sample to assess automated calculations with real data, covering many types of decay, as well as missing and filled surface patterns. To reduce the influence of possible sample patient types, six distinct subgroups of sample study patients were specified (Table 1).

Table 1.

Purposively Sampled Patient Subgroups.

Group Gender Age Group Level of Decay
A Male 13–18 Extreme (DMFS >30)
B Female 6–12 High (DMFS 15–25)
C Male 13–18 Moderate (DMFS 10–15)
D Female 6–12 Moderate (DMFS 10–15)
E Male 13–18 None (DMFS 0)
F Female 3–5 Moderate (DMFS >10), with missing teeth due to decay
Open in a new tab

Additional stratification for longitudinal validation included only patients with an initial visit (new patient or continuing care visit) (T0) and a regular continuing care visit (T1). An equal number of patients with two insurance payer types (Medicaid & private) were used for socioeconomic status generalizability. This resulted in 24 distinct subgroups of study patients with at least 10 patients each. One patient was selected from each subgroup to test the algorithm. In selecting the specific patient from each subgroup, attempts were made to identify some patients whose caries indices changed between the exam dates (except for group E as these patients were selected to have no decay at each timepoint). Additionally, cases with more than one finding category (e.g., primary/permanent, decayed/missing/filled) were given preference.

As in the first phase of testing, two trained and calibrated abstractors, a senior university dentist-researcher and a senior software developer, independently abstracted data from this sample of 24 patient odontograms and treatment histories at two different times, T0 and T1 (n=48), using CARIN; indices were calculated with SAS. Disagreements between abstractor results were resolved by consensus agreement. Each abstractor was again randomly assigned six repeat patient scenarios to estimate intra-rater reliability. The consensus for the 24 sample patient charts became the gold standard for this phase of testing. The VERDICT algorithm then processed the 48 patient charts (at T0 and T1) and the results were compared to the expert abstractor gold standard.

Results

VERDICT was successfully designed, tested, and validated to produce caries indices from an EHR. For each synthetic use case scenario, the odontogram and treatment history are shown in the Appendix Figures. The corresponding VERDICT (EHR-derived) caries indices for each use case are listed in Appendix Table 2. For the 45 synthetic test cases, agreement between the caries indices produced by VERDICT and the gold standard method are shown in Table 2.

Table 2.

Gold Standard vs. VERDICT Concordance for Synthetic Use Cases (N=45), CL=confidence limit.

Measure Description Lin’s
Concordance
Correlation		 Lower
95% CL		 Upper
95% CL
DMFS Number of decayed, missing, or filled permanent tooth surfaces 0.95 0.92 0.98
DMFT Number of decayed, missing, or filled permanent teeth 0.96 0.93 0.98
DS Number of decayed permanent tooth surfaces 1.00 1.00 1.00
DT Number of decayed permanent teeth 1.00 1.00 1.00
FS Number of filled permanent tooth surfaces 1.00 1.00 1.00
FT Number of filled permanent teeth 1.00 1.00 1.00
MS Number of missing permanent tooth surfaces 0.92 0.88 0.96
MT Number of missing permanent teeth 0.93 0.90 0.96
dmfs Number of decayed, missing, or filled primary tooth surfaces 1.00 1.00 1.00
dmft Number of decayed, missing, or filled primary teeth 1.00 1.00 1.00
ds Number of decayed primary tooth surfaces 1.00 1.00 1.00
dt Number of decayed primary teeth 0.99 0.98 1.00
fs Number of filled primary tooth surfaces 1.00 1.00 1.00
ft Number of filled primary teeth 0.99 0.98 1.00
ms Number of missing primary tooth surfaces 1.00 0.99 1.00
mt Number of missing primary teeth     1.00 1.00 1.00
Open in a new tab

Disregarding the positive and negative controls (i.e., extremes of no caries and all teeth carious, all teeth present and all teeth extracted, no restorations and all surfaces with restorations), among synthetic use cases 64% had decay or filled surfaces; their mean number of decayed, missing, or filled primary and permanent surfaces was 31 (dmfs +DMFS =31), while their mean number of decayed primary and permanent surfaces was 5 (ds +DS = 5). Investigators agreed that the synthetic use cases represented enough variation to initially test the algorithm well. VERDICT caries indices associated with missing teeth and surfaces due to decay had the lowest correlation with the gold standard in the first pass. Intra-rater reliability (n=6 use cases) was excellent for all 4 abstractors: LCC>0.93 for DMFS, DMFT, dmfs, dmft, ds, dt; LCC>0.85 for DT; and LCC>0.74 for DS. By design, the iterative development continued until the algorithm matched the gold standard consensus (i.e., perfect agreement between the gold standard and VERDICT caries indices for the 45 synthetic use case scenarios).

The EHR-derived VERDICT caries indices were successfully generated and validated from the purposive sample of 24 patient charts at two different time points (n=48). A representative odontogram and VERDICT caries indices are shown in Figure 2. Agreement between the VERDICT algorithm and the two abstractors was nearly perfect: LCCs>0.99, for all indices. The first abstractor overlooked a few missing surfaces, while the agreement between VERDICT and the second abstractor (gold standard) was perfect (LCCs=1.0; Table 3). Intra-rater reliability (n=6 patient charts) was perfect for both abstractors (LCCs=1.0).

Figure 2.

Figure 2.

Figure 2.

Open in a new tab

Example Patient Odontogram (a) and Caries Indices EHR-Derived Automated Procedures using VERDICT (b)

Table 3.

Abstractors vs. VERDICT Concordance for Patient EHR-Data at Two Visits (N=48), CL=Confidence Limit. Note there were no patients with missing teeth (MT) and missing surfaces (MS) for permanent teeth due to decay in this purposive sampling (n=0). Gold standard was abstractor 2.

Abstractor Measure Lin’s Concordance
Correlation		 Lower
95% CL		 Upper
95% CL
1 DMFS 1.00 1.00 1.00
1 DMFT 1.00 1.00 1.00
1 DS 1.00 1.00 1.00
1 DT 1.00 1.00 1.00
1 FS 1.00 1.00 1.00
1 FT 1.00 1.00 1.00
1 dmfs 1.00 0.99 1.00
1 dmft 1.00 1.00 1.00
1 ds 1.00 1.00 1.00
1 dt 1.00 1.00 1.00
1 fs 0.99 0.99 1.00
1 ft 1.00 1.00 1.00
1 ms 0.99 0.99 0.99
1 mt 1.00 1.00 1.00
2 DMFS 1.00 1.00 1.00
2 DMFT 1.00 1.00 1.00
2 DS 1.00 1.00 1.00
2 DT 1.00 1.00 1.00
2 FS 1.00 1.00 1.00
2 FT 1.00 1.00 1.00
2 dmfs 1.00 1.00 1.00
2 dmft 1.00 1.00 1.00
2 ds 1.00 1.00 1.00
2 dt 1.00 1.00 1.00
2 fs 1.00 1.00 1.00
2 ft 1.00 1.00 1.00
2 ms 1.00 1.00 1.00
2 mt 1.00 1.00 1.00
Open in a new tab

Discussion/Conclusion

The purpose of this study was to develop and validate an algorithm that automatically determines caries indices from an EHR. The study’s results show that the algorithm developed (VERDICT) is a valid instrument. Developing this algorithm was an essential part of an ongoing evaluation of a caries management program assessing oral health disparities in children. This study has demonstrated a valid and feasible process to produce accurate caries indices for large populations with the ability to follow those populations and the individuals within them longitudinally using EHR data. Creating a primary oral health outcome measure through a valid EHR algorithm for producing automated caries indices opens new doors to dental researchers:

  1. Caries outcomes research can now use caries indices obtained on a large scale from an EHR.

  2. Larger sample sizes for caries research can be collected quickly and inexpensively as compared to conventional randomized prospective clinical trials and epidemiologic studies.

  3. Longitudinal studies on individual patients and populations can now be conducted. This is especially important when tracking the therapeutic effects of behavioral and clinical preventive interventions.

  4. Extent of disease can be assessed in both primary and recurrent carious lesions.

  5. A more comprehensive variety of caries indices are now available, including tracking enamel lesions and incisal edge lesions. (However, when contrasting with national, state, and most other published data, comparable traditional caries indices should be used since the number of tooth surfaces at risk differ.)

  6. Other researchers can validate their caries index calculation software with this set of 45 synthetic use cases and results (see Appendix Table 2 and Figures).

  7. Real-time communication to providers on the caries experience of individual patients and the population of patients under their care is now possible, for example through data dashboards, allowing for real-time patient care decisions regarding caries interventions.

The EHR algorithm has the optional capability of producing caries indices that include enamel lesions, third molars, and incisal surfaces, as well as extent of decay, information normally not obtained in research using traditional caries indices, giving the opportunity for more granular data on extent of decay. This is due to the inclusion of the dental diagnostic interface terminology within the EHR. However, for comparability to state, national, and other published data, the traditional caries indices should be used to keep the number of teeth and number of tooth surfaces standardized.

The method used to test the EHR algorithm, comparing caries index outputs to more labor intensive, expertly determined caries indices in 45 synthetic use case scenarios and then using a subset of patient EHR data, could be replicated to disseminate and adopt the automated techniques in other EHR systems. As EHR use continues to expand throughout dentistry and medicine, algorithms to exploit the accumulating data are increasingly being explored, developed and validated. Validation procedures routinely and necessarily involve using expert examiners to create a gold standard against which algorithms can be tested [Barnado et al., 2017; Castro et al., 2015; Gruschow et al., 2016; Lingren et al., 2016; Sharma et al., 2017].

The algorithm developed and reported here was configured for the Willamette Dental Group axiUm EHR software implementation. Alternative implementations of axiUm would require modifying VERDICT mappings as not all implementations of axiUm contain the same configuration of dental conditions, procedures and diagnosis. Implementation at other institutions using axiUm with the standardized dental diagnostic terminology and standardized dental procedures would be straightforward, requiring only customization to include any unique caries findings and conditions but not customization of the algorithm. The portability of VERDICT to other axiUm EHR implementations allows for further disseminating the automated procedures to derive caries indices. The next steps are to disseminate VERDICT to a small number of dental institutions to refine the mapping processes required for these institutions to utilize VERDICT.

Other EHR systems could adapt the structure but would require modifications of the algorithm’s code for their EHR. For other dental EHR software, like Dentrix or Epic Wisdom, which have the same dental procedures, the algorithim could be readily adapted for the caries conditions and findings. Since these software programs do not utilize the standardized dental diagnostic terminology, the caries indices would include fillings due to decay and other reasons, until they incorporate dental diagnostic terms. Most dental EHR systems have sufficient detail to record findings, conditions, planned and performed procedures, which would provide for the capability to determine caries indices. The inclusion of dental diagnostic terms provides more accurate caries indices, as teeth filled due to fracture, trauma, and other non-caries reasons would not be included in the calculation. Currently, calibrated observers do not know the reason for fillings and assume that the restorations are due to decay. The algorithm reported in this paper can be configured with and without dental diagnostic terms. We believe that the most accurate caries indices utilize diagnostic terms where fillings due to non-caries diagnoses are excluded.

Study limitations

This study did not seek to assess reliability of provider EHR data input on oral conditions and diagnoses. An accurate caries index derived by the algorithm depends upon a valid assessment of oral conditions, properly recording findings, and using and inputting diagnostic codes, which is inherent in any caries measurement. Note that in WDG practices, oral conditions are charted using findings, conditions, and diagnostic terminology paired with the planned and completed procedures. Though inaccurate oral condition entries into the EHR can happen chair-side, such charting errors should be similar to those made in all other methods of data recording of oral conditions for clinical or research purposes. It was not the goal of this paper to evaluate the accuracy of clinician observations as compared to standard calibrated observers. Future studies could evaluate accuracy of clinician observations as compared to calibrated observers. Clinician behavior, motivation, incentives, biases, caries diagnosis training and reimbursement are all factors that may influence a clinician. In this study, we assume appropriate, accurate charting, diagnosis, and treatment. An honest mistake would be random error. Intentional “upcoding” would lead to higher caries indices. Similarly, intentional “downcoding” would lead to lower caries indices.

We believe that the EHR-derived caries indices reported here can be disseminated and utilized in large-scale observational studies. WDG EHR data are quite similar to national data; for WDG adults patients aged 35–44 the prevalence of untreated decay (DS>0) is 27.6% at the initial visit, which is very close to national estimate of 27.8% from Healthy People 2020 [U.S. Department of Health and Human Services, 2018].

Conclusion

This study has successfully demonstrated the validity of the designed and programmed automated procedures using EHR data to determine caries indices (VERDICT). Caries indices, important primary outcome measures for research and patient care, can be reliably derived from an EHR. These validated caries indices can be used for research purposes to study caries prevention interventions and examine program evaluations of caries prevention, in health disparities research, and produce longitudinal outcomes of dental caries in patient care.

Acknowledgement

We are grateful for the advice, expertise and dedication of our colleagues within our respective institutions who have donated their time and expertise to support this work, including the UCSF School of Dentistry Computer Informatics & Technology unit, Willamette Dental Group, Skourtes Institute, The University of Texas Science Center at Houston School of Dentistry, and Exan Corporation.

Funding Sources

Research reported in this publication was supported by the National Institute of Dental & Craniofacial Research (NIDCR) of the National Institutes of Health (NIH) under award numbers UH2DE025504, “Evaluating Standardized Preventive Care to Reduce Dental Disparities in Children,” and U01DE025507, “Coordinating Center to Help Eliminate/Reduce Oral Health Inequities in Children.” CARIN was developed and refined with support from award numbers R21DE08650, “PRIME,” and U54DE014251, “CANDO1,” and refined with support from award number U54DE019285, “CANDO2.” The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Manuscript preparation and editorial and publication assistance were supported by an unrestricted gift from the Sue Bloch and Igor Khandros family.

Appendix

The appendix includes lists of the VERDIT and CARIN caries indices output, and VERDICT output and the odontogram and treatment histories for each of the 45 synthetic use case scenarios.

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Appendix Figures:

Open in a new tab

Synthetic Use Case Scenario Odontograms and Treatment Histories (N=45)

Appendix Table 1: Caries Indices and Tooth/Surface Counts (N=36) using VERDICT and using CARIN with SAS (N=24).

VERDICT Output CARIN Output Description
DMFTdmft d2mft_a+ d2mft_c Overall combined tooth score
DMFT d2mft_a Overall permanent tooth score
dmft d2mft_c Overall deciduous tooth score
DMFSdmfs d2mfs_a+ d2mfs_c Overall combined surface score
DMFS d2mfs_a Overall permanent surface score
dmfs d2mfs_c Overall deciduous surface score
DMFS+I+dmfs+i Overall combined surface + incisal edges score
DMFS+I Overall permanent surface + incisal edges score
dmfs+i Overall deciduous surface + incisal edges score
DT d2t_a Decayed permanent tooth score
MT mt_a Missing permanent tooth score
FT ft_a Filled permanent tooth score
dt d2t_c Decayed deciduous tooth score
mt mt_c Missing deciduous tooth score
ft ft_c Filled deciduous tooth score
DS d2s_a Decayed permanent surface score
MS ms_a Missing permanent surface score
FS fs_a Filled permanent surface score
ds d2s_c Decayed deciduous surface score
ms ms_c Missing deciduous surface score
fs fs_c Filled deciduous surface score
DSI Decayed permanent surface + incisal edges score
MSI Missing permanent surface + incisal edges score
FSI Filled permanent surface + incisal edges score
dsi Decayed deciduous surface + incisal edges score
msi Missing deciduous surface + incisal edges score
fsi Filled deciduous surface + incisal edges score
Tt currToothCount_c+
currToothCount a		 Overall tooth count
t currToothCount_c Deciduous tooth count
T currToothCount_a Permanent tooth count
Ss currSurfaceCount_c+
currSurfaceCount a		 Overall surface count
s currSurfaceCou nt_c Deciduous surface count
S currSurfaceCount_a Permanent surface count
Sisi Overall surface + incisal edges count
si Deciduous surface + incisal edges count
Si Permanent surface + incisal edges count
Open in a new tab

Appendix Table 2: Synthetic Use Case Scenarios VERDICT-Calculated and Validated Caries Indices and Tooth and Surface Counts (N=45).

id Age First Last DMFT DT MT FT DMFS DS MS FS dmft dt mt ft dmfs ds ms fs T S t s
D283793 4 ALLdecayed DmftPEDO 0 0 0 0 0 0 0 0 20 20 0 0 88 88 0 0 0 0 20 88
D283796 3 ALLextracted DmftPEDO 0 0 0 0 0 0 0 0 20 0 20 0 88 0 88 0 0 0 0 0
D283791 6 ALLfilled DmftPEDO 0 0 0 0 0 0 0 0 20 0 0 20 88 0 0 88 0 0 20 88
D283795 3 ALLmissing DmftPEDO 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
D283745 5 Combo DmftPEDO 0 0 0 0 0 0 0 0 10 3 1 6 17 3 5 9 0 0 17 75
D283747 5 ComplexCombo DmftPEDO 0 0 0 0 0 0 0 0 11 4 1 6 27 8 5 14 0 0 17 75
D283739 5 DecayedBoth DmftPEDO 0 0 0 0 0 0 0 0 8 8 0 0 13 13 0 0 0 0 20 88
D283737 5 DecayedDX DmftPEDO 0 0 0 0 0 0 0 0 8 8 0 0 19 19 0 0 0 0 20 88
D283734 5 DecayFindings DmftPEDO 0 0 0 0 0 0 0 0 6 6 0 0 6 6 0 0 0 0 20 88
D283741 5 Filled DmftPEDO 0 0 0 0 0 0 0 0 10 0 0 10 14 0 0 14 0 0 20 88
D283771 4 IncisalDecay DmftPEDO 0 0 0 0 0 0 0 0 4 4 0 0 4 4 0 0 0 0 20 88
D283774 4 IncisalFilled DmftPEDO 0 0 0 0 0 0 0 0 3 0 0 3 3 0 0 3 0 0 20 88
D283743 5 Missing DmftPEDO 0 0 0 0 0 0 0 0 4 0 4 0 18 0 18 0 0 0 12 54
D283732 5 Nothing DmftPEDO 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 88
D283892 5 SilverOrderEntryCorrect DmftPEDO 0 0 0 0 0 0 0 0 3 3 0 0 5 5 0 0 0 0 20 88
D283812 10 ALLDecayed DmftMIXED 12 12 0 0 52 52 0 0 12 12 0 0 56 56 0 0 12 52 12 56
D283814 10 ALLextracted DmftMIXED 12 0 12 0 52 0 52 0 12 0 12 0 56 0 56 0 0 0 0 0
D283811 10 ALLFilled DmftMIXED 12 0 0 12 52 0 0 52 12 0 0 12 56 0 0 56 12 52 12 56
D283813 10 ALLMissing DmftMIXED 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
D283752 12 Combo DmftMIXED 5 2 0 3 8 5 0 3 5 1 1 3 11 1 5 5 9 39 10 47
D283754 12 ComboComplex DmftMIXED 9 3 0 6 20 5 0 15 3 1 1 1 9 3 4 2 18 83 7 33
D283755 12 DecayBoth DmftMIXED 3 3 0 0 6 6 0 0 3 3 0 0 5 5 0 0 12 52 12 56
D283753 12 DecayDx DmftMIXED 4 4 0 0 10 10 0 0 4 4 0 0 10 10 0 0 12 52 12 56
D283751 12 DecayFindings DmftMIXED 3 3 0 0 3 3 0 0 3 3 0 0 3 3 0 0 12 52 12 56
D283748 12 Filled DmftMIXED 6 0 0 6 9 0 0 9 4 0 0 4 6 0 0 6 12 52 12 56
D283773 12 IncisalDecay DmftMIXED 3 3 0 0 3 3 0 0 3 3 0 0 3 3 0 0 12 52 12 56
D283775 12 IncisalFillings DmftMIXED 3 0 0 3 3 0 0 3 2 0 0 2 2 0 0 2 12 52 12 56
D283750 12 Missing DmftMIXED 3 0 3 0 14 0 14 0 1 0 1 0 5 0 5 0 5 22 8 37
D283749 12 Nothing DmftMIXED 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 12 52 12 56
D283799 10 UndoMissing DmftMIXED 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 14 62 12 56
D283794 66 ALLdecayed DmftADULT 28 28 0 0 128 128 0 0 0 0 0 0 0 0 0 0 28 128 0 0
D283798 66 ALLextracted DmftADULT 28 0 28 0 128 0 128 0 0 0 0 0 0 0 0 0 0 0 0 0
D283792 66 ALLfilled DmftADULT 28 0 0 28 128 0 0 128 0 0 0 0 0 0 0 0 28 128 0 0
D283797 66 ALLmissing DmftADULT 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
D283891 18 ComboCorrectOrderEntry DmftADULT 7 3 0 4 11 6 0 5 0 0 0 0 0 0 0 0 27 123 0 0
D283746 18 ComplexCombo DmftADULT 8 3 0 5 18 4 0 14 0 0 0 0 0 0 0 0 24 108 0 0
D283738 18 DecayBoth DmftADULT 7 7 0 0 12 12 0 0 0 0 0 0 0 0 0 0 28 128 0 0
D283736 18 DecayDx DmftADULT 8 8 0 0 19 19 0 0 0 0 0 0 0 0 0 0 28 128 0 0
D283735 18 DecayFindings DmftADULT 6 6 0 0 6 6 0 0 0 0 0 0 0 0 0 0 28 128 0 0
D283742 18 Filled DmftADULT 10 0 0 10 14 0 0 14 0 0 0 0 0 0 0 0 28 128 0 0
D283772 66 IncisalDecay DmftADULT 3 3 0 0 3 3 0 0 0 0 0 0 0 0 0 0 28 128 0 0
D283776 66 IncisalFillings DmftADULT 5 0 0 5 5 0 0 5 0 0 0 0 0 0 0 0 28 128 0 0
D283740 18 Missing DmftADULT 4 0 4 0 16 0 16 0 0 0 0 0 0 0 0 0 17 77 0 0
D283733 18 Nothing DmftADULT 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 28 128 0 0
D283815 66 RecurrentandIncisalComplex DmftADULT 15 7 0 8 22 9 0 13 0 0 0 0 0 0 0 0 28 128 0 0
Open in a new tab

Footnotes

Supplementary Material

No Supplementary Material

Statements

Statement of Ethics

The study protocol has been approved by the research institute’s committee on human research (UCSF FWA IRB #00003471, protocol #15–17631), and a waiver of the requirement to obtain a signed consent form from study subjects (or their parents or guardians) was granted as (1) the research involves no more than minimal risk to the subjects; (2) the waiver or alteration will not adversely affect the rights and welfare of the subjects; (3) the research could not practicably be carried out without the waiver or alteration; and (4) whenever appropriate, the subjects will be provided with additional pertinent information after participation.

Disclosure Statement

J.M. Mullins, J.B. Even, T. Guy, and E. Blaga are employees of the Willamette Dental Group (WDG). R. Brandon is a paid consultant to WDG and UCSF, and is a UCSF affiliate. J.M. White was previously a paid consultant for Skourtes Institute (nonprofit foundation) for development and implementation of CAMBRA/PEMBRA/PDCP (clinic decision support and standardized care in the electronic health record, UCSF) and data analytics, and WDG for quality improvement. J.M. White also contributed toward Intellectual Property for DDS (dental diagnostic system, Harvard) and CAMBRA/PEMBRA/PDCP (UCSF). All other authors have no conflicts of interest to declare.

References (Alphabetical)

  1. Barnado A, Casey C, Carroll RJ, Wheless L, Denny JC, Crofford LJ: Developing electronic health record algorithms that accurately identify patients with systemic lupus erythematosus. Arthritis Care Res (Hoboken) 2017;69:687–693. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bratthall D: Introducing the Significant Caries Index together with a proposal for a new global oral health goal for 12-year-olds. Int Dent J 2000;50:378–384. [DOI] [PubMed] [Google Scholar]
  3. Castro VM, Minnier J, Murphy SN, Kohane I, Churchill SE, Gainer V, Cai T, Hoffnagle AG, Dai Y, Block S, Weill SR, Nadal-Vicens M, Pollastri AR, Rosenquist JN, Goryachev S, Ongur D, Sklar P, Perlis RH, Smoller JW: Validation of electronic health record phenotyping of bipolar disorder cases and controls. Am J Psychiatry 2015;172:363–372. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. de Lusignan S, Metsemakers JF, Houwink P, Gunnarsdottir V, van der Lei J: Routinely collected general practice data: goldmines for research? A report of the European Federation for Medical Informatics Primary Care Informatics Working Group (EFMI PCIWG) from MIE2006, Maastricht, The Netherlands. Inform Prim Care 2006;14:203–209. [DOI] [PubMed] [Google Scholar]
  5. Ditmyer M, Dounis G, Mobley C, Schwarz E: Inequalities of caries experience in Nevada youth expressed by DMFT index vs. Significant Caries Index (SiC) over time. BMC Oral Health 2011;11:12. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Gilbert GH, Richman JS, Gordan VV, Rindal DB, Fellows JL, Benjamin PL, Wallace-Dawson M, Williams OD: Lessons learned during the conduct of clinical studies in the dental PBRN. J Dent Educ 2011;75:453–465. [PMC free article] [PubMed] [Google Scholar]
  7. Gruebbel AO: A measurement of dental caries prevalence and treatment service for deciduous teeth. J Dent Res 1944;23:163–168. [Google Scholar]
  8. Gruschow SM, Yerys BE, Power TJ, Durbin DR, Curry AE: Validation of the use of electronic health records for classification of ADHD status. J Atten Disord 2016:1087054716672337. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Kahn MG, Kaplan D, Sokol RJ, DiLaura RP: Configuration challenges: implementing translational research policies in electronic medical records. Acad Med 2007;82:661–669. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Kalenderian E, Ramoni RL, White JM, Schoonheim-Klein ME, Stark PC, Kimmes NS, Zeller GG, Willis GP, Walji MF: The development of a dental diagnostic terminology. J Dent Educ 2011;75:68–76. [PMC free article] [PubMed] [Google Scholar]
  11. Klein H, Palmer CE, Knutson JW: Studies on dental caries: I. Dental status and dental needs of elementary school children. Public Health Rep (1896–1970) 1938;53:751–765. [Google Scholar]
  12. Kristianson KJ, Ljunggren H, Gustafsson LL: Data extraction from a semi-structured electronic medical record system for outpatients: a model to facilitate the access and use of data for quality control and research. Health Informatics J 2009;15:305–319. [DOI] [PubMed] [Google Scholar]
  13. Lingren T, Chen P, Bochenek J, Doshi-Velez F, Manning-Courtney P, Bickel J, Wildenger Welchons L, Reinhold J, Bing N, Ni Y, Barbaresi W, Mentch F, Basford M, Denny J, Vazquez L, Perry C, Namjou B, Qiu H, Connolly J, Abrams D, Holm IA, Cobb BA, Lingren N, Solti I, Hakonarson H, Kohane IS, Harley J, Savova G: Electronic health record based algorithm to identify patients with autism spectrum disorder. PLoS One 2016;11:e0159621. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Liu K, Acharya A, Alai S, Schleyer TK: Using electronic dental record data for research: a data-mapping study. J Dent Res 2013;92:90S–96S. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Mansson J, Nilsson G, Bjorkelund C, Strender LE: Collection and retrieval of structured clinical data from electronic patient records in general practice. A first-phase study to create a health care database for research and quality assessment. Scand J Prim Health Care 2004;22:6–10. [DOI] [PubMed] [Google Scholar]
  16. Marthaler TM: Changes in dental caries 1953–2003. Caries Res 2004;38:173–181. [DOI] [PubMed] [Google Scholar]
  17. Mertz E, Wides C, White J: Clinician attitudes, skills, motivations and experience following the implementation of clinical decision support tools in a large dental practice. J Evid Based Dent Pract 2017;17:1–12. [DOI] [PubMed] [Google Scholar]
  18. Monse B, Heinrich-Weltzien R, Benzian H, Holmgren C, van Palenstein Helderman W: PUFA--an index of clinical consequences of untreated dental caries. Community Dent Oral Epidemiol 2010;38:77–82. [DOI] [PubMed] [Google Scholar]
  19. National Health and Nutrition Examination Survey [Internet]: 2016 NHANES Oral Health Examiners Procedures Manual, Centers for Disease Control and Prevention[cited 2017 Apr 4]. Available from: https://wwwn.cdc.gov/nchs/data/nhanes/2015-2016/manuals/2016OralHealthExaminersProceduresManual.pdf.
  20. Pavlovic I, Kern T, Miklavcic D: Comparison of paper-based and electronic data collection process in clinical trials: costs simulation study. Contemp Clin Trials 2009;30:300–316. [DOI] [PubMed] [Google Scholar]
  21. Regents of the University of California: CARIN: CAries Research INstrument Software Package. Tech ID: 19030 / UC Case 2009–035-0. 2009.
  22. Schleyer T, Song M, Gilbert GH, Rindal DB, Fellows JL, Gordan VV, Funkhouser E: Electronic dental record use and clinical information management patterns among practitioner-investigators in the Dental Practice-Based Research Network. J Am Dent Assoc 2013;144:49–58. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Schleyer T, Spallek H, Hernandez P: A qualitative investigation of the content of dental paper-based and computer-based patient record formats. J Am Med Inform Assoc 2007;14:515–526. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Sharma NK, Pedreira C, Centeno M, Chaudhary UJ, Wehner T, Franca LGS, Yadee T, Murta T, Leite M, Vos SB, Ourselin S, Diehl B, Lemieux L: A novel scheme for the validation of an automated classification method for epileptic spikes by comparison with multiple observers. Clin Neurophysiol 2017;128:1246–1254. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Song M, Liu K, Abromitis R, Schleyer TL: Reusing electronic patient data for dental clinical research: a review of current status. J Dent 2013;41:1148–1163. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Sung NS, Crowley WF Jr., Genel M, Salber P, Sandy L, Sherwood LM, Johnson SB, Catanese V, Tilson H, Getz K, Larson EL, Scheinberg D, Reece EA, Slavkin H, Dobs A, Grebb J, Martinez RA, Korn A, Rimoin D: Central challenges facing the national clinical research enterprise. JAMA 2003;289:1278–1287. [DOI] [PubMed] [Google Scholar]
  27. Terry AL, Chevendra V, Thind A, Stewart M, Marshall JN, Cejic S: Using your electronic medical record for research: a primer for avoiding pitfalls. Fam Pract 2010;27:121–126. [DOI] [PubMed] [Google Scholar]
  28. U.S. Department of Health and Human Services [Internet]: Healthy People 2020: Oral Health of Adults: OH-3.1 Reduce the proportion of adults aged 35 to 44 years with untreated dental decay [cited 2018 Jul 22]. Available from: https://www.healthypeople.gov/2020/topics-objectives/topic/oral-health/objectives.
  29. van Dommelen P, Schuller AA: The amount of care delivered: challenges of indices in oral health studies. Community Dent Oral Epidemiol 2016;44:485–492. [DOI] [PubMed] [Google Scholar]
  30. Walji MF, Kalenderian E, Stark PC, White JM, Kookal KK, Phan D, Tran D, Bernstam EV, Ramoni R: BigMouth: a multi-institutional dental data repository. J Am Med Inform Assoc 2014;21:1136–1140. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Warren JJ, Weber-Gasparoni K, Tinanoff N, Batliner TS, Jue B, Santo W, Garcia RI, Gansky SA, Early Childhood Caries Collaborating C: Examination criteria and calibration procedures for prevention trials of the Early Childhood Caries Collaborating Centers. J Public Health Dent 2015;75:317–326. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. White JM, Kalenderian E, Stark PC, Ramoni RL, Vaderhobli R, Walji MF: Evaluating a dental diagnostic terminology in an electronic health record. J Dent Educ 2011;75:605–615. [PMC free article] [PubMed] [Google Scholar]
  33. World Health Organization: Oral Health Surveys: Basic Methods, ed 5th Geneva, Switzerland, WHO Press, 2013. [Google Scholar]

RESOURCES