Improving Caries Risk Prediction Modeling: A Call for Action

Abstract

Dentistry has entered an era of personalized/precision care in which targeting care to groups, individuals, or even tooth surfaces based on their caries risk has become a reality to address the skewed distribution of the disease. The best approach to determine a patient’s prognosis relies on the development of caries risk prediction models (CRPMs). A desirable model should be derived and validated to appropriately discriminate between patients who will develop disease from those who will not, and it should provide an accurate estimation of the patient’s absolute risk (i.e., calibration). However, evidence suggests there is a need to improve the methodological standards and increase consistency in the way CRPMs are developed and evaluated. In fact, although numerous caries risk assessment tools are available, most are not routinely used in practice or used to influence treatment decisions, and choice is not commonly based on high-quality evidence. Research will propose models that will become more complex, incorporating new factors with high prognostic value (e.g., human genetic markers, microbial biomarkers). Big data and predictive analytic methods will be part of the new approaches for the identification of promising predictors with the ability to monitor patients’ risk in real time. Eventually, the implementation of validated, accurate CRPMs will have to follow a user-centered design respecting the patient-clinician dynamic, with no disruption to the clinical workflow, and needs to operate at low cost. The resulting predictive risk estimate needs to be presented to the patient in an understandable way so that it triggers behavior change and effectively informs health care decision making, to ultimately improve caries outcomes. However, research on these later aspects is largely missing and increasingly needed in dentistry.

Introduction

Dental caries, one of the most common chronic noncommunicable diseases that expands throughout the life span (Kassebaum et al. 2015), is largely avoidable, with effective prevention and management measures available (Slayton et al. 2018). Despite successes in reducing caries in some groups, the distribution of disease remains skewed (Centers for Disease Control and Prevention [CDC] 2019). Targeting risk-based preventive/therapeutic approaches can improve caries-related and patient-centered outcomes while containing costs in an increasingly resource-constrained environment, where obtaining high value in return for health care interventions is paramount.

Multivariable caries risk prediction models (CRPMs), also referred to as prognostic models or prediction rules (Guyatt 2006), allow clinicians to use a combination of specific patient characteristics (predictors) to provide an absolute risk that a specific caries outcome (e.g., presence of new lesions or increasing severity or activity of existing lesions over a specific timeframe) will occur (Fontana and Zero 2006). Clinicians need individual rather than population-based risk estimates to affect clinical care (Alba et al. 2017). Estimating an individual’s risk is essential because, although caries manifests as a site-specific disease, the consequences affect treatment and prognosis at the individual level. The ultimate goal of a CRPM is to improve health outcomes through precise prognosis to cost-effectively target caries interventions and determine the periodicity of services.

This article presents a call to action to improve the scientific basis for CRPMs and discusses approaches to improve the existing body of evidence on the development, validation, and implementation of CRPMs, including use of emerging predictive analytics and big data methods. It also stresses the need for research associated with the impact of CRPMs on behavior change and patient-centered outcomes.

The Concept of Caries Risk Assessment

Because of the multifactorial and chronic nature of the caries disease process, studies on caries risk assessment (CRA) tend to be complex, with multiple factors influencing risk at the individual, family, and community levels throughout the lifetime (Fisher-Owens et al. 2007; Fontana 2015). Despite this complexity, caries experience is still the strongest predictor of individual risk (Mejare et al. 2014). This is less than desirable, considering that the ultimate goal is to prevent the disease. For a clinician, the overall subjective impression of a patient seems to have some predictive power (Disney et al. 1992) but is often omitted in existing tools (Twetman and Fontana 2009).

While there are numerous CRA tools, their choice appears to be based on tradition or “herding behavior” rather than their methodological rigor (Adibi et al. 2020). Most available tools are expert informed with limited formal evaluation and lacking validation (Tellez et al. 2013; Mejare et al. 2014; Cagetti et al. 2018). Despite this, many believe CRA has great potential to enhance patient care (Evans et al. 2016; Jørgensen and Twetman 2019). Methodological flaws in many CRPMs result in underestimating caries risk, resulting in inappropriate reduction of the intensity of preventive measures, leading to suboptimal outcomes, or overestimating the risk, with the implementation of unnecessary interventions, carrying with it the burden of costs and related adverse events.

There is a need to improve methodological standards and increase consistency in how CRPMs are developed and evaluated (Tellez et al. 2013; Mejare et al. 2014; Senneby et al. 2015; Cagetti et al. 2018; Halasa-Rappel et al. 2019). In addition, adequate reporting of CRPM studies, following guidelines such as TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis and Diagnosis) (Collins et al. 2015), can further facilitate assessment of these models. To date, most CRA tools have not been proven useful to inform clinical decision making (Halasa-Rappel et al. 2019).

Etiological versus Predictive Modeling

In the caries-related literature, it is common that etiological and predictive modeling are used interchangeably. Although they share some methodological commonalities (e.g., use of multivariable regression), they are not the same (van Diepen et al. 2017). Failing to understand their differences results in suboptimal methods, misguided interpretation, and poor reporting. Etiological studies aim to identify the relationship of factors causing the disease (i.e., risk factors) to the occurrence of the disease using relative measures (e.g., relative risk), with the underlying assumption that modifying them will improve disease outcomes. Controlling for confounders is essential to prevent obtaining spurious results. On the other hand, prediction modeling aims at prognostication—to determine the probability of a future outcome (e.g., 30% chance of experiencing carious lesion progression within the next year)—based on a set of “predictors.” Although risk factors can be included in prediction models, they are not necessary. For example, previous caries experience is a good caries predictor (Mejare et al. 2014), yet it is not part of the causal pathway of disease. A current challenge is that most CRA tools aim to predict, while also identifying modifiable etiological risk factors.

Predictive Analytics and Big Data

CRPMs are increasingly considering more data to improve accuracy and utility of their models. Predictive analytics and machine learning algorithms are being developed with the expectation of delivering precision care (Psaty et al. 2018). These methods require a large and diverse amount of high-quality data, generally collected for other purposes (e.g., electronic health records, wearable devices, biomarkers, genomics) (Divaris 2016), and linked at the individual level, overcoming technical, ethical, safety, and social challenges. The discussion of the role of big data, including genetic, microbiome, behavioral, and environmental factors, and their impacts on caries risk is ongoing. Some suggest that genetic associations are less relevant for caries risk than environmental factors, even when many environmental exposures may operate, in part, in a genetic context and are likely to vary with age (Silva et al. 2019). Others suggest that although genetic factors may explain approximately 50% of the variation in caries experience, their impact on biological factors may be more relevant. Research is needed to identify potential genetic-environmental risk profiles and how these could be used to better predict caries risk (Haworth et al. 2020).

Greater reliance on massive data sets to obtain clinical insights will require a shift from human to machine decision making. Traditionally, large cohort studies for predictive caries involving thousands of patients (e.g., Fontana et al. 2019) use classic statistical methods and intense human power to relate predictors to outcomes (Beam and Kohane 2018). The use of even larger data sets on the order of tens of millions of data points (e.g., Google AlphaGo Zero) will require the implementation of alternative methods, dramatically increasing the need for machine support (e.g., generative adversarial networks, convolutional neural networks, and random forest) (Appendix Fig. 1). Although not all of these strategies have been applied to disease risk models, some have been used on population health surveys to identify individuals with caries (Zanella-Calzada et al. 2018), with surface-specific lesions (e.g., root caries) (Hung et al. 2019), or to use time-series analyses to predict national prevalence of early childhood caries (Zhang et al. 2017).

Despite the excitement about the possibilities to improve caries prediction through artificial intelligence (AI), experts indicate that dynamic and highly flexible algorithms may not necessarily lead to a better estimation of risk (Van Calster et al. 2019). Recent systematic reviews have questioned the additional benefit of machine learning over logistic regression from traditional clinical prediction models (Christodoulou et al. 2019). Thus, improvements in CRA require the development of validated and evaluated CRPMs, while also assessing cost versus clinical utility derived from efforts to include larger data sets.

Development, Validation, and Implementation of CRPMs

The development of CRPMs should include the following steps (McGinn et al. 2015) (Fig. 1).

Figure 1.

Steps in the development, validation, and impact assessment of a prediction model according to level of evidence. Modified from McGinn et al. 2014.

Model Development

Although well-designed longitudinal cohort studies are traditionally used, natural experiments or mathematical modeling studies are increasingly being considered. During this stage, clinically relevant predictors are defined to determine the data for analysis, the strategy to derive the final model, and the measures of performance (i.e., predictive accuracy). At this stage, CRPMs are not suitable for implementation in practice (Royston et al. 2009).

Validation

This involves first assessment of the validity and usefulness of the CRPM for the underlying population from which the data originated (often referred to as “reproducibility”). If the study sample is large enough, it can be randomly split into 2 parts, one to develop the model and one to evaluate it. If it is not large enough to split, bootstrap resampling and cross-validation methods can be used. The final step is external validation (often referred to as “transportability”) to assess the performance on an independent but plausibly related population (Steyerberg and Vergouwe 2014). In general, except for the Cariogram, CRPMs have rarely been validated in independent populations, diminishing the generalizability of their results (Cagetti et al. 2018). For example, Mejare et al. (2014) identified only 1 study where a CRPM for children had been externally validated, with an impact on its accuracy.

Implementation and Impact Evaluation

After defining an implementation strategy, CRPMs need to be tested in randomized controlled trials or machine learning algorithms to determine the extent to which its use improves patient-centered outcomes, reduces costs, and improves clinical decision making (Moons et al. 2009). Unfortunately, this is largely lacking in CRA research.

Irrespective of the nature of the data (e.g., classic cohort study or big data), implementation of a CRPM requires that the absolute risk provided be displayed in a user-friendly interface that fits seamlessly into the clinical workflow to facilitate clinical decision making (Lipkus and Hollands 1999). For example, scoring systems assign points to each predictor based on their weight in the model. These systems are popular, as they are practical and allow a rapid assessment without the need for electronic devices (Austin et al. 2016). Prediction models can also be presented as web-based calculators (e.g., Cariogram), apps for mobile devices, or embedded in decision aids to support clinical decision making (Steyerberg and Vergouwe 2014). Success may depend on achieving a balance between prediction accuracy and simplicity, while considering implementation science factors, such as the role of champions, training, incentives, plan-do-study-act cycles, and electronic health record prompts. Some of the biggest CRPM implementations challenges are the need to reduce the burden of data collection and achieve interoperability. This will require a paradigm change for our current health information technology ecosystem, moving from primarily supporting documentation and billing of patient encounters to supporting clinical care.

Presenting personalized risk information does not lead, on its own, to changes in health behaviors (French et al. 2017). Behavioral research also needs to determine how we better translate precise caries risk predictions into effective messaging and communication to facilitate sustained patient behavior change, resulting in improved caries outcomes.

Evaluation of CRPMs

CRPMs must evaluate their predictive accuracy (Kalhan et al. 2020). An ideal CRPM will correctly discriminate those who will develop a new carious lesion from those who will not 100% of the time. As predictive models never reach that level of performance, 2 fundamental properties need to be considered and reported in CRPM studies: discrimination and calibration.

Discrimination refers to the extent to which the model differentiates between individuals at higher and lower caries risk (Alba et al. 2017). This is commonly measured with a concordance (c) statistic, which for a binary endpoint (at risk vs. no caries risk) is characterized using the receiver operating characteristic (ROC) curve (Royston et al. 2009). The greater the area under the ROC curve (i.e., the less the model is compromised in false positives when sensitivity is increased), the better is the discriminative property of the model (Fig. 2). Other proposed approaches to evaluate meaningful thresholds include the net benefit or decision curve analysis (Vickers and Elkin 2006) and the cost-effectiveness of decisions associated with different risk thresholds.

Figure 2.

Measurement of discrimination of clinical prediction models. Example of receiver operating characteristic (ROC) curve or “C statistic” and area under the curve (A–C). Measuring calibration of clinical prediction models: curves representing observed versus predicted values for a hypothetical caries risk score (D). Adapted from Alba et al. 2017.

CRPMs reflect how well they perform on a group of individuals (e.g., sensitivity, specificity). Positive and negative predictive values are often useful to evaluate this issue, as these depend on the prevalence of caries and vary from population to population (e.g., even an excellent CRPM will yield many more false than true positives if used in a population with low caries prevalence). Models often fail at the individual level if the individual is not represented (e.g., age, race, gender) in the study population from which the model was derived. Obtaining accurate absolute caries risk estimates is essential when determining the impact or effect of a caries intervention (relative effect) on a patient’s risk for experiencing the caries outcome of interest (Appendix Fig. 2).

Although discrimination is essential, it is not sufficient to evaluate the performance of a predictive model. In fact, a model can show a high area under the ROC curve (e.g., 0.7) and still provide incorrect absolute risks. For example, a model can correctly distinguish between 2 patients and predict an absolute risk for having a new carious lesion of 2% for one and 7% for the other, while in reality the real absolute risks were 20% and 70%. In this case, although the model discriminates, it is poorly calibrated.

Calibration or goodness of fit entails determining how similar the predicted absolute risk is to the observed risk in the population (Alba et al. 2017). Often, calibration is investigated by plotting the observed proportion of events against the predicted absolute risks (Fig. 2), assessing the intercept (“calibration-in-the-large”) or slope (“calibration slope”) of the calibration plot, or using the Hosmer-Lemeshow statistical test (Steyerberg et al. 2010).

Conclusions

In health care, improving prognosis is the ultimate goal of preventive and therapeutic decision making. However, the literature is filled with caries risk prediction studies that offer models that are rarely used, full of questionable predictors, and/or inconsistently measured and reported. Useful CRPMs must be able to “discriminate” between patients at higher and lower risk of developing caries lesions to allow clinicians and patients to make informed decisions about how to prevent the onset or improve the outcomes of existing disease. Selecting a clinically relevant threshold includes finding an optimal balance between benefits and harms of decisions associated with predicted risk, based on the accuracy of the model and costs. Future efforts should focus on improving and standardizing methods and reporting of prediction modeling in caries research, conducting implementation research to better incorporate validated models in practice and measure their utility, and considering new sources of predictors (e.g., omics, biomarkers, patient diaries). The implementation of a holistic approach to interoperability will permit the use of more sophisticated data mining and analytical methods. Ultimately, the use of validated CRPMs, driven by AI or not, within a supportive culture of routine data collection to improve health outcomes and a nurturing incentive structure (Emanuel and Wachter 2019) should facilitate a shift in caries management toward prevention and early intervention (i.e., a move from disease management to health management). The drive toward learning health systems (Institute of Medicine 2013) supporting the use of all data routinely collected during the care process of all patients may accelerate these developments. We not only need further development and more rigorous validation of CRPMs to accurately determine a future adverse caries-related event but also need to conduct research to assess how caries risk can be best communicated to affect behavior change, influence treatment decisions, and improve caries outcomes.

Author Contributions

M. Fontana, A. Carrasco-Labra, G. Eckert, B. Katz, contributed to conception, design, data acquisition, and interpretation, drafted and critically revised the manuscript; H. Spallek, contributed to design and data interpretation, drafted and critically revised the manuscript. All authors gave final approval and agree to be accountable for all aspects of the work.