Implementing a Preeclampsia Prediction Model in Obstetrics: Cutoff Determination and Health Care Professionals’ Adherence

Abstract

Background. Despite improved management, preeclampsia remains an important cause of maternal and neonatal mortality and morbidity. Low-dose aspirin (LDA) lowers the risk of preeclampsia. Although several guidelines recommend LDA prophylaxis in women at increased risk, they disagree about the definition of high risk. Recently, an externally validated prediction model for preeclampsia was implemented in a Dutch region combined with risk-based obstetric care paths. Objectives. To demonstrate the selection of a risk threshold and to evaluate the adherence of obstetric health care professionals to the prediction tool. Study Design. Using a survey (n = 136) and structured meetings among health care professionals, possible cutoff values at which LDA should be discussed were proposed. The prediction model, with chosen cutoff and corresponding risk-based care paths, was embedded in an online tool. Subsequently, a prospective multicenter cohort study (n = 850) was performed to analyze the adherence of health care professionals. Patient questionnaires, linked to the individual risk profiles calculated by the online tool, were used to evaluate adherence. Results. Health care professionals agreed upon employing a tool with a high detection rate (cutoff: 3.0%; sensitivity 75%, specificity 64%) followed by shared decision between patients and health care professionals on LDA prophylaxis. Of the 850 enrolled women, 364 women had an increased risk of preeclampsia. LDA was discussed with 273 of these women, resulting in an 81% adherence rate. Conclusion. Consensus regarding a suitable risk cutoff threshold was reached. The adherence to this recommendation was 81%, indicating adequate implementation.

Preeclampsia (PE) is an important cause of mortality and morbidity for both the mother and the fetus. Although management of PE has improved, a cure that would preserve the pregnancy remains unavailable. Therefore, preventive measures play a pivotal role in decreasing the burden of the disease.¹

In addition to an adequate calcium intake, diet, and lifestyle interventions, aspirin treatment receives an increasing amount of attention as a preventive measure.^1,2 Low-dose aspirin (LDA) prophylaxis, in a dosage of 80 to 150 mg daily, has been proven to reduce the risk of preeclampsia.³ Therefore, several professional authorities such as the American College of Obstetricians and Gynecologists (ACOG), the US Preventive Services Task Force, and the National Institute for Health and Clinical Excellence (NICE) recommend LDA prophylaxis in women at increased risk of PE.^4–6

These authorities all recommend LDA to women at increased risk by using a list of separate risk factors (e.g., a history of PE, or chronic hypertension). They however differ in their selection of risk factors and thus their definition of women at increased risk. Universal recommendation for all pregnant women has been proposed as well since LDA is inexpensive, widely available, and appears to be safe in pregnancy beyond the first trimester.⁷ However, this view is controversial due to a lack of understanding in the preventive mechanism of LDA, and a lack of proven benefits for women at low risk.⁷

Multivariable prediction models estimating the risk of PE weigh several risk factors simultaneously and can assist health care professionals in identifying women with increased risk. The results of a recent study comparing several PE prediction models simultaneously in 1 cohort⁸ indicated that some of these models are more efficient compared to a list of single risk factors. For a prediction model to serve as a decision tool, a cutoff has to be determined for the discrimination of low and increased risk.

Recently, the recommendation of LDA prophylaxis was adopted in the regional guidelines in the southeastern part of the Netherlands.⁹ Women with an elevated PE risk are identified using a prediction model. However, dissemination of guidelines or stating recommendations does not automatically result in adherence by health care professionals. Implementation of effective preventive interventions often suffers from low adherence rates.^10–12 Despite the increased attention of the role of LDA in the prevention of preeclampsia, a recent conference report showed that up to 42% of women considered high risk according to the NICE guidelines had not been offered LDA.¹³

This article reports on 1) the selection of a cutoff value by health care professionals for the identification of women at risk of PE using a prediction model and 2) results of health care professionals’ adherence to LDA recommendations in the local guidelines.

Methods

Definition of Women at Risk of Preeclampsia

The Limburg Obstetric Consortium (LOC) is a committee representing all obstetric health care professionals in the southeastern part of the Netherlands, which consists of 5 regions. Every region consists of a hospital providing secondary obstetric care (gynecologists and clinical midwives) and outpatient midwifery practices (autonomous midwives providing primary obstetric care). Each region provides 2 to 4 obstetric health care professionals as LOC representatives. In total, the LOC consists of independent midwives (n = 11), gynecologists (n = 10), maternity care nurses (n = 2), and researchers (n = 3).

The LOC developed risk-based care pathways that were implemented in 2017. These pathways consist of basic antenatal care for the low-risk group and additional recommendations for women at risk for several pregnancy-related complications, including PE. The methods of formulating these pathways and their content are reported elsewhere.^14,15

For women at risk of PE, additional risk-based care includes the recommendation of LDA prophylaxis (80–100 mg) from 12 to 36 weeks of pregnancy. The LOC agreed to use the prediction model of Syngelaki et al.,¹⁶ externally validated and recalibrated for their specific region by Meertens et al.⁸ This model was selected because it was the model with the highest discriminative performance, and its predictors are routinely collected in Dutch obstetric practice. Predictors included in the prediction model are age, body mass index, ethnicity, parity, assisted conception treatment, smoking during pregnancy, family history of PE, and medical history (regarding chronic hypertension, PE, and diabetes mellitus). The algorithm of the calibrated model, along with its discriminative performance, is provided in Supplementary File 1.

Consensus regarding the PE risk-threshold, the cutoff value at which health care professionals discuss the recommendation of LDA, was reached using a 3-step procedure. First, all obstetric health care professionals of the LOC region received a survey with statements regarding the implementation of a PE prediction model and possible risk-thresholds. Second, using the results of the survey, the preferences of health care professionals were discussed in regional meetings with the midwives and gynecologists of the region. Third, the results of both the survey and the regional meetings were discussed with the LOC committee. During a final meeting, the decision was made whether the prediction model should be adopted and which risk-threshold was preferred.

In the survey, 3 possible risk-thresholds were suggested: 1) a threshold with a high sensitivity and low specificity similar to the specificity of the ACOG guideline⁴ (risk-threshold 2.85%, sensitivity 79%, specificity 60%), 2) a threshold resulting in a relative risk of 2.0 upon PE for positive results (risk-threshold 3.90%, sensitivity 57%, specificity 80%), and 3) a threshold with a low sensitivity and high specificity similar to the specificity of the NICE guideline⁶ (risk-threshold 5.20%, sensitivity 30%, specificity 90%). Each suggested threshold was provided with additional information: sensitivity, specificity, and total number of test positives, test negatives, true positives, false positives, true negatives, false negatives, and numbers needed to treat. Data of the external validation study were used to calculate these test characteristics per risk-threshold.⁸

Health care professionals were asked to answer the statement “I agree using this cutoff value as threshold determining an elevated PE risk,” using a 10-point Likert scale (1 = totally disagree to 10 = fully agree).

The PE prediction model with corresponding threshold was embedded in the Expect prediction tool, which is available online for health care professionals. The LOC strongly encourages midwives and gynecologists to use the Expect prediction tool during the first antenatal visits. This was achieved by oral presentations, e-mails, regular phone calls, and in-person evaluations.¹⁴

Data Collection of Pregnant Women

When consensus regarding the threshold was reached, the Expect prediction tool was implemented. Participants, pregnant women, were enrolled in a multicenter prospective cohort study in the southeastern part of the Netherlands from April 2017 to August 2018 (Expect Study II). A more detailed description of the study design has been published elsewhere.¹⁴ Briefly, women were recruited at their first prenatal visit (<16 weeks of pregnancy) if their health care professional used the Expect prediction tool. In Dutch obstetric care, pregnant women visit either an autonomous midwife (outpatient clinic) or a gynecologist (hospital); both midwives and gynecologists recruited women for the Expect Study.

Women of at least 18 years with a singleton pregnancy were eligible for inclusion. Questionnaires and study information were provided in Dutch only. Eligible women were asked whether they agreed to provide their e-mail address to receive information regarding the Expect Study. When women agreed to participate and completed an online informed consent form, they received a personal link by e-mail to the web-based surveys. The first survey, collecting the data used for the analyses in this study, was disseminated at enrollment. Two automatic reminders were sent using 3-day intervals. Women were contacted by phone if no response was received. The survey embedded questions regarding the health care services women received from their midwife or gynecologist during the first visits. Women were specifically questioned whether their PE risk was discussed with them (yes, I have an increased risk; yes, I have an average risk; yes, I have a low risk; no, it was not discussed; I do not recall whether this was discussed). Furthermore, women were asked whether the option of LDA was discussed with them (yes/no/I do not recall).

Statistical Analysis

We cross-tabulated the proportions of women who reported discussing their PE risk and the option of LDA with respect to the predicted PE risk (low risk/increased risk). Furthermore, we plotted these proportions with respect to the predicted PE risk by categorizing PE risk predictions (≤1.0% to >6.0% using a binwidth of 0.5 percentage points). To analyze possible differences in health care professionals’ adherence rates to the risk-based recommendations, we plotted LDA discussion rates reported by women using the study duration as a continuous variable. A nonparametric local weighted regression (loess regression) was applied to fit the curves. We analyzed the correlation between the discussion rates and the predicted PE risk for women with a risk exceeding 3.0% by use of logistic regression with predicted PE risk as an independent variable (continuous, percentage).

All statistical analyses were performed using R statistical software version 3.6.0 along with the packages “foreign,”“dplyr,”“tidyr,”, and “ggplot2.”¹⁷

Ethical Approval and Funding

The Medical Ethical Committee of the Maastricht University Medical Centre evaluated the study protocol and declared that the Expect Study does not fall under the Medical Research Involving Human Subjects Act (METC-17-4-057). All participants gave informed consent.

Financial support for this study was provided entirely by a grant from ZonMw (The Netherlands Organization for Health Research and Development; federal funding). The funding agreement ensured the authors’ independence in designing the study, interpreting the data, writing, and publishing the report.

Results

The survey regarding the risk-threshold preference was sent to 136 health care professionals (53 midwives, 32 gynecologists, and 51 residents). In total, 43 (32%) health care professionals completed the questions regarding the PE risk-threshold. Response rates per type of health care professional were similar: midwives, 30% (n = 16); gynecologists, 31% (n = 10); and residents, 33% (n = 17). The boxplots, displayed in Figure 1, indicate that none of the risk-thresholds were clearly rejected but that there was no evident preference for a certain risk-threshold either.

Figure 1.

Boxplots of preferences of health care professionals for given risk-thresholds. Likert scale: 1 = totally disagree to 10 = fully agree. ACOG, American College of Obstetricians and Gynecologists; NICE, National Institute for Health and Clinical Excellence; RR, relative risk; Se, sensitivity; Sp, specificity.

During the regional discussions, health care professionals unanimously stressed that they preferred a prediction tool suitable for shared decision making. In their opinion, in the case of predicted risk exceeding the chosen cutoff value, the first step should be discussing the LDA recommendation with the pregnant woman. Furthermore, the prediction tool should provide relevant information and insight for both health care professionals and pregnant women regarding the predicted risk. When these conditions are met, using the prediction tool as a first step to start the discussion regarding LDA prophylaxis, a threshold with a high sensitivity (high detection rate) is preferred over one with a high specificity (low false-positive rate). However, regardless of the detection rate, specificity should be kept at an acceptable level.

The majority preferred either a threshold of 2.85% or 3.90%. At the same time, health care professionals strongly in favor of a 5.20% threshold did not agree with 2.85%. It was felt that the number of test positives should not exceed roughly a third of the population. On the other hand, health care professionals in favor of the 2.85% threshold stressed that at least everyone with an increased risk should be counseled. The observed incidence rate in the external validation study was 2.9%.⁸ Thus, it was decided that every woman with a PE risk above the population average should be informed regarding the option of LDA.

During the final LOC meeting, taking all considerations into account, it was decided that a threshold should be employed and that LDA treatment was to be discussed with the pregnant woman if estimated PE risk was greater than 3.0%. In the external validation study, this threshold corresponded with a sensitivity and specificity of 75% and 64%, respectively.⁸ To facilitate the shared decisional approach, the results of the prediction were visualized at a linear scale and provided together with relevant patient brochures.

Of the 866 women who provided informed consent, 850 (98%) completed the questionnaire at enrollment. Table 1 shows the characteristics of the women at enrollment of the Expect Study II, a flowchart of study enrollment is provided in Figure 2, and Supplemental Figure S1 shows the distribution of predicted PE risks of this population. Table 2 shows the results of the answers regarding whether PE risk prediction and LDA treatment were discussed during the prenatal visits. A total of 522 women (61%) stated that the results of their estimated PE risk were discussed during the antenatal visits. Estimated risks were not discussed with 265 women (31%), and 63 women (7%) could not recall whether they were discussed.

Table 1.

Baseline Characteristics of the Expect II Study Cohort^a

Characteristics	Expect II Cohort (n = 850)
Age, y	30.7 ± 4.0
University, or higher vocational education, n (%)	500 (58.8)
Body mass index, kg/m2	24.8 ± 4.8
Smoking during pregnancy, n (%)	38 (4.5)
History of chronic hypertension, n (%)	17 (2.0)
Family history of preeclampsia (biological mother), n (%)	42 (4.9)
Nulliparous, n (%)	415 (48.8)
Spontaneous conception, n (%)	772 (90.8)
History of preeclampsia, n (%)	50 (5.9)
Estimated preeclampsia risk percentage, median (interquartile range)	2.7 (1.1–4.3)
Estimated preeclampsia risk >3.0%, n (%)	364 (42.8)

^aData are expressed as mean ± standard deviation, median (interquartile range), or n (%).

Figure 2.

Flowchart participant enrollment for Expect Study II.

Table 2.

Reported Rates of Discussing Preeclampsia Risk and Low-Dose Aspirin Prophylaxis

Characteristic	Low Preeclampsia Risk, n (%)	Increased Preeclampsia Risk, n (%)	All Women, n (%)
Total	486 (100.0)	364 (100.0)	850 (100.0)
Preeclampsia risk discussed
Yes	249 (51.2)	273 (75.0)	522 (61.4)
No	199 (40.9)	66 (18.1)	265 (31.2)
Uncertain	38 (7.8)	25 (6.9)	63 (7.4)
Low-dose aspirin discussed
Yes	71 (14.6)	294 (80.8)	365 (42.9)
No	400 (82.3)	63 (17.3)	463 (54.4)
Uncertain	15 (3.1)	7 (1.9)	22 (2.6)

An estimated risk exceeding 3.0% was adopted as threshold for discussing LDA. In this subgroup of 364 women with an increased risk, PE risk and LDA prophylaxis were discussed with 273 (75%) and 294 (81%) women, respectively. Figure 3 shows the percentages of women who stated their health care professionals discussed the PE risk and LDA prophylaxis per risk category. This graph indicates a positive correlation between the predicted PE risk and discussion rates of both PE risk and LDA by health care professionals. For women identified with a risk exceeding 3.0%, predicted PE risk was a strong positive determinant of discussing PE risk (odds ratio per percent increase, 1.34; 95% confidence interval [CI], 1.18–1.56; P < 0.01) and of discussing LDA prophylaxis (odds ratio per percent increase, 1.28; 95% CI, 1.18–1.40; P < 0.01). Thus, health care professionals are significantly more likely to discuss both the predicted PE risk and LDA recommendation at increased PE risk estimates.

Figure 3.

Adherence rates of discussing preeclampsia risk and low-dose aspirin prophylaxis per estimated risk category.

Figure 4 shows health care professionals’ adherence rate throughout the study period. Therefore, we plotted LDA discussion rates reported by women using the study duration as a continuous variable. At the start of our implementation study, adherence rates ranged from 45% to 65% but eventually rose to approximately 85% and remained constant throughout the study period.

Figure 4.

Adherence rates of discussing low-dose aspirin prophylaxis during the study period.

Discussion

Although there is an enormous rise in models being published and an increasing amount is externally validated, only a few studies report the implementation of a prediction model.^18,19 To our knowledge, also reported by Kleinrouweler et al.,²⁰ this is the first study to describe the implementation and usage of a prediction model predicting absolute risks for preventive strategies in daily obstetric practice.

Strengths and Limitations

Before a prediction model can be used as a basis for clinical decision making, ideally, thresholds should be selected that indicate which risks are considered an increased risk.^21,22 Although the publication of prediction models increases rapidly, the amount of models applied in daily practice is still limited. As a result, most health care professionals may not be used to interpreting risk estimates. This may explain the low response rate and the lack of consensus in the survey regarding the threshold selection.

In this study, a 3-step process was used to select suitable risk-thresholds. Reilly et al.²¹ report the feelings of diminished autonomy by the health care professional as one of the potential barriers when applying a decision rule. In the final LOC meeting, the shared decisional approach was strongly stressed, which may have diminished this potential barrier.

A strength of our study is its prospective multicenter design. Particularly in the Netherlands, recruitment in multiple centers is essential because most pregnant women receive antenatal care at outpatient midwifery clinics.²³ Furthermore, by using our prediction tool as an inclusion method, we were able to link the received health care services to the estimated risk profiles of pregnant women.

The Expect Study II focused on analyzing the impact and results of risk-based care. As a result, only women for whom the prediction tool was used were eligible for inclusion. Usage of our prediction tool as an inclusion method enabled us to link the questionnaires completed by women to their individual PE risk prediction. The prediction tool was developed for usage in the general population and was promoted as such.¹⁴ Furthermore, all obstetric health care professionals of our region committed themselves to use the prediction tool. Nevertheless, this may have introduced some selection, since proactive health care professionals may be overrepresented among the professionals who use our prediction tool. The intensive usage of the prediction tool throughout the region and the multitude of collaborating centers diminish the amount of selection.

Recommendation of LDA treatment should preferably be based on the PE risk prediction by using a shared decision making approach. However, for most risk categories, more women reported that they discussed LDA prophylaxis than they discussed their PE risk. Thus, either their PE risk was not discussed or they did not recall the primary reason for discussing LDA prophylaxis. Our data do not allow analyzing possible reasons for this discrepancy. One possibility could be differences in women’s ability to recall both topics since aspirin is an easy, well-known word among nonprofessionals, whereas preeclampsia is not. This hypothesis may be supported by the fact that the proportion of women not recalling whether their PE risk was discussed (7.4%) is greater than the proportion of women not recalling whether aspirin was discussed (2.6%).

Interpretation

Discussion of LDA treatment was reported by 81% of women with an elevated PE risk. Compared to previous studies in obstetrics regarding protocol and guideline adherence, this percentage is relatively high.^10,12,13 In addition, a significant correlation was found between discussing LDA prophylaxis and the predicted PE risk. LDA prophylaxis was discussed more frequently with women having higher PE risk estimates; these women potentially have the highest individual benefit from LDA treatment.

As can be observed in Figure 4, the adherence rates tended to increase during the study period. At the start of the implementation of our prediction tool along with the selected threshold, LDA recommendation was at best mediocre and comparable to adherence rates previously reported.¹³ However, roughly after 9 months of implementation, adherence rates rose to 85% and remained consistent during the study period.

Recent research has emphasized the potential benefit of LDA treatment in women at high risk of PE. The ASPRE trial, a randomized clinical trial regarding the effect of LDA treatment in preventing preeclampsia, used a prediction model as well to identify the high-risk group.¹⁹ Compared to the model used by the LOC, the ASPRE model has a similar sensitivity but outperforms in specificity. However, the ASPRE model does not solely rely on routinely available predictors and uses biochemical markers as well as the uterine-artery pulsatility index. The addition of these predictors mainly reduces the false-positive rate.⁸ However, LDA prophylaxis from 12 weeks of gestation is inexpensive and does not result in adverse fetal effects, which reduces the disadvantages of a high false-positive rate. As a result, it is arguable whether the costs associated with these additional predictors are proportional to their benefits.²⁴

Currently, there is no consensus about the best screening method for identifying women at risk of PE. The advantage of a prediction model over a list of risk factors is that it provides both the health care professional and the pregnant women with the insight of the absolute risk. Moreover, prediction models weigh several risk factors and their possible interrelations simultaneously, allowing for a more personalized estimation of the absolute risk.²⁵ This information enables health care professionals to use a shared decisional approach. As a result, pregnant women have the opportunity to participate actively in the choices of additional health care services aimed at the prevention of PE.

Future research should focus on barriers that hamper the usage of a risk prediction tool by health care professionals. Moreover, reasons of nonadherence regarding recommendations provided by the prediction tool should be addressed. In addition, more insight is needed about the shared decisional approach regarding the choice of LDA prophylaxis. The contradictory results between reporting rates about whether PE risk and LDA prophylaxis were discussed (Figure 3) suggest that a substantial group of women may not correctly recall or understand the reasons of LDA prophylaxis. In that case, these women are unlikely to be able to make an informed choice.

Conclusion

Consensus regarding a suitable risk cutoff threshold to identify women at risk of PE was reached. Health care professionals agreed on employing a tool with a high detection rate (cutoff: 3.0%, sensitivity 75%, specificity 64%), followed by a shared decision between pregnant women and health care professionals on LDA prophylaxis. The adherence to this recommendation was 81%, indicating adequate implementation.