A Comparison of Prediction Models for Fractures in Older Women: Is More Better

Abstract

Background

A web-based risk assessment tool (FRAX®) using clinical risk factors with and without femoral neck bone mineral density (BMD) has been incorporated into clinical guidelines regarding treatment to prevent fractures. Our objective is to determine whether prediction with FRAX® models is superior to that based on parsimonious models.

Methods

We conducted a prospective cohort study in 6252 women aged ≥65 years and compared the value of FRAX® models that include BMD to parsimonious models based on age and BMD alone for prediction of fractures. We also compared FRAX® models without BMD to simple models based on age and fracture history alone. Fractures (hip, major osteoporotic [hip, clinical vertebral, wrist, or humerus], and any clinical fracture) were ascertained during 10 years of follow-up. Area under the curve (AUC) statistics from receiver operating characteristic (ROC) curve analysis were compared between FRAX® models and simple models.

Results

AUC comparisons revealed no differences between FRAX® models with BMD versus simple models with age and BMD alone in discriminating hip (AUC=0.75 for FRAX® model and 0.76 for simple model, p=0.26); major osteoporotic (AUC=0.68 for FRAX® model and 0.69 for simple model, p=0.51); or clinical fracture (AUC=0.64 for FRAX® model and 0.63 for simple model, p=0.16). Similarly, performance of parsimonious models containing age and fracture history alone was nearly identical to that of FRAX® models without BMD. The proportion of women in each quartile of predicted risk who actually experienced a fracture outcome did not differ between FRAX® and simple models (p≥0.16).

Conclusions

Simple models based on age and BMD alone or age and fracture history alone predicted 10-year risk of hip, major osteoporotic, and clinical fracture as well as more complex FRAX® models.

INTRODUCTION

Advanced age and low bone mineral density (BMD) are strongly associated with higher fracture risk in older women.¹ Data from randomized trials support the recommendation to initiate pharmacologic therapy to lower fracture risk among older women with osteoporosis as defined by BMD T scores of −2.5 or less (at least 2.5 SDs below average for a healthy young women). However, despite a lower risk of fracture, more fractures occur in the much larger group of older women without osteoporosis (BMD T-scores above −2.5) in whom there is much less certainty about the efficacy of drug therapy.

In addition to low BMD and older age, many other independent risk factors for fractures, in particular hip fracture, have been identified.^2;3 Thus, recent efforts by the World Health Organization Metabolic Bone Disease Group have focused on developing a risk assessment tool (FRAX®) using clinical risk factors with and without femoral neck BMD to enhance fracture prediction.⁴ To develop the Web-based FRAX® tool, Kanis and colleagues used data from nine epidemiologic cohorts on baseline BMD and common clinical risk factors easily determined by primary care clinicians that were identified from previous meta-analyses.^5-13 The performance characteristics of the FRAX® tool were then validated in eleven independent population based cohorts.¹⁴ The FRAX® algorithm¹⁵ is country-specific and uses clinical risk factor data, with or without consideration of femoral neck BMD measurement, to calculate an individual patient's 10-year probability of hip fracture and 10-year probability of major osteoporotic (hip, clinical vertebral, wrist, humerus) fracture.

The development of the FRAX® tool has been supported by organizations, including the International Osteoporosis Foundation (IOF) and the National Osteoporosis Foundation (NOF) in the United States, who have strongly advocated its use in clinical decision making. Based on the results of a U.S.-specific cost-effective analyses¹⁶, NOF recently modified its treatment guidelines¹⁷ to recommend pharmacologic therapy for adults aged 50 years and older meeting specific criteria, including those based on the presence of osteopenia (BMD T score between −1 to −2.5) and level of 10-year absolute probability of hip (3% or higher) or major osteoporotic (20% or higher) fractures as calculated by the FRAX® tool. As noted by NOF¹⁸, widespread adoption of these new guidelines would shift treatment approach from one based primarily on BMD measurement to a new approach based on absolute risk of fracture. This shift in approach may have major implications on the proportion of individuals who are recommended pharmacologic treatment to lower their risk of fracture.¹⁹ For example, it has been estimated that application of the new NOF guidelines would result in recommending pharmacologic treatment to a very large proportion of older white women in the United States (at least 72% of those ≥65 years and 93% of those ≥75 years).²⁰

Despite recommendations to incorporate fracture probabilities calculated by the FRAX® model into clinical decision making, it is uncertain whether fracture prediction with FRAX® models is superior to that based on parsimonious models. We utilized data collected in the Study of Osteoporotic Fractures to compare the value of FRAX® models with BMD to simple models based on age and BMD alone for prediction of fractures in older women. In addition, since BMD is not available in all practice settings, we performed a secondary analysis in which we compared FRAX® models without BMD to simple models based on age and fracture history alone.

METHODS

Participants

From 1986 to 1988, a total of 9704 women at least 65 years old were recruited for participation in the baseline examination of the prospective Study of Osteoporotic Fractures (SOF). Women were recruited from population-based listings in four areas of the United States.²¹ Black women were originally excluded from SOF because of their low incidence of hip fracture. In addition, women were excluded if they were unable to walk without assistance or had a history of bilateral hip replacement.

All surviving participants were invited to participate in a 2^nd examination between 1989 and 1990 that included measurement of femoral neck BMD. A total of 7963 (84% of survivors) had technically adequate femoral neck BMD measurements. Of these, 6252 women provided data for all FRAX® clinical risk factors and femoral neck BMD and are the subject of this analysis. The Institutional Review Board (IRB) at each center approved the study protocol, and written informed consent was obtained from all participants.

Clinical Risk Factors and Bone Mineral Density

Participants completed a questionnaire and were interviewed at the baseline examination and asked about race/ethnicity, prior history of fracture since the age of fifty years, physician diagnosis of rheumatoid arthritis, parental history of hip fracture, smoking status, alcohol intake, and use of oral glucocorticoids. Measurements of body weight and height were used to calculate a standard body mass index (BMI). BMD of the proximal femur including the femoral neck subregion was measured using dual energy x-ray absorptiometry (QDR 1000, Hologic, Waltham, MA). Details of the measurement methods and densitometry quality control methods have been published elsewhere.²²

FRAX® Tool

This analysis used the FRAX® tool²³ (October 2008 version) developed for Caucasians in the USA that included the following components: age, sex, fracture history, use of oral glucocorticoids, presence of rheumatoid arthritis, presence of specific conditions associated with secondary osteoporosis (type I diabetes mellitus, osteogenesis imperfecta in adults, untreated long-standing hyperthyroidism, premature menopause, chronic malnutrition or malabsorption, and chronic liver disease), parental history of hip fracture, smoking status (current smoker vs. nonsmoker), alcohol intake (≥3 drinks/day vs. <3 drinks/day), and body mass index. The algorithm calculated four fracture probabilities for each individual participant: the 10-year probability of hip fracture calculated with and without femoral neck BMD data and the 10-year probability of major osteoporotic fracture (defined as hip, clinical vertebral, wrist or humerus fracture) calculated with and without femoral neck BMD data.

Ascertainment of Fractures

After the baseline examination, we contacted (by postcard or telephone) participants every 4 months about fractures during the subsequent 10-year follow-up period. A 10-year time point was selected to match that specified by the FRAX® tool. Over 98% of these follow-up contacts were completed. Fractures were confirmed by review of radiographic reports. Incident fracture outcomes for this analysis included hip fracture, major osteoporotic fracture (defined as hip, clinical vertebral, wrist or humerus fracture), and any clinical fracture (nonvertebral and clinical vertebral fractures).

Statistical Analysis

We used logistic regression to examine receiver operating characteristic (ROC) curves for each model across a range of sensitivities and specificities. The model's ability to discriminate between women who did sustain a fracture during the 10-year follow-up period from those who did not was assessed by the area under the curve (AUC) statistic. Models with AUC statistics of 0.50 do no better than chance alone, while models with substantially higher AUC statistics do much better than chance. We compared AUC statistics between FRAX® models and simple models using the ‘roccomp’ procedure in STATA that uses a nonparametric approach.²⁴ In addition, we also used the 10-fold cross validation procedure²⁵ to generate AUC statistics for FRAX® models and simple models for each of the three fracture outcomes. The cross-validated AUCs were nearly identical in magnitude than the observed AUCs and findings regarding the comparison of the ability of the FRAX® models vs. simple models for prediction of each of these fracture outcomes were unchanged (results not shown).

The clinical usefulness of a model is greater as the proportion of individuals at high risk who actually experience the outcome increases, and as the proportion of individuals at low risk who actually experience the outcome decreases. Thus, we also compared across models the proportion of women in each quartile of predicted risk who actually experienced a fracture outcome. To use a more stringent criteria for identification of high and low risk groups, we performed a secondary analysis comparing across models the proportion of women in each decile of predicted risk who actually experienced a fracture outcome. Since the results of this analysis were similar to those of the primary analysis using quartiles, findings from the primary analysis are presented in this paper.

RESULTS

Characteristics of the Study Population

Characteristics of the cohort of 6252 women (average age 71.3 years) are shown in Table 1. During an average follow-up ranging from 7.7 ± 3.1 years for clinical fracture to 8.7 ± 2.5 years for major osteoporotic fracture to 9.2 ± 1.8 years for hip fracture, 389 (6%) of women experienced a hip fracture, 1037 (17%) experienced major osteoporotic fractures, and 1907 (30%) experienced clinical fractures. Compared with the 3452 women in the original SOF cohort excluded from this analysis due to missing data needed for calculation of FRAX® model probabilities (primarily parental history of hip fracture), the 6252 women included in the analytical cohort were, on average, slightly younger (mean age 71.4 vs. 72.3 years, p<0.001) and less likely to report poor to fair health status (15% vs. 20%, p<0.001) or prior history of fracture (35% vs. 42%, p<0.001). However, mean body mass index (26.4 vs. 26.4, p=0.99) and femoral neck BMD (0.65 vs. 0.65, p=0.37) were similar between the two groups.

Table 1.

Characteristics of the 6252 Participants

Characteristic	Cohort (n=6252)
Age, years, mean ± SD	71.3 ± 5.1
Prior fracture, n (%)	2155 (35)
Oral glucocorticoid therapy, n (%)	741 (12)
Rheumatoid arthritis, n (%)	429 (7)
Parental history of hip fracture, n (%)	925 (15)
Current smoker, n (%)	583 (9)
Alcohol intake, ≥3 drinks daily, n (%)	184 (3)
Body mass index, kg/m2, mean ± SD	26.4 ± 4.6
Femoral neck BMD, gm/cm2, mean ± SD	0.65 ± 0.11
Women with hip fracture, n (%)	389 (6)
Women with major osteoporotic fracture*, n (%)	1037 (17)
Women with any clinical fracture, n (%)	1907 (30)

^*Major osteoporotic fractures include hip, clinical vertebral, wrist, and humerus fractures

FRAX® Model with BMD vs. Model with Age and BMD for Prediction of Fractures

Across a range of sensitivities and specificities, the ROC curves for hip fracture (Figure 1A) were essentially superimposed for the FRAX® model with femoral neck BMD and the simple model containing age and femoral neck BMD alone. The AUC statistic was 0.75 for the FRAX® model with BMD and 0.76 for the model with age and BMD (p=0.26). While these models were more limited in their ability to discriminate major osteoporotic (AUC 0.68 for FRAX® model with BMD and 0.69 for model with age and BMD, p=0.51) and any clinical fracture (AUC 0.64 for FRAX® model with BMD and 0.63 for model with age and BMD, p=0.16), models containing age and BMD alone again performed similarly to FRAX® models with BMD (Figure 1B and C).

Figure 1.

Receiver Operating Characteristic (ROC) Curves for Prediction of (A) Hip Fracture, (B) Major Osteoporotic* Fracture, and (C) Clinical^† Fracture Using FRAX® Model with BMD and Model with Age and BMD^‡

*Major osteoporotic fractures include hip, clinical vertebral, wrist, and humerus fractures

^†Clinical fractures include non-vertebral and clinical vertebral fractures

^‡The black diagonal line indicates a reference AUC of 0.50 (no better than chance alone)

Abbreviations: BMD, bone mineral density; AUC, area under the curve

There was no evidence that the proportion of women in each quartile of predicted risk who actually experienced a fracture outcome differed between the FRAX® model with BMD and the model with age and BMD alone for the outcome of hip (p=0.72), major osteoporotic (p=0.78), or any clinical fracture (p=0.23).(Figure 2A) Among the women classified in the highest quartile of predicted risk, similar proportions actually experienced a hip (14% using FRAX® model with BMD and 15% using model with age and BMD), major osteoporotic (29% using FRAX® model with BMD and 30% using model with age and BMD), and any clinical (46% using FRAX® model with BMD and 45% using model with age and BMD) fracture during the 10-year follow-up period. In addition, among the women classified in the lowest quartile of predicted risk, nearly identical proportions actually sustained a fracture outcome (hip fracture: 1% using either model, major osteoporotic fracture: 6% using FRAX® model with BMD and 7% using model with age and BMD, and any clinical fracture: 19% using FRAX® model with BMD and 21% using model with age and BMD).

Figure 2A.

Proportion of Women Classified in Each Quartile of Predicted Risk Who Actually Experienced a Fracture Outcome Using FRAX® Model with BMD and Model with Age and BMD

Abbreviations: BMD, bone mineral density

FRAX® Model without BMD vs. Model with Age and Fracture History for Prediction of Fractures

The parsimonious model based on age and BMD alone was superior to the FRAX® model without BMD that had AUC statistics of 0.71 for hip fracture (p<0.001), 0.64 for major osteoporotic fracture (p<0.001), and 0.61 for any clinical fracture (p=0.01). However, BMD measurement is not universally available in all practice settings. We therefore performed a secondary analysis in which we compared the predictive validity of FRAX® models without BMD with that of simple models based on age and fracture history alone. In ROC analyses, the FRAX® model without BMD performed similarly to a model containing age and fracture history for prediction of hip fracture (Figure 3A, AUC statistic 0.71 for both models, p=0.91). AUC statistics were lower, but similar using either model, for major osteoporotic (AUC 0.64 both models, p=0.89, Figure 3B) and any clinical fracture (AUC 0.61 both models, p=0.70, Figure 3C). The model based on age and fracture history was superior to a model containing age alone that had AUC statistics of 0.69 for hip fracture (p=0.01), 0.60 for major osteoporotic fracture (p<0.001), and 0.56 for any clinical fracture (p<0.001).

Figure 3.

Receiver Operating Characteristic (ROC) Curves for Prediction of (A) Hip Fracture, (B) Major Osteoporotic* Fracture, and (C) Clinical^† Fracture Using FRAX® Model without BMD and Model with Age and Prior Fracture^‡

*Major osteoporotic fractures include hip, clinical vertebral, wrist, and humerus fractures

^†Clinical fractures include non-vertebral and clinical vertebral fractures

^‡The black diagonal line indicates a reference AUC of 0.50 (no better than chance alone)

Abbreviations: BMD, bone mineral density; AUC, area under the curve

There was no evidence that the proportion of women in each quartile of predicted risk who actually experienced a fracture outcome differed between the FRAX® model without BMD and the model with age and fracture history alone for the outcome of hip (p=0.26), major osteoporotic (p=0.69), or any clinical fracture (p=0.16) (Figure 2B). Among the women classified in the highest quartile of predicted risk, similar proportions actually experienced a hip (13% using either model), major osteoporotic (27% using either model), and any clinical (44% using FRAX® model without BMD and 43% using model with age and fracture history) fracture during the 10-year follow-up period. In addition, among the women classified in the lowest quartile of predicted risk, essentially identical proportions actually sustained a fracture outcome (hip fracture: 2% using either model, major osteoporotic fracture: 9% using FRAX® model without BMD and 10% using model with age and fracture history, and any clinical fracture: 23% using FRAX® model without BMD and 24% using model with age and fracture history).

Figure 2B.

Proportion of Women Classified in Each Quartile of Predicted Risk Who Actually Experienced a Fracture Outcome Using FRAX® Model without BMD and Model with Age and Prior Fracture

Abbreviations: BMD, bone mineral density

COMMENT

In this cohort of older community dwelling women, a simple model based on age and BMD alone predicted 10-year risk of hip, major osteoporotic, and any clinical fracture as well as more complex FRAX® models with BMD. Similarly, a parsimonious model based on age and fracture history alone predicted 10-year risk of these three fracture outcomes as well as more complex FRAX® models without BMD. These findings suggest that use of the FRAX® risk assessment tool does not enhance fracture prediction in older women beyond that provided by simple models based on age and BMD or age and fracture history alone.

Our results are in general agreement with published findings regarding the predictive validity of the FRAX® risk assessment tool. In the FRAX® development and validation study⁴, findings concerning hip fracture prediction indicated that a model combining BMD with FRAX® clinical risk factors vs. a model with BMD alone resulted in a small increment (5 to 19% increase depending on age group) in the gradient of risk per one standard deviation change in risk score. However, the performance of the model with BMD alone (assessed with ROC analysis) was nearly identical to that for the model with BMD plus FRAX® clinical risk factors suggesting that hip fracture prediction did not substantially improve by adding FRAX® clinical risk factors to the model with BMD alone. There was also a small increment (11 to 21% increase depending on age group) in the gradient of risk for prediction of other osteoporotic fractures (defined variably depending on the cohort, excluded hip fractures) when FRAX® clinical risk factors were used in combination with BMD; however, a comparison of the performance of the model containing BMD plus clinical risk factors with that of the model with BMD alone was not reported. Findings from this study and the FRAX® development and validation study both suggest that models for predicting hip fracture perform better than those for predicting other fracture outcomes. Neither this study nor the FRAX® development and validation study examined the ability of models to predict radiographic vertebral fracture, the most common manifestation of osteoporosis. However, a study using data collected in the Fracture Intervention Trial²⁶ reported that the FRAX® model with BMD and a simple model based on age and BMD alone were similarly accurate in terms of discrimination of new radiographic vertebral fracture.

While gradients of risk and areas under ROC curves are widely utilized methods for model comparison, the clinical usefulness of a risk prediction model may also be judged by the extent to which the risk calculated from the model reflects the fraction of individuals in the population who actually sustained the outcome of interest. To address this issue, we calculated the proportion of women in each quartile (and decile) of predicted risk who actually experienced a fracture outcome using FRAX® and simple models and compared these proportions between models. These results also suggest that use of FRAX® models does not enhance fracture prediction beyond that provided by more parsimonious models as there was no evidence that use of the FRAX® models improved the classification of high (and low) risk categories such that a higher (and lower) proportion of women who actually experienced a fracture outcome were identified. Similar findings have been reported in the study comparing FRAX® and simple models for the prediction of new radiographic vertebral fracture.²⁶

The FRAX® risk assessment tool represents a major advance in the field of osteoporosis for several reasons. The tool is based on data collected from cohorts in the USA, Europe, Australia and Asia and is applicable to both the developed and developing world. Modeling techniques incorporated into the FRAX® tool take into account country-specific fracture and death rates. Its aim to move forward risk assessment from a strategy based on BMD alone to an approach based on the absolute risk of fracture is appealing as absolute risk classification systems overcome several of the drawbacks posed by relative risk classification systems and may be more intuitive to both clinicians and patients.²⁷ However, despite these merits, findings from this study in older Caucasian women suggest that use of the FRAX® tool does not lead to substantial improvements in fracture risk prediction. Cost-effective analyses¹⁶ supporting use of the FRAX® tool to select men and women aged 50 years and older with osteopenia for pharmacologic therapy rely on a critical, but controversial assumption^28-31 that drug treatment is effective in reducing the risk of all clinical fractures regardless of BMD status. Moreover, application of the new NOF guidelines may result in recommending pharmacologic treatment to a very large proportion of women aged 65 years and older.²⁰ Thus, randomized trials evaluating the effectiveness of a treatment approaches based on absolute risk of fracture including the approach recommended by NOF (incorporating selected risk cutpoints based on FRAX® tool probabilities) should be conducted prior to widespread implementation of these approaches into clinical decision making regarding whether or not to initiate drug therapy.

This study has a number of strengths including the comprehensive set of measurements and duration and completeness of follow-up. However, this study has several limitations. Our findings are based on a single cohort of older Caucasian women. Although we cross-validated our findings regarding the performance of models, results concerning the predictive validity of FRAX® and simple models require confirmation in other cohorts. In particular, these findings may not be generalizable to younger women or men. Other than rheumatoid arthritis, data on six specific conditions associated with secondary osteoporosis that comprise an additional component in the FRAX® risk assessment tool were not collected in SOF. However, these medical conditions are uncommon in healthy older women and their association with increased fracture risk is in large part due to lower BMD among those with disease. For this reason, checking the “secondary osteoporosis” box in FRAX® does not alter the risk score once BMD is entered into the algorithm.³² For a given patient, FRAX® and simple models will not necessarily agree on the classification of risk status despite nearly identical ROC curves. Other methods are available to assess and interpret the utility of a new model such as measures of reclassification^33;34 that determine how often an alternative model successfully reclassifies individuals from one risk class to another and thus alters individual treatment decisions. However, the use of these measures require the existence of predefined treatment thresholds at which treatments would be altered, as well as the availability of effective treatments at different risk levels. In the case of osteoporosis, these requirements are not met at present, especially the availability of treatments at different risk levels. While model weights in the parsimonious models were derived from the SOF population and the potential for over-fitting exists, findings from this study are consistent with those reported in the FRAX® development and validation study¹⁴ and other cohorts.²⁶ Since the equations and algorithms that generate the fracture risk probabilities have not been published for the FRAX® tool, it was not feasible to directly quantify the change in the AUC statistic which accompanies adding each component of the FRAX® model. Finally, both the FRAX® models and simple models are limited in their ability to predict fracture, especially nonhip fractures. Thus, risk prediction is challenging, even when robust risk factors like BMD and age are available.

We conclude that a simple model based on age and BMD alone predicted 10-year risk of hip, major osteoporotic, and any clinical fracture as well as more complex FRAX® models with BMD. Similarly, a parsimonious model based on age and fracture history alone predicted 10-year risk of these fractures as well as more complex FRAX® models without BMD. These results suggest that use of the FRAX® risk assessment tool does not improve fracture prediction in older women beyond that provided by simple models based on age and BMD or age and prior fracture alone.