Developing and internally validating a prediction model for total knee replacement surgery in patients with osteoarthritis

Abstract

Objective

The objective of this study was to develop and internally validate a clinical algorithm for use in general practice that predicts the probability of total knee replacement (TKR) surgery within the next five years for patients with osteoarthritis. The purpose of the model is to encourage early uptake of first-line treatment strategies in patients likely to undergo TKR and to provide a cohort for the development and testing of novel interventions that prevent or delay the progression to TKR.

Method

Electronic health records (EHRs) from 201,462 patients with osteoarthritis aged 45 years and over from 483 general practices across Australia were linked with records from the Australian Orthopaedic Association National Joint Replacement Registry and the National Death Index. A Fine and Gray competing risk prediction model was developed using these data to predict the risk of TKR within the next five years.

Results

During a follow-up time of 5 years, 15,979 (7.9%) patients underwent TKR and 13,873 (6.9%) died. Predictors included in the final algorithm were age, previous knee replacement, knee surgery (other than TKR), prescribing of osteoarthritis medication in the 12 months prior, comorbidity count and diagnosis of a mental health condition. Optimism corrected model discrimination was 0.67 (95% CI: 0.66 to 0.67) and model calibration acceptable.

Conclusion

The model has the potential to reduce some of the economic burden associated with TKR in Australia. External validation and further optimisation of the algorithm will be carried out prior to implementation within Australian general practice EHR systems.

Abbreviations

ABS

Australian Bureau of Statistics

AIHW

Australian Institute of Health and Welfare

AOANJRR

Australian Orthopaedic Association National Joint Replacement Registry

ASGS

Australian Statistical Geography Standard

BMI

Body mass index

CCI

Charlson Comorbidity Index

CI

Confidence interval

DCA

Decision curve analysis

EHR

Electronic health record

EPV

Events per variable

SMC-FCS

Substantive model compatible fully conditional specification

GLA:D®

Good Life with steoarthritis: Denmark

GP

General practitioner

IRSAD

Index of Relative Socio-Economic Advantage and Disadvantage

MAR

Missing at random

MI

Multiple imputation

MNAR

Missing not at random

MP

Multi-processor

NDI

National Death Index

NPS

National Prescribing Service

OA

Osteoarthritis

SAP

Statistical Analysis Plan

SD

Standard deviation

SHR

Subdistribution hazard ratio

TKR

Total knee replacement

TRIPOD

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis

UK

United Kingdom

1. Introduction

Over nine percent of the Australian population are affected by osteoarthritis (OA) [1]. It is one of the ten most frequently managed health conditions by general practitioners (GPs) [2,3]. In 2015–16, an estimated $3.5 billion was spent on the management of OA in Australia with approximately 85% of these costs associated with total knee replacement (TKR) and total hip replacement (THR) surgeries [4]. Over the last decade and a half, the rate and cost of these procedures has risen, with TKR rates increasing by 38% and costs by 29% [1,5,6]. As the population continues to age, and obesity rates increase, OA expenditure in Australia is forecasted to reach $5.3 billion by 2030 [7].

GPs are often the first point of contact for patients with OA and can play a critical role in identifying early those patients likely to undergo TKR. Early identification of these patients may provide adequate time for non-surgical and non-pharmacological treatment strategies such as exercise and weight loss management programs to be adopted. Whilst these first-line treatment strategies are part of the current OA management guidelines and are proven efficacious in managing symptomatic OA [8,9], uptake has been low [10]. Knowing which patients are likely to undergo TKR in the future and discussing these predictions with the patient may help improve the uptake of first-line treatment strategies. For patients who have received first-line treatment but are still likely to undergo TKR, early identification provides a target audience for the development and testing of novel interventions which may prevent or delay the progression to TKR.

At the time of our literature review, only a handful of studies on prediction models for TKR were available in the published literature [11] of which none were conducted in the primary care setting, and none were based solely on predictors routinely collected in general practice. Since then, one study predicting TKR in 10 years utilising primary care electronic health record (EHR) data from the United Kingdom (UK) has been published [12]. Several of the predictors in the UK model are either not recorded or inconsistently recorded in Australian general practice EHRs and therefore applying this model in the Australian primary care setting would be difficult. This highlights the need for a clinical prediction model for TKR surgery developed from Australian general practice EHRs that could be easily embedded into Australian general practice workflow. The aim of this study was to develop and internally validate a prediction model for TKR, using data derived from Australian general practice EHRs.

2. Method

A detailed statistical analysis plan (SAP) for the development and validation of the prediction model has been published [11]. In this paper we briefly describe the methods and report on all aspects of model development and internal validation from the TRIPOD checklist [13]. Below we provide an overview of the data sources, outcomes, predictors and statistical methods.

2.1. General practice EHR data

The MedicineInsight data set, managed by NPS MedicineWise, consists of de-identified general practice EHRs from approximately 2.9 million patients from over 650 general practices across Australia [14,15]. For this study, NPS MedicineWise provided 475,870 patient EHRs with a recorded diagnosis of OA from 483 general practices. The coding used by NPS MedicineWise to identify patients with OA has been provided in Supplementary File 1. The data provided included patient clinical data recorded in the EHR by the December 31, 2017 and encounter data for the years 2013–2017.

2.2. Outcome and competing risk

The outcome was time to the first occurrence of a primary TKR on a particular side (e.g. right or left) within the five-year study period. The outcome was obtained by linking patient EHR data with TKR data from the Australian Orthopaedic Association National Joint Replacement Registry (AOANJRR) [16]. The AOANJRR includes data on TKRs performed in Australia since 1999 and has near complete capture of all TKRs in Australia from 2002 onwards. Data linkage was performed by BioGrid Australia [17].

Death was treated as a competing risk given the age range of patients with OA. Date of death was obtained through data linkage with the National Death Index (NDI) [18] conducted by the Australian Institute of Health and Welfare (AIHW) [18]. Details of the data linkage process and linkage assessment are included in our data quality assessment publication [19].

2.3. Study timeline and participants

The study timeline for model development was the five-year period between the January 1, 2014 (baseline) to the December 31, 2018 inclusive. The inclusion criteria were patients (i) with at least two visits to a GP from the same clinic for any reason in the year before baseline; (ii) aged 45 years and over at baseline (iii) alive at study baseline (i.e. no record of death in the NDI); and (iv) no recorded evidence of bilateral TKR prior to study baseline in the AOANJRR.

2.4. Predictors

We conducted a literature review to identify potential predictors of TKR and used an adapted Delphi process [20] to obtain consensus amongst experts in OA on potential predictors [11]. From this process, 32 predictors of TKR were identified [11]. However, only nine of these predictors were routinely collected in general practice and available in EHRs. These were age in years, body mass index (BMI), weight gain between early adulthood and middle age, overall health, prescribing of OA medications, mental health condition, previous/contralateral TKR, past knee surgery other than TKR and geographical residence of the patient. The coding of these predictors from the EHRs are provided in Supplementary File 1 and have been previously published [11,19].

From our published data quality assessment, the predictors BMI and weight gain between early adulthood and middle age had substantial amounts of missing data in the EHRs and therefore were excluded from model development [19]. Multimorbidity count was used as a proxy measure for overall health. This was a count of chronic conditions listed in the Charlson Comorbidity Index (CCI), which is a measure that predicts 10-year survival based on a weighted composite score for comorbidities [21]. We considered other measures of counting multimorbidity in our data quality assessment [19] but chose a count of conditions in the CCI for model development as it had the least amount of missing data.

2.5. Sample size

To date, sample size calculations for prediction models have been based on rules of thumb [22,23]. Recently however, more detailed sample size calculations specific to the type of prediction model have been proposed [24]. These are yet to be extended to the competing risk setting. Therefore, our sample size calculation published in our SAP [11] was based on estimating the events per variable (EPV) from the expected number of patients with the outcome and the expected number of predictors. The EPV was estimated to be over 1500 (15,000 patients underwent TKR/9 predictors≈1666 events per variable) and therefore sufficient to develop a stable model [22].

2.6. Statistical analysis

All statistical analyses were performed using Stata MP version 16.1 (StataCorp, College Station Tx, USA) and R Studio version 4.0.4 (“Lost Library Book”) [25,26]. Predictors and patient socio-demographics were summarised by outcome and competing risk. Age in years and BMI were summarised using the mean and standard deviation, and count of conditions from the CCI and number of patients per general practice using the median and interquartile range. All categorical data were summarised using counts and percentages. The proportion of missing data were also summarised for each variable.

2.7. Model specification and estimation

A Fine & Gray proportional hazards competing risk regression model was used to develop the prediction model [27,28]. Initially, a full model was fitted with the seven predictors and then predictors with subdistribution hazard ratios (SHRs) between 0.90 and 1.10 and p-values>0.1 were excluded [11]. A quadratic term for age was included in the model given the rate of TKR is known to increase with age until approximately 70 years, after which it decreases [1]. Two-way interactions between each predictor were tested. Robust standard errors for the regression estimates were calculated to account for clustering of patients within clinics.

2.8. Missing data

We performed multiple imputation (MI) of predictors by substantive model compatible fully conditional specification (SMC-FCS) using chained equations and created 27 imputed data sets [29]. The imputation model included all seven predictors identified. We included the number of clinic visits in the year prior to baseline as an auxiliary variable given those who attend general practice more often are less likely to have missing data in their EHR [30]. Due to the large number of clinics and GPs in our study, we were unable to include general practice clinic and GP as auxiliary variables to account for differences in recording practices between clinics and between practitioners. Instead, clinic state which was shown to be associated with missing predictor data (Supplementary File 2 Table 2) was included as an auxiliary variable to account for possible differences in recording practices of GPs between states. Convergence of the imputation process was assessed by comparing graphs of the distributions of the observed and imputed data. Rubin's rules were used to average regression coefficients across the 27 imputed data sets [31].

Table 2.

Final model regression coefficients and subdistribution hazard ratios.

Predictors	Regression coefficient from Fine & Gray competing risk model (95% CI)	Subdistribution hazard ratio (SHR) (95% CI)
Age (per unit increase), years	0.51 (0.49–0.53)	1.66 (1.63–1.70)
Age2	−0.0039 (−0.0040 to −0.0037)	0.9962 (0.9960–0.9963)
Prescribing of OA medications (yes)	0.41 (0.37–0.44)	1.50 (1.45–1.56)
Count of chronic conditions from CCI (per unit increase)	−0.14 (−0.16 to −0.12)	0.87 (0.85–0.89)
Mental health condition (yes)	−0.14 (−0.18 to −0.10)	0.87 (0.83–0.90)
Previous TKR (yes)	0.97 (0.91–1.02)	2.63 (2.49–2.77)
Past knee surgery (excluding TKR) (yes)	0.93 (0.87–0.99)	2.54 (2.39–2.69)

Abbreviations: CI ​= ​confidence interval; SHR ​= ​subdistribution hazards ratio from competing risk model; OA ​= ​osteoarthritis; CCI=Charlson Comorbidity Index; TKR ​= ​total knee replacement.

Note: Subdistribution hazard ratio obtained by exponentiating the regression coefficient from the Fine & Gray competing risk model.

2.9. Proportional hazards assumption and model goodness of fit

The proportional hazards assumption was checked for the final model in the first imputed data set for each categorical variable by plotting the log of the survival probability by survival time and performing a Chi-squared test of proportional hazards. When there was strong evidence against the null hypothesis for the proportional hazards assumption for a predictor, an interaction term between the predictor and time was tested in the model.

Possible influential observations were identified for each predictor by plotting DFBETAs which quantify the change in each predictor's coefficient if a subject was removed from the study [32]. Martingale residuals were plotted for age and count of conditions in the CCI to check the functional form of these predictors [33].

2.10. Model performance and internal validation

Harrell's overall and Wolber's adjusted c-statistics at 5 years were used to assess model discrimination [22,34]. Calibration was assessed by plotting the observed and predicted 5-year probabilities of TKR in deciles of predicted risk [35]. Bootstrapping was used to obtain optimism corrected measures of model performance. One hundred bootstrapped samples with replacement were created from each of the 27 imputed data sets and the final model fitted in each data set. The optimism corrected c-statistic and calibration were calculated as documented in Appendix E of our published SAP [11]. A decision curve analysis consistent with that proposed by Steyerberg and Vergouwe [36] was conducted.

2.11. Secondary analysis

The model was refitted using patients who had a recorded diagnosis of knee OA in their EHR to determine whether a better performing model could be developed in this subset of patients. The same modelling process (see model specification and performance sections) was applied using the first imputed data set only.

3. Results

3.1. Characteristics of study cohort

Of the 475,870 patient EHRs, and assuming each EHR uniquely represents a patient, 236,412 patients with OA had a general practice encounter in the year prior to baseline. Of these patients, 34,950 (14.8%) were excluded for not meeting the study inclusion criteria: n ​= ​28,069 (11.9%) patients had one clinic visit in 2013 and/or were less than 45 years old, n ​= ​491 (0.2%) patients died prior to study baseline, n ​= ​1626 (0.7%) patients had indeterminate or missing dates of death recorded in the NDI, and (iv) n ​= ​4764 (2.0%) underwent bilateral TKR prior to study baseline. Indeterminate dates of death in the NDI may have been due to deaths being discovered some time after the event or patients with common names and dates of birth having links to multiple records in the NDI and hence multiple dates of death. A total of 201,462 patients from 483 general practices across Australia fully met all study inclusion criteria and were included in the study. Approximately 9% (n ​= ​18,266) of these patients had a linked record from the NDI and 12.6% (n ​= ​25,321) a linked record from the AOANJRR. A small proportion of records from the NDI (0.05%) and AOANJRR (0.02%) were excluded during the data linkage process due to issues generating patient linkage keys (Fig. 1).

Fig. 1.

Flowchart for study cohort.

From linkage with the NDI and AOANJRR, 15,979 (7.9%) primary TKRs and 13,873 (6.9%) deaths were identified during the five years. The occurrence of TKR was approximately evenly distributed across the study period. That is, approximately 20% of TKRs occurred in each follow-up year. Patients had a mean age of 67.2 years and 61% were female (Table 1). Approximately half the patients were from major cities in Australia and had attended practices in these locations. Thirty-five percent of patients had attended a general practice clinic in New South Wales, 19% in Queensland and 18% in Victoria. The largest proportion of patients, 35%, were from the most advantaged areas (IRSAD 4th and 5th quintiles) in Australia according to the Index of Relative Socio-economic Advantage and Disadvantage [37]. Approximately 34% of patients had recorded evidence of being prescribed an OA medication, 5% had undergone TKR previously and 3% had undergone knee surgery other than TKR in the past. Given all patients were followed up at five years using data linkage, there were no missing outcome data.

Table 1.

Patient characteristics by total knee replacement (TKR) and death (N ​= ​201,462).

Patient characteristics	TKR (outcome) N ​= ​15,979	Death (competing risk) N ​= ​13,873	No TKR or death N ​= ​171,610	Missing
	n (%)	n (%)	n (%)	n (%)
Age (years), mean (SD)	66.7 (8.6)	76.3 (8.7)	66.5 (11.1)	3 (0.001)
Gender				–
Female	9478 (59.3)	7219 (52.0)	106,679 (62.2)
Aboriginal and/or Torres Strait Islander	185 (1.4)	185 (1.7)	2301 (1.6)	38,148 (18.9)
IRSAD quintiles				1228 (0.6)
1 (most disadvantaged)	3178 (20.0)	3306 (24.0)	35,116 (20.6)
2	3184 (20.1)	3053 (22.1)	33,302 (19.5)
3	3895 (24.6)	3402 (24.7)	40,825 (23.9)
4 and 5 (most advantaged)	5611 (35.4)	4029 (29.2)	61,333 (36.0)
Patient geographical locationa				1068 (0.5)
Major cities of Australia	8515 (53.6)	6933 (50.3)	95,627 (56.0)
Inner regional Australia	5189 (32.7)	4796 (34.8)	52,236 (30.6)
Remote Australia	2177 (13.7)	2068 (15.0)	22,853 (13.4)
Healthcare card	4153 (26.0)	4500 (32.4)	44,757 (26.1)	–
Member of Department of Veteran's Affairs	527 (3.3)	869 (6.3)	7141 (4.2)	–
BMI (kg/m2), mean (SD)	32.4 (6.2)	28.8 (6.7)	30.0 (6.4)	137,295 (68.2)
Prescribing of OA medication/s	5680 (42.6)	5634 (50.6)	45,776 (31.7)	32,591 (16.2)
Count of chronic conditions from CCI, median [IQR]	0 [0, 1]	1 [0, 2]	0 [0, 1]	15,875 (7.9)
0 conditions	9492 (63.5)	3392 (29.3)	100,213 (63.0)
1 condition	4006 (26.8)	3894 (33.6)	41,316 (26.0)
2 conditions	1132 (7.6)	2507 (21.7)	12,768 (8.0)
3 or more conditions	310 (2.1)	1778 (15.4)	4779 (3.0)
Mental health condition	3401 (22.1)	3751 (28.5)	39,707 (24.1)	8178 (4.1)
Previous knee replacement	1843 (11.8)	904 (6.6)	6685 (3.9)	1264 (0.6)
Past knee surgery on either knee (excluding TKR)	1324 (8.4)	219 (1.6)	4527 (2.7)	1007 (0.5)
Smoking status				13,504 (6.7)
Smoker	750 (5.0)	1395 (11.1)	14,803 (9.2)
Ex-smoker	5570 (37.3)	5534 (43.9)	57,135 (35.6)
Non-smoker	8603 (57.7)	5687 (45.1)	88,481 (55.2)
Number of practice visits in year prior to study baseline, median [IQR]	11 [6, 18]	19 [11, 30]	10 [6, 18]
General practice clinics (N ​= ​483)
Number of clinics	454	449	483
Median [IQR] number of patients per clinic	339 [205, 572]	350 [214, 575]	327 [181, 566]
Clinic state				–
New South Wales	6175 (38.6)	5280 (38.1)	59,942 (34.9)
Queensland	2992 (18.7)	2526 (18.2)	32,980 (19.2)
Victoria	2200 (13.8)	2413 (17.4)	31,576 (18.4)
Western Australia	2189 (13.7)	1388 (10.0)	18,993 (11.1)
Tasmania	1596 (10.0)	1647 (11.9)	18,004 (10.5)
South Australia	375 (2.4)	290 (2.1)	5353 (3.1)
Australian Capital Territory	329 (2.1)	255 (1.8)	3584 (2.1)
Northern Territory	123 (0.8)	74 (0.5)	1178 (0.7)
Clinic geographical locationa				71 (0.04)
Major cities of Australia	8771 (54.9)	7055 (50.9)	98,377 (57.4)
Inner regional Australia	5145 (32.2)	4802 (34.6)	51,554 (30.1)
Remote Australia	2055 (12.9)	2013 (14.5)	21,619 (12.6)

Abbreviations: TKR ​= ​total knee replacement; SD ​= ​standard deviation; IRSAD=Index of Relative Socio-economic Advantage and Disadvantage (37); BMI ​= ​body mass index; OA ​= ​osteoarthritis; CCI=Charlson comorbidity index; IQR ​= ​interquartile range.

Notes: Counts and percentages presented unless otherwise stated.

^aBased on Australian Bureau of Statistics (ABS) Australian Statistical Geography Standard (ASGS) remoteness areas [44].

3.2. Model development

Results from the inspection of missing data and convergence of the imputation process are detailed in Supplementary File 2. All predictors had less than 10% missing data except for the prescribing of OA medication (16%). Approximately 26% of patients had a least one predictor with missing data. The most common missing data patterns were prescribing of OA medications (14%), count of chronic conditions listed in the CCI (5%), recording of a mental health condition (2%) and both prescribing of OA medications and count of chronic conditions (1%) in combination. Patient characteristics were similar between those with incomplete and complete predictor data except for the median number of clinics visits (median [IQR]: 10 [5, 17] vs 14 [8, 23] respectively). There was good convergence of the imputation process across the imputed data sets (Supplementary File 2).

The same six predictors were found to be predictive of TKR in all imputed data sets (Supplementary File 3). These were age, prescribing of OA medications, count of chronic conditions listed in the CCI, recording of a mental health condition, previous TKR and past knee surgery other than TKR. Patient geographical location was not predictive of time to TKR. The regression coefficients for the final model are shown in Table 2. Prescribing of OA medications, previous TKR and previous knee surgery (other than TKR) were associated with an increase in the rate of TKR in subjects who were event free. The reverse association was true for recording of a mental health condition and increasing count of chronic conditions. Age was found to be quadratically predictive of TKR. That is, as age increased from 45 years to approximately 66 years, the rate of TKR increased. After 66 years, the rate of TKR decreased. The strongest predictors were past TKR and past knee surgery (excluding TKR). The rate of TKR (in those who were event free) was approximately 2.6 times higher (95%CI 2.49 to 2.77) for those who had undergone previous TKR and 2.5 times higher (95%CI 2.39 to 2.69) for those who had undergone past knee surgery (excluding TKR).

3.3. Model specification

The estimated probability of TKR at five years (Table 3) was calculated using the final model regression coefficients from Table 2, patient baseline predictors and the baseline cumulative subdistribution hazard function [28].

Table 3.

Estimated probability of total knee replacement (TKR) at five years.

Probability of TKR at five years	$=1-\text{exp}(-\text{exp}({\text{X}}_{\text{J}}\beta )\cdot \underset{0}{\overset{5}{\int }}{\overline{\lambda }}_{\mathrm{1,0}}(s)ds)$
$X_J\beta$	= ​0.51∗Age-0.0039∗Age2+0.41∗Prescribing_OA_medication-0.14∗CCI_condition_count-0.14∗Mental_health_condition+0.97∗Past_knee_replacement+ 0.93∗Other_knee_surgery Where, Age is in years (range between 45 and 113) Prescribing_OA_medication ​= ​1 if prescribed an OA medication in the year prior to baseline and 0 otherwise, CCI_condition_count ​= ​count of chronic conditions listed in the Charlson comorbidity index (range between 0 and 8), Mental_health_condition ​= ​1 if recording of a mental health condition and 0 otherwise, Past_knee_replacement ​= ​1 if history of previous TKR and 0 otherwise, Other_knee_surgery ​= ​1 if history of past knee surgery (excluding TKR) and 0 otherwise.
$\underset{0}{\overset{5}{\int }}{\overline{\lambda }}_{\mathrm{1,0}}(s)ds$	= ​baseline cumulative subdistribution hazard at five years  ​= ​5.6 ​× ​10−9

AbbreviationsTKR ​= ​total knee replacement; OA ​= ​osteoarthritis; CCI ​= ​count of conditions from the Charlson Comorbidity Index; and “exp” ​= ​exponential function.

For instance, a patient aged 65 years, prescribed OA medications in the past 12 months, with diabetes, without a diagnosis of a mental health condition, who has previously undergone TKR and other knee surgery has a 58% probability of undergoing TKR within the next five years.

3.3.1. Proportional hazards assumption and model goodness of fit

Statistical tests performed on the first imputed data set indicated possible violation of the proportional hazards assumption by three predictors: prescribing of OA medications, previous TKR and age (squared). Including interactions between each of these three predictors and time in the model resulted in small regression coefficients for the interaction terms (Supplementary File 4) which had little impact on the predicted probabilities. The inclusion of two-way interactions did not improve model fit or performance.

DFBETA plots indicated no influential observations. Martingale residuals plots for age and count of chronic conditions were approximately quadratic and linear respectively (Supplementary File 4).

3.4. Model performance

The apparent and optimism corrected c-statistic were similar, 0.667 (95% CI 0.660 to 0.673) and 0.666 (95% CI 0.660 to 0.673) respectively (Supplementary File 5). The c-statistic implies that the model correctly predicts who will undergo TKR earlier between two patients 67% of the time. Wolber's adapted c-statistic [34] based on the cumulative incidence function was similar (66%).

Model calibration and optimism corrected calibration are shown as dots and crosses respectively in Fig. 2. Overall, model calibration was acceptable. Due to the large sample size and small differences in the estimated optimism corrected and apparent performance measures, shrinkage of the model regression estimates was not performed [22].

Fig. 2.

Optimism corrected calibration- Apparent predictive accuracy shown as dots and optimism corrected predictive accuracy shown as crosses. Error bars represent the 95% confidence intervals of the observed cumulative incidence function for total knee replacement (TKR) at five years at each predicted risk decile.

Results from the decision curve analysis (DCA) are shown in Fig. 3. The model provides a net benefit between threshold probabilities 0.01 and 0.28. Compared to referring all patients with OA to a novel non-surgical intervention program aimed at preventing or delaying TKR, the maximum number of extra true TKRs identified by the model (for the same false-positive rate) is approximately 16 per 1000 patients. This occurs at a threshold probability of 0.08. Fig. 4 shows a maximum net reduction in referrals of 72 per 100 patients if the model was used instead of all patients with OA being referred to the novel non-surgical intervention.

Fig. 3.

Decision curve- Solid line ​= ​net benefit from treating/referring all patients with osteoarthritis (OA) to a non-surgical intervention, Dotted line ​= ​net benefit from treating/referring no one, Dashed line ​= ​smoothed net benefit from the prediction model.

Fig. 4.

Net reduction in interventions per 100 patients- Line represents net reduction in interventions per 100 patients for a particular threshold probability.

3.5. Secondary analysis

Approximately 3% of patients with a recorded diagnosis of OA in their EHR had a recorded diagnosis of knee OA specifically. The same predictors were found to be predictive of TKR for patients with a recorded diagnosis of knee OA compared with any form of OA. Approximately 18% of the knee OA sample underwent TKR and 7% died during the five-year study period. Compared to the first model, discrimination was poorer (c-statistic ​= ​0.63, 95% CI 0.61 to 0.65) and regression coefficients for previous TKR and other knee surgery reduced (Appendix A).

4. Discussion

We developed a five-year risk prediction model for TKR in patients with OA aged 45 years and over using general practice EHR data and linked registry data. The discriminative ability of the model was likely restricted by the lack of inclusion of potentially strong predictors such as BMI and health insurance status [11]. BMI was excluded from model development because almost 70% of patients had missing data [19] and health insurance status was not available in the EHR data set.

The model showed net benefit between threshold probabilities 0.01 to 0.28. The threshold probability is a weighting of the relative harm of a false-positive and false-negative result [38]. A threshold probability or risk of 10% for example, is an odds of 1:9 meaning that the harm of a false-negative is 9 times worse than the harm of a false-positive [38]. Assuming the cost of a novel non-surgical intervention program aimed at delaying or preventing TKR is approximately equivalent to the cost of an existing evidence-based program such as The GLA:D® program [39,40], then the harm of unnecessarily referring a patient to such an intervention (harm of a false-positive) is at an estimated cost of $1500 [41]. The harm of a false-negative is the cost of TKR surgery, approximately $23,000 [42], hence the harm of a false-negative is 15 ($23,000/$1500) times worse than the harm of a false-positive. This corresponds to a threshold probability of approximately 6% (odds 1:15) which translates to a net benefit of 27 true TKRs being identified per 1000 OA patients through use of the model. Compared with referring all OA patients to the novel intervention program, the model identifies an extra 9 true TKRs per 1000 OA patients for the same false-positive rate and can reduce the number of referrals by 14 per 100 OA patients.

Despite moderate model performance, the DCA suggests potential financial benefit. For example, if 34%–68% of true TKRs identified by the model can be avoided [41], approximately $AUD2.0-$AUD3.9 million may be saved per 10,000 OA patients visiting the GP. The potentially large financial benefit is likely due to the relative expense of TKR surgery compared with non-surgical interventions. Clinical benefits may include improved quality of life, pain and function in those unlikely to respond to TKR [43].

Our model is the first to be developed using predictors from Australian general practice EHR data and linked outcome data from gold standard national registries. Of the predictors (age, OA prescriptions, CCI, mental health condition and previous knee surgery) common to our model and the model by Yu et al. [12], the direction of predictor effects were consistent. However, we were unable to directly compare the magnitude of predictor regression estimates given slight differences in how predictors and the outcome, 10-year risk of TKR were defined in Yu et al. [12]. The predictive performance of our model was worse than the model by Yu et al. (c-statistic 0.79; 95% CI 0.78 to 0.79) [12] and likely due to limitations in the quality of data in Australian general practice EHRs, specifically the completeness of data fields and consistency in diagnoses recording which limited the number of predictors available for model development [19]. Further, restricting model development to patients with a recorded diagnosis of knee OA in their EHR resulted in a model with slightly poorer predictive performance (Appendix A). A possible explanation is that the sample of patients with a recorded diagnosis of knee OA in their EHR may underrepresent the target population of patients with knee OA attending Australian general practices. Given only 3% of OA patients had a recorded diagnosis of knee OA in their EHR instead of an expected 30%, and fewer patients in this cohort underwent TKR than expected (18% vs approximately 41%) (44,45), the predictor effect estimates may be attenuated, thus possibly leading to poorer predictive performance. Further, the candidate predictors selected for model development were identified from studies predicting TKR in patients with OA, and may not be the strongest predictors of TKR for the subset of patients where knee OA was identified in the EHR. The main model was specifically developed to be used in Australian general practices, where it can be embedded within the EHR system and automated to identify patients with OA who are at high risk of TKR. Our model generates shorter-term (five-year) risk prediction estimates, which may be more relevant and motivational for patients than their 10-year risk estimate when adopting non-surgical management options such as weight management and exercise therapy. Lastly, from our DCA, use of our model has the potential to delay or prevent TKR surgery in patients with OA and alleviate some of the economic burden associated with knee OA in Australia.

4.1. Strengths and limitations

Strengths of our study included a rigorous and methodological approach to model development consistent with that proposed by Steyerberg [22] and no significant departures from our SAP [11]. The sample size was large allowing predictor effects to be estimated with good precision and the cohort was representative of the wider Australian OA population [19]. MI by SMC-FCS was used to ensure the imputation model was compatible with the competing risk outcome model so that unbiased estimates of the predictors could be obtained [29].

Limitations of our study included poor data quality in general practice EHRs for potentially important predictors, such as BMI which may have limited the predictive performance of our model. Whilst MI was used to impute missing data for predictors, these models assume missing data are missing at random (MAR) and missingness is conditional on variables included in the models. It is possible that there are other auxiliary variables that may explain the missing data that we did not have access to and inclusion of practice state as an auxiliary variable did not fully capture the potential relationship between missing data and individual clinics/GPs. If this is the case, imputations may be bias and result in poor model performance during external validation. However, it seems unlikely that our MI models are missing strong predictors of missing data and hence the impact on predictive performance of missing predictor data that may be missing not at random (MNAR) is expected to be minimal.

The estimated financial benefit of the model is based on a simple calculation that does not account for social or personal costs to the patient from undergoing TKR or a novel non-surgical intervention, and does not account for the cost of surgeries that may be deferred to later years. Also, novel non-surgical interventions may be more expensive than existing first-line treatment and hence actual cost savings may be less than that estimated.

Lastly, we were unable to determine from the EHRs which prescriptions were specifically for OA management, and it is possible that patients were taking pain medication for other purposes. This may have negatively impacted the predictive ability of our model if the prescribing of such medication specifically for OA is a strong predictor of TKR.

4.2. Future work

External validation will be performed using another EHR data set and the model updated to include any other important predictors that may be available. The study cohort may be restricted to those with a recorded diagnosis of knee OA should this be accurately recorded. Pending external validation, pilot testing of the model will be conducted in Australian general practices. A knowledge translation strategy that aims to maximise uptake and appropriate use of the model will be implemented.

Ethics approvals

We have obtained the necessary ethics approvals from the following: The University of Melbourne ID 1852593, St Vincent's Hospital ID 46036 LRR 202/18, The University of South Australia ID 201840 and Australian Institute of Health and Welfare EO2018/5/509.

The analyses are based on secondary data sources and consent to use the data for this research purpose has been granted by participants (AOANJRR) or general practice clinics (MedicineInsight EHR data). This study (DG2018-022) was approved by the NPS MedicineWise data governance committee on the August 29, 2018.

Author contributions

ST, MD, PChondros, JMN contributed to the study conception, design, acquisition of data, revising of draft manuscript and final manuscript. ST carried out the analyses, drafted the manuscript and interpreted the results. PChondros and TS contributed to the interpretation of analyses results. TS, JG and Pchoong contributed to the conception and design of the study and revising of the manuscript. All authors reviewed and approved the final version of the manuscript.

Role of the funding source

The researchers gratefully acknowledge the Royal Australian College of General Practitioners (RACGP) Foundation and HCF Research Foundation (HCF18-04) for their support of this project and Centre for Research Excellence in Total Joint Replacement.

Availability of data

The data sets used to develop the prediction model in this study are not publicly available due to patient privacy regulations. Data may be available from the authors upon reasonable request if permission is granted by all data providers: NPS MedicineWise, Australian Orthopaedic Association and Australian Institute of Health and Welfare.

Declaration of competing interest

The authors declare that they have no competing interests in relation to this study.

Appendix A. Secondary Analysis

Approximately 3% of patients with a recorded diagnosis of OA had a recorded diagnosis of knee OA in the EHR (Table A1). Of these patients, approximately 18% (n ​= ​1095) underwent TKR during the five-year study period (Table A2). Patient characteristics were similar in those with a recorded diagnosis of OA and those with knee OA except for the proportion prescribed OA medications (33.8% vs 44.4%) and who underwent previous knee surgery (TKR 4.7% vs 12.4% or other knee surgery 3.0% vs 7.4%). The final model developed using patients with a recorded diagnosis of knee OA included the same predictors as the model developed from patients with a recorded diagnosis of OA, however, model performance was worse (c-statistic ​= ​0.63, 95% CI 0.61 to 0.65 vs c-statistic ​= ​0.67, 95% CI 0.66 to 0.67). Subdistribution hazard ratios (Table A3) for previous TKR and other knee surgery were smaller in the model developed from patients with a recorded diagnosis of knee OA.

Whilst there are limited published data on the characteristics of patients with knee OA in Australia, estimates suggest closer to 30% of patients with OA are expected to have knee OA and approximately 41% of these patients would be expected to undergo TKR within five-years (44,45). The data used to conduct this secondary analysis likely underrepresent the population of Australian patients with knee OA. This may explain the reduced regression coefficients for previous knee surgery and hence poorer predictive performance of this model compared with the main model developed in this paper. Further, the predictors used in model development were identified from studies predicting TKR in patients with OA and therefore may not be the strongest set of predictors of TKR in patients with knee OA.

Table A.1.

Comparison of patient characteristics between patients with a recorded diagnosis of osteoarthritis (N ​= ​201,462) and knee osteoarthritis (N ​= ​6043) in their EHR

Patient characteristics	Recorded diagnosis of OA (N ​= ​201,462)	Missing	Recorded diagnosis of knee OA (N ​= ​6043)	Missing
	n (%)	n (%)	n (%)	n (%)
Age (years), mean (SD)	67.2 (11.1)	–	68.7 (10.7)	–
Gender
Female	123,376 (61.2)	–	3633 (60.1)	–
Aboriginal and/or Torres Strait Islander	2671 (1.6)	38,148 (18.9)	86 (1.7)	1112 (18.4)
IRSAD quintiles		1228 (0.6)		44 (0.7)
1 (most disadvantaged)	41,600 (20.8)		1350 (22.5)
2	39,539 (19.7)		1120 (18.7)
3	48,122 (24.0)		1452 (24.2)
4 and 5 (most advantaged)	70,973 (35.4)		2077 (34.6)
Patient geographical location∗		1068 (0.5)		41 (0.7)
Major cities of Australia	111,075 (55.4)		3274 (54.6)
Inner regional Australia	62,221 (31.0)		1878 (31.3)
Remote Australia	27,098 (13.5)		850 (14.2)
Healthcare card	53,410 (26.5)	–	1908 (31.6)	–
Member of Department of Veteran's Affairs	8537 (4.2)	–	272 (4.5)	–
BMI (kg/m2), mean (SD)	30.1 (6.5)	137,071 (68.0)	31.8 (7.0)	3826 (63.3)
Prescribing of OA medication/s	57,090 (33.8)	32,591 (16.2)	2205 (44.4)	1074 (17.8)
Count of chronic conditions from CCI, median [IQR]	0 [0, 1]	15,875 (7.9)	0 [0, 1]	289 (4.8)
0 conditions	113,097 (60.9)		3105 (54.0)
1 condition	49,216 (26.5)		1703 (29.6)
2 conditions	16,407 (8.8)		649 (11.3)
3 or more conditions	6867 (3.7)		297 (5.2)
Mental health condition	46,859 (24.2)	8178 (4.1)	1606 (27.3)	157 (2.6)
Previous knee replacement	9432 (4.7)	1264 (0.6)	748 (12.4)	23 (0.4)
Past knee surgery on either knee (excluding TKR)	6070 (3.0)	1007 (0.5)	447 (7.4)	29 (0.5)
Smoking status		13,504 (6.7)		292 (4.8)
Smoker	16,948 (9.0)		385 (6.7)
Ex-smoker	68,239 (36.3)		2037 (35.4)
Non-smoker	102,771 (54.7)		3329 (57.9)
Number of practice visits in year prior to study baseline, median [IQR]	11 [6, 18]		12 [7, 20]
Number of general practice clinics	483		426

AbbreviationsTKR ​= ​total knee replacement; SD ​= ​standard deviation; IRSAD=Index of Relative Socio-economic Advantage and Disadvantage (37); BMI ​= ​body mass index; OA ​= ​osteoarthritis; CCI=Charlson comorbidity index; IQR ​= ​interquartile range.

Notes: Counts and percentages presented unless otherwise stated.

∗Based on Australian Bureau of Statistics (ABS) Australian Statistical Geography Standard (ASGS) remoteness areas (47).

Table A.2.

Characteristics of patients with a recorded diagnosis of knee osteoarthritis in their electronic health record by outcome (N ​= ​6043)

Patient characteristics	TKR (outcome) N ​= ​1095	Death (competing risk) N ​= ​437	No TKR or death N ​= ​4511
	n (%)	n (%)	n (%)
Age (years), mean (SD)	67.1 (8.8)	77.2 (7.9)	68.3 (11.0)
Patient geographical location∗
Major cities of Australia	598 (54.6)	225 (51.5)	2473 (54.8)
Inner regional Australia	346 (31.6)	154 (35.2)	1390 (30.8)
Remote Australia	151 (13.8)	58 (13.3)	648 (14.4)
Prescribing of OA medication/s	543 (49.6)	262 (60.0)	1897 (42.1)
Count of chronic conditions from CCI, median [IQR]	0 [0, 1]	1 [1, 2]	0 [0, 1]
0 conditions	672 (61.4)	103 (23.6)	2465 (54.6)
1 condition	300 (27.4)	149 (34.1)	1352 (30.0)
2 conditions	92 (8.4)	123 (28.2)	468 (10.4)
3 or more conditions	31 (2.8)	62 (14.2)	226 (5.0)
Mental health condition	278 (25.4)	141 (32.3)	1236 (27.4)
Previous knee replacement	165 (15.1)	76 (17.4)	509 (11.3)
Past knee surgery on either knee (excluding TKR)	138 (12.6)	12 (2.8)	297 (6.6)

AbbreviationsTKR ​= ​total knee replacement; SD ​= ​standard deviation; CCI=Charlson comorbidity index; IQR ​= ​interquartile range.

Notes: Counts and percentages presented unless otherwise stated.

∗Based on Australian Bureau of Statistics (ABS) Australian Statistical Geography Standard (ASGS) remoteness areas (47).

Table A.3.

Regression coefficients and subdistribution hazard ratios for model based on patients with a recorded diagnosis of knee osteoarthritis in their electronic health record (N ​= ​6043)

Predictors	Regression coefficient from Fine &">& Gray competing risk model (95% CI)	Subdistribution hazard ratio (SHR) (95% CI)
Age (per unit increase), years	0.42 (0.34–0.50)	1.52 (1.40–1.65)
Age2	−0.0032 (−0.0038 to −0.0026)	0.9968 (0.9962–0.9974)
Prescribing of OA medications (yes)	0.32 (0.20–0.45)	1.38 (1.22–1.56)
Count of chronic conditions from CCI (per unit increase)	−0.19 (−0.27 to −0.11)	0.82 (0.76–0.89)
Mental health condition (yes)	−0.13 (−0.27 to 0.008)	0.88 (0.76–1.01)
Previous TKR (yes)	0.24 (0.07–0.42)	1.27 (1.07–1.52)
Past knee surgery (excluding TKR) (yes)	0.55 (0.37–0.74)	1.74 (1.45–2.09)

AbbreviationsCI ​= ​confidence interval; SHR ​= ​subdistribution hazards ratio from competing risk model; OA ​= ​osteoarthritis; CCI=Charlson Comorbidity Index; TKR ​= ​total knee replacement.

Note: Subdistribution hazard ratio obtained by exponentiating the regression coefficient from the Fine & Gray competing risk model.

Appendix BSupplementary data

The following are the Supplementary data to this article: