OARSI–OMERACT definition of relevant radiological progression in hip/knee osteoarthritis

Summary

Background:

Joint space width (JSW) evaluated in millimeters on plain X-rays is the currently optimal recognized technique to evaluate osteoarthritis (OA) structural progression. Data obtained can be presented at the group level (e.g., mean ± standard deviation of the changes). Such presentation makes difficult the interpretation of the clinical relevance of the reported results. Therefore, a presentation at the individual level (e.g., % progressors) seems more attractive but requires to determining a cut-off. Several methodologies have been proposed to define cut-offs in JSW: arbitrary chosen cut-off, cut-off based on the validity to predict a relevant end-point such as the requirement of total articular replacement or cut-off based on the measurement error such as smallest detectable difference (SDD).

Objectives:

The objective of this OARSI–OMERACT initiative was to define a cut-off evaluated in millimeters on plain X-rays above which a change in JSW could be considered as relevant in patients with hip and knee OA.

Methods:

The first step consisted in a systematic literature research performed using Medline database up to July 2007 to obtain all manuscripts published between 1990 and 2007 reporting a cut-off value in JSW evaluated in millimeters at either the knee or hip level. The second step consisted in a consensus based on the best knowledge of the 11 experts with the support of the available evidence.

Results:

Among the 506 articles selected by the search, 47 articles reported cut-off of JSW in millimeters. There was a broad heterogeneity in cut-off values, whatever the methodologies or the OA localization considered (e.g., from 0.12 to 0.84 mm and from 0.22 to 0.78 mm for Knee (seven studies) and hip (seven studies), respectively when considering the data obtained based on the reliability). Based on the data extracted in the literature, the expert committee proposed a definition of relevant change in JSW based on plain X-rays, on an absolute change of JSW in millimeters and on the measurement error e.g., calculation of the SDD using the Bland and Altman technique. The results of the analysis of JSW should be expressed in terms of a dichotomous variable (e.g., progressors yes/no): a patient with a change in JSW during the study over such SDD will fulfill the definition of “progressor”. Moreover, the pilot study aimed at evaluating the measurement error should be designed to reflect the different characteristics of the primary study in which the analysis of the radiological findings will be based on (patient’s characteristics, centers characteristics, readers).

Conclusion:

This initiative based on both an Evidence Based Medicine (Systematic Literature Research) and Expert Opinion approach resulted in a proposal of definition of relevant radiological progression in OA to be used as end-point in clinical trials and also recommendations on the conduct of the reliability study allowing such definition.

Osteoarthritis (OA) is a major cause of disability worldwide and has a significant impact on the patients’ health-related quality of life. With the development of disease modifying OA drugs (DMOADs) with a potential structure-modifying effect, technologies such as plain radiographs and/or magnetic resonance imaging (MRI) are used to assess structural progression in hip and knee OA. Using these imaging techniques, several therapeutical trials have recently reported significantly slower structural progression for patients receiving various DMOADs compared to placebo. The medical community is not yet convinced of the clinical relevance of such results.

The capacity of a drug to prevent or to delay a hard clinical end-point such as the requirement of total articular replacement might be considered more relevant. This end-point has been criticized because the decision to perform total articular replacement is influenced by the severity of pain and functional limitation due to OA, covariates such as age, co-morbidities, and the patient’s willingness to undergo the procedure. This criticism has led to the suggestion that the appropriate end-point might be the time to fulfill the criteria for considering total articular replacement rather than time to the actual surgery¹. In 2004, OARSI (Osteoarthritis Research Society International) and OMERACT (Outcome Measures in Rheumatology) agreed to set-up a task force to establish a set of criteria to determine if a patient qualified for total articular replacement (knee/hip). The set of criteria would be used as an end-point in clinical trials evaluating potential DMOADs¹. It was anticipated that these criteria would reflect the three main domains, i.e., pain, functional impairment and structural damage² and would be combined into an overall index. Consequently, the task force was sub-divided among three sub-tasks:

• a sub-task force to propose a tool to evaluate pain;

• a sub-task force to propose a tool to evaluate function;

• a sub-task force to propose a tool to evaluate structure.

The proposed tools to evaluate these criteria would have to fulfill the OMERACT filter i.e., truth, discrimination and feasibility. In December 2006, the sub-task force on structure finished evaluating validated measures of joint damage and concluded that, at the present time, joint space width (JSW) on a plain X-ray expressed in millimeters was optimal for use in OA clinical trials, at least for the next few years, to evaluate the severity of structural damage³. However, JSW expressed in millimeters is a continuous variable that does not make it possible to classify patients as “progressors” or non-”progressors” based on changes in JSW. In fact, it is more clinically relevant to express the results of analyses in terms of individual patients. For example, the percentage of individual patients who have structural progression during a study is more meaningful than the average change in JSW. Therefore the composite index based on the set of criteria should be defined at the individual level, i.e., a cut-off for change in JSW above which a patient would be classified as a “progressor”.

Based on the conclusion of the above mentioned sub-task force, the objective of the present OARSI–OMERACT initiative was to determine a cut-off expressed in millimeters above which a change in JSW evaluated on plain X-rays would be considered relevant in patients with hip and knee OA.

Methods

Relevant change in JSW in patients with hip or knee OA was determined in a two-step procedure. The first step was a systematic literature search and the second step was a consensus based on the opinion of the OARSI experts after reviewing the available evidence.

SYSTEMATIC LITERATURE SEARCH

Selection process

The Medline database was systematically searched through July 2007 to obtain all manuscripts published between 1990 and 2007 reporting a cut-off value for JSW expressed in millimeters for either the knee or hip. The initial search was performed in May 2007 using the following combination of key words “Osteoarthritis AND (Hip or Knee) AND joint space”. The following limits were used: human and reports published between 1990 and 2007. In addition, Reference sections of the papers initially detected were further searched manually to identify additional relevant reports. Subsequently, regular updates were performed up to July 2007. From the resulting initial pool of abstracts, only articles reporting a cut-off for JSW expressed in millimeters on plain X-rays over any set interval of time were finally retained. Reviews, guidelines, editorials and case reports were excluded as well as articles based on secondary hip or knee OA.

Cut-offs methodologies

Several approaches can be used to determine a cut-off point above which changes in JSW observed during a study would be considered indicative of radiological progression. Significant progression can be defined on the basis of:

• An intuitive global assessment of disease progression given by an expert, based on knowledge of and experience with disease progression.

• Statistical methods based on measurement error involved calculating the mean of the differences between two analyses to evaluate reliability, expressed as either:

  • The coefficient of variation (CV), defined as the standard deviation (SD) of a set of JSW measurements multiplied by 100 and divided by their mean.
  • The smallest detectable difference (SDD): in the Bland and Altman method, for each patient, the change from the baseline to the post-treatment JSW assessment is plotted against the mean of the two assessments including the limits of agreement (defined as mean change ±1.96 × SD of the change). A change in JSW over time greater than the SDD, i.e., 1.96 × SD of the change from baseline to post-treatment measurements (cut-off point), probably reflects organic change⁴. Thereafter, a radiological relevant change is defined by a change which is over this measurement error.

• Predictive models based on evidence derived through the predictive validity of an outcome measure (on the basis of maximal sensitivity and specificity using receiver operating characteristic (ROC) curve analysis): for example, determination of a cut-off point above which a change in JSW could be considered clinically relevant in patients with knee/hip OA on the basis of the prediction of subsequent total articular replacement.

Moreover, it was anticipated that cut-offs used might differ between randomized controlled trials (RCT) and longitudinal epidemiological studies.

Data extraction

The full text of each retained article was reviewed and the reviewer (PO) extracted the following information: first author, year of publication, journal name, impact factor, localization (hip/knee), proposed cut-off and methodologies used to determine the cut-off. For RCT and epidemiological studies, study duration and percentage of progressors were also extracted. For articles in which the cut-off was based on a measurement error approach, methods for evaluation of reliability (e.g., SDD or CV) and radiographic procedure were also extracted.

Because of the heterogeneity of the studies, populations and radiological techniques by which JSW was measured, the results (population data and cut-off value) were not pooled.

EXPERT OPINION

The steering committee comprised 11 experts (i.e., senior researchers), one librarian (GUW) and one research fellow (PO) from four countries. All of the experts have been involved in OA longitudinal epidemiological studies and/or OA clinical trials and in radiographic assessment of OA for many years. The results of the literature search were summarized and disseminated to the expert committee. Based on their discussion of the literature review, the OARSI experts proposed a cut-off expressed in millimeters above which a radiological change in JSW could be considered relevant in patients with hip and knee OA participating in a clinical trial.

Results

SYSTEMATIC LITERATURE SEARCH

The results of the manuscript selection process are reported in Fig. 1. Results from RCT and epidemiological studies (Tables I and II) where developing a cut-off (based on intuitive expert opinion or based on reliability) was secondary to other objective are separated from results of studies whose only purpose was to establish a cut-off (based on predictive validity (Table III) or reliability (Tables IV and V)).

Fig. 1.

Flow chart: manuscripts selection process.

Table I.

Proposed cut-off values of radiological JSW in millimeters used in RCT and epidemiological studies in hip OA

Author	Year	Study*	Cut-off mm	Rationale†	Study duration months	Groups
						Placebo			Active
						N	% Progressors		N (‡)	% Progressors
							Compl	ITT		Compl	ITT
Pavelka	2000	1	0.5	1	60	60	62	22	60 (3)	24	10
Chevalier	2001	2	0.6	2	12				29 (3)	34.1
Dougados	2001	1	0.5	2	36	225	62.3	60.4	221 (3)	50.7	47
Lane	2004	2	0.5	1	96				745 (2)	46
Reijman	2005	2	1	1	72				1905 (1)	13.1
Reijman	2007	2	1	1	72				1676 (1)	8.7

ITT: intent-to-treat population; Compl: completers’ population.

^*Study: 1 = randomized placebo controlled trial; 2 = epidemiological study.

^†Rationale: 1 = arbitrary proposal; 2 = proposal based on the reliability of technique of measurement of JSW.

^‡Patients: 1 = population; 2 = hip OA population; 3 = hip OA painful-population.

Table II.

Proposed cut-off values of radiological JSW in millimeters used in RCT and epidemiological studies in knee OA

Author	Year	Study*	Cut-off mm	Rationale†	Study duration months	Groups
						Placebo			Active
						N	% Progressors		N (‡)	% Progressors
							Compl	ITT		Compl	ITT
Sharif	1995	2	2	1	60				94 (3)	27.6
Dieppe	1997	2	2	1	36				145 (3)	14.4
Pavelka	2000	1	0.5	1	60	140	31	20.8	140 (3)	36	26
Reginster	2001	1	0.5	1	36	106	42.2	30	106 (3)	22	15
Pavelka	2002	1	0.5	1	36	101	25.5	14	101 (3)	7.6	5
Mazzuca	2006	1	0.5	1	16	191	21.5		188 (3)	15.4
					30	180	31.7		181 (3)	26.0
Mazzuca	2006	1	1.0	1	16	191	7.3		188 (3)	3.7
					30	180	7.2		181(3)	9.4
Pham	2004	1	0.5	1	12	85	20.3	19.1	122 (3)	17.7	16.5
Pham	2004	1	0.5	1	24	74	37.2	26	82 (3)	43	29
Spector	2005	1	0.75	2	12	99	6	5	90 (3)	0	0
Mikesky	2006	1	0.5	2	30	60	41	?	45 (2)	42	?
Mazzuca	2006	2	0.5	2	30				267 (2)	31
Raynauld	2006	2	0.6	2	24				110 (3)	13
Bingham	2006	1	0.6	1	24	622	13	11.5	1861 (3)	13	12
Bruyere	2007	2	0.3	1	12				62 (2)	20.5
Reijman	2007	2	1	1	72				532 (2)	21.8

ITT: intent-to-treat population; Compl: completers’ population.

^*Study: 1 = randomized placebo controlled trial; 2 = epidemiological study.

^†Rationale: 1 = arbitrary proposal; 2 = proposal based on reliability of technique of measurement of JSW.

^‡Patients: 1 = population; 2 = knee OA population; 3 = knee OA painful-population.

Table III.

Proposed cut-off values of radiological JSW in millimeters used in studies based on predictive validity in hip and knee OA

Author	Year	Location	Cut-off mm	X-rays interval*, weeks	Study Duration†, weeks	End-Point‡
Maillefert	2002	Hip	0.4	104	156	1
Maillefert	2002	Hip	0.4	104	0	2
Bruyere	2005	Knee	0.7	36	60	1

^*X-ray interval = duration in weeks between the two films.

^†Study duration = duration in weeks between the date of the second X-ray and the time the end-point is collected.

^‡End-point: 1 = total articular replacement; 2 = expert opinion.

Table IV.

Proposed cut-off values of radiological JSW in millimeters based on the measurement error (e.g., Bland and Altman method with SDD) in hip OA

Author	Year	Analysis*	Film†	Bland and Altman method
				N	Mean change	SD change	SDD mm (cut-off)	Limits of agreement
								Min	Max
Dougados	1996	3(52)	1	30	0.01	0.28	0.56	−0.55	0.57
Auleley	1998	1	1 (standing)	46	0.02	0.11	0.22	−0.2	0.24
		1	1 (supine)	46	−0.002	0.13	0.26	−0.262	0.258
Auleley	2001	2	1	37	−0.02	0.23	0.46	−0.48	0.44
Conrozier	2001	1	1	28	0	0.15	0.3	−0.3	0.3
Maillefert	2002	3 (156)	1	35	0.028	0.39	0.78	−0.752	0.808
		3 (156)	1 (computer)	35	0.077	0.34	0.67	−0.593	0.747
Hilliquin	2002	1	1	100	−0.03	0.22	0.44	−0.47	0.41
Maheu	2005	3 (156)	1	50	0.02	0.18	0.37	−0.35	0.39
		3 (156)	2	50	0.03	0.16	0.31	−0.28	0.34
		3 (156)	3	50	0.03	0.16	0.31	−0.28	0.34

^*Analysis: 1 = score of a single film in a single patient; 2 = score of two films in a single patient taken less than 2 days a part; 3 = score of the changes in a single patient between two visits and () the number of weeks between these two visits.

^†Film: 1 = pelvic; 2 = hip AP; 3 = hip lateral.

Table V.

Proposed cut-off values of radiological JSW in millimeters based on the measurement error (e.g., Bland and Altman method with SDD in knee OA)

Author	Year	Analysis*	Film†	Bland and Altman method
				N	Mean change	SD change	SDD mm (cut-off)	Limits of agreement
								Min	Max
Ravaud	1996	3 (52)	1	55	−0.01	0.43	0.84	−0.85	0.83
		1	1	55	0.2	0.33	0.64	−0.44	0.84
Ravaud	1998	2	1	36	−0.03	0.32	0.62	−0.65	0.59
Mazzucca	2002	2	1	76	0	0.3	0.6	−0.6	0.6
Buckland-Wright	2003	2	1	266	−0.012	0.169	0.33	−0.342	0.318
Vignon	2003	1	2	36	0	0.25	0.5	−0.5	0.5
Conrozier	2004	1	1 (semi-auto)	127	0.01	0.14	0.27	−0.26	0.28
		1	1 (auto)	127	0.005	0.06	0.12	−0.115	0.125
Lavalley	2005	2	3	355	−0.01	0.43	0.84	−0.85	0.83

^*Analysis: 1 = score of a single film in a single patient; 2 = score of two films in a single patient taken less than 2 days a part; 3 = score of the changes in a single patient between two visits and if yes precise () the number of weeks between these two visits.

^†Film: 1 = antero-posterior view; 2 = flexed view; 3 = lateral view.

The systematic search identified 504 publications related to hip or knee OA and joint space. After abstract and title review, 169 manuscripts were further evaluated concerning radiological change in JSW. They included reports of RCT (43), cohort studies (65), and studies evaluating reliability (58) or predictive validity (3). Another 122 articles were withdrawn during this process (Fig. 1), most of which did not report a JSW cut-off in millimeters, leaving a final group of 47 articles which reported a JSW cut-off in millimeters, among which half were based on the measurement error approach (reliability). The manual search of the Reference sections did not identify any additional reports.

Table I (hip OA^5–10) and Table II (knee OA^6,7,11–23) show the cut-offs reported in the articles on RCT and epidemiological studies. Table III shows the cut-offs based on predictive validity^24–26. Table IV (hip^27–33) and Table V (knee OA^34–49) show SDDs based on the Bland–Altman reliability method. Only studies reporting SDD are shown. Studies that reported cut-offs based on the CV are not shown because this procedure is not recommended as discussed below. The quality of selected manuscripts (reporting a cut-off for JSW expressed in millimeters) was assessed using the 2006 impact factors. These articles were published in various journals with a mean ± SD impact factor of 6.39 ± 2.12 (hip) and 6.78 ± 4.96 (knee).

EXPERT’S OPINION

Table VI summarizes the outcome of the panel’s discussion, that is, a consensus based on the expert opinion of the researchers after reviewing the results of the systematic literature search.

Table VI.

OARSI–OMERACT definition of relevant structural progression in hip/knee OA

1	Structural progression is optimally defined by plain radiological evaluation of JSW in millimeters
2	The results of the analysis of JSW should be expressed in terms of a dichotomous variable (e.g., progressors yes/no)
3	An absolute change in JSW over a predefined threshold is defining a progressor.
4	The threshold above which a change in JSW can be considered as relevant should be based on the evaluation of the measurement error of the radiological technique.
5	The Bland and Altman technique is the most appropriate one to evaluate the measurement error of the radiological technique evaluating JSW.
6	The pilot study aimed at evaluating the measurement error should be designed to reflect the different characteristics of the primary study in which the analysis of the radiological findings will be based on (patient’s characteristics, centers characteristics, readers …).

Discussion

CHOICE OF MODALITY

The panel of experts confirmed a previous decision by the sub-task force on structure that structural progression of OA should be evaluated via plain X-rays. At this time, only radiography has been adequately validated to assess OA structural progression in large multi-center phase III trials. The initial sub-task force concluded in 2006³ that other techniques such as MRI pose difficult practical issues in multi-center international trials, such as lack of a validated scoring system available in the public domain and insufficient data concerning metrological properties. The panel of experts agrees that MRI might eventually be recommended to evaluate structural change in hip and knee OA if these difficulties can be resolved. The panel of experts also confirmed the previous decision to quantify change in JSW in millimeters.

CHOICE OF JSW CRITERIA

It was agreed that results should be presented at the individual level as a dichotomous variable (e.g., percentage of “progressors”) rather than at the group level using a continuous variable (e.g., mean changes in JSW) as being more meaningful to clinicians. It was recognized that a categorical variable based on a cut-off in JSW would have less statistical power than a continuous variable. However, a dichotomous variable (Radiological Progression: Yes/No) is needed for the domain structure of the final composite index (planned set of criteria) even though it decreases statistical power.

Thereafter, the panel of experts focused the discussion on the relative interest of choosing an absolute cut-off value at a certain point of time vs a change in the JSW vs baseline. The group of experts further agreed that the set of criteria to qualify for total articular replacement in knee/hip OA should be based on change in JSW (e.g., JSW change >0.5 mm) rather than absolute JSW (e.g., JSW <1 mm) or relative change (e.g., change >30%). This decision was mainly due to the potential lack of clinical relevance of relative change from a small baseline value (for example, a 33% decrease in two patients with a baseline JSW of 2.4 and 1.2 mm would be 0.8 and only 0.4 mm, respectively).

CHOICE OF CUT-OFF METHODOLOGY

The panel felt that predictive validity would be the best method to establish a cut-off defining relevant structural progression. However, they agreed that at the moment there are not currently sufficient published results on predictive validity, in particular for knee OA. Thus, although clinically relevant, it is therefore too early to recommend change in JSW based on predictive validity as a criterion to assess the relevance of structural progression. Consequently, the panel recommended that radiological progression should be defined in terms of measurement error, i.e., a change greater than the combined variability in repeating the imaging procedure and in the measurement process itself.

CALCULATION OF SDD

The committee recommended a cut-off for radiological progression in terms of the SDD determined from the Bland and Altman method that requires calculation of the mean and SD deviation of the difference in JSW between two successive analyses. Radiological progression is defined as a change in JSW greater than the upper bound of the two-sided 95% confidence interval around the mean difference. Because of the numerous pitfalls associated with the CV, the experts agreed to not recommend this statistical technique.

However, the experts noted that several different procedures used to measure JSW have been used to calculate the SDD in the literature, all based on the Bland–Altman approach, and that the resulting SDD values vary widely among well-designed studies (0.22–0.78 for hip OA, 0.12–0.84 for knee OA). Such heterogeneity might be explained by the fact that some studies evaluated the whole process of data collection (i.e., from the patient’s positioning to the scoring of the film) while others evaluated a single step of the procedure (e.g., scoring system). Furthermore, the huge difference in the proposed cut-offs in epidemiological studies vs in RCT is probably explained by the quality of the film.

According to the experts, the SDD to define a cut-off for radiological progression should reflect the variability of all facets of the radiological measurement (i.e., patient’s positioning, radiographic procedure and scoring system) and not only one facet (e.g., scoring system). The patient should come twice to the same department of radiology with a repositioning of both the machine and the patient between the two examinations. Cut-offs defined by such reproducibility study are closely dependent on all sources of variability in JSW measurement. Therefore, standardization of radiographic procedure and joint positioning or improvement in the scoring process are key factors in order to minimize such cut-offs.

In light of the observed heterogeneity of SDD from study to study, the panel of experts felt that a cut-off should apply only to the population that was sampled to determine the SDD, using the same radiographic procedure, the same measuring instrument and reader. It is therefore appropriate, before beginning any study of a DMOAD, to conduct a pilot study to evaluate the SDD in a sub-group of patients with similar characteristics to the patients who will be enrolled in the study, particularly in terms of baseline JSW. The centers participating in the SDD pilot study should also be representative of those participating in the main study, and should not all be the most experienced centers.

As regards progression defined as the JSW exceeding some cut-off based on the SDD, the experts underlined that the probability of progressing clearly increases with longer patient follow-up. Given the potentially low probability of progression with a short follow-up, there are two options to demonstrate a statistically significant effect of an experimental DMOAD:

• A larger sample size and a short follow-up. This scenario has the advantage of minimizing loss of patients to follow-up.

• A longer follow-up. This option is more consistent with the natural history of the disease (i.e., a slow process which can be modified only by a “substantial” period of treatment).

This study was conducted to develop a definition of radiological progression of hip and knee OA based on both the Evidence Based Medicine (Systematic Literature Search) and Expert Opinion approach. This initiative is part of a larger initiative to develop a set of criteria that would qualify a subject for total articular replacement. Satisfaction of these criteria could be an end-point in clinical trials evaluating potential DMOADs.

Neither the systematic literature research nor consideration of expert’s opinion yielded unique and universal JSW change cut-offs for either hip OA or knee OA that clearly identified progressors. Instead, the panel proposed that a cut-off should be determined for each study based on a pilot study that assesses the inherent variability of the measurement process in a representative sample of the studied population. The cut-off should be the SDD calculated from measurements using X-rays repeated over a short period.

This approach to defining progression should be revisited as relevant results are obtained in ongoing and/or planned studies in this field.