Abstract
Objective: To assess the reproducibility of the Outerbridge and the French Society of Arthroscopy classifications between different observers, and to establish a comparison between them. Method: Thirty videos on randomly selected knee arthroscopy procedures demonstrating chondral lesions were used. These were classified by six observers: two third-year orthopedics residents and four orthopedic surgeons, of whom two were knee surgery specialists. The intraobserver and interobserver reliability was evaluated by means of the kappa index. Results: The result from the complete evaluation on the Outerbridge classification with all the observers gave a kappa index of 0.434411. For the classification proposed by the French Society of Arthroscopy, the kappa index was 0.45166. Conclusion: The Outerbridge and French Society of Arthroscopy classifications for chondral lesions are moderately reproducible between observers. Comparing the two classifications, the proposal from the French Society of Arthroscopy was shown to be more reproducible, and the authors suggest that this classification should be used preferentially in clinical practice for evaluations on chondral lesions of the knee.
Keywords: Knee Injuries/classification, Knee Injuries/pathology, Arthroscopy
INTRODUCTION
A combination of mechanical factors and loss of the structural proteins that make up the cartilage is responsible for osteoarthrosis(1), which is a highly prevalent pathological condition of degenerative nature that is especially seen in the knee joint.
The joint cartilage is a type of tissue with the capacity to deal with large forces applied over many cycles, but it has a low capacity for regeneration after injury(1).
Traditionally, osteoarthrosis of the knee is diagnosed through clinical signs resulting from the inflammatory and/or mechanical process, accompanied by specific radiological abnormalities such as diminished joint space, subchondral sclerosis and osteophytes. However, there may be very little evidence of such abnormalities, particularly during the initial stages of the degenerative process, thus leading to delays in diagnosing this condition.
Studies have revealed the presence of advanced femorotibial and/or femoropatellar arthrosis through arthroscopy of the knee, in cases with normal radiological findings(2).
There are some radiographic classifications for osteoarthrosis of the knee, but they have been shown to be imprecise, especially in the initial stages.
Arthroscopic evaluation of the knee presents the characteristic of direct viewing of the joint surface, thus enabling greater detailing of the chondropathy, such as size, depth, consistency and lesion location(3).
Some arthroscopic classification systems for chondral lesions of the knee have been described in the literature. Among these is the Outerbridge system, which was originally described through direct viewing using arthrotomy, and the classification of the French Society of Arthroscopy(4).
In 1961, Outerbridge classified patellar chondral lesions into four grades, such that grade I was softening; grade II was fragmentation/fissure of 1.25 cm or less; grade III was fragmentation/fissure greater than 1.25 cm; and grade IV was bone erosion(5).
In 1994, the French Society of Arthroscopy proposed the classification: grade I, softening; grade II, superficial fissure; grade III, deep fissure; and grade IV, bone exposure(6).
Reproducibility between observers is essential for any type of classification(6), and arthroscopy should be considered to be an important measurement method for assessing osteoarthrosis of the knee(7).
The purpose of this study was to assess the reproducibility of the Outerbridge and French Society of Arthroscopy classifications between observers.
MATERIALS AND METHODS
Between the months of October 2008 and May 2009, at the Orthopedics and Traumatology Service of São José University Hospital, 30 arthroscopy procedures on knees were performed in order to treat ligament or meniscal lesions, with inventories of the medial, lateral and femoropatellar compartments. All the procedures were recorded on DVD and edited to form 30 clips of five seconds each, totaling 30 images. These images were assessed and classified by six observers, of whom two were titular members of the Brazilian Society of Knee Surgery, two were titular members of the Brazilian Society of Orthopedics and Traumatology and two were third-year medical residents, using the Outerbridge and French Society of Arthroscopy classifications for chondral lesions of the knee.
These assessments were performed on a special form that was filled out by one of the researchers. In order to minimize any bias due to difficulty in interpretation or any possible forgetfulness, the classifications were described on the response sheets, together with schematic drawings of the respective classifications, which were handed over to each observer at the time of the image assessments. There was no time limit on making the image classifications.
The data obtained were analyzed by a statistician, using the chi-square test and kappa coefficient, which were first used by Fleiss and Cohen(8). The kappa coefficient was used to assess the concordance between the observers within the same sample unit. This ranged from -1, when all the observers disagreed regarding all the assessments to +1, when there was complete agreement.
RESULTS
The series of images were analyzed using the Outerbridge classification (four types) and the French Society of Arthroscopy classification (four types).
The result from the complete evaluation of the Outerbridge classification with all of the observers was an inter-observer kappa index (k) of 0.434411, which produced a p-value = 0. Analysis on the two residents alone gave k = 0.395973. Analysis on the orthopedists alone gave k = 0.165379. Analysis on the knee specialists alone gave k = 0.140127 (Table 1).
Table 1.
Inter-observer kappa concordance index for the Outerbridge classification.
| Orthopedist 1 | Orthopedist 2 | Resident 1 | Resident 2 | Knee specialist 1 | Knee specialist 2 | |
|---|---|---|---|---|---|---|
| Orthopedist 1 | 1 | 0.165379 | 0.363636 | 0.217791 | 0.049296 | 0.437500 |
| Orthopedist 2 | 1 | 0.010989 | 0.041553 | 0.220183 | 0.269406 | |
| Resident 1 | 1 | 0.395973 | 0.127907 | 0.361702 | ||
| Resident 2 | 1 | 0.122807 | 0.212963 | |||
| Knee specialist 1 | 1 | 0.140127 | ||||
| Knee specialist 2 | 1 | |||||
| Descriptive statistics for the 15 indices found | ||||||
| Minimum | Median | Mean | Maximum | Standard deviation | ||
| 0.01099 | 0.21296 | 0.20915 | 0.43750 | 0.13408 | ||
The highest value for the intra-observer concordance index was for orthopedist 1, with k = 0.509002, and the lowest was for resident 2, with k = -0.064516, as presented in Table 2. The mean intra-observer concordance was k = 0.2955 (Table 2).
Table 2.
Intra-observer concordance index for the Outerbridge classification.
| Intra-observer kappa concordance index (calculated by comparing the observer’s first and second evaluations on the same 30 patients) | ||
|---|---|---|
| Observer | Kappa coefficient (K) | P-value |
| Orthopedist 1 | 0.509002 | 2.5207e-05 |
| Orthopedist 2 | 0.40458 | 0.000145 |
| Resident 1 | 0.233227 | 0.026750 |
| Resident 2 | -0.064516 | 0.700497 |
| Knee specialist 1 | 0.244604 | 0.044259 |
| Knee specialist 2 | 0.447853 | 3.48437e-05 |
In relation to the classification proposed by the French Society of Arthroscopy, we found the following data. The inter-observer kappa was k = 0.45166, thus resulting in p = 0, among all the six observers. Among the different groups, the index was greatest for the orthopedists, with k = 0.339623, followed by the knee surgeons, with k = 0.338983. For the residents, the concordance coefficient was k = 0.22619. These results are presented in Table 3.
Table 3.
Inter-observer kappa concordance index for the classification of the French Society of Arthroscopy
| Orthopedist 1 | Orthopedist 2 | Resident 1 | Resident 2 | Knee specialist 1 | Knee specialist 2 | |
|---|---|---|---|---|---|---|
| Orthopedist 1 | 1 | 0.339622 | 0.303483 | 0.062500 | 0.148936 | 0.360655 |
| Orthopedist 2 | 1 | 0.145729 | 0.090909 | 0.117647 | 0.296875 | |
| Resident 1 | 1 | 0.226190 | 0.038462 | 0.150000 | ||
| Resident 2 | 1 | 0.062500 | 0.018182 | |||
| Knee specialist 1 | 1 | 0.338983 | ||||
| Knee specialist 2 | 1 | |||||
| Descriptive statistics for the 15 indices found | ||||||
| Minimum | Median | Mean | Maximum | Standard deviation | ||
| 0.01818 | 0.14894 | 0.18004 | 0.36066 | 0.12045 | ||
The highest value for the intra-observer concordance index was for orthopedist 1 (k = 0.540034), and the lowest value was for resident 2 (k = 0.033742), as shown in Table 4. The mean kappa index for the intra-observer evaluation was k = 0.3165.
Table 4.
Intra-observer concordance index for the classification of the French Society of Arthroscopy.
| intra-observer kappa concordance index (calculated by comparing the observer’s first and second evaluations on the same 30 patients) | ||
|---|---|---|
| Observer | Kappa coefficient (K) | P-value |
| Orthopedist 1 | 0.540034 | 2.55338e-05 |
| Orthopedist 2 | 0.39441 | 0.000306 |
| Resident 1 | 0.170732 | 0.084768 |
| Resident 2 | 0.033742 | 0.382217 |
| Knee specialist 1 | 0.440299 | 0.001714 |
| Knee specialist 2 | 0.322581 | 0.00428 |
DISCUSSION
Several authors have expressed the opinion that in assessing the reliability of inter-observer concordance, there is a need to incorporate concordance due to chance in the evaluation9, 10. Kappa is a coefficient of concordance that corrects for the error due to chance and is used to determine the intra and interobserver variation. It is used when two observers separately classify a sample of objects using the same category scale.
Landis classified concordance as: poor (less than 0), slight (0-0.2), weak (0.21-0.4), moderate (0.41-0.6), substantial (0.61-0.8), almost perfect (0.81-1.0).
We found some articles that assessed the reproducibility of the Outerbridge and French Society of Arthroscopy classification. Cameron et al(11) concluded that the Outerbridge classification is moderately accurate when used to grade chondral lesions arthroscopically.
In the present study on the Outerbridge and French Society of Arthroscopy classifications, we found kappa values of 0.434411 and 0.45166 respectively. According to Landis, this indicates that the classifications have moderate concordance.
In comparing the means within the groups, we could see that in the groups using the French Society of Arthroscopy classification, the means were higher than in the other groups.
CONCLUSION
On the basis of the present assessment, it can be concluded that the French Society of Arthroscopy and Outerbridge classifications have moderate concordance between observers.
Comparing the two classifications, the proposal from the French Society of Arthroscopy was shown to be more reproducible, and the present authors suggest that this classification should be used preferentially for assessing chondral lesions of the knee in clinical practice.
Footnotes
Work performed in the Orthopedics and Traumatology Service, São José University Hospital (FCMMG-BH).
The authors declare that there was no conflict of interest in conducting this work
REFERENCES
- 1.Fife RS, Brandt KD, Braunstein EM, Katz BP, Shelbourne KD, Kalasinski LA. Relationship between arthroscopic evidence of cartilage damage and radiographic evidence of joint space narrowing in early osteoarthritis of the knee. Arthritis Rheum. 1991;34(4):377–382. doi: 10.1002/art.1780340402. [DOI] [PubMed] [Google Scholar]
- 2.Brismar BH, Wredmark T, Movin T, Leandersson J, Svensson O. Observer reliability in the arthroscopic classification of osteoarthritis of the knee. J Bone Joint Surg BR. 2002;84(1):42–47. doi: 10.1302/0301-620x.84b1.11660. [DOI] [PubMed] [Google Scholar]
- 3.Brandt KD, Fife RS, Braunstein EM, Katz B. Radiographic grading of the severity of knee osteoarthritis: relation of the Kellgren and Lawrence grade to a grade based on joint space narrowing, and correlation with arthroscopic evidence of articular cartilage degeneration. Arthritis Rheum. 1991;34(11):1381–1386. doi: 10.1002/art.1780341106. [DOI] [PubMed] [Google Scholar]
- 4.Collins DH. The pathology of articular and spinal diseases. Edward Arnold; London: 1949. [Google Scholar]
- 5.Outerbridge RE. The aetiology of chondromalacia patellae. J Bone Joint Surg BR. 1961;43:752–757. doi: 10.1302/0301-620X.43B4.752. [DOI] [PubMed] [Google Scholar]
- 6.Flikkilä T, Nikkola-Sihto A, Kaarela O, Pääkkö E, Raatikainen T. Poor interobserver reliability of AO classification of fractures of the distal radius. J Bone Joint Surg Br. 1998;80(4):670–672. doi: 10.1302/0301-620x.80b4.8511. [DOI] [PubMed] [Google Scholar]
- 7.Dougados M, Ayral X, Listrat V, Gueguen A, Bahuaud J, Beaufils P. The SFA system for assessing articular cartilage lesions at arthroscopy of the knee. Arthroscopy. 1994;10(1):69–77. doi: 10.1016/s0749-8063(05)80295-6. [DOI] [PubMed] [Google Scholar]
- 8.Fleiss JL, Cohen J. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Meas. 1973;33(3):613–619. [Google Scholar]
- 9.Felson DT, Naimark A, Anderson J, Kazis L, Castelli W, Meenan RF. The prevalence of knee osteoarthritis in the elderly. The Framingham Osteoarthritis Study Arthritis Rheum. 1987;30(8):914–918. doi: 10.1002/art.1780300811. [DOI] [PubMed] [Google Scholar]
- 10.Dillon CF, Rasch EK, Gu Q, Hirsch R. Prevalence of knee osteoarthritis in the United States: arthritis data from the Third National Health and Nutrition Examination Survey 1991-94. J Rheumatol. 2006;33(11):2110–2112. [PubMed] [Google Scholar]
- 11.Cameron ML, Briggs KK, Steadman JR. Reproducibility and Reliability of the Outerbridge Classification for Grading Chondral Lesions of the Knee Arthroscopically. Am J Sports Med. 2003;31(1):83–86. doi: 10.1177/03635465030310012601. [DOI] [PubMed] [Google Scholar]