Interobservational variation in determining fusion rates in anterior cervical discectomy and fusion procedures

Abstract

The fusion rate represents one of the most commonly used criteria for evaluating the efficacy of spinal surgical techniques and the effectiveness of newly developed instrumentation and spinal implants. Reported fusion rates are not frequently supported by adequate information regarding by whom and how fusion was defined. In our prospective study we examined the fusion rate in patients undergoing first time anterior cervical discectomy and fusion for degenerative disease. Separate, well-defined radiographic fusion criteria were used and the 12-month post-operative X-rays were reviewed independently by a neurosurgeon, a neuroradiologist and an orthopedic surgeon, who were not involved in the patients’ management. The observed fusion rates were 77.3, 87.8 and 84.7% respectively. Statistical analysis demonstrated concordance rates of 87.8, 91 and 91.4% and Kappa coefficients of 0.585, 0.620 and 0.723 for each pair of evaluators. Another set of ratings of the same radiographs, by the same interviewers, was obtained 6 weeks after the initial one. The reported fusion rates were 78.2% for the neurosurgeon, 87.4% for the orthopedic surgeon, and 86.1% for the neuroradiologist. Statistical analysis demonstrated intra-observer concordance rates of 98.7, 92.2 and 97.9% respectively, while the Kappa coefficients were 0.963, 0.677 and 0.907 for each reviewer. Our findings confirm the necessity of defining and describing criteria for fusion whenever this rate is reported in clinical series. The lack of widely accepted, well-defined criteria makes comparison of these results difficult. The development of a well organized, prospective clinical study in which fusion and outcome will be assessed by both clinical and radiographic parameters could significantly contribute to a more accurate evaluation of overall outcome of cervical spinal procedures.

Introduction

In modern spinal surgery, bony fusion is a major goal of stabilizing surgery and one of the most important criteria, if not the most important in predicting patient outcome [34, 42]. The accomplishment of a solid osseous fusion is of major concern in spinal procedures since unsuccessful spinal fusions result in significant morbidity and often require re-operation [37]. The term “fusion” is one of the most commonly used terms in spinal articles and has been widely used for assessing the efficacy of novel surgical spinal techniques, the biomechanical properties of new instrumentation and the safety of newly developed implants [2–4, 6, 7, 11, 13, 23, 25, 30, 32, 33, 35, 36, 38, 39, 42, 43, 46, 49, 50, 52–54].

Although “fusion” is a meritorious term in the spinal literature, its definition has remained quite vague. Bony union is usually implied by the term “fusion” without strict defining criteria, in clinical or radiographic terms [34, 48]. While the vast majority of spinal articles report “fusion rate” or the “degree of fusion” as important factors in determining the success or good outcome, only rarely is the methodology of defining fusion reported; most descriptions are vague and superficial, e.g., “strict radiologic criteria” with no references to by whom and how fusion was defined [3, 6, 11, 12, 30, 33, 36, 41, 42]. No definition of osseous fusion is provided even in classical neurosurgical textbooks [21, 24, 39, 44, 51]. In the absence of strict defining criteria with expert validation and broad training, interobservational variation in assessment of fusion might well represent a significant determining factor in conclusions regarding outcome, especially when physicians of different specialty and background (e.g. neurosurgeons, orthopedic surgeons, and neuroradiologists) have to define fusion, sometimes without information regarding the patient’s clinical outcome [48]. The problem of insufficient inter-observer agreement on the radiographic evaluation of postoperative fusion has been previously identified and remains a complex one with significant clinical implications [15, 48]. Additionally, it has been previously demonstrated that two-dimensional radiographs give a reliable result regarding the presence or absence of fusion in only 70% of cases [5, 9, 15, 31].

In our current communication, we present our experience regarding the magnitude of interobservational and intraobservational variation in determining fusion and our thoughts in utilizing “fusion rate” as an outcome criterion.

Materials and methods

In a prospective clinical study, we examined 303 consecutive patients, during a 6 month period, undergoing anterior cervical discectomy and fusion for cervical spondylosis due to spinal degenerative disease. There were 187 men and 116 women with a mean age of 49.3 years (range 28–65 years). The inclusion criteria were: adult patients up to 65 years undergoing first-time one to three-level anterior cervical discectomy and fusion with autologous bone graft only. Patients older than 65 years, active cigarette smokers, patients with osteopenia evident on their preoperative imaging studies, or history of osteoporosis or osteopetrosis, patients with systemic connective tissue diseases, Paget’s disease, renal dystrophy, chronic renal failure, rickets, osteomalacia,hyperparathyroidism, Addison’s disease, and patients on steroids were excluded from our study. A total of 476 discectomies and fusion procedures were performed with the standard Smith–Robinson anterior approach, utilizing an autologous bone graft harvested from the patient’s anterior iliac crest; 291 patients (96.0%) had also an anterior restricted, constrained cervical plate implanted. All the procedures were performed by the same surgeons (KNF, HFS) who were not involved in the reviewing of any postoperative X-rays.

All of our patients were evaluated at twelve months post-operatively for assessment of fusion. A neurosurgeon, an orthopedic surgeon and a neuro-radiologist, not involved in the patient’s management, independently evaluated the patient’s post-operative plain X-rays; these included static lateral and antero-posterior cervical spine views. None of the examiners had access to the evaluation of the other examiners during the study. Their evaluations were recorded and stored. Six weeks later, the same X-rays were re-evaluated by the same reviewers utilizing the previously used fusion criteria.

The criteria for defining fusion by each of the involved examiners were: The neurosurgeon (JSR) considered fusion as more than five bony bridges crossing the intervertebral space. The orthopedic surgeon (LGN) called fused any level with bone crossing the intervertebral disc and no evidence of fibrous tissue or radiolucency between the vertebral bodies and the implanted allograft. Finally, the involved neuro-radiologist (EZK) called fused levels with at least three bony bridges or a single bony bridge wider than 3 mm crossing the intervertebral space. There were cases in which fusion was observed by all reviewers (Fig. 1) or cases that nonunion was unanimously documented (Fig. 2), while in other cases, reviewers had contradictory interpretations (Fig. 3).

Fig. 1.

Characteristic examples of post-operative lateral X-rays of the cervical spine obtained 12 months post-operatively, demonstrating solid fusion

Fig. 2.

Characteristic examples of post-operative lateral X-rays of the cervical spine obtained 12 months post-operatively, demonstrating non-union

Fig. 3.

Characteristic examples of post-operative lateral X-rays of the cervical spine obtained 12 months post-operatively, demonstrating questionable fusion

Results

The reported fusion rates based on the first reading were 77.3% (368/476) for the neurosurgeon, 84.7% (403/476) for the orthopedic surgeon while the involved neuroradiologist found that fusion had occurred in 87.8% (418/476) operated levels. The respective fusion rates based on the second reading were 78.2% (372/476) for the neurosurgeon, 87.4% (416/476) for the orthopedic surgeon and 86.1% (410/476) for the neuroradiologist (Fig. 4). Statistical analysis of our results was performed by calculating the concordance rates and simple Kappa coefficients, to measure the agreement among the evaluators, a pair at a time, in determining fusion. The analysis was performed on the number of procedures and not the number of patients. In the pair comparing the fusion rating of the orthopedic surgeon and the neuro-radiologist, there were 433 concordant pairs out of 476 ones (91.0%) and the Kappa coefficient was calculated to be 0.620. As regards the pair of the neurosurgeon’s and the orthopedic surgeon’s ratings, the concordant pairs were 435/476 (91.4%) while the Kappa coefficient was 0.723. In contrast, the comparison of the neurosurgeon’s and the neuro-radiologist’s ratings showed concordant pairs in 418/476 levels (87.8%) and the simple Kappa coefficient was 0.585. The same X-rays were interpreted by independent physicians using different criteria, but the results for each pair showed moderate agreement in determining fusion. Furthermore, statistical analysis of the results of each reviewer after the first and the second evaluation of the obtained follow-up radiographs were performed by calculating concordance rates and simple Kappa coefficients in order to measure the intra-observer agreement in radiographically determining fusion. The analysis was again performed on the number of procedures and not on the number of patients. In the evaluations performed by the neurosurgeon, there were 470 concordant pairs out of 476 ones (98.7%) and the Kappa coefficient was calculated to be 0.964. As regards the evaluations performed by the orthopedic surgeon, the concordant pairs were 439/476 (92.2%) and the simple Kappa coefficient was 0.677, while in the ratings performed by the neuroradiologist, there were 466/476 concordant pairs (97.9%) and the simple Kappa coefficient was 0.907.

Fig.4.

Comparative fusion rates for each reviewer, based on the first and second postoperative X-ray evaluations

Discussion

Fusion is traditionally considered as solid osseous union in spinal surgery [34]. Surgical spinal fusion was introduced by Albee [1], for providing mechanical support to vertebrae affected by tuberculosis; at approximately the same time, Hibbs published his technique for spinal fusion for treating progression of scoliosis [26]. The creation of adequate spinal fusion is of significant interest to spinal surgeons, because unsuccessful spinal fusions result in significant morbidity and often, re-operations [37]. Mostly, the overall outcome of a spinal procedure depends upon a solid fusion between selected intervertebral segments [24]. It is generally accepted that for ACDF procedures, radiographically proven fusion is required to obtain ideal clinical outcome [42]. The development of fusion depends on several factors related to the host locally and systemically [25, 32, 34, 35, 37–39, 49, 50]. Numerous clinical series have adequately identified the factors influencing fusion in anterior cervical discectomy and fusion [8, 10, 16, 17, 19, 20, 27–29, 40, 41].

Even though fusion represents a common target for spinal surgeons, the ideal radiographic criteria for defining it remain unclear. The lack of universally accepted radiographic criteria is indicated by the total absence of such criteria in spinal text books [21, 24, 44, 51]. Some attempts have been made in the past for establishing more elaborate radiographic criteria but none of these have been generally accepted [2, 4, 8, 12, 13, 15, 52]. The range of fusion definitions is impressively wide [2, 4, 8, 12, 13, 52]. Bishop et al. [4], in their prospective clinical study defined fusion in ACDF cases as occurring, when bony trabeculae were seen crossing the involved interspace. Delayed union was defined as the failure of the bone to bridge the interspace and the persistence of a linear lucency on the 3-month follow-up radiograph [4]. Nonunion was defined by the same criteria exhibited on the 1-year follow-up X-ray film [4]. Brown et al. [12], in their clinical study, defined fusion as complete bridging of trabeculae between adjacent vertebral bodies and the bone graft. Bose in his retrospective clinical study defined fusion as trabecular bony bridging across the disc space and lack of motion in flexion/extension post-operative X-rays, interchangeably using the terms stability and fusion [8]. An et al. [2] in their prospective multicenter clinical study, described a three tier grading system regarding fusion, in which grade one represented a clear cut pseudoarthrosis, any evidence of a partial or complete radiographic line that is left without motion on flexion-extension views was categorized as grade two ,while grade three represented solid arthrodesis. Likewise, Cauthen et al. [13], in a retrospective study, defined fusion as the presence of bony trabeculae or perigraft lucency without motion on flexion/extension plain X-rays. Interestingly, in their study, three spine radiologists interpreted fusion in a blinded fashion regarding the clinical status of the patients. Similarly, Christensen et al. [15], in their well-designed study regarding inter-observer and intra-observer agreement of radiograph interpretation for postero-lateral lumbar fusion in patients with and without pedicle screw implants, emphasized the need for developing an accurate and quantitive fusion classification system based on radiographic criteria. In fact, they introduced such a system by simplifying the actual area of view and establishing a reliable and reproducible method for the evaluation of postoperative fusion [15]. However, their well-designed and easily applicable fusion classification system, which could be adapted for the evaluation of cervical interbody fusion too, perhaps with a few minor modifications, has not been widely used. Finally, Tuli et al. [48], in their study, emphasized the existence of significant variation in the reported fusion criteria, which are ill-defined, and mostly vaguely descriptive.

Analysis of our results confirmed the insufficient inter-observer agreement in radiographically determining fusion. The relatively low Kappa coefficient (0.585) observed in the comparison between the ratings of the neurosurgeon and the neuroradiologist is indicative of the diverse interpretations of these radiographs due to the different fusion criteria used by these two reviewers. The comparison between the Kappa coefficients of the neurosurgeon/orthopedic surgeon (0.723) and the neuro-radiologist/orthopedic surgeon ratings (0.620) to the one of the neuro-surgeon/neuro-radiologist ratings (0.585) shows that the criteria used by the neurosurgeon and the neuro-radiologist were stricter than the ones applied by the orthopedic surgeon. Indeed, the criteria used by the neurosurgeon and the neuro-radiologist were more quantitive than the descriptive ones used by the orthopedic surgeon. The subjectivity of those criteria used by the orthopedic surgeon was also reflected in the analysis of the intra-observational agreement among the ratings of each reviewer. The orthopedic surgeon’s ratings demonstrated the lowest Kappa coefficient among the three reviewers. The radiographic fusion criteria used by each reviewer were the same criteria they were using in their long-term practices. Christensen et al. in their study, aptly pointed out that inter-observer agreements are often presented in the literature as pair-wise observations (as in our study), which results in higher agreement rates than when agreement among all three observers is demanded [15].

Differing interpretations of diagnostic studies significantly affect patient management in many areas of medicine [14, 18, 45, 47, 55]. The problem of disagreement in the interpretation of diagnostic tests has been nicely outlined in previous publications [14, 18, 45, 47, 55]. Similar to our findings, Stein et al. [45], reported the existence of inter-observational variation in the interpretation of pulmonary angiograms, performed for the detection of pulmonary embolism. The reported interpreter variation in the assessment of pulmonary angiography for the diagnosis of pulmonary embolism may be important in evaluating the extent to which conventional pulmonary angiography can be used as a benchmark for the evaluation of newer diagnostic techniques [45]. Furthermore, criteria for a positive exercise stress test range from 0.5  to 2.0 mm of ST segment depression; criteria utilized directly affects the percentage of patients sent for cardiac catheterization [22]. Likewise, Tandberg. et al. [47], reported the existence of observer variation in measured ST-segment elevation in isolated ECG complexes. This variation could result in the misclassification of candidates for prompt thrombolytic therapy [47]. Similar disagreements among interpreters of diagnostic tests have also been reported in the case of coronary angiograms by other investigators; when the decision for a coronary bypass operation is considered, any degree of diagnostic inconsistency is the cause of major concern [18, 55]. Chamberlain et al. [14], in their study, emphasized the existence of significant disagreement in the interpretation of performed mammograms, a crucial test with significant implications in the early diagnosis of breast cancer. Finally, Christensen et al. [15] examined in a study similar to ours, the intra- and inter-observer agreements in the evaluation of postoperative postero-lateral lumbar fusion in patients with and without pedicle screw implantation by examining postoperative two-dimensional X-rays (antero-posterior and lateral lumbar radiographs). Although their intra-observer agreement was 93% (Kappa 0.78), their inter-observer agreement was only 86% (Kappa 0.53) [15]. It is apparent, that clear, consistent, reproducible standards for interpretation are rare and are desperately needed.

The importance of radiographic fusion in the overall outcome of patients undergoing ACDF has been emphasized [37, 43]. Differences in interpretation of post-operative X-rays, as happened in our study, could lead to further imaging tests (such as CT-scan with sagittal and coronal reconstruction, which can accurately estimate the fusion mass) with substantial financial consequences for patients and the health system. Furthermore, inter-observational difference could potentially lead to a re-operation. The observed variability in our study mandates the use of clear, universally accepted radiographic criteria in defining osseous fusion. These criteria should include some quantitative grading scale such as degree of bony trabeculation. In the meanwhile, in reports of fusion, the issues of by whom and how it is defined, need to be consistently addressed. We strongly believe that in all the spinal clinical studies reporting fusion rates, the disclosure of information regarding the least number of reviewers assessing fusion could be beneficial for understanding how powerful the conclusions of each study could be. Moreover, the subjectivity of defining fusion makes the importance of clinical outcome more important for evaluating the surgical techniques and biomechanical properties of implants or instrumentation. This has been nicely pointed out by An et al. [2] in their study, which demonstrated that there is a discrepancy between the radiographically proven fusion and clinical success rate. Although the original purpose of their study was to compare the fusion rates of allograft–demineralized bone matrix and autograft, they found that the success of surgery depends mostly on the patient selection [2]. Despite the belief that “fusion” represents a more objective, radiographically proven parameter, inter-observational variation and background-related biases can easily alter the observed results and lead to erroneous interpretations with significant impact. While the concordance rates and Kappa values in our study represent good agreement, improvement is possible and needed.

Given the inter-observational variability that we found, the question raised is; how can the precision of the employed radiographic tests and their interpretation be improved?. The need for a well-organized, well-defined, prospective clinical study, evaluating fusion in ACDFs cannot be overemphasized. In this study, evaluation of fusion should be performed by both clinical and radiographic criteria and outcome assessment in a double-blinded fashion; the clinical interpreters should not be involved in the patient’s management and the outcome assessment should be done by using a range of criteria. This should allow the development of criteria for assessment of fusion which combine clinical and radiographic parameters.

Conclusions

The problem of insufficient inter-observer agreement as this, was demonstrated by our study (Kappa coefficient 0.58–0.72) on the radiographic evaluation of postoperative fusion in ACDFs, remains quite perplexing. Although the intra-observer agreement was quite satisfactory for each reviewer in our study, the usage of stricter, quantitative radiographic fusion criteria resulted in higher concordance rates. The necessity for accepting an existent quantitative classification fusion system or designing a new one, especially for cervical procedures, is apparent. In the meanwhile, the clear description of the used radiographic criteria for defining fusion is imperative for interpreting the fusion rates and results of the reported spinal series.