Interobserver agreement for the spine instability neoplastic score varies according to the experience of the evaluator

Abstract

OBJECTIVES:

To evaluate the interobserver agreement for the Neoplastic Spine Instability Score (SINS) among spine surgeons with or without experience in vertebral metastasis treatment and physicians in other specialties.

METHODS:

Case descriptions were produced based on the medical records of 40 patients with vertebral metastases. The descriptions were then published online. Physicians were invited to evaluate the descriptions by answering questions according to the Neoplastic Spine Instability Score (SINS). The agreement among physicians was calculated using the kappa coefficient.

RESULTS:

Seventeen physicians agreed to participate: three highly experienced spine surgeons, seven less-experienced spine surgeons, three surgeons of other specialties, and four general practitioners (n = 17). The agreement for the final SINS score among all participants was fair, and it varied according to the SINS component. The agreement was substantial for the spine location only. The agreement was higher among experienced surgeons. The agreement was nearly perfect for spinal location among the spine surgeons who were highly experienced in vertebral metastases.

CONCLUSIONS:

This study demonstrates that the experience of the evaluator has an impact on SINS scale classification. The interobserver agreement was only fair among physicians who were not spine surgeons and among spine surgeons who were not experienced in the treatment of vertebral metastases, which may limit the use of the SINS scale for the screening of unstable lesions by less-experienced evaluators.

INTRODUCTION

The spine is the most frequent site of bone metastases. In up to 20% of patients, the symptoms related to vertebral metastases are the initial manifestation of cancer (1). Vertebral metastases can cause severe complications, including compression of the spinal cord, which is found in 5-14% of patients with cancer over the course of the disease (2,3). The spinal cord compression may result from the growth of the tumor mass in the epidural space or may be associated with pathological fracture of the vertebra, which leads to compression by bone fragments or mechanical instability secondary to fracture (4,5).

Many studies have examined the options for operative treatment of spinal cord compression caused by solid tumors (2,6,7) and found that surgery contributes to a better quality of life and improves the ability to walk for patients with vertebral metastases (8).

The indication for surgical intervention is based on the health of the patient, survival prognosis, histology of the primary tumor, expectation of improvement with the use of other methods of treatment, and presence of spinal instability (9). The criteria for defining instability of the spine are well accepted for spinal injuries. However, there is controversy regarding the criteria for definition of instability arising from metastatic involvement (10).

Spinal instability is a key element in decision-making regarding the need for surgical treatment. However, the lack of objective criteria for vertebral instability is one of the reasons why many patients are unnecessarily referred to specialists for spinal evaluation, which results in increased healthcare costs, increased length of stay in hospitals and a delay in the start of cancer treatment. Additionally, many patients are treated insufficiently to correct the instability and they suffer worsening of the fracture, deformity, pain, and neurological deficits because the severity of the instability was not recognized.

Spinal instability, according to the Spine Oncology Study Group (SOSG), is defined as a loss of spinal integrity resulting from a neoplastic process that is associated with movement-related pain, symptomatic or progressive deformity, and/or neural compromise under a physiologic load (11). Despite the established definition of this concept of stability, its application in clinical practice is difficult. Therefore, the SOSG published the Neoplastic Spine Instability Score (SINS) based on the association of the best literature available with a consensus of expert opinions (11).

The SOSG classification uses parameters such as the location of the lesion and clinical characteristics of pain, quality of the matrix of the bone lesion, radiographic alignment of the spine, collapse of the vertebral body, and involvement of posterior spine structures. The minimum score is 0, and the maximum score is 18 points. A score between 0 and 6 indicates stability, a score between 7 and 12 indicates indefinite stability, and a score between 13 and 18 indicates instability. An expert evaluation is recommended for patients with a score of 7 points or higher (11) (Table 1).

Table 1.

Spine Instability Neoplastic Score (SINS) (11).

	Score
Spine location
Junctional (occiput-C2, C7–T2, T11–L1, L5–S1)	3
Mobile spine (C3–C6, L2–L4)	2
Semi-rigid (T3–T10)	1
Rigid (S2–S5)	0
Mechanical or postural pain
Yes	3
No (occasional pain but not mechanical)	1
Pain-free lesion	0
Bone lesion quality
Lytic	2
Mixed lytic/blastic	1
Blastic	0
Radiographic spinal alignment
Subluxation/translation present	4
De novo deformity (kyphosis/scoliosis)	2
Normal alignment	0
Vertebral body involvement
>50% collapse	3
<50% collapse	2
No collapse with >50% of the body involved	1
None of the above	0
Posterior involvement
Bilateral	3
Unilateral	1
None of the above	0

One possible application of the SINS score is the screening for spinal instability in the emergency room for quick decision-making. There is a consensus on the need for expert opinion in cases of spinal cord compression. However, in some patients, the main complaint is axial pain caused by metastatic disease where there is a risk for tumor-related spinal instability. A recent study by Fourney et al. (12) verified good interobserver reliability in determining stability using SINS. However, the study participants only comprised experienced spinal surgeons, and agreement among less-experienced attending physicians was not evaluated.

The objective of this study was to evaluate the interobserver agreement in the Neoplastic Spine Instability Score (SINS) among spine surgeons with or without experience in vertebral metastasis treatment and physicians in other specialties.

METHODS

This study was based on the medical records of 40 patients with spinal metastatic lesions that were treated at a public referral cancer center (Instituto do Câncer do Estado de São Paulo). The symptoms and history of the disease were described (Figure 1), and the case descriptions and imaging (computed tomography [CT] and magnetic resonance [MRI]) were published in an online system created for this study. The online system allowed the study participants to evaluate the case descriptions by answering questions according to the Neoplastic Spine Instability Score (SINS). In total, 40 cases that represented all SINS categories were included in the system.

Figure 1.

Example of a clinical case used in the evaluations. “Forty one-years-old patient with a diagnosis of metastatic colon adenocarcinoma. He has a complaint of progressive dorsal pain which is worse at night and with movement. The patient has a limited ability to move on the bed due to dorsal pain.” A. Anteroposterior and profile radiographs. B. Axial cut in computed tomography showing the lesion site.

Thirty physicians from all of the departments in one of the largest public university hospitals in Latin America (Hospital das Clínicas da Faculdade de Medicina da Universidade de São Paulo) were invited to participate in this study to evaluate the 40 cases online. In the invitation, the physicians were asked to declare how many cases of metastatic lesions they had surgically treated in the prior year (2011) and to fill in the SINS online questionnaire for ten cases every week. They were not asked to examine patients clinically. Rather, they had to evaluate the case descriptions online (which were based on medical records) and respond to the questionnaire.

The identity of the patients was not revealed to the study participants. Only sex, age, and clinical history and imaging were made available to the participants. This study did not require informed consent, but it was approved by the local ethics committee before the participants were invited.

The interobserver agreement for the final SINS score among the participants was calculated according to the kappa coefficient and the percentage of agreement. The kappa coefficient was also calculated for each component of the SINS: spine location, pain, spinal alignment, vertebral involvement, and bone lesion quality.

The null hypothesis (kappa equals zero, i.e., that there is a lack of agreement or that any observed agreement is purely by chance) was tested by using statistical methods. The reliability was evaluated as proposed by Landis and Koch (13): 0 to 0.2 indicated poor agreement; 0.21 to 0.4 indicated fair agreement; 0.41 to 0.6 indicated moderate agreement; 0.61 to 0.8 indicated substantial agreement; and 0.81 to 1.0 indicated very good agreement. An online collection of statistical programs was used for statistical analysis, and the tools that we used are available at http://www.stattools.net/CohenKappa_Pgm.php.

RESULTS

Of the physicians invited, 17 agreed to participate in this study and responded to ten questionnaires per week. Seven of the physicians were not spine surgeons, and they had graduated 3 to 39 years ago. These seven participants were orthopedic surgeons (two), a neurosurgeon (one), and general practitioners (four). The ten other participants were spine surgeons that had graduated 4 to 23 years ago, and three of them were highly experienced, as they had operated on 20 or more cases of spinal metastasis in the year prior to the study. Seven were less experienced: three had operated on three to six cases, and four surgeons had no spinal metastasis cases in 2011.

The agreement for the final SINS score among all participants was fair. For the spinal location only, the agreement was substantial. The agreement for each of the score components is shown in Table 2.

Table 2.

Agreement for the final results and components of the Spine Instability Neoplastic Score (SINS) among all evaluators.

	% of overall agreement	Fixed-marginal kappa	95% CI	Agreement
SINS score	62.44%	0.375379	0.3563-0.3944	Fair
Spinal location	79.76%	0.718683	0.7040-0.7332	Substantial
Mechanical or postural pain	72.54%	0.419348	0.3952-0.4434	Moderate
Bone lesion quality	56.32%	0.219671	0.1989-0.2403	Fair
Spinal alignment	77.22%	0.552618	0.5316-0.5735	Moderate
Vertebral body involvement	58.79%	0.42824	0.4138-0.4426	Moderate
Posterior involvement	61.78%	0.42375	0.4059-0.4415	Moderate

The agreement among the seven physicians who were not spine surgeons was fair for the final SINS and substantial only for the spine location component. The kappa coefficient for these participants is shown in Table 3. None of these seven physicians had previous knowledge of the SINS scoring methodology.

Table 3.

Agreement for the final result and components of the Spine Instability Neoplastic Score (SINS) among physicians who were not spine surgeons.

	% of overall agreement	Fixed-marginal kappa	95% CI	Agreement
SINS score	61.07%	0.346549	0.2975-0.3955	Fair
Spinal location	71.79%	0.610494	0.5739-0.6470	Substantial
Mechanical or postural pain	70.36%	0.387745	0.3268-0.4486	Fair
Bone lesion quality	45.24%	0.092664	0.0424-0.1428	Slight
Spinal alignment	68.57%	0.419333	0.5791-0.8695	Moderate
Vertebral body involvement	59.40%	0.426548	0.3888-0.4642	Moderate
Posterior involvement	57.50%	0.35992	0.3146-0.4051	Fair

The spine surgeons were divided into those with low and high surgical experience in vertebral metastasis treatment. All of the spine surgeons reported that they knew the SINS scoring methodology, but they did not use it in their daily routine. The less-experienced spine surgeons demonstrated fair agreement for the final SINS score and substantial agreement for the spine location and spinal alignment components, as shown in detail in Table 4. The agreement was higher among experienced surgeons and was substantial for the final SINS score. The agreement was also substantial for spinal alignment and nearly perfect for spinal location (Table 5).

Table 4.

Agreement for the final result and components of the Spine Instability Neoplastic Score (SINS) among spine surgeons with low experience for vertebral metastasis treatment.

	% of overall agreement	Fixed-Marginal Kappa	95% CI	Agreement
SINS score	60.12%	0.32285	0.2738-0.3718	Fair
Spinal location	83.57%	0.768719	0.7306-0.8067	Substantial
Mechanical or postural pain	72.38%	0.391383	0.3299-0.4528	Fair
Bone lesion quality	62.86%	0.311485	0.2585-0.3644	Fair
Spinal alignment	82.14%	0.629295	0.5731-0.6854	Substantial
Vertebral body involvement	57.98%	0.424512	0.3888-0.4601	Moderate
Posterior involvement	61.67%	0.42154	0.3761-0.4669	Moderate

Table 5.

Agreement for the final result and components of the Spine Instability Neoplastic Score (SINS) among experienced spine surgeons.

	% of overall agreement	Fixed-Marginal Kappa	95% CI	Agreement
SINS score	76.67%	0.631256	0.5069-0.7555	Substantial
Spinal location	93.33%	0.908047	0.8114-1.004	Near perfect
Mechanical or postural pain	78.33%	0.554796	0.3913-0.7181	Moderate
Bone lesion quality	69.17%	0.333134	0.1727-0.4935	Fair
Spinal alignment	86.67%	0.724376	0.5791-0.8695	Substantial
Vertebral body involvement	70.00%	0.578209	0.4799-0.6764	Moderate
Posterior involvement	71.67%	0.570617	0.4501-0.6911	Moderate

DISCUSSION

The benefits of surgery in the treatment of spinal compression caused by to metastasis of solid tumors are well known (2,14). However, when there is no spinal cord compression and the main complaint is axial mechanical pain it is difficult to recognize the patient population that would benefit from surgical treatment and spinal fixation. The presence of instability is an independent indication for surgery (15) or percutaneous cement reinforcement (6,16,17).

The instability of the spine associated with metastasis is still judged by the attending physician and is based on clinical experience. Criteria that have been developed for traumatic injuries of the spine are often used in these cases. However, the pathophysiology of traumatic fracture of the spine is different from that of metastatic involvement with respect to the pattern of bone and ligament involvement, as well as bone quality.

The development of more appropriate criteria for evaluating the instability of the spine can lead to improvement in the quality of care. In 2010, the SOSG published the Spine Instability Neoplastic Score (SINS) based on the best literature available and a consensus of expert opinions (11). The SINS can be used by attending physicians who are not spine surgeons when screening patients with metastatic disease of the spine prior to referral for a specialized evaluation. However, factors among the components evaluated by the SINS may be influenced by the evaluator's skills. These components include the quality of the bone matrix, alignment parameters and degree of vertebral impairment. Part of the scale requires technical knowledge, which most likely makes it difficult to use by professionals who are not used to evaluating the spine based on imaging studies. Thus, it is very important to study the interobserver agreement among groups of physicians with varying experience in the evaluation and treatment of patients with vertebral metastases.

In 2011, the SOSG published a study to evaluate the reliability and consistency of the SINS scale among spine surgeons who were considered oncology experts (12). The interobserver agreement in this study was nearly perfect. In the present study, the results of the final interpretation of the SINS score were fair when considering all examiners. The results were also fair among physicians who were not spine surgeons and among spine surgeons with low experience in surgery for spinal metastases. However, the agreement was substantial when the evaluation was performed among spine surgeons who were highly experienced in surgery for vertebral metastases. These findings suggest that the examiner's experience influences the agreement of the SINS final score.

A closer analysis of the SINS components in the present study shows that the agreement among observers is not the same for all of the domains. In our study, spinal location was a component of the SINS that showed substantial agreement among evaluators when considering all groups. However, among highly experienced spine surgeons, the agreement was nearly perfect, which corroborates the findings from the SOGS study by Fourney et al. (12). Therefore, spinal location appears to be an easy to use evaluation factor for imaging studies, even among inexperienced professionals.

However, the other components were not so easily evaluated. Similar to the results of Fourney et al. (12), the agreement among all participants for spinal alignment was moderate in our study. Spine surgeons (with high or low experience in oncology) had substantial agreement for spinal alignment, which provides evidence for the need for normality criteria and definitions in the SINS to help the less-experienced physicians. Agreement was also considered to be low for vertebral body involvement and posterior involvement.

In the present study, the agreement among participants with respect to mechanical or postural pain was never higher than "moderate", even among experienced spine surgeons. In Fourney et al. (12), the agreement was nearly perfect among participants who were oncology experts. Pain evaluation depends on the interpretation of the patient history, which involves some subjectivity, and on the physical exam. However, in the present study, only imaging exams could be safely uploaded online. In this study, pain was described by the attending physician in the medical record while taking the clinical history from the patient. The participants in this study had to rely on the notes of other clinicians because it was not possible to examine all patients again to evaluate pain (even if re-examination were possible, the pain would be different after treatment). Therefore, the agreement among observers could have been high or low according to the way in which the case was described. Asymmetry in the severity of cases may have also led to discrepancies in the SINS score.

Bone lesion quality was the component of the SINS with the lowest agreement in our study; the agreement for this parameter was also fair among the oncology experts evaluated by Fourney et al. (12) This finding indicates the need for a revised bone lesion quality score component. One possible improvement would be to divide the lesion quality into two categories: predominantly lytic or predominantly blastic lesions.

One limitation of this study is the low number of participants. Many physicians refused to participate because of a lack of time. It was also difficult to find balanced numbers of specialists from all fields. Ten spine surgeons and seven other specialists participated in this study. Among the ten spine surgeons, only three had previous significant experience with the surgical treatment of metastatic lesions. Despite this imbalance, the number of specialists and non-specialists in this study allowed us to calculate interobserver agreement. Most of the participants in this study were not specialized in metastasis treatment, and the agreement among these physicians was low, which suggests that that either the SINS is not a good screening tool for the emergency room or it requires training prior to use. Additional studies are needed to answer this question.

This study demonstrated that the experience of the evaluator has an impact on the SINS scale classification. The interobserver agreement was only fair among physicians who were not spine surgeons and among spine surgeons who were not experienced in the treatment of vertebral metastases, which may limit the use of the SINS scale for the screening of unstable lesions by less-experienced evaluators.