Interobserver Reliability in the Classification of Thoracolumbar Fractures Using the AO Spine TL Injury Classification System Among 22 Clinical Experts in Spine Trauma Care

Abstract

Study Design

Reliability study utilizing 183 injury CT scans by 22 spine trauma experts with assessment of radiographic features, classification of injuries and treatment recommendations.

Objectives

To assess the reliability of the AOSpine TL Injury Classification System (TLICS) including the categories within the classification and the M1 modifier.

Methods

Kappa and Intraclass correlation coefficients were produced. Associations of various imaging characteristics (comminution, PLC status) and treatment recommendations were analyzed through regression analysis. Multivariable logistic regression modeling was used for making predictive algorithms.

Results

Reliability of the AO Spine TLICS at differentiating A3 and A4 injuries (N = 71) (K = .466; 95% CI .458 – .474; P < .001) demonstrated moderate agreement. Similarly, the average intraclass correlation coefficient (ICC) amongst A3 and A4 injuries was excellent (ICC = .934; 95% CI .919 – .947; P < .001) and the ICC between individual measures was moderate (ICC = .403; 95% CI .351 – .461; P < .001). The overall agreement on the utilization of the M1 modifier amongst A3 and A4 injuries was fair (K = .161; 95% CI .151 – .171; P < .001). The ICC for PLC status in A3 and A4 injuries averaged across all measures was excellent (ICC = .936; 95% CI .922 – .949; P < .001). The M1 modifier suggests respondents are nearly 40% more confident that the PLC is injured amongst all injuries. The M1 modifier was employed at a higher frequency as injuries were classified higher in the classification system.

Conclusions

The reliability of surgeons differentiating between A3 and A4 injuries in the AOSpine TLICS is substantial and the utilization of the M1 modifier occurs more frequently with higher grades in the system.

Introduction

Traumatic injuries of the thoracolumbar spine represent a significant diagnostic and therapeutic challenge for treating physicians.^1-3 As such, in 2013 the AO Spine Spinal Cord Injury and Trauma Knowledge Forum generated the AO Spine Thoracolumbar Injury Classification System.^4,5 While this new system undoubtedly succeeded in many regards, it still lacked utility when it came to delineating the accurate diagnosis/classification and optimal treatment for several common pathologies, namely A3 and A4 type injuries, without neurologic involvement (TL fractures).^6-15

Randomized controlled trials (RCTs) represent the gold standard for guiding evidence when it comes to delineating treatment algorithms. ¹⁶ However, in the field of spine trauma, specifically with regards to TL fractures, this represents both a logistic, ethical and technical challenge. One of the primary concerns associated with evaluating TL fractures via an RCT is simply defining relevant inclusion criteria because it is currently unclear which morphologic features associated with these injures are guiding treatment.

In the setting of TL fractures, the range of treatment options for a given injury extends from operative to nonoperative management. Similarly, the range of operative and nonoperative treatment offered for a given injury also varies significantly.^8-14,17 Proponents of operative strategies aimed at addressing this pathology highlight the improving safety profile of modern perioperative care, surgical technique, and implants in the setting of radiologic and biomechanical data that suggests a higher incidence of sagittal deformity progression in patients treated nonoperatively.^11,18-27 Often quoted in this “operative school” is the Load Sharing Classification published by McCormack et al which suggest that increasing degrees of anterior column comminution imparts greater instability and likelihood of sagittal plane deformity progression, even with short segment posterior fixation. ²⁷ Members of the “nonoperative school” highlight the numerous clinical studies that suggest, regardless of comminution, the potential for sagittal deformity progression imparts no clinical significance and as such does not justify the risk of surgical stabilization.^17,28-35 Therefore, like that of many challenging problems in the realm of spine surgery, the optimal treatment of TL fractures represents a true ‘clinical equipoise,’ as there is “genuine uncertainty within the expert community on the optimal approach” to treat this pathology. ³⁶

Inherent to challenging clinical scenarios like that of the optimal treatment for TL fractures is the opportunity for justification bias. ³⁶ In the setting of TL fractures, the M1 modifier may inherently provide treating surgeons with this opportunity. ⁴ As originally designed, the M1 modifier imparts a subjective level of “instability” to a given fracture by suggesting that although the osseous aspect of the injury is “stable,” the inherent energy required to induce said injury undoubtedly altered the integrity of the posterior ligamentous complex (PLC), rendering the injury “unstable”. ⁴ Therefore, the M1 modifier represents a clear opportunity for opinion, rather than evidence, to justify different treatment decisions for a given injury.

The purpose of this study was to first assess the reliability of expert providers at classifying TL fractures. Secondly, we aimed to identify which, if any, morphologic aspects associated with these injuries most accurately divide the 2 groups, operative and non-operative proponents. Lastly, we aimed to assess the inherent biases providers have when assigning the M1 modifier.

Methods

The detailed methodology is available in the article of Dandurand et al ³⁷ “Understanding Decision Making as it Influences Treatment in Thoracolumbar Burst Fractures Without Neurological Deficit: Conceptual Framework and Methodology” in this focus issue. The AO Spine Knowledge Forum Trauma completed consent and recruitment for a multicenter prospective observational study of TL Fractures; the Spine A3/A4 study. Each enrolling center obtained local approval from their institutional review board.

The expert panel of 22 surgeons included surgeons that were representative of a variety of geographic regions and reflected a heterogeneity in training backgrounds as well as patterns of clinical practice. Eight expert panel members represented the sites that recruited patients to the Spine TL A3/A4 study.

From that study, baseline CT scans and plain radiographs of 183 patients were available for review. Of those patients 44 were classified as having A3 and A4 injuries. All patients met the criteria of the Spine TL A3/A4 study and were neurologically intact and had injuries between T11 and L2 (Table 1) (Table 2).

Table 1.

Inclusion Criteria for the AO Spine A3/A4 Study.

• Skeletally mature, age 18 to 65 years old inclusive

• Diagnosis of AO type A3 and A4 fractures on a CT scan with or without a suspected PLC injury from T10 to L2.*

• TLICS score between 2-5 [x]

• Acute fracture – diagnosis and treatment within 10 days of injury

• Ability to understand the content of the patient information/informed consent form

• Willingness and ability to participate in the clinical investigation

* Fractures may have an associated (suspected) B injury but must have an A3 or A4. This was to prevent selective exclusion knowing that distinction between A and.

Table 2.

Exclusion Criteria for the AO Spine A3/A4 Study.

• Severe systemic disease that would exclude the patient from surgery

• Recent history of substance abuse

• Pregnancy or women planning to conceive within the study period

• Prisoner

• Participation in any medical device or medicinal product study within the previous month that could influence, the results of the present study

• Any neurological deficit associated with the fracture

• Spontaneous fractures due to pathologic processes or neoplasia

• Head injuries causing inability to cooperate during hospital admission

• Open or penetrating spinal lesion due to gun, stab, or projectile

• Prior spinal surgeries in thoracic or lumbar spine

• Additional musculoskeletal, head, or other injuries which would preclude rapid mobilization

• Multiple trauma or ISS >16

• Other co-morbidities precluding the patient to be considered as a surgical or non-surgical candidate

* ISS, Injury Severity Score; BMI, Body Mass Index; AIDS, Acquired Immunodeficiency.

Statistical Analysis

Frequency tables were produced for the distribution of each injury type for each member of the expert panel. Fleiss multi-rater Kappa scores were produced for analyzing the agreement of all expert panel raters for injury classification. Inter-class correlation coefficients were produced to measure the reliability whenever data were continuous or ordinal, using a two-way mixed effects model. The estimator is the same, whether the interaction effect is present or not. Type C intraclass correlation coefficients were used for reliability analysis, using a consistency definition.

Associations of various imaging characteristics (comminution, PLC status) and treatment recommendations were analyzed through a process of regression analysis and development of predictive modeling equations. We also employed multivariable logistic regression modeling for making a predictive algorithm whenever it was necessary.

Statistical significance was defined as a P-value of .05 or less.

Results

Based on the data from 183 effective cases, and 21 of the 22 surgeons, interrater reliability assessment revealed moderate overall agreement when classifying 183 effective cases using the AO Spine Thoracolumbar Injury Classification System (K = .430; 95% CI .422 – .437; P = .000). Conversely, the intraclass correlation coefficient (ICC) averaged across all measures was high (ICC = .932; 95% CI .916 – .946; P = .000) despite the weak ICC between individual measures (ICC = .383; 95% CI .332 – .442; P = .000). This discrepancy exists because of the lack of heterogeneity in case distribution with most injuries being classified as either A3 or A4 (3465/4026). When only the A3 or A4 injuries are assessed, and the remainder of the cases are removed (N = 71), the interrater reliability improves modestly between the reviewers (K = .466; 95% CI .458 – .474; P < .001) (Table 3). Nonetheless, when we evaluate the ICC for only the A3 and A4 injuries, the ICC, averaged across all measures, was excellent (ICC = .934; 95% CI .919 – .947; P = .000) and the ICC between individual measures was moderate (ICC = .403; 95% CI .351 – .461; P < .001) (Table 4).

Table 3.

Overall Agreement and Kappa Analysis for Only A3 and A4 Injuries.

	Kappa	Asymptotic			95% Confidence Interval
		Standard Error	z	P Value	Lower Bound	Upper Bound
Overall agreement	.466	.004	114.909	<.001	.458	.474

Table 4.

Intraclass Correlation Coefficient for Only Identifying A3 and A4 Injuries.

	Intraclass Correlation	95% Confidence Interval		Value	P Value
		Lower Bound	Upper Bound
Single measures	.403	.351	.461	15.156	<.001
Average measures	.934	.919	.947	15.156	<.001

Evaluation of the PLC and the use of the M1 modifier was also assessed in our classification questionnaire (Figure 1). The interrater reliability of the M1 modifier demonstrated weak overall agreement when all cases were included (K = .157; 95% CI .147 – .166; P = .000) and like the above data, when only the A3 and A4 injuries were included, there was only modest improvement in the agreement (K = .161; 95% CI .151 – .171; P < .001) (Table 5). Like the ICC for the entire classification system noted above, the ICC for all cases averaged across all measures for the assessment of the PLC was high (ICC = .940; 95% CI .953 – .927; P = .000) despite the weak ICC between individual measures (ICC = .418; 95% CI .365 – .477; P = .000). When we evaluate the ICC for only the A3 and A4 injuries, the ICC, averaged across all measures, was excellent (ICC = .936; 95% CI .922 – .949; P < .001) and the ICC between individual measures was moderate (ICC = .411; 95% CI .359 – .469; P < .001) (Table 6). In addition, when evaluating respondents’ assessment of the PLC in conjunction with their use of the M1 modifier, when respondents utilize the M1 modifier, they are nearly 40% more confident that the PLC is injured compared to when it is not employed (mean 54.6 vs mean 15.0%; P = .000).

Figure 1.

Expert survey questionnaire.

Table 5.

Overall Agreement and Kappa Analysis for the Utilization of the M1 Modifier.

		Kappa	Asymptotic			95% Confidence Interval
			Standard Error	z	P Value	Lower Bound	Upper Bound
Overall agreement	All cases	.157	.005	32.202	<.001	.147	.166
	A3/A4	.161	.005	31.608	<.001	.151	.171

Table 6.

Intraclass Correlation Coefficient for the Assessment of the PLC in A3 and A4 Injuries.

	Intraclass Correlation	95% Confidence Interval		Value	P Value
		Lower Bound	Upper Bound
Single measures	.411	.359	.469	15.637	<.001
Average measures	.936	.922	.949	15.637	<.001

The utilization of the M1 modifier was also assessed in the setting of primary injury classification and this can be seen in Table 5. As expected, the M1 modifier was employed in a greater proportion of those injury patterns with seemingly greater osseous involvement in an almost stepwise fashion. For example, in those injuries classified as A0 through A2, only 8.3% of respondents utilized the modifier, whereas the respondents utilized the modifier in 37.9% of cases classified as A3 and A4 (Table 7). Likewise, the modifier was also employed more frequently as injuries were classified as higher in the entitre alphanumeric classification system. For example, 46.3% of respondents used the modifier in A4 injuries vs only 30% in A3 injuries.

Table 7.

Crosstabulation and the Use of the M1 Modifier.

			Modifiers
			Other	M1	Total
Grouped injury	A3/A4	Count	2178	1287	3465
		% Within injury grouped	62.9%	37.1%	100.0%

Lastly, respondents were also asked to quantify the degree of vertebral body comminution they thought was associated with a given injury to discern whether certain bias was likely to predict a given classification. The results of this aspect of the survey suggest that respondents subjectively associate higher degrees of vertebral body comminution with higher alphanumeric scores in the AO Spine Thoracolumbar Injury Classification System. For example, respondents on average classified A4 injuries as having nearly 25% more comminution relative to their A3 counterparts (34.6% vs 58.1%; P = .000). The topic of comminution will be further explored in another subsection of the equipoise study.

Discussion

The AO Spine Thoracolumbar Injury Classification System classifies A3 and A4 fractures as “incomplete” or “complete” burst fractures based on whether a given injury involves 1 or both vertebral endplates, respectively. ⁴ Prior literature delineates that while the classification system is generally reliable, the interrater reliability for specifically identifying A3 and A4 fractures is relatively weak. ³⁸ Kepler et al. demonstrated that this system has a weak interobserver reliability with a Kappa of .19 for A4 fractures and .42 for A3 fractures. ³⁸ Similarly, a 2016 study by Schroeder et al. highlighted that only 60 and 30% of international survey respondents could correctly identify A3 and A4 fracture, respectively. ¹⁵ Similar to the prior studies, the results of our study indicate moderate interrater reliability with a Kappa of .466 when comparing the ability of observers to correctly identify A3 vs A4 injuries. While it is unclear why there was not better agreement in our analysis, there was a trend amongst our reviewers to correlate more vertebral body comminution with a higher alphanumeric score in the classification system and because our system does not take this metric into account this represents an opportunity for disagreement.

The Load Sharing Classification system initially published in 1994 suggested that those thoracolumbar fractures with greater vertebral body comminution are more likely to progress from a sagittal deformity standpoint if treated with short vs long segment posterior fixation. ²⁷ While several clinical studies, and even the 2005 Thoracolumbar Injury Classification and Severity Score (TLICS) classification system, highlighted that this phenomenon does not impact clinical outcomes, our study suggests that observers still correlate the degree of comminution with injury severity from an alphanumeric classification standpoint.^14,28-35 Our study also suggests that those injuries classified as A4 are thought to have nearly 25% more comminution relative to their A3 counterparts. Though there is no direct correlation with the degree of vertebral comminution and the correct classification of TL fractures, observers still seem to associate comminution with increased injury severity. This association will be further explored in another subsection of the equipoise study. In addition, those injuries that were classified as more severe from an alphanumeric classification standpoint, were also more likely to be assigned the M1 modifier (46.3% vs 30.0%) despite no objective evidence of any PLC injury. This represents an important finding in our analysis as it points to comminution, rather than endplate involvement, as 1 of the primary driving factors for treating physicians to deem an injury “unstable” and in need of operative stabilization, giving credence to the original Load Sharing Classification findings.

The PLC associated with the thoracolumbar spine represents a significant static stabilizer to the entire vertebral column.^1,39,40 Biomechanical studies indicate that there is a significant increase in vertebral column range of motion and instantaneous axes of rotation when only 1 component of this complex is released. ⁴⁰ As such, treatment decisions surrounding thoracolumbar spine injuries, dating back to the 3 Column Spine Classification by Denis, have emphasized its importance when deciding what treatment strategy to employ. ¹ However, as several prior studies have highlighted, even with advanced imaging in the form of MRI, evaluating the integrity of the PLC is challenging and often subject to bias. ⁴¹ While our study found an excellent for average measure and moderate for single measures interrater reliability for evaluating the integrity of the PLC, even when advanced imaging in the form of MRI was available, raters were still nearly 40% more confident that the PLC was disrupted in cases that were also assigned an M1 modifier. Again, this represents a significant finding in our analysis because the M1 modifier acts as a surrogate for instability and if it is being employed at a greater frequency with increasing alphanumeric classification severity, it can be deduced that the experts were directly correlating the degree of anterior column comminution with the integrity of the PLC, despite there being no clear association in between compression type injuries and disruption of the PLC. Therefore, in those injuries where the integrity of the PLC is unclear, our study demonstrates that the M1 modifier is assigned at higher frequency with greater vertebral body comminution, which again will be further explored in another subsection of this equipoise study. While treatment decisions based on injury classification was not specifically reviewed for this manuscript, this finding undoubtedly represents a significant opportunity for justification bias to be introduced into our treatment algorithms. ⁴² This is especially true if 1 considers the TLICS score, as the M1 modifier would increase a score of 2, which indicates nonoperative treatment, to 4 or 5, which introduces the option of surgical stabilization. ⁴²

There were several limitations of this study design, but none were more impactful than the fact the results of this study are based on expert opinon, of which 1 had to be eliminated due to considerable variation, which undoubtedly introduces substantial bias into our observations. This limitation is inherent to challenging topics like those of TL fractures, where logistical and ethical issues limit our ability to impose clincal trials. The second limitation had to do with the homogenous nature of the entire case volume that was reviewed. This limitation had a significant impact on the interrater reliability findings of our result sections for all injuries as there was only universal aggreemnt on 71/183 cases. While these findings in isolation do suggest a reliability issue with the classification system as a whole, this study was designed to specifically evaluate A3 and A4 injuries. Therefore, when just the A3 and A4 cases were reviewed the Kappa coefficient demonstrated modest improvement from the overall agreement findings.

Conclusion

This study highlights the intrinsic challenges associated with classifying TL fractures and ultimately designing comparative trials to develop evidence-based guidelines for their treatment. Our findings illustrate that there are inherent difficulties associated with differentiating between all TL fractures but that treating physican demonstrate moderate agreement when it comes to differentiating between A3 and A4 injuries. In addition, our results delineate that treating physicians may be utilizing the degree of anterior column comminution to distinguish between the fracture patterns delineated in this system. Lastly, our data suggests that providers may infer that an increasing amount of vertebral body comminution is associated with a greater likelihood of PLC disruption and increased overall spine instability.

ORCID iDs

Jose A. Canseco https://orcid.org/0000-0002-2152-5725

Taylor Paziuk https://orcid.org/0000-0002-6083-4176

Lorin M. Benneker https://orcid.org/0000-0002-5320-5638

Emiliano Vialle https://orcid.org/0000-0003-1157-4889

Shanmuganathan Rajasekaran https://orcid.org/0000-0001-6043-006X

Mohammad El-Sharkawi https://orcid.org/0000-0001-6177-7145

Rishi M. Kanna https://orcid.org/0000-0001-5817-4909

Martin Holas https://orcid.org/0000-0002-3763-8767

Eugen Cezar Popescu https://orcid.org/0000-0001-5732-5402

Charlotte Dandurand https://orcid.org/0000-0002-4520-6454

Gaston Camino-Willhuber https://orcid.org/0000-0002-5684-7679

Mohamed M. Aly https://orcid.org/0000-0001-5221-8288

Andrei Fernandes Joaquim https://orcid.org/0000-0003-2645-0483

Sebastian Bigdon https://orcid.org/0000-0002-4649-0610

Ulrich J. Spiegl https://orcid.org/0000-0002-5179-4192