Validation of the AO Spine Sacral Classification System: Reliability Among Surgeons Worldwide

Supplemental Digital Content is Available in the Text.

Abstract

Objectives:

To (1) demonstrate that the AO Spine Sacral Classification System can be reliably applied by general orthopaedic surgeons and subspecialists universally around the world and (2) delineate those injury subtypes that are most difficult to classify reliably to refine the classification before evaluating clinical outcomes.

Design:

Agreement study.

Setting:

All-level trauma centers, worldwide.

Participants:

One hundred seventy-two members of the AO Trauma and AO Spine community.

Intervention:

The AO Sacral Classification System was applied by each surgeon to 26 cases in 2 independent assessments performed 3 weeks apart.

Main Outcome Measurements:

Interobserver reliability and intraobserver reproducibility.

Results:

A total of 8097 case assessments were performed. The kappa coefficient for interobserver agreement for all cases was 0.72/0.75 (assessment 1/assessment 2), representing substantial reliability. When comparing classification grading (A/B/C) regardless of subtype, the kappa coefficient was 0.84/0.85, corresponding to excellent reliability. The kappa coefficients for interobserver reliability were 0.95/0.93 for type A fractures, 0.78/0.79 for type B fractures, and 0.80/0.83 for type C fractures. The overall kappa statistic for intraobserver reliability was 0.82 (range 0.18–1.00), representing excellent reproducibility. When only evaluating morphology type (A/B/C), the average kappa value was 0.87 (range 0.18–1.00), representing excellent reproducibility.

Conclusion:

The AO Spine Sacral Classification System is universally reliable among general orthopaedic surgeons and subspecialists worldwide, with substantial interobserver and excellent intraobserver reliability.

INTRODUCTION

The management of high-energy sacral fractures remains challenging for both spine surgeons and traumatologists. Difficulties with clinical decision making are often attributable to the lack of a comprehensive sacral fracture classification system. Current classification schemes are mainly descriptive, lack clinical correlations regarding prognosis and treatment, and are nonvalidated.^1–7 The lack of an appropriate conceptual framework in classifying sacral fractures has resulted in barriers to communication, research, education, standardized care, and evaluation of outcomes.

Classification systems are instruments used by surgeons to improve communication, guide treatment, and understand prognosis and are ideal when simple, reproducible, and comprehensive information can be transmitted to guide clinical management.⁸ Current sacral fracture classification schemes are based on various independent parameters such as injury location, fracture morphology, mechanism of injury, and associated injuries. The existence of numerous systems, however, is an indication that none have been universally adopted. The first developed classifications of sacral fractures focused on the location and pattern of injury.^4,6,9 The most notable and widely used, the Denis 3-zone classification, is based on the location of the fracture with reference to the sacral neuroforamina.² The Roy-Camille classification and its most recent modification were developed to further characterize transverse fractures of the sacrum, the most common type of Denis zone 3 injuries.^1,4 Finally, the Isler³ classification was devised to address lumbosacral instability pertaining to fractures involving the superior S1 facet. All these classifications, however, are suboptimal regarding true purpose of a classification system and lack associated management recommendations. The Denis classification remains overly simplistic and is likely still used primarily because of its ease of application. Although the zone of sacral fracture was noted to have implications regarding neurologic injury, this relationship has been called into question as of late.¹⁰ The modified Roy-Camille and Isler classifications are aimed to help guide management but are too specific regarding fracture type and are not sufficiently generalizable. Most recently, the Lumbosacral Injury Classification System incorporated fracture morphology, posterior ligamentous complex integrity, and neurologic injury status regarding lumbosacral injuries.⁵ Although this system does incorporate a severity score to help guide surgical decision making, it focuses specifically on those fractures resulting in lumbosacral instability and has not achieved widespread adoption.

In an attempt to overcome the current limitations of sacral fracture classifications, all members of AO Spine and AO Trauma were surveyed by researchers from the AO Spine Knowledge Forum Trauma regarding management of the most controversial aspects of sacral fractures.¹¹ The result was the AO Spine Sacral Classification System, a concise and comprehensive scheme to be used universally among surgeons (see Supplemental Digital Content 1, http://links.lww.com/JOT/B416). This sacral classification is separated into 3 subtypes based on morphological characteristics: type A (lower sacrococcygeal injuries), type B (posterior pelvic injuries), and type C (spinopelvic injuries). It is a hierarchical system in which each morphologic type is subdivided into increasing numerical subtypes based on the energy of injury, with higher numbers ascribed to increased injury severity. Case-specific and neurologic modifiers, not evaluated in this study, use clinical information to help further plan management based on relevant clinical information.¹² Regional differences in the care and treatment of patients, including varying indications for operative management, must be taken into consideration during the development of a universal classification scheme. The proposed classification scheme, therefore, is based on morphology and was created independent of treatment algorithms to account for such international variability of surgical indications.

Validation of classification systems remains crucial because a poorly validated system will be a biased predictor of patient outcomes.¹³ Incorrect classification of fracture severity may result in inappropriate surgical management and can potentially lead to undue harm. Furthermore, lack of appropriate validation can result in bias in scientific research that relies on categorization. The classification system presented in this study has completed pilot agreement validation as previously published, demonstrating excellent interobserver reliability for fracture severity and substantial reliability for all fracture subtypes among expert spine and trauma surgeons from around the world.^11,12 Although promising, these results must be verified in the broader clinical context, incorporating surgeons of all types from around the world to evaluate generalizability. This study, therefore, is essential to determine whether a sacral fracture classification system can be reliably adopted on a worldwide scale.

Accordingly, the aims of this study were to (1) establish that the AO Spine Sacral Classification System can be reliably applied by surgeons of all types universally across the world and (2) delineate those injury types that are most difficult to classify reliably to refine the classification before evaluating clinical outcomes.

METHODS

The background methodology for the development and description of the AO Spine Sacral Classification System has been previously described.^11,12 High-resolution computed tomography images from 26 cases with sacral fractures representing all subtypes of the classification were assessed by 172 investigators. An email request was sent out to all members of the AO Spine and AO Trauma community for surgeons who routinely treat patients with sacral fractures and would like to be involved in the validation of the sacral fracture classification. The investigators represented 6 world regions (Europe, North America, Latin and South America, the Middle East, Africa, and Asia). The investigators included orthopaedic spine surgeons, general orthopaedic surgeons, neurosurgeons, and orthopaedic traumatologists. Before the assessment, all participants participated in a video tutorial of the classification and a 10-case training session. The assessments were then performed through web conference, where both key high-resolution images of the fracture and the entire axial computed tomography scan, and, if necessary, the sagittal and coronal sequences, were analyzed. Two assessments were performed by each investigator independently 3 weeks apart from one another. The case order was scrambled so that a consecutive series was not presented, given the hierarchical nature of the classification system.

For reliability analysis, surgeons were provided with a video instruction of the classification system and a reference guide to injury types, which included both written and iconic descriptions with drawings of clinical images (see Supplemental Digital Content 1, http://links.lww.com/JOT/B416). To minimize linguistic and semantic confusion across different regions, verbal descriptions of the injury patterns were combined with standardized iconic images of each injury type to establish a rigorous visual and linguistic descriptive understanding of each injury pattern.

Statistical analysis used agreement percentages to evaluate the correct classification of each rating in comparison with the gold standard, as defined by a panel of expert spine surgeons and traumatologists. Cohen kappa (k) statistic was used to assess the reliability of classification between independent observers (interobserver agreement) and reproducibility between classifications of the same observer on separate assessments (intraobserver reproducibility). The k coefficients were interpreted using the Landis and Koch¹⁴ grading system. A k coefficient of less than 0.2 was defined as slight agreement or reproducibility, between 0.2 and 0.4 as fair agreement or reproducibility, between 0.4 and 0.6 as moderate agreement or reproducibility, between 0.6 and 0.8 as substantial agreement or reproducibility, and greater than 0.8 as excellent agreement or reproducibility. The k coefficients were calculated for fracture morphology type (A, B, or C) and classification of subtypes (eg, A1, A2, or A3).

RESULTS

A total of 172 surgeons were included in the study, resulting in 8097 case assessments. Practice setting for surgeons included 41% academic, 48% hospital employed, and 11% private practice. Baseline characteristics including region, years of practice, and surgical specialty are summarized in Table 1. The 26 cases for review consisted of 7 (26.9%) type A, 8 (30.8%) type B, and 11 (42.3%) type C fractures. Overall, 94.0% of assessments were correctly classified by morphology type. When stratifying by morphologic group, 98.3% were correctly classified as type A, 92.5% were correctly classified as type B, and 92.3% were correctly classified as type C fractures. Subtype A1 had the highest agreement percentage (95.4%), whereas fracture subtypes C1 and C3 had the lowest agreement percentages (67.6% and 77.5%, respectively). Correct classification stratified by years of experience, specialty, and region are provided in Table 2.

TABLE 1.

Baseline Characteristics of Observers

Region	Responses	172
	Africa	5.2%
	Asia	20.3%
	Europe	42.4%
	Latin and South America	19.2%
	Middle East	8.1%
	North America	4.7%
Years of practice	Responses	171
	<5 y	22.8%
	5–10 y	24.0%
	11–20 y	33.3%
	>20 y	19.9%
Specialty	Responses	172
	General orthopaedics	4.7%
	Neurosurgeon	23.3%
	Orthopaedic spine Surgery	66.3%
	Orthopaedic trauma Surgery	5.2%
	Other	0.6%

TABLE 2.

Correct Classification of Fracture Stratified by Years of Experience, Specialty, and Region

	% Correct
Years of experience
<5	84.6
5–10	82.8
11–20	87.4
>20	82.0
All	84.7
Specialty
General orthopaedics	86.0
Neurosurgery	84.3
Orthopaedic spine	83.1
Orthopaedic trauma	84.5
All	84.7
Region
Africa	80.6
Asia	82.1
Europe	89.3
Latin and South America	81.2
Middle East	87.5
North America	92.0
All	84.7

Interobserver Reliability

Because observers evaluated 26 cases and performed 2 separate assessments, k values are presented as assessment 1 k/assessment 2 k. The overall k coefficient for all cases was 0.72/0.75, representing substantial reliability. When comparing classification grading (A/B/C) regardless of subtype, k was 0.84/0.85, corresponding to excellent reliability. The k coefficients for interobserver reliability were 0.95/0.93 for type A fractures, 0.78/0.79 for type B fractures, and 0.80/0.83 for type C fractures (Table 3). These values represent substantial reliability for type B fractures and excellent reliability for type A and C fractures. When evaluating by subtype, A1 had the highest level of agreement with k of 0.88/0.89, and fractures C1 and C0 had the lowest level of agreement with k of 0.58/0.66 and 0.60/0.68, respectively (Table 4).

TABLE 3.

Interobserver Reliability for Classification by Morphology Type

Sacral Group	Kappa Value
	Assessment 1	Assessment 2
A	0.95	0.93
B	0.76	0.79
C	0.80	0.83
Overall	0.84	0.85

TABLE 4.

Interobserver Reliability by Fracture Subtype

Sacral Classification	Kappa Value
	Assessment 1	Assessment 2
A1	0.88	0.89
A2	0.74	0.72
A3	0.77	0.71
B1	0.68	0.63
B2	0.78	0.82
B3	0.77	0.83
C0	0.60	0.68
C1	0.58	0.66
C2	0.79	0.84
C3	0.59	0.73
Overall	0.72	0.75

Intraobserver Reliability

The overall k was 0.82 (range 0.18–1.00), representing excellent reproducibility. When only evaluating morphology type (A/B/C), the average k was 0.87 (range 0.18–1.00), representing excellent reproducibility.

DISCUSSION

This article demonstrates the applicability of the AO Spine Sacral Classification System among surgeons from a variety of subspecialties in orthopaedics and neurosurgery from various regions of the world. The overall interobserver reliability of the classification scheme was found to be substantial at minimum, and the overall intraobserver reproducibility was found to be excellent. Those fractures types that had the highest interobserver reliability were A1, B2/3, and C2 fractures, demonstrating excellent reliability. Most challenging in classifying were C0/1/3 fractures. C1/3 fractures had the lowest agreement percentages for correct classification; however, the reliability of C3 fractures improved from moderate to substantial from the first to the second assessments. C0/1 fractures demonstrated substantial reliability on both assessments. Given the lack of displacement and the difficulty in identifying nondisplaced fracture lines of C0/1, these fractures may be more challenging to identify. This subtype of fracture may require particular emphasis in teaching to facilitate knowledge translation. However, no fracture subtypes were found to have poor reliability of classification.

There were no major differences when analyzing fracture classification by years of experience or by surgical subspecialty, underscoring the utility of the classification scheme to the less experienced surgeon. Classification of fracture pattern based on geography showed regional variations. Observers from Africa, Latin and South America, and Asia had a greater proportion of incorrect classifications compared with those from Europe, the Middle East, and North America. This may be due to worldwide variations in access to advanced imaging, with decreased availability in rural areas of middle-income and low-income countries.^15,16 When comparing the first and second assessments, there was a slight “learning effect” in which interobserver reliability improved during the second assessment in most of the fracture subtypes. This is most notable in type C fractures. For example, 77.4% of C0 fractures were correctly identified on the first assessment, whereas the second assessment demonstrated correct identification of 94.1% of C0 fractures. Despite such learning effect, there remains high overall consistency between assessments. Even when taking into account the first assessment alone and negating the learning effect, there remains substantial interobserver reliability.

This is the first study for the advanced validation of the AO Spine Sacral Classification System. In addition, this is also the first study to validate a comprehensive sacral fracture classification system. There are no previous studies to compare reliability or demonstrate noninferiority. However, previous validation of this classification among expert spine surgeons and traumatologists demonstrated interobserver k values of 0.58 for overall classification and 0.75 for classification of morphology only.¹² Intraobserver reproducibility among experts had a k value of 0.71 for overall classification and 0.83 for morphology only. Interestingly, this study notes superior reliability when the classification system was applied among a larger group of nonexpert surgeons from across the globe. This underscores the generalizability of the classification to less experienced surgeons despite demographic, cultural, or training differences. When comparing the current reliability results with other widely adopted classification systems for fractures outside the sacrum, the AO Spine Sacral Classification System has either similar or superior reliability grading. For instance, the Thoracolumbar Injury Classification and Severity Score widely used for fracture care of the thoracolumbar spine was found to have an overall k of 0.63 (substantial) among an expert group of surgeons.^17,18 Thoracolumbar Spine Injury Classification and the AO Spine Subaxial Cervical Spine Injury Classification systems have overall interobserver reliabilities with k of 0.56 (moderate) and 0.64 (substantial), respectively.^19–21 Therefore, in this worldwide sample of surgeons of all types, we demonstrate at minimum substantial reliability that is comparable with current widely adopted classification schemes used in the treatment of spine patients.

Communication, research, education, and appropriate evidence-based treatment of patients are all dependent on the ability to comprehensively classify their injuries or conditions, as is the accuracy of information that surgeons are able to impart to patients about prognosis. Reliability is essential to communicate, classify, and appropriately guide treatment of relevant pathology. Without establishing such reliability, the education of surgeons becomes impractical. Furthermore, the validity of any research regarding the treatment and outcomes of sacral fractures can be called into question. The development process for the AO Spine classification systems is based on the evaluation of numerous cases in multiple sessions by a group of investigators and follows an iterative methodological pathway geared toward developing a validated system. The goal is for all surgeons to be able to accurately and reliably describe injury morphology and severity to facilitate communication between clinicians and researchers, with the ultimate goal of providing universal evidence-based algorithms for the treatment of fractures. Sacral fractures are rare, composed of both low-energy insufficiency and high-energy fractures, where controversy ensues regarding management.²² Adding to the uncertainty surrounding the most appropriate treatment of sacral fractures is the lack of a reliable and comprehensive means to classify these injuries. Our results suggest that the AO Spine Sacral Classification System may be reliably used to accurately classify sacral fractures despite user subspecialty or level of experience.

Despite encouraging results, there are limitations to our study. This study was performed in a retrospective manner based on previously obtained images. Previously obtained imaging was reviewed and classified by members of the AO Knowledge Forum Trauma. Those cases with complete agreement were deemed acceptable for use in the validation as the gold standard. Cases with incomplete or poor-quality imaging were excluded from use. Although this is consistent with an intermediate phase validation study, a true pragmatic approach would require a prospective application of the classification system, including controversial cases. The cases chosen in this study may have inflated the reliability of the classification system. However, further validation studies evaluating the clinical application of the classification system will account for such challenging cases, providing further insight for classification refinement. The surgeons were each provided with an educational video and reference guide readily available for proper classification during assessments, deviating from the true clinical scenario. Moreover, study participants were all members of AO Spine and/or AO Trauma and may not represent a true cross-section of surgeons globally. Surgeon practice setting for respondents suggests a participation bias toward academic and hospital-employed surgeons, which would be expected for members of an academic global community. It may be surmised that academic surgeons may be more familiar with AO classification systems, and therefore, using a population of academic surgeons may overstate the reliability of the classification system. In addition, there was an uneven regional distribution of observers worldwide and a higher proportion of Orthopaedic Spine respondents, which is consistent with AO Spine membership demographics. However, high agreement regarding management of patients has been previously shown between orthopaedic and neurosurgical spine surgeons.^23,24 Of the investigators involved in the study, 24 of the 172 (14%) of investigators did not perform a second assessment. We would expect, however, that interobserver agreement would likely increase because of the “learning effect,” as was seen in most of the fracture subtypes in the second assessment. Finally, understanding that each surgeon's assessments would be reviewed and evaluated also has possible implications on surgeon scrutiny of the imaging (Hawthorne effect).

Overall, this intermediate phase validation suggests that the AO Spine Sacral Classification System exhibits substantial interobserver and excellent intraobserver reliabilities. The results of this study are promising, demonstrating acceptable accuracy in the classification of fractures among surveyed orthopaedic, spine, and trauma subspecialists with various levels of experience. Further studies are required to assess the reliability of the classification system among a more generalizable population. Future clinical studies, including application of the classification system prospectively, will need to be conducted to further evaluate the clinical relevance and the usefulness of classification categories in a true pragmatic approach.