Reliability and Validity of the English Version of the AOSpine PROST (Patient Reported Outcome Spine Trauma)

The AOSpine PROST (Patient Reported Outcome Spine Trauma) is a new disease-specific outcome instrument for spine trauma patients. Very good results were obtained for reliability and validity of the English version. It has the potential to contribute to the reduction of ongoing controversies in spine trauma care.

Abstract

Study Design.

Multicenter validation study.

Objective.

The aim of this study was to translate and adapt the AOSpine PROST (Patient Reported Outcome Spine Trauma) into English, and test its psychometric properties among North-American spine trauma patients.

Summary of Background Data.

In the absence of an outcome instrument specifically designed and validated for traumatic spinal column injury patients, it is difficult to measure the effect size of various treatment options. The AOSpine Knowledge Forum Trauma initiated a project and developed the AOSpine PROST consisting of 19 items.

Methods.

Patients were recruited from two level-1 North-American trauma centers. For concurrent validity, next to AOSpine PROST also 36-item Short-Form Health Survey (SF-36) was filled out by patients. Patient characteristics were analyzed using descriptive statistics. Floor and ceiling effects as well as the number of inapplicable and missing questions were analyzed for content validity. Cronbach α and item-total correlation coefficients (ITCCs) were calculated for internal consistency. Spearman correlation tests were performed within AOSpine PROST items and in correlation to SF-36. Test-retest reliability was assessed using intraclass correlation coefficients (ICCs). Factor analysis was performed to explore any dimensions within AOSpine PROST.

Results.

The AOSpine PROST was translated adapted into English using established guidelines. Of 196 enrolled patients, 162 (82.7%) met the inclusion criteria and provided sufficient data. Content validity showed good results, and no floor and ceiling effects were seen. The internal consistency was excellent (Cronbach α = 0.97; ITCC 0.50–0.90) as well as test–retest reliability (ICC = 0.97). Spearman correlations were good (0.29–0.85). The strongest correlations of AOSpine PROST with SF-36 were seen with the physical components (0.69–0.82; P < 0.001). Factor analysis revealed two possible dimensions (Eigen values >1), explaining 75.7% of variance.

Conclusion.

The English version of AOSpine PROST showed very good validity and reliability. It is considered as a valuable tool, and has the potential to contribute to the reduction of ongoing controversies in spine trauma care.

Level of Evidence: 2

Spine trauma remains a global public health concern, in terms of both care and costs. A subset of spine-injured patients may deal with an associated neurologic injury, which can contribute to substantial disability with long-term consequences and considerable associated health care-related costs.¹,²,³,⁴ The reported annual incidence of traumatic spine fractures in various articles varies between 19 and 88 per 100,000 population.⁵,⁶ The estimated global incidence-rate of traumatic spinal cord injury is reported as 23 per million, with a worldwide variation of 3.6 to 195.4 per million.⁷,⁸,⁹

In the absence of a validated disease-specific outcome instrument for traumatic spinal column injury patients, a wide range of tools have been used including generic outcome measures and instruments designed for polytrauma patients and patients with chronic degenerative back pain.¹⁰,¹¹,¹²,¹³ This makes it difficult to compare outcomes of different treatments of the spinal column injury within and between studies, and may contribute to treatment variation among these patients.¹⁴,¹⁵,¹⁶ Therefore, an initiative was undertaken by AOSpine Knowledge Forum (KF) Trauma to develop a disease-specific outcome measure for spine trauma patients: the AOSpine PROST (Patient Reported Outcome Spine Trauma). Based on the systematic approach and Core Set development methodology of the International Classification of Functioning, Disability, and Health (ICF) of the World Health Organization (WHO) the tool was developed.¹⁷,¹⁸ Four preparatory studies, followed by an international consensus conference, led to the development of a Dutch version of AOSpine PROST.¹⁹ In the developmental process of the AOSpine PROST we sought to focus on patients sustaining injuries to their spinal column and excluded completely paralyzed (ASIA impairment grade A or B at discharge from hospital) and polytrauma patients (Injury Severity Score [ISS] >15), to identify specific problems related to spine trauma. A validation study among traumatic spinal column injury patients in the Netherlands showed very good results.²⁰

The aims of this study are to translate and cross-culturally adapt the AOSpine PROST into English, and test its psychometric properties among North-American spine trauma patients.

MATERIALS AND METHODS

Translation and Cross-cultural Adaptation

Established guidelines were used for the translation and cross-cultural adaption of the AOSpine PROST from Dutch into English.²¹ The translation process consisted of four stages: forward translations, synthesis, back translations, and expert committee review, which was followed by validation.

In the first stage, two native English language speakers translated the AOSpine PROST from Dutch into English. The translators used English as their mother tongue and translated independently of each other. The first translator was a local Masters study program manager and the second translator a professor in Orthopedic biomechanics. During a meeting including the translators and a recording observer the translations were discussed, leading to one synthesis of the two translations. Subsequently, two independent uninformed bilingual translators with Dutch as their mother tongue performed a translation back from English into Dutch. These back-translators were Orthopedic researchers, one with medical and another technical medicine background. They were blinded to the original AOSpine PROST to avoid information bias. In the next step, an expert committee meeting (including forward-translators, back-translators, two recording observers, a professor of spinal surgery, and a professor in spinal cord injury rehabilitation) reviewed all translations and written reports. Equivalence of the prefinal version and original Dutch AOSpine PROST was examined for semantics, idioms, and conceptual meaning. After discussion, the expert committee developed a pre-final English version of the questionnaire. This was sent to the North-American AOSpine KF Trauma Steering Committee members for a final review.

Patients and Study Procedures

Adult traumatic spinal column injury patients (≥18 years of age) within 13 months post-trauma were included. They had to be capable of understanding and adequately filling out the questionnaires (e.g., no cognitive impairments). Completely paralyzed patients (ASIA impairment grade A or B at discharge from hospital) and polytrauma patients (ISS >15) were excluded.

Patients were recruited from two level-1 trauma centers in North-America: Rothman Institute (Philadelphia) and University of British Columbia (Vancouver, Canada). Patients in follow-up at the outpatient clinic within 13 months post-trauma were checked for eligibility. After informed consent, they received an email with an online link or postal mail with a login code to complete the questionnaire. If the questionnaire was not completed within 3 days, a reminder via email or telephone was sent. For the purpose of test–retest, 1 week after completing the questionnaire a part of the included patients were asked to fill out the same questionnaire once more. Ethics approval was obtained from the institutional review board of both participating centers.

Instruments

The questionnaire that was administered to the patients through an online system consisted of different parts. The first part was the AOSpine PROST consisting of 19 questions on a broad range of aspects of functioning (Appendix 1). Each item has a 0 to 100 numeric rating scale, with 0 indicating no function at all and 100 the pre-injury level of function. The scale is supported by smileys at the ends of the ruler. For the purpose of concurrent validity, the AOSpine PROST should be compared to a validated outcome instrument designed for patients with traumatic spinal column injuries. However, as no such instrument is available, the AOSpine PROST was followed by a generic health-related quality of life measure, the Medical Outcomes Study 36-item Short-Form Health Survey (SF-36) as reference standard.²² The SF-36 is the most frequently used generic instrument to measure outcomes in studies including spine trauma patients.¹²,²³,²⁴ It consists of 36 questions measuring eight health subscales from which the physical component summary (PCS) and mental component summary (MCS) are calculated. Scores range from 0 to 100 with higher scores indicating better outcome. Finally, patients were asked to answer additional questions: any difficulties in filling out the AOSpine PROST, any inapplicable questions, any relevant question missing, and self-reported degree of recovery.

Another part of the online system was completed by health professionals. They were asked to complete background data (sociodemographic characteristics and trauma-related variables), and to make an assessment of patients’ degree of recovery based on clinical and radiological findings.

Statistical Analysis

Descriptive statistics and frequency analysis were used to analyze patient characteristics. Content validity was assessed by evaluating the responses to the additional questions (difficulties in filling out the AOSpine PROST, inapplicable questions, and relevant questions missing). Also floor and ceiling effects were investigated, which could occur if >15% of the patients achieve the lowest or highest possible score, respectively. Furthermore, correlation between the mean total scores with the degree of recovery as reported by patients and assessed by clinicians was analyzed using Welch analysis of variance.

Concurrent validity explored the Spearman correlation coefficient (r_s) between AOSpine PROST and SF-36. The r_s ranges from + 1 to−1, with + 1 indicating a perfect association, 0 no association, and −1 perfect negative association.²⁵ Concurrent validity is supported if the coefficient is at least 0.70.²⁶

For internal consistency Cronbach α and item-total correlation coefficients were calculated. An α >0.70 is accepted as good results.²⁶,²⁷ Pairwise Spearman correlations between AOSpine PROST items was also investigated.

Intraclass correlation coefficients (ICCs) were calculated to assess test–retest reliability. An ICC of 0.70 to 0.85 and > 0.85 indicate good and excellent reliability, respectively.²⁶

Finally, factor analysis was performed to explore the dimensionality of AOSpine PROST items. Based on the MINEIGEN criterion, an Eigen value >1 would indicate a dimension within the questionnaire. The loading of the items on identified dimensions was also analyzed.

RESULTS

Translation and Cross-cultural Adaptation

The translation and adaptation process proceeded without any difficulties. During the expert committee meeting, there was some discussion on item 14 (concerning emotional functioning). This item was translated as “Emotions and moods” and “Emotions” by the forward-translators. The synthesis was defined as “Emotions and moods.” Discussions during the expert committee meeting led to the definition as “Emotional state.” Ultimately, after a review by the North-American AOSpine KF Trauma members, the item was stated “Emotional function” which brings it in line with the overall aim of AOSpine PROST, that is measuring patients’ functional status. Another discussion point was how to indicate the phrase in the response scale, freely translated as “I do NOT function at all.” Although the forward-translators chose for “I do not function” and “I am completely disabled,” ultimately consensus was reached on “I am NON-FUNCTOINAL.”

Patient Characteristics

A total of 196 patients consented to take part in the study, of which 162 (82.7%) were enrolled: 95 (58.6%) USA and 67 (41.4%) Canadian patients. Nineteen USA and 15 Canadian patients were excluded: 5 were ineligible, 9 did not provide any data at all, 15 did not complete any patient questionnaire, and 5 were >13 months post-trauma. The majority of the included patients were male (67.5%) and the mean age 51.2 years. Table 1 shows the basic patient and clinical characteristics.

TABLE 1.

Sociodemographic and Clinical Characteristics of the Study Population^∗

Content Validity

The mean time to complete AOSpine PROST as registered by the online system was 7.6 minutes (median = 7.0; Q1 = 5.0, Q2 = 10.0).

Nine (5.6%) patients indicated they experienced some difficulties when filling out AOSpine PROST. In their further explanations, however, they described the limitations that hey experienced in more detail. Thus, no misunderstanding of the questions was mentioned. No patient indicated an item to be inapplicable for measuring their health status and function. Twenty-one (13.0%) patients answered positive to any item missing, but they mainly mentioned items that are already incorporated in AOSpine PROST. Only one patient had a new suggestion: “spiritual practice which supports the inner peace.” Two other patients wondered whether the scores should also apply to other trauma-related injuries.

AOSpine PROST scores relative to the degree of recovery were more strongly related (P < 0.001) to the indication by patients compared to the clinicians’ assessments (Table 2). No floor and ceiling effects were seen.

TABLE 2.

Mean AOSpine PROST Scores Relative to the Degree of Recovery, Both as Reported by Patients and as Assessed by Clinicians (Mean ± SD [Range])^∗

Concurrent Validity

Table 3 shows AOSpine PROST and SF-36 Spearman correlations. The most strong correlation was seen with the physical components of SF-36 (P < 0.001): role physical (0.82), physical functioning (0.78), and PCS (0.69). A negative correlation was observed with General Health subscale (−0.08; P < 0.001).

TABLE 3.

Spearman Correlation (r _s) Between AOSpine PROST and SF-36, Both for the Subscales and Summary Scales

Internal Consistency

The internal consistency of AOSpine PROST total score was excellent (Cronbach α = 0.97). With a range of 0.50 to 0.90, item-total correlation showed good results (Table 4). “Bowel movement” and “Urinating” had the lowest values (0.50 and 0.52, respectively), whereas the highest values (0.90) were seen for “Energy level” and “Lifting and Carrying.” Cronbach alpha remained excellent and did not change (0.97) when an item was removed. Good results were also obtained with Spearman correlations (Appendix 2) within AOSpine PROST items (range: 0.29–0.85).

TABLE 4.

Internal Consistency and Test–Retest Reliability Results. Internal Consistency Is Shown for Each AOSpine PROST Item, Both Item-total Correlation (Rho) and Cronbach α if Item Deleted. Test–retest Reliability Was Assessed by Using ICC Along With Its 95% CI

Test–retest Reliability

Initially, 128 patients were included in the test–retest arm (USA n = 74, Canada n = 54). However, 80 (USA n = 49, Canada n = 31) could not be included in the test–retest analysis (Table 5). The basic patient characteristics were comparable to the overall study population (Appendix 3). The main reasons were not completing the questionnaire twice (n = 36) or not within an interval of 4 to 14 days (n = 27). A total of 48 patients could be included for test–retest analysis. The interval between the first and second administration was 8.9± 1.9 days (range: 7–14). Excellent test–retest reliability was seen (ICC = 0.97, 95% confidence interval 0.95–0.98). When looking into detail (Table 4), all items had acceptable to excellent reliability results, except for “Urinating” (ICC = 0.66).

TABLE 5.

Reasons of Exclusion From Test–retest Analysis^∗, n (%)

Factor Analysis

Two factors were found that had an Eigen value >1, indicating two possible dimensions across AOSpine PROST items. These factors had an Eigenvalue of 12.9 and 1.6, and explaining 67.6% and 8.1% of the variance, respectively. As shown in Appendix 4 and Figure 1, rotated factor analysis revealed that most items load high on Factor 1 and low on Factor 2, that is those items contribute considerably to the dimension represented by Factor 1. “Urinating” and “Bowel movement” seemed to load high on Factor 2 and low on Factor 1. A few items loaded high on both factors, that is, “Walking,” “Personal care,” “Emotional function,” and “Sexual function.” No item showed low loadings on both factors indicating that a possible third factor is not expected.

Figure 1.

Rotated factor pattern of AOSpine PROST factor analysis for the English version.

DISCUSSION

The present study describes the translation and cross-cultural adaptation of AOSpine PROST into English following established guidelines,²¹ and showed excellent psychometric properties among patients from 2 North-American centers.

In the translation and cross-cultural adaptation process it was chosen to incorporate stage V (prefinal testing) in the validation study. We found this to be a grounded decision as pretesting was already performed for the Dutch version in a previous study.¹⁹ As hypothesized, patients had no misunderstanding of the questions and no question was indicated as inapplicable. Patients provided very useful information on the general concept of AOSpine PROST: it was questioned whether they should also take the trauma-related injuries into account when scoring an item. This will be taken into account in the further development and refinement of the instrument.

Slight differences were observed in characteristics between American and Canadian patients (Table 1). The Canadian patients were younger, had more often no comorbidities, cervical spine fractures, neurologically impaired, and surgically treated. These characteristics together with the finding of sports or recreation as the main cause of trauma might indicate a healthier lifestyle among Canadian patients in the present study.²⁸ However, increasing spine injuries have been seen in various thrill-seeking sports.²⁹,³⁰,³¹,³² A fall as cause of injury was frequently seen in our study population, which has shown to be a major public health problem in North-America and many other industrialized countries.²⁹,³³ Violence as the cause of injury was only seen in American patients. In previous literature it has been described that act of violence is not as common in Canada.³⁴,³⁵ Nevertheless, the AOSpine PROST showed very good results. This may be related to the unique approach of the tool by asking the patients to compare their current function with the pre-injury level of function (Appendix 1). Undoubtedly, besides patient-reported outcomes also clinical and radiological parameters are deemed relevant to have a holistic view on patients’ health and functional status. Therefore, the AOSpine KF Trauma is also making efforts on the development of an outcome measure from the perspective of the treating surgeons: AOSpine CROST (Clinician Reported Outcome Spine Trauma).³⁶

With both Cronbach α and ICC being 0.97, very good internal consistency and test–retest reliability was seen. Alpha values remained high when an AOSpine PROST item was removed (Table 4), indicating that the tool might have some item redundancy. In a next phase including a larger patient sample, a stricter test of unidimensionality could be performed to identify redundant items. We did perform a factor analysis with the available data in the current study which showed somewhat contradictory results with the identification of two possible dimensions. All items loaded high to one specific dimension with the exception of “Urinating” and “Bowel movement.” This finding may be related to our patient selection with no or only mild neurological impairment. Bladder and bowel dysfunctions may be major impairments in patients with severe or complete neurological deficit.³⁷,³⁸ A remarkable finding from the concurrent validity analysis was the negative correlation with SF-36 General Health subscale. We could not find an explanation for this result. A positive correlation (r _s = 0.58) was found in another study that investigated the psychometric properties of the Dutch version of AOSpine PROST.²⁰

We do recognize this study has several limitations. The patient characteristics of the Canadian and USA patients were slightly different. Nevertheless, we believe this is an adequate reflection of the North-American spine trauma patients, and an adequate intent to include both populations in the present study. Another limitation was a higher than expected number of patients that were excluded from the test–retest analysis; however, even with these exclusions, the results of the study are still convincing. Furthermore, no prospective responsiveness analysis was performed, as it has already been done for the Dutch version and showed excellent results.²⁰

In conclusion, the English version of AOSpine PROST showed to be a valid and reliable tool to measure outcomes in spine trauma patients. In the next phase, its applicability and psychometric properties among completely paralyzed patients will be tested. The AOSpine PROST is considered as a valuable tool for use in the clinical setting and research. It has the potential to contribute to the reduction of ongoing controversies in spine trauma care when used in a global setting, allowing us to compare different treatment options in a valid and reliable fashion.

Key Points

• The AOSpine PROST (Patient Reported Outcome Spine Trauma) is a new disease-specific outcome measure for spine trauma patients.

• Very good results were obtained for reliability and validity of the English version of AOSpine PROST.

• The AOSpine PROST has the potential to compare different treatment options in a valid and reliable fashion, and contribute to the reduction of ongoing controversies in spine trauma care.