Validation of the Foot and Ankle Outcome Score for Hallux Rigidus

Abstract

Background

There is a clear call for improved patient-centered outcomes. The Foot and Ankle Outcome Score (FAOS) is a region-specific patient-reported measure that has been validated for a number of foot and ankle diagnoses, but not hallux rigidus.

Questions/Purposes

The aim of this study was to validate the FAOS in patients with hallux rigidus.

Methods

From 2007 to 2013, 211 patients with hallux rigidus (HR) were included in the study. For the construct validity portion of the study, 125 patients completed a Short-Form 12 (SF-12) and FAOS survey. Forty additional HR patients were prospectively given questionnaires to assess the relevance of each FAOS question as it pertained to their HR. Reliability was assessed in 36 HR patients via administration of a second FAOS an average 1 month following the first. In 55 patients, preoperative and postoperative FAOS scores were compared to determine responsiveness.

Results

All FAOS subscales demonstrated moderate correlation coefficients with the physical functioning, role physical, bodily pain, and physical health component scores of the SF-12, with all subscales demonstrating poor correlation with the SF-12 mental health-related domains. Content validity was high for all FAOS scores, with the exception of the daily activities subscale. All subscales achieved acceptable test–retest reliability with correlation coefficients of ≥0.72. Furthermore, all subscales were rated as responsive to change in postoperative patients (p < 0.001).

Conclusion

This study demonstrates the acceptable construct and content validity, reliability, and responsiveness of the FAOS for hallux rigidus. Due to its broad applicability and proven validation across multiple foot and ankle pathologies, the FAOS represents a patient-centered outcome measure that can be reliably used for the assessment of patients with hallux rigidus.

Electronic supplementary material

The online version of this article (doi:10.1007/s11420-015-9466-4) contains supplementary material, which is available to authorized users.

Introduction

Hallux rigidus occurs in 2.5% of the population over 50 years of age and is the most common osteoarthritic condition in the foot [8, 9]. A variety of treatments for hallux rigidus have been well-described in the literature with the most frequently performed operative procedures, cheilectomy and arthrodesis, demonstrating good long-term outcomes [6, 8]. However, none of the patient-reported outcome instruments utilized in prior studies on hallux rigidus have been validated specifically for this pathology. The American Orthopaedic Foot and Ankle Society (AOFAS) clinical rating scale, which is the outcome score most often reported, has been specifically questioned for its use in forefoot disorders [1, 20]. One of its major criticisms has been the heavy emphasis on pain in the final weighted score [3, 17]. This may potentially affect the validity of the AOFAS score for a condition such as hallux rigidus which is not always associated with pain.

In contrast, the Foot and Ankle Outcome Score (FAOS) has been validated previously in the forefoot for hallux valgus and in other conditions of the ankle and hindfoot including flatfoot, ankle instability, and ankle arthritis [5, 11, 12, 18]. Derived from the Knee Injury and Osteoarthritis Outcome Score (KOOS), the FAOS is a questionnaire consisting of five subscales relevant to the foot and ankle: pain, symptoms, activities of daily living, sport and recreational activities as well as quality of life. The goal of this study was to validate the FAOS for use in patients with hallux rigidus. The psychometric properties that were evaluated included content validity, construct validity, reliability, and responsiveness. Validation of the FAOS in the context of hallux rigidus could provides an alternative tool to assess patient outcomes, both in clinical practice and for future research. Given the ever increasing focus on patient-centered outcomes in the current health care environment, it is paramount to establish an outcome instrument that is validated among a broad range of foot and ankle disorders.

The purpose of this study was therefore to validate the FAOS for use in patients with hallux rigidus. We studied the psychometric properties including construct validity, content validity, reliability, and responsiveness of this legacy outcome tool.

Patients and Methods

Both the FAOS and the Short-Form 12 (SF-12) are administered as standard of care to all patients within the Department of Foot and Ankle Surgery at the investigators’ institution. This data is collected and stored as part of the Foot and Ankle Surgery Patient Registry. Both questionnaires are completed at the initial, preoperative, 6-month follow-up, and 1-year follow-up visits. All subsequent statistical analyses of outcome scores were carried out using SAS version 9.2.

Foot and Ankle Outcome Score

The FAOS is a region-specific outcome instrument adapted from the KOOS, which has been previously validated [2, 19]. It contains 42 question items organized within five subscales: pain, other symptoms, activities of daily living (ADLs), sports and recreational activities, and foot- and ankle-related quality of life (QoL). The FAOS was designed as a purely patient-reported and self-administered questionnaire in order to eliminate observer bias. Patients respond to each question on a scale from 0 to 4, and a score for each subscale is calculated according to a unique formula designed for this particular instrument. The subscale scores range from 0 (extreme symptoms) to 100 (no symptoms) [18]. Invariably, some patients may not complete all questions for a subscale. In the event a patient does not complete more than two (>2) questions in a subscale, that respective subscale is not scored. Typically, patients finish the questionnaire within 10 min.

The Medical Outcome Study 12-Item Short Form Health Survey

While the FAOS assesses conditions of the foot and ankle, the SF-12 (as well as its predecessors the SF-36 and the SF-36v2) is designed as a general health outcome instrument. It has been previously validated [16, 23], is also self-administered, and contains 12 questions subdivided into eight domains: physical functioning, role-physical, bodily pain, general health, vitality, social functioning, role-emotional, and mental health [24]. Each domain is scored independently and contributes to varying degrees to two summary scores: the physical health component score and the mental health component score. The SF-12 serves as a necessary comparative tool in this study as it is validated and used widely in the literature, specifically in foot- and ankle-related studies [4, 7, 13, 15, 21].

Subjects

Approval from the institutional review board was obtained in order to conduct this study. Patients were identified through the Department of Foot and Ankle Surgery Patient Registry by searching the database for the appropriate diagnosis and procedure codes. Those diagnosed with hallux rigidus (ICD-9 code 735.2) or who underwent first metatarsophalangeal joint fusion (CPT CODE 28750) or cheilectomy (CPT CODE 28289) between January, 2007 and December, 2012 were eligible. Patient charts from eight orthopedic foot and ankle surgeons were reviewed to ensure the absence of any incorrect diagnoses. Content validity questionnaires are not administered as standard of care through the patient registry and therefore were collected prospectively by postal mail with informed consent. Patients under the age of 18 or with incomplete FAOS or SF-12 questionnaires were excluded. A total of 211 patients were included in one or more of four major components of this study: construct validity, content validity, reliability, and responsiveness.

Postoperative Protocols

Patients who underwent first metatarsophalangeal joint fusion were seen at 2, 6 weeks, 3, and 6 months after surgery. AP and lateral radiographs were obtained at each follow-up visit. Patients were advised to remain non-weight-bearing on the operative foot for the first 6 weeks after surgery. At the 2-week follow-up appointment, the below knee splint was removed and the foot was placed in a tall CAM walker boot. Six weeks after surgery, patients began progressive weight-bearing over the next 4 weeks.

Those patients who underwent dorsal cheilectomy at the first metatarsophalangeal joint began weight-bearing as tolerated immediately after surgery with a hard-soled, postoperative shoe. All patients returned within 2 weeks for suture removal. Patients who did not undergo a combined proximal phalangeal osteotomy transitioned into a supportive tennis shoe at that time. Patients who did undergo an osteotomy remained in the hard-soled shoe until 6 weeks after surgery. Most patients progressed to all activities as tolerated at 3 months. We did not quantify how many patients underwent osteotomy as a part of their surgical treatment.

Construct Validity

Construct validity assesses the extent to which an instrument measures what it is designed to measure. This is accomplished by comparing one instrument against a previously validated instrument. One hundred and twenty-five patients who had completed both the FAOS and SF-12 at the same time point were included in this component of the study. The mean subject age was 55.9 years (range, 25 to 84 years). The study group contained 87 females (69.6%) and 38 males (30.4%). A Spearman’s correlation (SRCC) was calculated to compare the FAOS against the SF-12. As suggested in the literature, a correlation coefficient between 0.3 and 0.7 was considered to indicate moderate correlation [14, 20]. A correlation coefficient of 0.7 or greater indicated good correlation but that the instruments were likely too similar and possibly redundant. A correlation coefficient below 0.3 demonstrated that the two instruments were poorly related to one another. We hypothesized that all of the FAOS subscales would demonstrate moderate correlation with the four physical health domains of the SF-12 (physical functioning, role-physical, bodily pain, and general health) and poor correlation with the four mental health domains of the SF-12 (vitality, social functioning, role-emotional, mental health). The presence of floor and ceiling effects was also analyzed to obtain an overall view of the FAOS score distribution. Floor and ceiling effects occur if more than 20% of subjects achieved the worst or best possible score on the FAOS within a particular subscale [22]. The presence of any floor or ceiling effects prohibits the confirmation of the hypothesis, as scores would fall well below or above the minimum and maximum values.

Content Validity

Content validity measures the patient’s own perception of the relevance of each FAOS question as it pertains to their diagnosis of hallux rigidus. Forty patients received a “Relevance Survey” by mail, which asked them to rate each FAOS question as 1 (not relevant/important), 2 (somewhat relevant/important), or 3 (very relevant/important). The mean age of each subject was 58.0 years (range, 38 to 76 years). There were 35 females (87.5%) and five males (12.5%) in the group. Twenty-four patients (60.0%) completed the survey preoperatively, and 16 patients (40.0%) completed it postoperatively. Within each subscale, patient responses were summed and then a mean score was calculated across the study group. A mean score of 2 or greater was considered to demonstrate acceptable content validity [5].

Reliability

Test–retest reliability measures the consistency of an instrument between two separate administrations over a determined period of time. If no treatment or intervention occurs in between the two administrations, the patient should report comparable or similar responses. Thirty-six patients completed a second FAOS questionnaire after completing a standard of care FAOS questionnaire without any interim intervention. Of the FAOS questionnaire pairs analyzed in this component of the study, 31 (86.1%) were completed preoperatively and five (13.8%) were completed postoperatively. The mean patient age was 55.6 years (range, 26 to 75 years). The group consisted of 30 females (83.3%) and six males (16.6%). The mean time interval between the first and second administration of the FAOS was 44.8 days (range, 4 to 92 days). Intraclass correlation coefficients (ICC) were calculated, and a value of at least 0.7 was deemed acceptable to indicate reliability.

Responsiveness

The responsiveness of an instrument refers to its ability to detect a change in the patient’s health status following intervention (i.e., surgery). Sixty patients who underwent first metatarsophalangeal joint fusion or cheilectomy were included in this component. The average patient age was 57.4 years (range, 33 to 84 years), and there were 43 females (71.7%) and 17 males (28.3%). Patients completed their postoperative FAOS surveys at a mean 16.7 months after surgery (range, 6.0 to 61.2 months). Analysis of FAOS scores was carried out using the Student’s paired t test (significance set at p < 0.05).

Responsiveness was also evaluated by calculating the effect size (ES) and standard response mean (SRM) for each FAOS subscale. A value greater than 0.5 was deemed to demonstrate acceptable responsiveness of the respective subscale.

Results

Construct Validity

All FAOS subscales demonstrated moderate correlation with the physical health and function component scores of the SF-12. All of the FAOS subscales demonstrated poor correlation with the SF-12 mental health and mental health component score (Table 1).

Table 1.

Construct validity between the Short-Form 12 (SF-12) and the foot and ankle outcome score (FAOS)

SF-12 subdomain	Stat	FAOS pain	FAOS symptoms	FAOS ADLs	FAOS sports/rec	FAOS QoL
Physical function	Correlation coefficient	0.5101	0.4185	0.7191	0.3506	0.634
	p Value	<0.0001	<0.0001	<0.0001	<0.0001	<0.0001
	N	123	123	123	123	123
Role-physical	Correlation coefficient	0.4512	0.2964	0.4805	0.3198	0.5638
	p Value	<0.0001	0.0008	<0.0001	0.0003	<0.0001
	N	124	124	124	124	124
Bodily pain	Correlation coefficient	0.5427	0.3276	0.5138	0.3406	0.5489
	p Value	<0.0001	0.0002	<0.0001	0.0001	<0.0001
	N	121	121	121	121	121
General health	Correlation coefficient	0.1682	0.0904	0.2373	−0.037	0.1382
	p Value	0.0618	0.318	0.008	0.6771	0.1257
	N	124	124	124	124	124
Vitality	Correlation coefficient	0.3734	0.1485	0.3337	0.1029	0.3037
	p Value	<0.0001	0.0984	0.0001	0.2531	0.0006
	N	125	125	125	125	125
Social functioning	Correlation coefficient	0.4148	0.3563	0.3842	0.2465	0.4124
	p Value	<0.0001	<0.0001	<0.0001	0.0062	<0.0001
	N	122	122	122	122	122
Role-emotional	Correlation coefficient	0.3386	0.2948	0.2767	0.1653	0.3276
	p Value	0.0001	0.001	0.0021	0.0699	0.0002
	N	121	121	121	121	121
Mental health	Correlation coefficient	0.2695	0.0921	0.2042	0.0766	0.2239
	p Value	0.0024	0.3067	0.0223	0.3958	0.0121
	N	125	125	125	125	125
Physical health component	Correlation coefficient	0.5773	0.4298	0.6577	0.4028	0.6444
	p Value	<0.0001	<0.0001	<0.0001	<0.0001	<0.0001
	N	115	115	115	115	115
Mental health component	Correlation coefficient	0.2769	0.1472	0.1555	0.0379	0.1684
	p Value	0.0027	0.1164	0.097	0.6869	0.0719
	N	115	115	115	115	115

Construct validity assesses whether the obtained FAOS scores accurately reflect the outcomes they are intended to measure though comparison with the SF-12. ADLs activities of daily living, QoL quality of life

Score Distribution

Floor effects were not observed in the subject population. Ceiling effects were found in the ADLs and sports/recreation subscales, with 25.6 and 24% of respondents achieving the maximum score, respectively.

Content Validity

Overall, all FAOS subscales except ADLs were found to contain relevant questions from the patient’s perspective (Table 2). The preoperative group had a mean relevance score of 1.84, but when combined with postoperative scores, it was greater than 2 which was the level deemed necessary to conclude relevance.

Table 2.

Mean relevance scores for each foot and ankle outcome score (FAOS) subscale

FAOS subscale	Overall	Pre	Post
Pain	2.28	2.29	2.28
Symptoms	2.0	1.84	2.24
Daily activities	1.9	1.84	1.99
Sports/recreation	2.21	2.23	2.16
Quality of life	2.63	2.72	2.5

Mean relevance scores for each FAOS subscale as reported by patients. Overall, preoperative and postoperative relevance scores are indicated

Test–Retest Reliability

Intraclass correlation coefficients for the FAOS pain, symptoms, ADLs, sports/recreation, and QoL subscales were 0.871, 0.723, 0.834, 0.867, and 0.905, respectively, indicating acceptable test–retest reliability for all subscales (Table 3). It is important to note that the actual number of patients used for each subscale ranged from 29 to 36 as not all patients completed questions for all five subscales; thus, subscale score comparison could not be determined.

Table 3.

Reliability of foot and ankle outcome score (FAOS) subscales

FAOS subscale	N	Mean ± SD first FAOS	Mean ± SD second FAOS	ICC
Pain	34	62.23 ± 24.65	62.64 ± 24.65	0.882 (0.780, 0.940)
Symptoms	35	70.06 ± 20.76	67.12 ± 20.09	0.707 (0.495, 0.840)
Daily activities	32	79.28 ± 23.13	79.51 ± 18.88	0.797 (0.624, 0.896)
Sports/recreation	29	56.01 ± 34.22	53.74 ± 32.83	0.855 (0.715, 0.929)
Quality of life	36	37.38 ± 27.97	36.57 ± 27.43	0.908 (0.827, 0.952)

Reliability evaluates whether FAOS scores remain stable when the underlying condition is stable. All five subscales exhibited good test–retest reliability with ICCs of 0.723 and above. The N value differs among subscales because some patients did not complete all questions of that particular subscale; thus, subscale score comparison could not be determined

Responsiveness

Mean baseline scores were compared with the mean postoperative scores; there were statistically significant improvements (p < 0.05) in all five FAOS subscales (Table 4).

Table 4.

Responsiveness of foot and ankle outcome score (FAOS) subscales

FAOS subscale	N	Preoperative score ± SD	Postoperative score ± SD	p Value	ES	SRM
Pain	55	58.59 ± 17.30	79.00 ± 18.07	<0.0001	1.180	0.929
Symptoms	53	66.18 ± 19.90	77.69 ± 19.85	0.0013	0.578	0.466
Daily activities	44	75.18 ± 17.60	86.05 ± 16.08	0.0013	0.617	0.519
Sports/recreation	41	51.28 ± 28.08	72.59 ± 25.06	0.0004	0.759	0.608
Quality of life	55	36.40 ± 21.86	56.93 ± 25.65	<0.0001	0.939	0.839

Responsiveness reflects whether the instrument changes as the underlying condition changes. All FAOS subscales were responsive to change after surgery. The scores shown are expressed as mean values with standard deviations. The effect size (ES) and standard response mean (SRM) indicate that QoL and pain were the most responsive FAOS subscales. Significance set at p < 0.05. The N value differs among subscales because some patients did not complete all questions of that particular subscale; thus, subscale score comparison could not be determined

The lowest ES and SRM values were seen in the symptoms subscale, at 0.578 and 0.466, respectively. All other ES and SRM values were greater than 0.5 in the other subscales. It was again observed that not all patients completed all questions for a respective FAOS subscale, which resulted in a variable N value across subscales (range 41–55) (Table 4).

Discussion

There is a clear need and push for the use of valid and reliable outcome measures for foot and ankle surgery [10]. The variable clinical presentation and severity of symptoms associated with hallux rigidus, also considered hallux osteoarthritis, supports the need for a validated, diagnosis-specific, assessment of outcomes. This study demonstrates the acceptable construct and content validity, reliability, and responsiveness of the FAOS for hallux rigidus.

Construct validity provides an assessment of an instruments’ ability to measure what it was intended to measure. When compared to the general health questionnaire, SF-12, all FAOS subscales demonstrated good overall correlation with the physical health component. There was poor correlation with all mental health component domains of the SF-12, which is consistent with our hypothesis and findings of prior studies evaluating the validity of the FAOS [5, 11, 18]. These findings suggest that the FAOS can be administered in combination with the SF-12 and provide similar, but not redundant, patient outcome data.

In our assessment of the content validity, or relevance, of the FAOS from the patient’s perspective, we found that the FAOS was deemed relevant in all subscales with the exception of ADLs. Though the ADL subscale showed a trend toward relevance with a preoperative and postoperative average of 1.9, this finding is consistent with those found in recent FAOS validation studies for hallux valgus and flatfoot deformity [5, 11]. Though it is, however, important to note that in the validation of Roos et al. of the FAOS for ankle ligament reconstruction all subscales were deemed relevant [18]. These consistent findings may represent an overall inability of the ADLs subscale questions to play a significant role in a patient’s assessment of their outcome.

Consistency between administrations is essential for any validated outcome measure. All five FAOS subscales demonstrated acceptable test–retest reliability when administered to hallux rigidus patients. The second FAOS was completed between 4 and 92 days following initial administration. This broad range does carry a concern regarding the ability of the FAOS to provide consistent assessment in both the immediate and long term. However, this concern is tempered by our reliability findings being consistent with those proven for the FAOS validation for both hallux valgus and flatfoot deformity with each also found the FAOS reliable on all subscales [5, 11].

We also found that the FAOS showed robust responsiveness across all five subscales. This was most significant for both the pain and QoL subscales (p < 0.0001). This result further supported the utilization of the FAOS as a region-specific outcome measure, especially for hallux rigidus, as this is the first validation study to exhibit responsiveness across all subscales. It should be noted that Roos et al. did not assess responsiveness in their initial FAOS validation study [18].

Despite our findings pointing toward the validation of the FAOS for hallux rigidus, there are several limitations to our study. First of all, this study pulls from a single population of patients from one institution. Furthermore, we only included patients with the diagnosis of hallux rigidus (735.2) which potentially excluded patients with diagnoses such as first metatarsophalangeal joint arthritis due to other means (i.e., osteoarthritis, posttraumatic, rheumatoid arthritis, etc.). It is not possible to know what effect the inclusion of these diagnoses would have had on our data collection and results.

We also did not correlate the radiographic grade or severity of hallux rigidus with FAOS scores as our primary goal of the study was to determine performance of the scales and be able to justify their broad use. Furthermore, the minimum length of follow-up of an average of 6 months could be viewed as a limitation in comparison to the traditionally recommended 2-year follow-up period. Our evaluation and validation included patients over 6 years. Since there was a significant improvement (i.e., responsiveness) at 6 months, there likely would be at other time points as well. Also, previous published studies have shown a precedent for 6-month follow-up. In addition, for both reliability and responsiveness, some subscales were not fully completed by all patients, potentially leading to that subscale not being scored for that respective patient and resulting variable N values for that subscale as denoted in Tables 3 and 4. Particularly, ADLs and sports/recreation subscales were observed to have the lowest rate of completion for both reliability and responsiveness. It is important to note that this observation is consistent with prior FAOS validation studies [11]. Furthermore, this may represent patients simply feeling as though these questions did not apply to their specific problem and left them blank. In the end, however, our relevance scores were good across all subscales with the exception of ADLs, which also showed a trend toward relevance (avg. 1.9).

Also, though no floor effects were identified, as mentioned previously, ceiling effects were found in the ADLs and sports/recreation subscales, with 25.6 and 24% of respondents achieving the maximum score, respectively. These findings are concerning but consistent with those demonstrated by Roos et al. and Chen et al., with ceiling effects identified in both studies [5, 18]. It is the perspective of our research group that the consistency of such ceiling effects across several FAOS validation studies should not lead to the non-use of the FAOS but serve as a known limitation of an otherwise valid and useful patient outcome measure going forward.

Despite being a region-specific tool, we have studied the FAOS now in three different conditions affecting the foot and ankle to the current study for hallux rigidus, hallux valgus, and flatfoot deformity, since it was originally described and validated for use in patients with lateral ankle instability [5, 11, 18]. The reason to study these conditions, the most common pathologies requiring surgeries seen by foot and ankle orthopedic surgeons, is that such a tool may be valid for one condition and not another. This is particularly true with regards to relevance (i.e., patients may feel that the tool is not important depending on their pathology) and responsiveness (i.e., a patient may or may not show an improvement with surgical treatment based upon their condition). In addition, the goal of the series of studies is to provide a reference against which researchers and clinicians can base conclusions of outcomes in studies utilizing the FAOS.

In the future, we plan to validate the FAOS for osteochondral defects of the talus and other foot and ankle pathologies. To our knowledge, no other legacy scale has been validated for such a diverse number of foot and ankle pathologies which will enable its routine use in foot ankle research and registries. Future directions will be to also assess the FAOS with more modern response theory models (i.e., Rasch models, as opposed to classical models used in the current study). Finally, efforts should be made in general to assess patient-reported outcome scales such as computer adaptive tests (CATs) which are more efficient and effectively eliminate floor and ceiling effects.

In conclusion, the treatment of the varying stages of HR differs widely among surgeons, further complicating the measurement of outcomes. Our results demonstrate that the FAOS is a valid, reliable, and responsive patient-reported outcome tool for hallux rigidus. Furthermore, the FAOS has been validated for lateral ankle ligament reconstruction, hallux valgus, and adult-acquired flatfoot deformity [5]. Despite this, in a recent review, the FAOS ranked 11th among most commonly used outcome measures in the foot and ankle literature over the past decade [10]. We attest that due to its broad applicability and proven validation across multiple foot and ankle pathologies, the FAOS represents a patient-centered outcome measure that can be reliably used for the assessment of patients with hallux rigidus and that its use be considered as we move into the next era of clinical outcomes and patient-centered research.