Inter-rater reliability of affected-side weight bearing ratio measurement in postoperative patients with proximal femoral fractures

Abstract

[Purpose] We examined the inter-rater reliability of the affected-side weight-bearing ratio in patients who have undergone proximal femoral fracture surgery. [Participants and Methods] Ten patients with hip fractures were included in this study. Weight-bearing ratios were measured five times daily on postoperative days 3, 5, and 7. Three physical therapists independently assessed the patients. [Results] High intra-class correlation coefficients were observed across all days and measurement trials. The coefficient of variation (CV) showed no significant difference between days. CV was negatively associated with weight-bearing ratio but was not influenced by pain, body-mass index, age, sex, or surgical procedure. [Conclusion] In patients with proximal femoral hip fractures, weight-bearing ratio measurements on the affected side demonstrated high inter-rater reliability, even in a single trial from postoperative day 3. Pain during weight-bearing may contribute to variability in measurements, particularly in patients with low weight-bearing ratios. This method is feasible, minimally burdensome, and can be implemented in various rehabilitation environments without the need for specialized equipment. Future studies should examine its generalizability across multiple facilities and assess its reliability in patients with cognitive impairment.

INTRODUCTION

As the Japanese population grows older, the incidence of proximal femoral fractures has been increasing concurrently. According to Takusari et al., approximately 190,000 such injuries occurred in 2017¹⁾. In Asia, Cheung et al. projected an increase in the number of cases from 1,124,060 in 2018 to 2,563,488 by 2050, representing a 2.28-fold rise²⁾. Nikkel et al. reported that in patients undergoing surgery for proximal femoral fractures, longer hospital stays were associated with a higher mortality rate within 30 days after discharge, highlighting the importance of the early postoperative period³⁾. Therefore, ensuring mobility from the early postoperative phase onwards is an urgent concern for patients after proximal femoral fracture surgery. Various prognostic factors related to the recovery of walking ability have been investigated. For example, Fukuda et al. reported that walking ability on postoperative day 7 was significantly associated with the likelihood of discharge in patients with proximal femoral fractures⁴⁾. They also found that patients who returned home had significantly higher preinjury mobility than those who were transferred to another facility. Tsushima et al. classified postoperative patients with proximal femoral fractures into two groups: 35 who were able to walk at discharge and 19 who were not⁵⁾. They compared scores on the revised Hasegawa’s Dementia Scale (HDS-R) and found that the walking group had significantly higher scores than the non-walking group, highlighting the importance of cognitive function⁵⁾. Furthermore, Kang et al. conducted a multivariate regression analysis in patients with fragility hip fractures and identified factors influencing physical function and quality of life (QOL) at six months postoperatively⁶⁾. They reported that early 4-meter gait speed tests, the Korean version of the Mini-Mental State Examination (K-MMSE), Korean Instrumental Activities of Daily Living (K-IADL), and the Korean version of the Modified Barthel Index were independent predictors⁶⁾. In addition, physical function assessments such as muscle strength and balance tests have been utilized as effective tools for early postoperative prognostic prediction in patients with proximal femoral fractures. Tamura et al. reported that in older patients with proximal femoral fractures, the Berg Balance Scale score at admission was a significant predictor of walking independence at discharge⁷⁾. Katoh and Kaneko also reported that isometric knee extension strength of the quadriceps is useful for predicting independent mobility four weeks after surgery in patients with proximal femoral fractures⁸⁾. They further reported that the intra-rater reliability of the handheld dynamometer used for this measurement was >0.9. However, muscle strength testing may pose challenges in clinical settings due to the need for specialized equipment, procedural complexity, and associated costs. There is an increasing need to establish simple and low-cost physical function tests that can be performed in the early postoperative period to predict patient prognosis. Therefore, we focused on the weight-bearing ratio (WBR) on the affected side as a simple and inexpensive physical function assessment tool, and examined its relationship with walking acquisition in patients after proximal femoral fracture surgery. We had previously reported that WBR on postoperative day 7 was significantly associated with walking independence, with a cutoff value of 72.3%⁹⁾. Based on our previous study, WBR on the affected side is an important early predictor of walking independence in patients after proximal femoral fracture surgery. However, to promote the widespread clinical use of this indicator, especially across multiple institutions, it is essential to establish its reliability, particularly its inter-rater reliability.

Hoang-Kim et al. recommended the use of standardized patient-reported outcomes to evaluate functional recovery after femoral neck fracture surgery, emphasizing the importance of ensuring both generalizability and reproducibility of clinical assessments¹⁰⁾. In this study, we examined the inter-rater reliability of WBR measurements on the affected side during the early postoperative period following proximal femoral fracture surgery. It is important to note that the purpose of the present study was to examine the inter-rater reliability of the affected-side weight-bearing ratio rather than to evaluate its prognostic ability. Therefore, this study should be positioned as a reliability investigation aimed at supporting the clinical applicability of WBR in the early postoperative phase.

PARTICIPANTS AND METHODS

This observational study aimed to examine the inter-rater reliability of WBR measurements. The participants were 10 patients (three men and seven women) who underwent either hemiarthroplasty or internal fixation for proximal femoral fractures (femoral neck or intertrochanteric fractures) at our hospital between April and July 2023 and received rehabilitation provided by physical therapy professionals. The participants had a mean age of 75.7 ± 6.5 years, height of 155.2 ± 3.6 cm, body weight of 54.3 ± 8.4 kg, body mass index (BMI) of 22.5 ± 3.2, and a mean waiting period before surgery of 4.0 ± 5.2 days. Exclusion criteria included a score of 20 or lower on the HDS-R; a history of neurological, musculoskeletal, or internal disorders that could interfere with movement; and cases in which postoperative weight-bearing restrictions were imposed. Survey items included disease diagnosis and surgical procedures that were obtained from medical records. WBR and weight-bearing pain were evaluated using a physical function assessment. For WBR measurements, two scales were placed on the floor. Each participant held onto parallel bars at the level of the greater trochanter with both hands, and stood with each foot on a separate scale. They were instructed to load their affected limb maximally for 3 seconds, which was repeated five times. No physical assistance or verbal guidance was provided by the examiners during the measurements.

WBR was defined as the load on the affected side divided by the participant’s body weight. Each session consisted of five trials, and the coefficient of variation (CV) was calculated from the mean and standard deviation (SD) of the five WBR measurements. Weight-bearing pain was assessed after each session using the Numerical Rating Scale (NRS). All examiners and participants practiced the procedure adequately before the formal measurements were carried out.

The detailed evaluation method and the background of the raters are described below. The verbal instructions during the assessment were standardized as follows: “Please place as much weight as possible on your right/left leg [depending on the affected side] while holding onto the parallel bars”, followed by, “Please maintain that position for 3 seconds”. Participants lowered their weight after each trial, stepped off the scales, and then resumed the standing position for the next measurement to ensure that each trial was performed independently. Six physical therapists from the orthopedic team at our department served as raters in this study. Their clinical experience included one therapist with 15 years, one with 10 years, two with 7 years, one with 2 years, and one with 1 year of experience. On each measurement day, three therapists who were on duty conducted the assessments; therefore, the combination of raters differed among the three days. Before data collection, all raters received standardized training regarding the verbal instructions, patient positioning, weight-shift timing, and the measurement protocol to ensure consistency across assessors.

Statistical analyses were conducted using R (version 4.4.0) and Python (version 3.11). To assess the number of trials required to ensure reliability, and whether reliability differed depending on the measurement day, ICC (2, k) values were calculated using the average of one, two, three, four, and five trials.

Sample size was determined by using the method proposed by Walter et al.¹¹⁾ and calculated using the R software (version 4.4.0) with the pwr package. Assuming a significance level of 0.05, statistical power of 0.80, minimum acceptable intraclass correlation coefficient (ICC) of 0.60, an expected ICC of 0.85, five raters, and five measurements per rater, the required sample size was calculated to be 10 participants. To prevent information bias and intentional manipulation of weight-bearing values, the raters were prohibited from sharing measurement results with each other.

When evaluating inter-rater reliability, where multiple raters assess multiple participants, appropriate statistics include ICC (2, k), ICC(3, k), Cohen’s kappa, and Kendall’s W^{12, 13)}. Cohen’s kappa is typically used when two raters evaluate two or more participants and is applied to nominal or ordinal data; therefore, it was not suitable for this study. Furthermore, Craigie et al. stated that, although kappa statistics are used for nominal variables in questionnaire-based studies, Kendall’s W is more robust for ordinal scales¹⁴⁾. Hence, W was also deemed inappropriate for use in this study.

As WBR is considered a ratio scale variable, the ICC was determined to be an appropriate statistical method. ICC (2, k) treats both raters and subjects as random effects, and is suitable for generalizing the reliability of the measurement method itself. In contrast, ICC (3, k) treats raters as fixed effects and assesses agreement only within a specific group of raters. Therefore, ICC (2, k) was used in this study.

Next, the CV for each 5-trial session was calculated as (SD / mean) × 100, and its relationship with the NRS score was analyzed using Spearman’s rank correlation coefficient. The Friedman test was used to compare CVs across postoperative days 3, 5, and 7. The significance level was set at 5% for all statistical analyses.

To examine the association between CV and various explanatory variables, a linear mixed-effects model (Hereinafter, LMM) was employed, as multiple measurements were obtained from the same participant and the independence of observations could not be assumed. In this model, CV was set as the dependent variable, and the mean WBR, NRS score, age, sex, BMI, and surgical procedure were included as explanatory variables.

To account for inter-individual variability, all CV values from postoperative days 3, 5, and 7 were pooled into one LMM model, and the ICC was calculated using the estimated between-subject variance and residual variance. Specifically, ICC was computed using the following formula:

ICC=Between-subject variance / (Between-subject variance + Residual variance),

which quantifies the proportion of the total variance attributable to individual differences.

All statistical tests were conducted with a significance level of 5%.

This study was approved by the Ethics Committee of the Koshigaya-Seiwa Hospital (Approval No. 06). Prior to participation, all patients received a detailed explanation of the study’s purpose, procedures, and handling of the results. Written informed consent was obtained. No financial support or conflicts of interest were associated with this study.

RESULTS

Results for inter-rater reliability are shown in Table 1. Significant intraclass correlations were observed on postoperative days 3, 5, and 7. The ICC with 95% confidence intervals (CIs) on postoperative day 3 were as follows: 0.970 [0.936, 0.986] for a single trial, 0.985 [0.966, 0.993] for the mean of two trials, 0.993 [0.987, 0.996] for three trials, 0.995 [0.991, 0.997] for four trials, and 0.996 [0.993, 0.997] for five trials.

Table 1. Demographic characteristics of the study participants (n=10), including mean age and other clinical parameters.

Variable	Mean ± SD
Age (years)	75.7 ± 6.5
Sex	Male: 3/Female: 7
Height (cm)	155.2 ± 3.6
Weight (kg)	54.3 ± 8.4
Body mass index (kg/m²)	22.5 ± 3.2
Time from injury to surgery (days)	4.0 ± 5.2

On postoperative day 5, the ICCs were 0.955 [0.904, 0.978] for a single trial, 0.977 [0.950, 0.989] for two trials, 0.989 [0.979, 0.994] for three trials, 0.992 [0.986, 0.996] for four trials, and 0.994 [0.989, 0.996] for five trials.

On postoperative day 7, ICCs were 0.977 [0.952, 0.988] for a single trial, 0.988 [0.975, 0.994] for two trials, 0.992 [0.986, 0.996] for three trials, 0.994 [0.990, 0.997] for four trials, and 0.996 [0.992, 0.997] for five trials (Table 2).

Table 2. Intraclass correlation coefficients (ICC) and 95% confidence intervals for each number of trials on postoperative day 3, day 5, and day 7.

Measurement day	1st ICC	95% CI	2nd ICC	95% CI	3rd ICC	95% CI	4th ICC	95% CI	5th ICC	95% CI
Day3	0.970	[0.936, 0.986]	0.985	[0.966, 0.993]	0.993	[0.987, 0.996]	0.995	[0.991, 0.997]	0.996	[0.993, 0.997]
Day5	0.955	[0.904, 0.978]	0.977	[0.950, 0.989]	0.989	[0.979, 0.994]	0.992	[0.986, 0.996]	0.994	[0.989, 0.996]
Day7	0.977	[0.952, 0.988]	0.988	[0.975, 0.994]	0.992	[0.986, 0.996]	0.994	[0.990, 0.997]	0.996	[0.992, 0.997]

CVs on each postoperative day were 5.84 ± 3.07 on day 3, 3.61 ± 2.15 on day 5, and 4.12 ± 2.70 on day 7, with no statistically significant differences observed between measurement days.

LMM analysis showed that CV was significantly negatively associated with WBR, whereas NRS had no significant effect on CV. No significant associations were found for BMI, age, sex, or surgical procedure (Table 3).

Table 3. Results of a linear mixed-effects model examining factors influencing weight-bearing ratio variability.

Variance	Coefficient (β)	95% CI
Age (years)	0.067	[−0.127, 0.261]
Body mass index (kg/m²)	0.429	[−0.307, 1.166]
WBR	−0.216	[−0.362, −0.070]
Pain	0.074	[−0.433, 0.581]
Sex (male=1 female=0)	−1.592	[−5.346, 2.163]
Surgical procedure (ORIF=1, Hemiarthroplasty=0 )	1.655	[−1.977, 5.286]
Group Var	3.652

*p<0.05; **p<0.01.

WBR: weight bearing ratio; β: regression coefficient; ORIF: open reduction and internal fixation.

Furthermore, when constructing an LMM combining all measurements from postoperative days 3, 5, and 7, the estimated between-subject variance was 147.89 and the residual variance was 30.08, yielding an ICC of 0.831.

DISCUSSION

In this observational study, we aimed to determine the inter-rater reliability of the weight-bearing ratio of the affected side in patients with proximal femoral fractures. Reliability was evaluated on postoperative days 3, 5, and 7 using single measurements and the mean values of two, three, four, and five repeated trials. The ICC values were all >0.90 for every measurement day and method, indicating excellent reliability. These findings suggest that early postoperative assessment of the fractured-side weight-bearing ratio provides highly reliable data in patients with proximal femoral fractures.

An ICC value of ≥0.75 is considered to indicate moderate reliability, while a value of ≥0.90 is regarded as excellent reliability¹²⁾. In the present study, ICC exceeded 0.90 across all measurement days and trial counts, suggesting that weight-bearing ratio assessment in patients with proximal femoral fractures may provide valuable information, even with a single trial. Eng and Chu reported good reliability in five repeated trials of average weight-bearing measurements during static standing in post-stroke patients for both the paretic and non-paretic sides¹⁵⁾. Although the patient populations differed, our results were broadly consistent with their findings. Data acquisition from a single trial reduces the patient burden and shortens the evaluation time for therapists.

Furthermore, no significant differences were found in the coefficients of variation across postoperative days 3, 5, and 7, suggesting that the measurement accuracy of the weight-bearing ratio was consistent across these early postoperative time points. Fukuda et al.⁴⁾ investigated the relationship between walking ability one week after surgery for proximal femoral fractures and the discharge destination. They reported that patients with a Functional Ambulation Category (FAC) score of 3 or higher at postoperative day 7 were significantly more likely to be discharged home, suggesting that walking ability at one week post-surgery may serve as a useful clinical indicator for discharge planning. In contrast, the affected-side WBR demonstrated high reliability even on postoperative day 3, indicating that this assessment method is applicable regardless of postoperative acuity or measurement day. Given the trend toward shorter hospital stays and the increasing demand for reliable early prognostic indicators, the measurement of the affected-side weight-bearing ratio on postoperative day 3 may serve as a dependable early evaluation tool for patients after proximal femoral fracture surgery. Numerous previous studies have demonstrated that the affected-side weight-bearing ratio is a crucial factor for predicting prognosis after surgery for proximal femoral fractures. Although the present study does not directly investigate prognosis, it is valuable in that it successfully standardized the protocol for this important assessment, making it independent of the rater’s skill. The Five Times Sit to Stand (FTSTS) Test is widely used outside Japan as an assessment tool and is recommended for evaluating mobility and fall risk in patients in the acute postoperative phase¹⁶⁾. However, it has also been reported in the same journal that 88% of patients in the acute postoperative phase were unable to rise from a chair. While both assessments were performed during the acute phase, the measurement of the affected-side WBR may be more widely applicable because it can be conducted in patients who are able to maintain a standing position with support. In addition, Kristensen et al. reported that the inter-rater reliability of the Cumulated Ambulation Score (CAS), a chart for assessing mobility in patients after proximal femoral fracture, was high when evaluated by two physical therapists¹⁷⁾. In contrast, the present study examined the reliability of measurements conducted by six randomly assigned raters, which enhances generalizability by reducing rater-specific bias.

Background factors such as pain, BMI, age, sex, and surgical method did not influence CV. Only the weight-bearing ratio had an impact. It has been generally reported that increased weight bearing leads to greater muscle activation¹⁸⁾, and that greater muscle activation is associated with reduced CV¹⁹⁾. In the present study, it was also observed that an increase in the WBR was accompanied by increased muscle activity, which in turn stabilized muscle output and reduced variability in the weight-bearing measurements. These findings suggest that special attention should be paid to the accuracy of measurements in patients or at time points where the WBR is low. Furthermore, when data from postoperative days 3, 5, and 7 were pooled and analyzed using a LMM model, the resulting ICC was 0.83, indicating that 83% of the variability in the WBR was attributable to inter-individual differences. These results support the conclusion that even a single measurement of the WBR can provide high inter-rater reliability; however, it should be noted that most of the variation originates from individual differences. It is also possible that lower WBR are associated with greater variability in the results.

This study has some limitations. First, inter-rater reliability was examined only within a single institution. Therefore, future studies should include multicenter investigations to enhance generalizability. Second, the raters were not consistent across all the measurement days. Although we attempted to balance clinical experience as much as possible, the measurements were conducted by different assessors. However, this variation may also reflect real-world clinical conditions involving multiple therapists. Third, Bai et al.²⁰⁾ classified patients into three groups—those with femoral neck fractures, trochanteric fractures, and unstable trochanteric fractures—and conducted physical therapy evaluations accordingly. The study reported that isometric knee extension strength at 1, 2, 3, and 4 weeks postoperatively was significantly lower in the group with unstable trochanteric fractures. Additionally, according to the Japanese Orthopaedic Association (JOA) score, which includes pain-related items, patients with femoral neck fractures had significantly higher scores than those with trochanteric fractures. As our study did not classify patients according to fracture type, it is necessary to consider the potential impact of fracture severity on weight-bearing pain and the ability to maintain a standing position. Fourth, although five raters were assumed in the sample size calculation based on the method of Walter et al., the actual measurements were conducted by three raters per day due to manpower constraints. Therefore, the number of raters did not fully meet the theoretical assumption, which may have influenced the estimates of inter-rater reliability. Patients with a score below 20 on the Hasegawa Dementia Scale-Revised (HDS-R) were excluded. Arnold et al.²¹⁾ investigated the intra- and inter-rater reliability of handheld dynamometry in community-dwelling older adults aged 65–92 years and reported that those with Mini-Mental State Examination (MMSE) scores below 26 tended to demonstrate a 15% decrease in measurement reproducibility. Given that the prevalence of cognitive impairment among patients with proximal femoral fractures has been reported to be 27.2%²²⁾, it is necessary in future studies to examine whether the presence of dementia affects the reliability of weight-bearing assessments on the affected side performed in a standing position.

In patients with proximal femoral hip fractures, WBR measurements on the affected side demonstrated high inter-examiner reliability, even in a single trial from postoperative day 3. Pain during weight-bearing may contribute to variability in measurements, particularly in patients with low WBR. This method is feasible, minimally burdensome, and can be implemented in various rehabilitation settings without the need for specialized equipment.

Conflict of interest

The authors declare no conflicts of interest associated with this manuscript.