Use of and Confidence Administering Outcome Measures among Clinical Prosthetists: Results from a National Survey and Mixed-Methods Training Program

Abstract

Background

Outcome measures can be used in prosthetic practices to evaluate interventions, inform decision making, monitor progress, document outcomes, and justify services. Strategies to enhance prosthetists' ability to use outcome measures are needed to facilitate their adoption in routine practice.

Objective

To assess prosthetists' use of outcome measures and evaluate the effects of training on their confidence administering performance-based measures.

Design

Cross-sectional and single group pretest-posttest survey

Methods

Seventy-nine certified prosthetists (mean of 16.0 years of clinical experience) were surveyed about their experiences with 20 standardized outcome measures. Prosthetists were formally trained by the investigators to administer the Timed Up and Go and Amputee Mobility Predictor. Prosthetists’ confidence in administering the Timed Up and Go and Amputee Mobility Predictor was measured before and after training.

Results

The majority (62%) of prosthetists were classified as non-routine outcome measure users. Confidence administering the TUG and AMP prior to training was low-to-moderate across the study sample. Training significantly (p<0.0001) improved prosthetists' confidence administering both instruments.

Conclusion

Prosthetists in this study reported limited use of and confidence with standardized outcome measures. Interactive training resulted in a statistically significant increase of prosthetists' confidence in administering the TUG and AMP and may facilitate use of outcome measures in clinical practice.

II. Clinical Relevance

Frequency of outcome measure use in the care of persons with limb loss has not been studied. Study results suggest that prosthetists may not regularly use standardized outcome measures and report limited confidence in administering them. Training enhances confidence and may encourage use of outcome measures in clinical practice.

III. Background

Standardized outcomes measurement has long been advocated as a strategy for enhancing the quality of healthcare provided to those who require prosthetic services.^1-3 When appropriately selected and administered, self-report and performance-based outcome measures can be used for a variety of purposes, including determining current level of function, predicting future ability, determining contribution of an intervention to function, informing clinical decisions, measuring changes over time, and fulfilling documentation requirements (i.e., to provide supporting evidence about patients' functional abilities for reimbursement purposes). Despite the value for use of outcome measures in prosthetic practices, it is unknown to what extent they are routinely administered or how they are commonly used.

Although standardized outcome measures have not historically been employed in prosthetics practice, prosthetists may soon be required to regularly administer outcome measures and document the results from such instruments in order to justify selected prosthetic interventions. Since 2013, the Centers for Medicare and Medicaid Services (CMS) have required enhanced documentation from providers of physical, occupational, and speech-language therapies describing Medicare beneficiaries' initial function and condition, furnished therapy services, and achieved outcomes.^{4, 5} To be considered for reimbursement, therapy providers must include non-payable G codes (and modifiers) on the claim forms to document the beneficiary's functional impairments and activity limitations at the outset of therapy, during treatment, and at discharge. G codes are functional classifiers, based on the World Health Organization's International Classification of Functioning, Disability, and Health (ICF)⁶, used to describe a patient's current impairment and associated activity limitations. In short, healthcare providers are now being required to document outcomes that demonstrate the functional value of the interventions selected. It is reasonable to assume that prosthetists may soon be required to provide G codes or similar documentation of outcomes assessment in the provision of prosthetic services.

Practitioners' lack of familiarity and experience with standardized outcome measures may be a barrier to their routine use in prosthetics practice.^7-9 While contemporary prosthetic and orthotic educational standards¹⁰ require competency in clinical evaluation and documentation (skills which typically include standardized procedures for outcome assessment), academic programs have only recently adopted standardized outcomes measurement in their curricula. Therefore, a large majority of practicing prosthetists may be unfamiliar with appropriate use of outcome measures in clinical practice. This gap in knowledge and experience could be addressed by introducing outcome measure training through professional continuing education courses. Specifically, mixed-method education strategies, which have been shown to produce more effective learning than didactic or interactive methods alone,¹¹ may be employed to enhance prosthetists' comfort with standardized measurement of patient outcomes. Such training may help overcome prosthetists' limited exposure to and familiarity with outcome measures. However, the effects of mixed-method training on prosthetists' confidence in administering outcomes measures have not been examined.

The purpose of this study was to assess prosthetists' routine use of outcome measures and to evaluate the effects of a training program on prosthetists' confidence in administering two common performance-based outcome measures. We hypothesized that the majority of prosthetists do not routinely use outcome measures when evaluating patients and that an interactive training program would significantly improve prosthetists' confidence in administering the trained outcome measures. Support of these hypotheses would indicate the presence of barriers to outcomes measurement with prosthetic patients and a strategy (i.e., training) to facilitate their use in clinical practices.

III. Methods

Study Design

This study was conducted to determine prosthetists' experience with performance-based and self-report outcome measures and to assess the impact of formal training on prosthetists' confidence in administering selected performance-based measures. Prosthetist participants agreed to attend a hands-on, interactive training program and complete pre- and post-training surveys. Participants were trained to appropriately administer two performance-based measures, the Timed Up and Go (TUG)¹² and the Amputee Mobility Predictor (AMP).¹³ Both the TUG and AMP have been reported to be reliable and valid measures of mobility in persons with lower limb loss^{13, 14} and are recommended for use in clinical practice and research.¹⁵ Pre-training surveys were used to assess prosthetists' typical use of outcome measures in clinical practice. Pre-training and post-training surveys were used to measure confidence in administering the TUG and the AMP. The surveys were developed by the investigators for the purpose of this study.

Training sessions were conducted at prosthetic clinics across the United States. Clinics were identified by the study investigators through professional contacts. Participants were certified and licensed (if applicable) to provide prosthetic services and regularly engaged in the prosthetic care of persons with lower limb loss. All participants met a minimum requirement of providing at least five prosthetic prescriptions per year. Prosthetists were screened for eligibility by phone or email prior to participating in the study. All study procedures were reviewed and approved by the University of Washington and Miami Veterans Affairs Healthcare System institutional review boards.

Study Procedures

Pre-training Survey

Prosthetists completed a pre-training survey to collect basic demographic information (i.e., gender, race, ethnicity, degree and/or certification attained, years of clinical practice, additional training and education in another profession). Participants were also asked three questions pertaining to their use of outcome measures in clinical practice:

1. “Have you ever administered any of the following standardized outcome measures in your daily clinical practice outside of this study?”

2. “How frequently do you use standardized outcome measures to accomplish the following clinical tasks?”

3. “How confident are you in your current ability to administer the following outcome measures?”

Question 1 included a list of 20 performance-based (e.g., 6-Minute Walk Test¹⁶, Berg Balance Scale¹⁷) and self-report outcome measures (e.g., Prosthesis Evaluation Questionnaire¹⁸, Activities Specific Balance Confidence Scale¹⁹) that have been used to assess persons with lower limb loss. The list of measures (Table 1) was compiled by the investigators from various sources, including a recent review of lower limb prosthetic outcome measures¹⁵. Question 2 included six potential outcome measure applications (i.e., document patient status, inform clinical decisions, justify selection of interventions, monitor patients over time, communicate information to others, and predict patient outcomes). The applications included in the survey were adapted from intended uses of standardized measures described by Potter et al.²⁰ Question 3 included reference to the Timed Up and Go (TUG)¹² and the Amputee Mobility Predictor (AMP).¹³ Questions 1 and 2 were assessed on a 5-level ordinal scale with responses that described frequency of use (i.e., never, rarely, sometimes, often, and always). Question 3 was assessed on a 5-level ordinal scale with responses that described degree of confidence (i.e., not at all, a little bit, somewhat, quite a bit, and very much).

Table 1. Outcome measures polled in the pre-training survey.

Outcome Measure Name
Amputee mobility predictor (AMP)
Patient Assessment Validation Evaluation Test (PAVET)
Distance Walk Test (DWT) (e.g., 10-Meter Walk Test)
Timed Walk Test (TWT) (e.g., 2-Minute Walk Test)
Prosthesis Evaluation Questionnaire (PEQ)
Activities Specific Balance Confidence Scale (ABC)
Timed Up and Go (TUG)
Prosthetic Profile of the Amputee (PPA)
Borg Rating of Perceived Exertion (RPE)
Functional Mobility of the Amputee (FMA)
Orthotic and Prosthetic Users Survey (OPUS)
Short-Form 8, 12, or 36 (SF-8, SF-12, or SF-36)
Step Activity Monitor (SAM)
Berg Balance Scale (BBS)
Houghton Questionnaire
Locomotor Capabilities Index (LCI)
Lower Extremity Functional Scale (LEFS)
Rivermead Mobility Index (RMI)
Nottingham Health Profile (NHP)
Trinity Amputation and Prostheses Experience Scales (TAPES)

Training Program

Study investigators (i.e., certified and licensed prosthetists and physical therapists (PT)) conducted the training program. The training program combined didactic (e.g., presentation and period of question and answer) and interactive (e.g., role play and practice administering measures) instructional methods. Mixed methods courses have been shown to produce more effective learning than those based on either method alone.¹¹ The investigators began each training session with a PowerPoint presentation that reviewed the clinical value of outcome measures, general principles of outcome measures as they apply to prosthetic patients, and the psychometric properties, administration methods, and scoring guidelines for the TUG and the AMP. The study investigators then demonstrated set-up, administration, and scoring on volunteer patients with lower limb amputation (if available), a training program participant, or a colleague. Following the demonstration, the prosthetists were required to practice administration, interpretation, and scoring of the measures under observation of study investigators. Study investigators gave the prosthetists verbal, tactile, and positional feedback, as needed, throughout the training. Instructional videos of both set-up and administration of the TUG and AMP were provided to prosthetists on tablet computers (Apple iPad, Cupertino, CA) and reviewed prior to the conclusion of the training. Prosthetists were also provided with written instructions on how to administer and score the TUG and AMP for later reference.

Post-training Survey

Immediately after training, prosthetists completed a brief survey that asked them to again rate their confidence in performing the TUG and AMP (i.e., Question 3, above). Results of the pre- and post-training surveys were compared to evaluate effectiveness of the training program in increasing prosthetists' confidence in administering the measures.

Analyses

Data were analyzed using SAS 9.3 statistical software (SAS Institute Inc., Cary, NC). Response frequencies and means were calculated for demographic information and survey results. The responses for Questions 1 and 2 were collapsed into two categories (i.e., “routine users” and “non-routine users”) for ease of interpretation. “Routine users” of outcome measures were defined as those prosthetists who indicated that they “often” or “always” used a measure. Conversely, “non-routine users” were defined as those who indicated that they “sometimes,” “rarely” or “never” used a measure. Routine and non-routine users' demographic characteristics, frequency of outcome measure use, and reasons for use of outcome measures were compared using the Chi Square test for categorical variables (e.g., frequency of outcome measure use) and the Wilcoxon signed-rank test for continuous variables (e.g., years of practice). Differences in prosthetists' pre- and post-training confidence scores (i.e., Question 3) were assessed using the Wilcoxon signed-rank test with a threshold for significance set at α = 0.05. This nonparametric test was used because responses to the confidence items were not normally distributed.

IV. Results

A total of 79 prosthetists from 37 clinics participated in the study (Table 2). Clinics were located in 12 states (Florida, Illinois, Massachusetts, Maryland, Michigan, New Hampshire, New Jersey, Oregon, Pennsylvania, Tennessee, Texas, and Washington). Prosthetists had a mean ± standard deviation (SD) age of 42.9 ± 9.8 years and had been in practice for 16.0 ± 10.6 years at the time of the survey. The majority of the participants (76%) had completed either a prosthetic certificate or a bachelor's degree in prosthetics. Few participants (5%) completed a Master's degree. Most were certified in both prosthetics and orthotics (65%).

Table 2. Characteristics of study participants.

	All Participants (n = 79)		Routine Usersa (n = 30)		Non-routine Usersb (n = 49)
	n	%	n	%	n	%
Gender
Male	60	75.9%	23	76.7	37	75.5
Female	19	24.1%	7	23.3	12	24.5
Race/Ethnicity
NH White	68	86.1%	27	90.0	41	83.7
Asian	7	8.9%	2	6.7	5	10.2
Hispanic	3	3.8%	1	3.3	2	4.1
More than 1 race	1	1.3%	0	0	1	2.0
Educationc
Certificate only	23	29.1%	12	40.0%	11	22.4%
Bachelor's only	37	46.8%	9	30.0%	28	57.1%
Certificate & Bachelors	12	15.2%	5	16.7%	7	14.3%
Certificate & Masters	4	5.1%	3	10.0%	1	2.0%
Certificate and Other	3	3.8%	1	3.3%	2	4.1%
Clinical Certification
CP	24	30.4%	13	43.3	10	20.4
CPO	51	64.6%	15	50.0	35	71.4
Other	4	5.1%	1	3.3	3	6.1
Other Clinical Experiencec
Physical or Occupational Therapy	6	7.6%
Exercise physiology, Personal Training or Athletic Training	5	6.3%
Other	3	3.8%

CP: certified prosthetist; CPO: certified prosthetist/orthotist.

^aParticipants answered Question 1 (any item) with “often” or “always.”

^bParticipants answered Question 1 (all items) with “never,” “rarely,” or “sometimes.”

^cParticipants may check more than 1 option (i.e., cumulative percent may be > 100%)

The majority of prosthetists (62%) were categorized as “non-routine” users, as they reported using none of the 20 surveyed outcome measures in their daily clinical practices. There were no statistically significant differences (p>0.05) between routine and non-routine outcome measure users in terms of age, years in clinical practice, gender, ethnicity, education, or certification. Measures most commonly administered by routine users included the AMP (22%), Patient Assessment Validation Evaluation Test (17%), and distance walk test (10%) (Figure 1).

Figure 1. Instruments and surveys used routinely (i.e., always or often) by participants in clinical practice. Measures not shown were not routinely used by any participants.

AMP = amputee mobility predictor, PAVET = patient assessment validation evaluation test, DWT = distance walk test, TWT = timed walk test, PEQ = prosthesis evaluation questionnaire, ABC = activities-specific balance confidence scale, PPA = prosthetic profile of the amputee, TUG = timed up and go, RPE = rating of perceived exertion, FMA = functional mobility of the amputee, OPUS = orthotic and prosthetic users survey

Routine users of outcome measures indicated that they used them for multiple reasons. Outcome measures were most commonly (77%) used to justify selection of prosthetic interventions and they were used least often (50%) to predict patient outcomes (Table 3).

Table 3. Participants' reasons for using outcome measures in clinical practice.

I use standardized outcome measures to…	Routine Users (n = 30)		Non-routine Users (n = 49)
	n	%	n	%
Justify selection of interventions	23	76.7	6	12.2
Document patient status	22	73.0	5	10.2
Inform clinical decisions	19	63.3	6	12.2
Monitor patients over time	18	60.0	5	10.2
Communicate information to others	17	56.7	4	8.2
Predict patient outcomes	15	50.0	4	8.2

Routine users’ pre-training confidence in administration of the TUG and AMP was significantly higher (p = 0.016 and p = 0.0004, respectively) than non-routine users (Table 4). Results from the Wilcoxon signed-rank test indicate that confidence in administration of the TUG and AMP increased significantly (p < 0.0001) in both groups after formal training. No significant differences was found between routine and non-routine users in TUG confidence (median = 5 and 5, range = 4-5 and 3-5, p = 0.79) or in AMP confidence (median = 5 and 5, range 3-5 and 1-5, p = 0.31) after training.

Table 4.

Participants reported confidence (median and range) in the administration of the TUG and AMP.

Measure	Timed Up and Go (TUG)			Amputee Mobility Predictor (AMP)
	Routine Users (n = 30)	Non-routine Users (n = 48)	All Participants (n = 78)	Routine Users (n = 30)	Non-routine Users (n = 48)	All Participants (n = 78)
Pre-training	3 (1-5)	2 (1-5)	2 (1-5)	4 (1-5)	2 (1-5)	3 (1-5)
Post-training	5* (4-5)	5* (3-5)	5* (3-5a)	5* (3-5a)	5* (1-5b)	5* (1-5c)

TUG: Timed Up and Go; AMP: Amputee Mobility Predictor.

Confidence is rated as 1 = “not at all,” 2 = “a little bit,” 3 = “somewhat,” 4 = “quite a bit,” and 5 = “very much.”

^aOne respondent reported 3, all others reported 4–5.

^bOne respondent reported 1, one reported 3, all others reported 4–5.

^cOne respondent reported 1, all others reported 3–5.

^*Significantly different from pre-training confidence (p < 0.0001).

The majority of prosthetist participants (64%) improved two or more confidence categories after receiving TUG training. In the post-training survey, these prosthetists all reported that they were “quite a bit” or “very much” confident in administering the TUG. Another 21% of participants reported a one-category improvement in confidence and 16% reported no change. We observed similar improvements in prosthetists' confidence in administering the AMP, as 52% of prosthetists improved two or more confidence categories. Similar to the TUG, 26% reported a one-category improvement and 20% reported no change in confidence administering the AMP after training. Only 3% reported worse confidence after AMP training.

V. Discussion

The primary purpose of this study was to examine the frequency of use and the type of outcome measures used by prosthetists in clinical practice. Additionally, we aimed to assess the impact of a didactic and interactive training program on prosthetists' confidence in the administration of two performance-based outcome measures. We hypothesized that the majority of prosthetists do not routinely use outcome measures. We found that only a minority of prosthetists (38%) in this study were routine users of outcome measures in their clinical practices. These results are consistent with reports from other allied health fields (e.g., physical therapy, occupational therapy, and psychology) that have identified limited adoption of outcome measures into routine practices. Jette et al., reported that less than half (48%) of 498 practicing physical therapists (PTs) reported using standardized outcome measures in their clinical practice.⁹ Stapleton and McBrearty found that between 15% and 39% of 109 practicing occupational therapists (OTs) frequently or consistently used any of five common outcome measures.²¹ Hatfield and Ogles reported that among a national sample of 874 psychologists, only 37% (n = 324) reported using outcome measures in their clinical practice.⁸

The AMP and PAVET were the measures most frequently used by prosthetists enrolled into this study. A common trait between these measures is that both of them are intended to guide selection of appropriate prosthetic components. The AMP is a valid and reliable measure of current and future functional capabilities of persons with lower limb amputation.¹³ AMP scores have been show to significantly differ among persons classified by different Medicare Functional Classification Levels (MFCLs)¹² and is recommended for use in determining a patient's appropriate MFCL.²² The PAVET is a proprietary, patient- and clinician-report survey developed by Hanger Orthopedic Group (Austin, TX) to assist clinicians in the evaluation of patients with transfemoral limb loss who may be candidates for microprocessor-controlled prosthetic knees. Evidence to support the reliability, validity, or responsiveness of the PAVET as a standardized outcome measure is currently unavailable. The reported frequency of use of the AMP and PAVET is therefore consistent with survey results which indicate that standardized outcome measurement is most often undertaken for justification purposes. As employers and payers increasingly require documentation of patient outcomes, it is expected that use of these (or similar) measures will grow.

A number of barriers to the routine use of outcome measures in clinical practice have been identified in the literature. Limited knowledge about (selecting, using, and interpreting) standardized outcome measures, insufficient time to administer measures in practice, low organizational priority, and lack of perceived value in outcome measurement have all been reported as barriers across numerous allied health disciplines.^{7, 9, 23, 24} These same barriers almost certainly exist in the field of prosthetics.

Recent changes to prosthetic and orthotic educational standards¹⁰, which now mandate outcome measurement education, are intended to promote greater familiarity and regular use of outcomes measures in clinical practice. However, changes to educational standards alone may not be sufficient to change prosthetists' frequency of outcome measure use in routine clinical practice. Therefore, complimentary or alternative methods of providing outcome measure education and experience are likely needed to promote their use. For example, providing practitioners with greater time and resources for administration, scoring, and analysis at the clinic level or fast-tracking reimbursement of prosthetic service claims that include pre- and post-intervention measurements made with approved instruments could facilitate prosthetists' routine use of outcome measures. Similarly, availability of measures that easily integrate data into patients' electronic medical records, such as those that can be administered and scored via a handheld computer, may also increase acceptance and adoption.

A secondary purpose of this study was to develop, implement, and test a focused training program designed to familiarize prosthetists with two performance-based outcome measures (i.e., the TUG¹² and the AMP¹³). The training program developed for this study incorporated mixed didactic and interactive techniques, as recommended by Forsetlund et. al in a recent Cochrane review¹¹. Mixed techniques, often utilized in academic instruction, were employed to provide both empirical knowledge (e.g., information of the measures' psychometric properties) as well as practical experience (e.g., opportunities to practice administration under the guidance of experienced trainers). The mixed-method training employed in this study was found to be an effective strategy for improving prosthetists' confidence in administering the TUG and AMP. In general, non-routine user's confidence improved two full categories after they received the training and routine user's confidence improved one full category. Furthermore, after training, the non-routine user's confidence in administering the TUG and AMP reached the routine user's confidence level. Similar techniques could be used to teach clinical prosthetists to administer, document, and interpret a range of different outcome measures. Such methods may be a means to address barriers to routine measurement in prosthetics and other professions.

Professional organizations, such as the International Society for Prosthetics and Orthotics (ISPO), American Academy of Orthotists and Prosthetists (AAOP), and American Orthotic and Prosthetic Association (AOPA) have advanced outcomes measurement in O&P by hosting interactive sessions at recent O&P conferences.^25-27 The mixed method training provided in this study expanded on the content of these sessions to include periods of focused individual attention by the investigators and provision of various forms of feedback (e.g., verbal, tactile, and positional) that may be atypical in traditional conference sessions or workshops. Although mixed-method training may be impractical in larger settings, such as professional conferences, it may be useful for providing focused outcome measure training to practicing clinicians. Mixed-method outcome measure training workshops may also provide an ideal opportunity for prosthetists to obtain continuing education credits needed to maintain certification and/or licensure while gaining experience and familiarity with outcome measures they can use in their clinical practices.

Study Limitations

Convenience sampling was used to recruit the prosthetists in this study. Therefore, the generalizability of the observed results may be limited. Participants were generally motivated prosthetists who were interested in contributing to research, had experience with prior research projects, and/or wanted additional experience using outcome measures in their clinical practices. As such, we believe that this sample included prosthetists with a range of exposure to and experience with outcome measures and represents a reasonable snapshot of the profession at-large. However, it is possible that our sample may over- or under-represent the outcome measure experience of practicing prosthetists nationwide. Future research may include a larger study to examine if the frequency and use of outcome measures in prosthetic clinical practice reported in this study is representative of routine use across the profession.

The training results described in this study may also be limited to the selected outcome measures. The provided training focused on two performance-based measures, the TUG¹² and the AMP.¹³ These instruments were selected because they are reliable and valid measures of prosthetic mobility and have been recommended for use in clinical care.¹⁵ It is acknowledged that the change in prosthetists' confidence observed after AMP and TUG training may not apply to other measures or other forms of measurement, like self-report. As well, it is unknown if the improvements in confidence observed in this study will lead to greater use of these measures in clinical practice. Seminal literature has demonstrated that improvement in confidence can lead to behavioral changes.²⁸ However, a follow-up study would be needed to confirm whether the training provided promoted increased use of the studied measures. Similarly, prosthetists' confidence may change over time. Additional research is needed to determine if confidence is maintained after training (i.e., at 1 week, 1 month, or 1 year post-training).

Finally, a control group was not utilized to determine the effectiveness of the mixed method training approach used in this study. It is unknown if using just a didactic or interactive approach to training would be more or less effective as the combined approach used in this study. Although evidence strongly suggests that mixed-mode education is superior to single mode (i.e., interactive or didactic) education¹¹, a between-groups study would be needed to confirm if it is similarly effective for training clinicians to administer outcome measures in their daily practices.

VI. Conclusion

Prosthetists participating in this study reported limited use of and comfort with outcome measures in clinical practice prior to training. A didactic and interactive training program that provided study participants with the knowledge and opportunity to practice administering the TUG and AMP while receiving performance feedback was found to significantly increase prosthetists' confidence in administering these measures. Study results suggest that training of prosthetists in the performance and interpretation of outcome measures can lead to greater confidence in administration and may facilitate increased use of outcome measures in routine clinical practice. Professional organizations and educators may consider using the mixed method training techniques utilized in this study to expose prosthetists to standardized outcome measures that may be suited to clinical practice.