The Development and Usability of the AMPREDICT Decision Support Tool: A Mixed Methods Study

Abstract

Objective:

Amputation level decision making in patients with chronic limb threatening ischaemia is challenging. Currently, evidence relies on published average population risks rather than individual patient risks. The result is significant variation in the distribution of amputation levels across health systems, geographical regions, and time. Clinical decision support has been shown to enhance decision making, especially complex decision making. The goal of this study was to translate the previously validated AMPREDICT prediction models by developing and testing the usability of the AMPREDICT Decision Support Tool (DST), a novel, web based, clinical DST that calculates individual one year post-operative risk of death, re-amputation, and probability of achieving independent mobility by amputation level.

Methods:

A mixed methods approach was used. Previously validated prediction models were translated into a web based DST with additional content and format developed by an expert panel. Tool usability was assessed using the Post-Study System Usability Questionnaire (PSSUQ; a 16 item scale with scores ranging from 1 to 7, where lower scores indicate greater usability) by 10 clinician end users from diverse specialties, sex, geography, and clinical experience. Think aloud, semi-structured, qualitative interviews evaluated the AMPREDICT DST’s look and feel, user friendliness, readability, functionality, and potential implementation challenges.

Results:

The PSSUQ overall and subscale scores were favourable, with a mean overall total score of 1.57 (standard deviation [SD] 0.69) and a range from 1.00 to 3.21. The potential clinical utility of the DST included (1) assistance in counselling patients on amputation level decisions, (2) setting outcome expectations, and (3) use as a tool in the academic environment to facilitate understanding of factors that contribute to various outcome risks.

Conclusion:

After extensive iterative development and testing, the AMPREDICT DST was found to demonstrate strong usability characteristics and clinical relevance. Further evaluation will benefit from integration into an electronic health record with assessment of its impact on physician and patient shared amputation level decision making.

INTRODUCTION

Investigations from Veteran Health Administration (VHA) data sources suggest that between 2005 and 2014, the proportion of transmetatarsal (TM) amputations of all chronic limb threatening ischaemia (CLTI) amputations tripled from 10% to 30%, with a corresponding decrease in the proportion of transtibial (TT) and transfemoral (TF) amputations.^1,2 The increase in TM amputations may in part be driven by the goal of preserving mobility compared with TT or TF amputations.³ Mobility after lower extremity amputation has been shown to positively impact patient quality of life,⁴ which is a key consideration in patient centred decision making.^5,6 Conversely, TM amputations are associated with compromised healing, need for ongoing wound care, and a risk of re-amputation to a higher level.^1,5,7 Among failed TM amputations, the majority (72%) will result in a higher level amputation, with 55% and 17% being converted to TT and TF amputation, respectively.² Of those who require a re-amputation, 25% will undergo more than one additional procedure. The consequences of failed primary healing and need for additional re-amputation surgery can lead to restricted ambulation.^8–11 Balancing the risks of these outcomes in the CLTI population is at the core of a complex, shared decision making process between physicians and patients as they determine the “best” level of amputation for specific patients.²

Shared decision making (SDM) occurs when an informed clinician involves an informed patient as a partner in making choices about the patient’s care, and balances the risks and benefits of different care options with the patient’s values and preferences, based on available evidence.¹² Clinical decision support tools (DSTs) efficiently integrate published literature and guidelines into this process.^13,14 DSTs provide clinicians with up to date clinical knowledge and patient specific risks for key outcomes in real time. Therefore, they inform decision making discussions with intelligently filtered and personalised data.¹⁵

Currently, amputation level decisions are made on the basis of surgeon clinical experience underpinned by immediate knowledge, but without objective information about patient specific long term outcomes. No DST exists to assist physicians with amputation level determination in CLTI patients. The AMPREDICT prediction models for mobility, mortality, and re-amputation have been published;^1,16,17 however, anecdotally, clinicians find applying the published formulas and coefficients to aid decision making at the point of care to be challenging and even prohibitive. This research aimed to overcome this, based on a three phase continuum of work over the past decade. Phase 1 encompassed the development and validation of the three AMPREDICT prediction models, which were published from prior work. Phase 2 is represented by the work presented in this manuscript, which incorporates the AMPREDICT models into a DST with rigorous development and usability testing to arrive at the final AMPREDICT DST. Phase 3 will include integration of the tool into an electronic health record (EHR) and the assessment of its usability as a point of care tool on physician and patient shared decision making outcomes. This phase will also include external validation in other populations. Therefore, the goal of this study (phase 2) was to take the previously validated AMPREDICT prediction models (phase 1) and develop an online risk calculator using the coefficients from the models, then having physicians test its usability through hypothetical and real world case scenarios, to create the final AMPREDICT DST.

MATERIALS AND METHODS

Study design

This mixed methods (quantitative and qualitative) study included data from developers of the AMPREDICT prediction models,^1,16,17 an expert panel of physicians, and U.S. Veteran’s Health Administration (VHA) physician end users.

Validated AMPREDICT models informing the decision support tool

The AMPREDICT prediction models were developed and validated on VHA patients, using national data from 2005 to 2014.^1,16,17 The models were validated on patients 40 years of age or older, undergoing their first major lower extremity amputation (defined as TM, TT, or TF), as a result of diabetes and/or peripheral artery disease (PAD). The three models predict the one year mortality risk, one year reamputation risk (at the same or higher amputation level), and one year probability of achieving independent ambulatory mobility. The mortality model was externally validated in different VHA regions and the re-amputation models were internally validated in the same VHA population. There are 21 predictors across the three models (Table 1).

Table 1.

The 21 predictors that are entered into the AMPREDICT decision support tool to obtain one year risks of death, reamputation, and achieving an independent level of basic ambulation

Demographics:	Age	Sex	BMI (height and weight)	Race	Marital status	Highest education level	Functional status
Comorbidities:	CHF	Diabetes	COPD	Anxiety or depression	Prior revascularisation	Currently on dialysis
Labs and meds:	BUN level	Platelet count	WBC count	eGFR value	Outpatient anticoagulants
Health behaviours:	Current smoker	Current alcohol misuse	Self rated health

BMI = body mass index (requires the entry of two variables: height and weight); CHF = congestive heart failure; COPD = chronic obstructive pulmonary disease; BUN = blood urea nitrogen; WBC = white blood cell; eGFR = estimated glomerular filtration rate.

AMPREDICT decision support tool content development (qualitative)

The initial AMPREDICT DST underwent comprehensive iterative development with refinement at multiple stages based on input from study investigators, an expert panel, and physician end users (Fig. 1).

Figure 1.

The comprehensive iterative development of the AMPREDICT decision support tool (DST) involving input from study investigators, an expert panel, and physician end users translating to the final clinical application. PSSUQ = Post-Study System Usability Questionnaire.

The first step in content development involved extracting the coefficients from the validated mortality, re-amputation, and mobility prediction models^1,16,17 and programming them into an online risk calculator. Next, an expert panel of four vascular surgeons, a podiatric surgeon, and a physical medicine and rehabilitation physician interacted with the DST and provided feedback with respect to its visual components (i.e., look and feel), user friendliness, readability, functionality, and potential implementation challenges. The expert panel represented different geographic regions of the United States and the United Kingdom. This content development included three expert panel iterations resulting in a beta version of the AMPREDICT DST, which underwent usability testing (Fig. 1).

AMPREDICT decision support tool usability testing (quantitative)

After the development of a beta version through iterative expert panel feedback, approval was obtained from the Institutional Review Board (IRB) to have an online version of the AMPREDICT DST evaluated by 10 potential VHA end users recruited through Regional Amputation Centre directors. The goal was to identify end users with diversity in practice (vascular and podiatric surgeons, as well as physical medicine and rehabilitation physicians), gender, and geography who met a minimum requirement of ongoing care of amputees resulting from CLTI.

End users gave their informed consent to undergo an individual, virtual, structured interview with a research coordinator with expertise in qualitative methods (Fig. 1). The end users were asked to navigate the web portal of the beta version of the DST for two clinical scenarios: (1) one hypothetical provided by a research coordinator, and (2) one real world scenario selected by the provider from their own experience. Participants were encouraged to “think aloud” as they navigated through the tool in a virtual structured interview, with responses recorded and analysed. After completing the DST, participants were asked to complete the Post-Study System Usability Questionnaire (PSSUQ), third version.¹⁸ The PSSUQ-3 is a 16 item measure, with all items on a 7 point Likert scale (1 strongly agree to 7 strongly disagree). Lower scores on the PSSUQ indicate greater satisfaction with the tool. The score can be broken down into an overall score (the mean of all 16 items): a system usefulness subscale, how easy the system is to use and learn (mean of items 1 – 6); an information quality subscale, the feedback the system provides to the user (mean of items 7 – 12); and an interface quality subscale, how much the user liked the system and if it has expected functionality (mean of items 13 – 16). In instances where an individual indicated “not applicable”, mean scores were calculated using a smaller denominator. PSSUQ data were analysed via SPSS 19.0 (IBM SPSS Statistics for Windows, Version 19.0. Armonk, NY: IBM Corp).

AMPREDICT decision support tool usability testing (qualitative)

The investigators listened to recordings of all “think aloud” interviews and notes were taken. These notes were presented to the expert panel for review when making final modifications to the tool. After interacting with the AMPREDICT DST using clinical scenarios, researchers conducted qualitative interviews with the end users with respect to the look and feel of the home, input, and output pages, and to potential application and implementation challenges of the DST. The number of end users that participated in this stage of the development and testing was determined by consistency of PSSUQ scores and thematic saturation of qualitative input. Once themes appeared to achieve saturation, recruitment was discontinued.

RESULTS

AMPREDICT decision support tool usability testing (quantitative)

Ten VHA end users were successfully contacted and consented to participate. This included five vascular surgeons, three physical medicine and rehabilitation physicians, and two podiatric surgeons (Table 2). There was excellent diversity in gender (50% male and female) and geography, with physicians from the west, mountain regions, southwest, northeast, east, and southeast regions. The average experience with amputation surgery or consultation was nearly 14 years (Table 2) and the average number of transmetatarsal and transitibial amputation procedures combined per year over 10 with an overage of over 50 consultations (Table 3). The PSSUQ overall and subscale scores were very favourable, with a mean overall total score of 1.57 (standard deviation [SD] 0.69) and a range from 1.00 to 3.21. Mean system usefulness, information quality, and interface quality subscale scores were 1.6, 1.27, and 1.85 respectively (Table 4).

Table 2.

Usability testing physician end user demographic information

Characteristic	End user (n = 10)
Provider type
Vascular surgeon	5 (50)
Podiatric surgeon	2 (20)
Physical medicine and rehabilitation physician	3 (30)
Male	5 (50)
Number of years in practice	13.70 ± 10.47 (6–37)
Home facility is designated as operatively complex*	9 (90)
Location of practice
Colorado	1 (10)
Georgia	1 (10)
Massachusetts	1 (10)
Michigan	1 (10)
New Mexico	3 (30)
North Carolina	1 (10)
Washington	1 (10)
West Virginia	1 (10)

Data are presented as n (%) or mean ± standard deviation (range).

^*In Veterans Health Administration facilities, “operative complexity” references the infrastructure that is required at a facility in relationship with the complexity of surgical procedures performed at that site. Operatively complex facilities have infrastructure to support complex operations (e.g., coronary artery bypass graft).

Table 3.

Amputations and consultations performed by the 10 usability testing end users in past year

Amputations/consultations	Performance in past year
Transmetatarsal amputations*	2.86 ± 4.14 (0–11)
Transtibial amputations†	9.57 ± 11.09 (0–25)
Transtibial/transmetatarsal consults‡	51.3 ± 48.01 (4–100)

Data are presented as n ± standard deviation (range).

^*Among vascular and podiatric surgeons, the number of transmetatarsal amputations performed in the past year.

^†Among vascular and podiatric surgeons, the number of transtibial amputations performed in the past year.

^‡Among physical medicine and rehabilitation physicians, the number of transtibial or transmetatarsal amputations on which the provider consulted in the past year.

Table 4.

Post-Study System Usability Questionnaire (PSSUQ) total and subscale scores of AMPREDICT Decision Support Tool from 10 end user interviews

PSSUQ	Score
Total score	1.57 ± 0.69 (1.00–3.21)
System usefulness subscale	1.60 ± 0.95 (1.00–4.00)
Information quality subscale	1.27 ± 0.43 (1.00–2.25)
Interface quality subscale	1.85 ± 1.01 (1.00–4.25)

Data are presented as score ± standard deviation (range).

AMPREDICT decision support tool usability testing (qualitative)

With respect to look and feel, and intuitiveness, end users provided very detailed suggestions and recommendations for the home, input, and output pages (Table 5). Pertinent key quotes on the potential application and implementation recommendations are included in Table 6. In summary, most providers said they were eager to use the AMPREDICT DST and some requested to use it as soon as possible. In addition to assisting with amputation level selection, provider input also suggested the AMPREDICT DST had the potential for the following broader utility: (1) assisting in counselling patients on the amputation level decision, (2) setting expectations for future outcomes, and (3) using as a teaching tool in the academic environment to better understand the factors that influences risks.

Table 5.

End user input on the AMPREDICT Decision Support Tool look and feel and intuitiveness with respect to the home, input, and output pages

Home page	Input pages	Output pages
Improved presentation with larger fonts and less text	Add pagination by domain so that predictors fit on one page and to improve flow of navigation	Improved presentation with larger fonts and less text
Emphasis on the benefits of the tool	Radio buttons for selecting predictors	Improved risk output visuals
Links to the prediction model publications	Hover features to provide clarity on predictor definitions	Changes in ordering of outcome risk presentation.
Hover features for definitions	Ensuring predictors cannot be left blank or values outside of acceptable ranges	Hover features to provide clarity on outcome definitions
Extra details moved to an “About this Tool” page	Adding height and weight separately for those who do not have body mass index values
Disclaimer that the tool is intended for health professionals only who will share the individual risks with their patients

Table 6.

Summary of quotes evolving from provider interviews after using the AMPREDICT Decision Support Tool

Potential application	Implementation recommendations
“As soon as this is available, I will use it.”	“App would make it easy, but if it takes long time to access the tool, it would make it harder to use regularly.”
“I don’t see why this couldn’t be incorporated into our daily consult service.”	“A quick in service (could be webinar) of background for tool, validation and demo for implementation.”
“I think it would be helpful, anything that we can provide that is individualised to the patient. They say, ‘that’s them, not me.’ Anyway, we can empower patients.”	“App or add to CPRS/Cerner toolkit.”
“The more patients know, and the more transparency help engender trust in the provider and empowering them [patients].”	“Engagement with professional societies and clinicians at site with high number of amps. Targeted training, not a fan of online training tools. Getting in at the provider level of investment.”
“I would potentially use this weeks prior to a surgery in the outpatient setting.”	“Identifying advocates in each institution to act as points of contact to spread the word—perhaps those who have participated in the study.”
“Use with patients deciding between levels [of amputation], in- or outpatient, probably outpatient.”	“In clinic go through the whole tool in the presence of the patient; more challenging to do in wards…helpful for patients with distrust to show them numbers, that they are not made up.”
“Other providers could administer the tool, as long as they could quantify the functional status of the patient. A tech or nurse could use the tool but would not necessarily be able to implement results into a medical treatment plan, I would be okay with someone else inputting and then I review result with the patient.”	“Do along with the patient there so they are actively participating in the answering of questions; prior to surgery, either same time as consent form or prior visit.”
	“Done with patient at bedside. Has to be a discussion, not fair to be ‘I know better,’ get them into the mindset of being engaged in their recovery and PT. “

CPRS = computerised patient record system; PT = physical therapy.

In addition, the following constructive feedback was provided: clinicians preferred to see the DST integrated into the EHR to allow for auto-population of the predictor variables. However, there was concern that some predictive variables would need to be gathered through patient interview, as they are not usually recorded as objective variables in the patient chart. Several end users felt the tool could be used along a broad clinical continuum, from those who were being cared for in wound clinics and were at high risk of amputation, to inpatient settings where the amputation was imminent. Based on feedback from end users, a final iteration that included only minor changes was approved by the expert panel (Fig. 1).

Final AMPREDICT decision support tool

A web based user friendly portal was designed for easy global access and use (Fig. 2). Physicians can access the portal and enter content for the necessary predictive variables (Fig. 3A–D). Immediate calculations then produce visual and objective results, which are the predicted risks of the outcomes of interest: re-amputation, mobility, and mortality (Fig. 3E,F). The tool is currently available at https://www.ampredict.org.

Figure 2.

AMPREDICT decision support tool home page.

Figure 3.

AMPREDICT decision support tool predictor inputs for (A) patient demographic information, (B) comorbid conditions, (C) medications and current laboratory values, and (D) health factors. (E) Risk of mortality within the first year after lower extremity amputation and (F) risks of re-amputation and ability to achieve at least basic ambulation within first year after lower extremity amputation.

DISCUSSION

The AMPREDICT DST was successfully developed using previously validated and published prediction models,^1,16,17 expert panel iterative feedback and clinician end user evaluation. This represents the second phase of this body of work, which has resulted in a risk calculator available for widespread use. It is ready to move to the third phase of evaluation in an implementation environment where it is integrated into the EHR and evaluated in a shared decision making context at the point of clinical care.

Clinical DSTs play an important role in complex clinical decision making by presenting personalised outcomes at the point of care during physician/patient shared decision making encounters. In a systematic review, features of clinical decision support systems that succeeded in improving clinical practice were available to clinicians automatically at the point of care, while those that did not improve patient care required physicians to seek information outside of the typical workflow environment.¹⁹ Similarly, tools that were integrated into medical record systems were significantly more likely to be used to improve clinical care than standalone tools.

The output of the AMPREDICT DST was structured so that mortality risk is presented first. This is crucial, because conventional wisdom is that the amputation level has a significant impact on future mortality. Mortality risk can vary significantly by individual depending on their composite of baseline risk factors. In fact, it was found that the difference in one year mortality by amputation level was relatively small compared with other studies, suggesting that mortality risk is explained by other risks that were controlled for in the AMPREDICT models. By presenting the mortality risk first, the physician and patient may first consider a patient’s probability of surviving the year after the amputation before considering the risks they may be willing to take with respect to the re-amputation/mobility tradeoffs. For example, a patient with a 70% risk of death in the year following amputation may not wish to endure the possible risk of requiring long term wound care or additional surgery associated with a distal amputation, with the hope of achieving ambulatory mobility. The choice of a more proximal amputation may reduce the risk of wound healing complications but at the expense of mobility outcome. In contrast, a patient with a 10% risk of death may be more willing to incur the risks of failure of primary wound healing because of its potential mobility benefits.

The tool performed well on formal usability testing by a national cohort of VHA physicians that interacted with it using hypothetical and real patient examples. The mixed methods approach allowed for both quantitative and qualitative feedback to improve the overall potential clinical impact of the tool. The wide range of physician diversity in background, experience and opinions, ensured the overall iterative development of the tool was both comprehensive and tailored to the broad range of needs and challenges these physicians face at the point of care. The usability scores on the PSSUQ by the clinician end users were excellent. They also provided valuable feedback and almost unanimously reported they would use the tool in clinical practice if it were available today. Recommendations for improving the future use of the tool centred on making it easily accessible in the EHR and to possess the capacity to auto populate the predictors, thereby minimising key obstacles to clinical use.

The usability of the DST compared favourably with prior studies evaluating decision support tools. Sauer and Lewis²⁰ compiled the PSSUQ scores from 21 published usability studies to serve as benchmarks for other studies using the PSSUQ. From these published studies, the mean overall PSSUQ score was 2.82 (range 2.61 – 3.02). The mean system usefulness subscale score was 2.8 (range 2.57 – 3.02), the mean information quality subscale score was 3.02 (range 2.79 – 3.24) and the mean information quality subscale score was 2.49 (range 2.28 – 2.71). Mean scores for the AMPREDICT DST were 1.57, 1.60, 1.27, and 1.85, respectively, which were considerably better than the average reported in prior studies evaluating DSTs.

This patient population is very heterogenous with respect to their pre-amputation risk factors, which may profoundly influence their individual mortality, re-amputation, and mobility outcomes. Clinical practice guidelines for CLTI are broad in regard to amputation level decision making because risk factors for individual patients are so varied.²¹ The AMPREDICT DST overcomes these obstacles by determining individual patient and amputation level outcomes, assisting physicians and their patients in making informed decisions.

Like other clinical care innovations, DSTs need to overcome significant challenges to be adopted in routine clinical use. A systematic review by Kawamoto et al. identified 15 DSTs that were deemed to have improved clinical practice in specific trials.¹⁹ This review concluded that the following features associated with DSTs improved patient care significantly: integrating decision support in the workflow, providing both assessment and recommendations, providing support at the time the decision is made, and systems that are electronic and computer based. Other successful features of DSTs include (1) being derived from high quality evidence,²² (2) promoting disease management (and patient expectations) by generating patient specific risks for clinicians,¹⁵ (3) overcoming the problem of translating research findings to practice,²³ (4) adaptability to new evidence,²⁴ and (5) availability as part of the EHR for point of care access.²³

The AMPREDICT DST has limitations. While the tool demonstrated strong usability and utility, the risks produced by the tool are generated from models developed in patients undergoing their first major amputation (defined as a TM, TT, or TF) who did not have severe comorbidities (e.g., coma, paraplegia, quadriplegia, disseminated cancer, tumour of the central nervous system, ventilator dependent) that would put them at high risk of death and more likely a candidate for TF amputation only. Further, the models were developed and validated in a U.S. Veteran population with CLTI (comprising primarily males); therefore, its use in other populations should be considered with caution, since the management and risks of patients in VHA hospitals may differ from other populations. While the prediction models that underpin the DST were developed with access to thousands of patients and a large selection of candidate predictors, like other large database studies, they are limited by the predictors available. For example, prior revascularisation was an important predictor in the reamputation model; however, the granularity of successful or failed revascularisation was not available. The tool is currently undergoing a process of integration in the VHA electronic health record to be used at the point of care. Physicians outside the VHA have chosen to use the tool with caution and processes are being developed to validate the models in non-VHA populations both within and outside the United States. Future development and research efforts will include models with predictors that can be auto populated within the EHR and externally validated within and outside the U.S. The ideal DST would use a systems dynamic modelling approach where the model is improved over time as new data become available.²⁵

Conclusions

The AMPREDICT DST was successfully developed, refined and tested for usability and clinical relevance. Physicians found it to be helpful, meaningful, and are asking for it to be applied in clinical practice. The tool is ready to move to the third phase of development and evaluation with integration into the VHA EHR and testing to determine its effectiveness in the informed shared decision making process.

WHAT THIS PAPER ADDS.

Currently, available evidence on amputation level decision making in patients with chronic limb threatening ischaemia relies on published average population risks rather than individual patient risks. The result is significant variation in the distribution of amputation levels across health systems, geographical regions, and time. Prediction models have been published to address this; however, translation to clinical application has not been undertaken. After extensive iterative development and testing, the AMPREDICT DST was found to demonstrate strong usability characteristics and clinical relevance. Further evaluation will benefit from integration into an electronic health record with assessment of its impact on physician and patient shared amputation level decision making.