Using Implementation Science to Evaluate the Implementation of Patient-Reported Outcome Measures (PROMs) in a Clinical Heart Failure Care Setting

Abstract

Background

Patients with heart failure (HF) can experience poor quality of life, recurring hospitalizations, and progressive disease symptoms. Patient-reported outcome measures (PROMs) include patients’ voices in clinical care by assessing patient symptoms, function, and quality of life. In 2022, PROMs were implemented into the electronic health record system (Epic) at a large academic hospital in Toronto, Canada. The purpose of this study was to use implementation science frameworks to systematically evaluate the uptake and integration of PROMs into clinical HF care.

Methods

The Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework guided this mixed-methods, 1-year quality-improvement project. Data sources included clinician use of PROMs, patient-level data on completed PROMs, and semistructured interviews with clinicians. The PROM was the Kansas City Cardiomyopathy Questionnaire-12, which captures 4 domains related to HF: symptom frequency, physical limitations, social limitations, and quality of life. Quantitative data were analyzed using descriptive statistics, and qualitative data were analyzed using behaviour-change frameworks and latent content analysis.

Results

Over the course of 1 year, more patients were assigned to PROMs, a higher proportion of patients completed PROMs, and approximately 60% of patients had high questionnaire scores. Clinicians experience barriers related to attention and decision processes, environmental context, and their professional role, in integrating PROMs into practice. Suggested resources include adding language licenses for PROM translations, reducing cognitive load for clinicians assigning and interpreting PROMs in Epic, and champions modelling PROMs in practice.

Conclusions

This study demonstrates the benefit of using implementation science frameworks to evaluate the implementation of PROMs in practice and provide actionable recommendations to health systems.

Nearly 780,000 people across Canada had a diagnosis of heart failure (HF) in 2021-2022, accounting for 3.2% of the population.¹ HF is a complex chronic condition that occurs when the heart muscle does not function properly and fluid builds up in the lungs, causing shortness of breath.² Individuals with HF typically experience progressive disease, poor quality of life, frequent hospitalizations, and a high mortality incidence.^3,4 HF also is associated with a higher prevalence of depressive symptoms and anxiety, in addition to coexisting cognitive issues.⁵ Whereas death and hospitalizations are easy to measure, health status (including quality of life) is more difficult to capture in a valid and reliable manner.⁶ Patient-reported outcome measures (PROMs) enable clinicians to incorporate the patient’s voice in assessing the impact of HF on function, symptoms, and quality of life, to optimize treatment and planning.⁷ PROMs are necessary to capture outcomes that are important to patients with HF and to support a patient-centred care model.^8,9

Using PROMs in clinical practice can improve patient health outcomes, provide indicators of quality care, and advance research in clinical care.¹⁰ Although they are used regularly for research purposes, historically, PROMs have not been incorporated routinely into clinical practice. Understanding strategies to implement use of PROMs successfully therefore are required to improve the patient care experience. Indeed, which implementation strategies for PROMs adoption and scaling are the most effective is poorly understood.¹¹ Using implementation science frameworks to assess the implementation of PROMs can help identify factors influencing the integration of PROMs into care settings.¹² Frameworks can vary in typology depending on the aim: describing the process, understanding influences on implementation, and/or evaluation implementation.¹³ However, implementation science frameworks are rarely used and applied in research on PROMs and HF.¹¹ In spring of 2022, PROMs were deemed a priority within HF care pathways as part of provincially funded demonstration projects. At the same time, University Health Network (UHN) in Toronto, Ontario, a large and multisite academic hospital, launched Epic as its new electronic health record system. Epic was part of the clinical and digital transformation made to allow patient data to flow across providers and increase patient access to medical information. Given the alignment in timing and funder expectations, the decision was made to integrate PROMs into the new Epic system in fall 2022, as part of standard of care in the heart function programs at 2 UHN sites. The availability of PROMs in Epic was communicated to the heart function program via e-mail, by the program lead, in fall 2022, and was supported by the following: cardiology rounds with Dr. John Spertus (author of the Kansas City Cardiomyopathy Questionnaire-12 [KCCQ-12], one of the PROMs implemented); “lunch and learn” sessions; a Best-Practice Advisory within Epic prompting clinicians to use PROMs; and a PROMs User Guide in Epic.

This study used implementation science frameworks to evaluate implementation of PROMs at UHN Heart Function Clinics (Reach, Effectiveness, Adoption, Implementation, Maintenance [RE-AIM] framework)¹⁴; and understand influences on implementation outcomes (Theoretical Domains Framework [TDF]).^13,15 The overall evaluation was guided by the RE-AIM framework, which was initially designed to evaluate interventions and public health programs,¹⁴ but it has since evolved to report results in diverse healthcare areas.¹⁶ The RE-AIM framework provides a comprehensive, logical, and systematic conceptualization of assessment of internal and external validity.¹⁴ As part of the RE-AIM evaluation, the TDF¹⁵ was used to understand influences on implementation outcomes, and it was selected to systematically identify factors related to healthcare provider behaviour in integrating PROMs into patient care. These factors can be linked to behaviour change interventions to support implementation in the future.¹³ As a quality-improvement project to enhance uptake and integration into clinical practice, implementation science frameworks (the RE-AIM framework and TDF) allowed for systematic and comprehensive evaluation of PROMs implementation. We evaluated the implementation of electronically delivered PROMs (specifically the KCCQ-12) in the 2 HF Clinics at UHN, using implementation science frameworks.

Methods

This mixed-methods project was conducted as part of a quality-improvement initiative (QIRC 23-0624). The overall framework guiding the year-long evaluation of PROMs was the RE-AIM framework (see Table 1 for an overview). Data sources included data from Epic on clinician use of PROMs, patient-level data on completed PROMs, and semistructured interviews with clinicians. Epic data were collected on a quarterly basis between September 1, 2022 and September 30, 2023. Interviews were conducted in fall 2023.

Table 1.

Overall Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) evaluation framework and data analysis plan

Domain	Indicators	Data sources	Timepoint
Reach: number, proportion, and representativeness of people receiving PROMs	The characteristics of patients (according to site) who are assigned to complete PROMs	Epic	Quarterly
	The rate of success of patients who are assigned compared to patients who qualify to complete PROMs	Epic	Quarterly
Effectiveness: patient-level health status according to PROMs and perceived utility in heart failure care	The number of patients who have completed PROMs	Epic	Quarterly
	The characteristics of patients who have completed PROMs by site and gender	Epic	Quarterly
	The overall patient symptom frequency, overall patient physical limitations, overall patient social limitations, and overall summary score from the KCCQ-12	Epic	Quarterly
Adoption: number, proportion, and representativeness of possible settings and staff participating in PROMs	The characteristics of healthcare practitioners who are assigning PROMs for patients (gender, provider type)	Epic	Quarterly
Implementation: extent to which PROMs are implemented as intended	The success rate of being assigned PROMs to completing PROMs	Epic	Quarterly
	The barriers and facilitators to practitioners’ use and integration of PROMs into clinical practice according to the Theoretical Domains Framework17	Interviews	Once
Maintenance: assessment beyond 6 mo at the patient and setting level	The resources needed to sustain PROMs implementation in clinical practice long-term	Interviews	Once

Epic, University Health Network (Toronto, Ontario, Canada) electronic health record system; KCCQ-12, Kansas City Cardiomyopathy Questionnaire, 12-item; PROM, patient-reported outcome measure.

Participants

Patients with HF who completed PROMs were seen in the outpatient clinics at 2 UHN hospitals—Toronto General Hospital (TGH) and Toronto Western Hospital (TWH). Several methods, both clinician-directed and automated, were used to assign PROMs to patients,. New patients with HF were assigned a PROM automatically in advance of their first appointment. For existing patients with HF, a system prompt (a Best-Practice Advisory) appeared in the clinician-view of the patient electronic chart for all qualifying patients (ie, patients with an appointment in outpatient heart function and cardiac clinics with an established diagnosis of HF or cardiomyopathy). Clinicians then decided, at their discretion, whether to assign the PROM. Once PROMs were assigned, patients completed PROMs through their “MyChart” (the secure app/website for patients to use to manage appointments, access results, update information, and complete questionnaires). They could be assigned and could complete more than one PROM in the period of study, to coincide with appointments, but never more frequently than every 2 weeks (the timeframe used in the KCCQ-12).

Clinicians included UHN Peter Munk Cardiac Centre cardiologists, internists, registered nurses, and nurse practitioners who saw patients with HF in the outpatient clinic. Clinicians were invited to be interviewed if they had high patient volumes (> 10 patients per month) and were part of the HF clinic. The goal was to interview an equal number of clinicians specifically in the HF clinic in each of the following categories: never assign (provider never assigned PROMs between October 2022 and September 2023); sometimes assign (provider assigned PROMS to ≥ 1 patient in ≥ 1 month between October 2022 and September 2023); and usually assign (between October 2022 and September 2023, on average, provider assigned PROMs to > 1 patient per month).

Measures

The Epic data

Data from Epic were extracted by the Decision Support Team at UHN. Data included the number of patients who were assigned (and not assigned) PROMs, the number of patients who completed (and did not complete) PROMs, and which clinicians assigned (and did not assign) PROMs. Data also included information about the site (TGH vs TWH) and patient gender.

KCCQ-12

The KCCQ-12 is the short form of the KCCQ and is meant for use in clinical care.¹⁸ The HF questionnaire assesses 4 domains over the preceding 2 weeks: symptom frequency; physical limitations; social limitations; and quality of life. Participants’ responses are converted into domain scores, ranging from 0 (indicating more severe symptoms or limitations and very poor quality of life) to 100 (representing no symptoms, no limitations, and excellent quality of life). The Overall Summary Score is calculated as the average of these domain scores. Additionally, the Clinical Summary Score is derived from the average of the symptom frequency, the physical limitations domain score, and the social limitations domain score. The average of the symptom frequency, the physical limitations domain score, and the social limitations domain score is the Clinical Summary Score. The KCCQ-12 is a mandated PROM for use with patients with HF in Ontario, as determined by the provincial funder, Ontario Health.

Semistructured interviews

Interviews were conducted with clinicians to explore their experiences with PROMs; barriers and facilitators to assigning, interpreting, and integrating PROMs into practice; perceived benefits and consequences of PROMs; and resources required for sustainable PROMs implementation. Interview guides (see Supplemental Appendix S1) were developed to understand clinician thought processes and workflows related to PROMs and were informed by the TDF.¹⁵ For example, factors related to the environmental context and resources, knowledge, and skills for PROM implementation were explored. Interviews were conducted using Microsoft Teams, and they were audio-recorded and transcribed verbatim.

Analysis

The KCCQ-12 results and Epic data were analyzed using descriptive statistics. The mean, standard deviation, median, and interquartile range are presented for each domain and the Overall Summary Score for every quarter.

For the semistructured interviews, a pragmatic approach was undertaken to prioritize the research use, emphasizing the translation, co-production of knowledge, and applicability of findings to the clinical setting.¹⁹ Using Microsoft Word (Microsoft, Redmond, WA) to manage the data, the transcripts were analyzed using latent content analysis.²⁰ After transcripts were reviewed and initial codes were created, codes were applied to texts—both deductively, based on the TDF,¹⁵ and inductively, to capture additional categories. After coding, groups of codes were organized into different categories to create meaningful clusters of information.²⁰

Results

Reach

The number of individual patients assigned PROMs by a healthcare provider, according to site and the rate of patients who are assigned PROMs, compared to those who were not assigned PROMs is found in Table 2. Overall, among eligible patients, a higher percentage were assigned PROMs at TGH, with increasing numbers of patients assigned PROMs over the course of the year.

Table 2.

The rate of assignment for patients who are assigned vs patients who are not assigned patient-reported outcome measures (PROMs), according to site

Quarter	Patients assigned PROMs			Patients not assigned PROMs	Rate of assignment, %
	TGH	TWH	Total	Total
October–December 2022	121	4	125	2442	5.1
January–March 2023	184	23	207	2014	10.3
April–June 2023	206	17	223	2009	11.1
July–September 2023	167	9	176	1879	9.4

TGH, Toronto General Hospital; TWH, Toronto Western Hospital.

Effectiveness

Table 3 shows the number of unique patients and their characteristics (according to site and gender) who completed PROMs over time, and the total number of PROMs completed. Similarly, the total number of PROMs completed over time increased substantially from launch, with most being completed at TGH and by men. The mean and standard deviation for the KCCQ-12 domain and summary scores for each quarter are presented in Table 4. The percentage of quality-of-life and summary scores according to cut points are shown in Tables 5 and 6, respectively. Overall, most patients (consistently around 70%) have fair-to-good quality-of-life and summary scores. Between April and September 2023, more patients had lower quality-of-life and summary scores (between 25 and 50).

Table 3.

The characteristics (according to site and gender) and total number of unique patients who completed patient-reported outcome measures (PROMs) and total number of PROMs completed

Category	October–December 2022	January–March 2023	April–June 2023	July–September 2023
Site
TGH	41	113	142	170
TWH	6	15	19	13
Gender
Male	24	89	94	113
Female	23	39	67	70
Total unique patients	47	128	161	183
Total PROMs	48	131	167	187
Mean PROMs per unique patient	1.02	1.02	1.04	1.02

TGH, Toronto General Hospital; TWH, Toronto Western Hospital.

Table 4.

Kansas City Cardiomyopathy Questionnaire, 12-item, domain and summary scores per quarter

Quarter	Symptom frequency score	Physical limitations score	Social limitations score	Quality of life score	Summary score
October–December 2022	80.2 (23.3)	76.9 (24.8)	69.7 (33.9)	64.6 (32.7)	73.0 (25.7)
January–March 2023	82.8 (21.6)	78.5 (26.2)	75.6 (31.0)	70.6 (31.6)	77.0 (24.6)
April–June 2023	80.6 (22.6)	76.1 (27.1)	73.1 (30.2)	67.8 (30.5)	74.2 (24.8)
July–September 2023	81.5 (21.9)	76.6 (28.1)	76.6 (29.4)	68.4 (29.7)	75.5 (24.2)

Values are mean (standard deviation).

Table 5.

Percentage of KCCQ-12 quality-of-life scores according to cut-points

Quarter	< 25	25–49	50–74	75–100
October–December 2022 (N = 48)	18.8	4.2	16.7	60.4
January–March 2023 (N = 131)	9.2	14.6	10.8	65.4
April–June 2023 (N = 167)	7.8	17.5	19.3	55.4
July–September 2023 (N = 187)	5.9	17.6	21.4	55.1

Cut-points are as follows: very poor to poor (< 25); poor to fair (25–49); fair to good (50–74); and good to excellent (75–100).

Table 6.

Percentage of KCCQ-12 summary scores according to cut-points

Quarter	< 25	25–49	50–74	75–100
October–December 2022 (N = 48)	8.3	12.5	18.9	60.4
January–March 2023 (N = 131)	4.6	11.5	20.6	63.4
April–June 2023 (N = 167)	1.8	19.3	22.3	56.6
July–September 2023 (N = 187)	4.3	15.5	19.8	60.4

Cut-points are as follows: very poor to poor (< 25); poor to fair (25–49); fair to good (50–74); and good to excellent (75–100).

Adoption

Not all clinicians who see patients with HF at UHN were included in this analysis. A total of 62 clinicians (including HF fellows) who regularly see patients with HF, primarily in the Heart Function Clinics at UHN were prioritized as most relevant to PROMs implementation. Of these providers, 53 are physicians, 8 are nurse practitioners, and 1 is a registered nurse; 37 are men and 25 are women. Only 7 providers (11%) regularly assign PROMs (see definition above; 6 physicians, 1 nurse practitioner; 2 men, 5 women); 13 providers (21%) sometimes assign PROMs (see definition above; all physicians; 9 men, 4 women); and the rest (42 providers; 68%) have never assigned PROMs.

Implementation

Quantitative results

Although a small number of patients were assigned PROMs, the ratio of those who completed PROMs to those who were assigned PROMs increased substantially over time: 37.6% from October to December 2022; 61.8% from January to March 2023; 72.2% from April to June 2023; and 100% from July to September 2023. Note that some patients were assigned PROMs in previous quarters but completed the PROMs in later quarters, resulting in higher numbers of completed PROMs than assigned PROMs.

Qualitative results

Interviews were conducted with 6 clinicians in the HF clinic and lasted an average of 26 minutes, 43 seconds (standard deviation = 1 minute, 29 seconds). Four clinicians never assigned PROMs, and 2 clinicians usually assigned PROMs.

Clinicians discussed barriers and facilitators related to PROMs implementation, and these factors were categorized according to the TDF.¹⁵ All TDF domains were addressed, except beliefs about capabilities, optimism, intentions, goals, emotions, and behavioural regulation (see Table 7).

Table 7.

Factors influencing use of patient-reported outcome measures (PROMs) in clinical practice, according to the Theoretical Domains Framework

Domain	Definition	PROMs context	Quotes
Knowledge	An awareness of the existence of something	Clinicians unaware of PROMs existence in Epic; do not know that they should use PROMs in practice	“. . . I don't know how to access them [PROMs] in Epic, and I haven't triggered them” (C1) “Nope, . . . I've seen the prompts for PROMs, but I haven't used any data from prompts, and I don't know if that's something that's available to us . . . what happens is there's a prompt to send the survey in Epic on patients who have a history but that just then vanishes into outer space. And I don't know what happened with it?” (C2)
Skills	An ability or proficiency acquired through practice	Clinicians do not know how to interpret PROMs or incorporate PROMs into practice	“. . . It's never clear, at least to me, what the action item should be arising from that. The exception, of course, is that patients are very depressed or anxious because then we have mental health professionals that would refer them to but outside of that I don't know how to make them more able and realize a better quality of life” (C4)
Social/ professional role and identity	A coherent set of behaviours and displayed personal qualities of an individual in a social or work setting	Clinicians unclear about role in assigning, interpreting, and integrating PROMs; unclear about whether clinicians need to help patients in completing PROMs	“I just ignored it or didn't know if I had any responsibility to enroll the patient” (C5) “It's a very intensive case management model and so the nurses that that do that are well positioned and well skilled to be able to address that” (C4)
Beliefs about consequences	Acceptance of the truth, reality, or validity about outcomes of a behaviour in a given situation	Clinicians support PROMs but worried about follow-up; supports patient engagement and research opportunities	“What are my next steps? I don't feel that I have a great look (sic) . . . here's when and why to do it . . . From an evidence-based perspective, how does this lead to my next steps?” (C2) “I think it's anything you can do to (sic) enhance patient engagement is huge and anything you can do to if you can build in opportunities for patients to participate in research or educational initiatives, that's an added bonus. So, you know, identifying patients that may be interested in participating in research . . . You know, it's an example of how greater patient engagement leads to better outcomes” (C4)
Reinforcement	Increasing the probability of a response by arranging a dependent relationship, or contingency, between the response and a given stimulus	Clinicians lack incentives to use PROMs; may be punishments for addressing PROMs (eg, liability concerns)	“I don't know if there is a liability issue. For example, right now my patients may be completing the PROMs and I don't even know it exists, so I don't know who is receiving that information, who is acting on that information?” (C1) “I think you need to answer the questions for physicians, probably delivered by a colleague or someone in a leadership position—What's in it for me as the clinician, what's in it for my patients as a clinician? . . . One of the themes I keep harping on is like the currency for a lot of positions is time and so and so (sic) if you're asking more of someone's time, then the payoff for them or their patients has to outweigh that; Must be worth the investment if somehow this was time neutral and there was a payoff, even better” (C4)
Memory, attention, and decision processes	The ability to retain information, focus selectively on aspects of the environment, and choose between 2 or more alternatives	Clinicians experience cognitive overload related to Epic and PROMs; clinicians do not pay attention to or forget about PROMs	“There's a prompt to assign it which I think could be automated actually, if it's part of our QPS, then the physician shouldn't need to trigger it. . . . I don't actually get a survey back. I haven't yet, and I haven't gotten a survey actually pop up (sic) during my meeting with my page, so I don't know where to actually.” (C2) “The only thing is it's just sometimes it feels like you open Epic and there's 7000 things that are popping up at you and I think sometimes there's like, you know, clinician fatigue with things like that. And they're like, go away. Like there's an awful lot of stuff that, that (sic) pops up and I would bet that the majority of people just ignore it.” (C3)
Environmental context and resources	Any circumstance of a person’s situation or environment that discourages or encourages the development of skills and abilities, independence, social competence, and adaptive behaviour	Clinicians are busy and lack time to incorporate PROMs into practice; limited by resources available (eg, allied support, language barriers); context of internal medicine	“Lack of allied health support . . . we don't have nurses like the heart function clinic . . . We weigh our own patients [at Toronto Western]. So we have no additional supports, so it's just us seeing our high workload of a mixed bag of patients. So, it's also cognitive load not only because not only all heart failure (sic), so you're not in an algorithmically approach (sic), you're OK. Switching from heart attack, atrial fibrillation, valve disease. [PROMs] is lower on the checklist of priorities, because I'm focusing on GDMT and then moving on to the next” (sic) (C2) “I feel like truthfully, I feel busy enough like I feel really busy and it's, you know, enough to just get through the work that I have to get through with the clinic visit . . . I don't feel like I have extra time to review and talk to patients or encourage them” (C3)
Social influences	Those interpersonal processes that can cause individuals to change their thoughts, feelings, or behaviours	Lack of social pressure and norms to integrate PROMs; modelling is important	“I think it would be up to sort of the [heads of departments] to lead that. It's leading by example, right or them sort of championing it. Yeah, some like leadership role model” (C3) “No, I doubt that any of my Toronto Western Hospital folks are doing it other than (sic) [physician]. I'd be curious to know, but I don't think it's also been well advertised necessarily. And so we never met about it . . . it's not been in any of our site meetings and no one from the heart failure team has come to meet with us, to talk about it either.” (C2)

C1–C5, clinicians 1–5; University Health Network (Toronto, Ontario, Canada) electronic health record system; GDMT, guideline-directed medical therapy; QPS, quality patient safety.

Maintenance

Resources for implementation

The TDF domains listed above are linked to intervention functions outlined by the Behaviour Change Wheel.²¹ Relevant intervention functions that are supported by participant interviews include the following: education (knowledge, professional and/or social role and identity, beliefs about consequences); training (skills, memory, attention and decision processes); environmental restructuring (memory, attention, and decision processes, environmental context and resources); modelling (professional and/or social role and identity, social influences); and persuasion (professional and/or social role and identity, beliefs about consequences).

These intervention functions are linked to commonly used behaviour-change techniques²¹ and provide a starting point for interventions and/or resources to support the sustainability of PROMs in clinical HF care. See the list of suggested resources, the intervention functions, and behaviour-change techniques, and the evidence from participants, in Table 8.

Table 8.

Resources for maintenance of patient-reported outcome measures (PROMs) in clinical care, according to the Behaviour Change Wheel²¹

Resource	Description	Intervention function	Behaviour-change techniques	Participant quote
PROMs presentation during academic rounds	•Provide basic information on PROMs (what they are, how to use them) •Demonstrate how to assign PROMs and integrate them into practice •Have physicians practice assigning PROMs in Epic •Provide information on why PROMs are beneficial and how they will improve care	Education, Training	Demonstration of the behaviour, instruction on how to perform the behaviour, information about consequences	“ . . . there probably would be a slot of an hour where you could give a session [at the academic rounds]. Or you could do it at our business meeting once a month. For 10 minutes you could introduce the concept to the staff. You probably get more uptake at the business meeting.” (C6) “ . . . some education would be great and ‘how does it inform our practice?’ is going to be the most important part . . . Because it being part of QBP is not necessarily going to resonate. How does this actually improve the care we provide?” (C2)
Language translation	Translate questionnaires to patients’ first language to support equity	Environmental Restructuring, Enablement	Restructuring the physical environment	“The other barrier would be language. A good proportion of patients at Toronto Western, probably more than at Toronto General, would not have good command of English . . . and it's fell down online (sic).” (C6)
Add PROMs score to letter template	•Add PROMs score to letter template to ease interpretation of questionnaires when patients complete PROMs •Supports integration of PROMs into clinical practice	Environmental Restructuring, Enablement	Restructuring the physical environment, prompts/cues	“ . . . if there was some way to . . . have a letter template in Epic . . . like if you were able to pull the score directly into your letter . . . if it popped right up into your letter automatically, then you'd sort of see it in that moment.” (C3)
Change PROMs prompts in Epic	•Notify clinicians about low PROM scores only •Facilitate better notification in digital system with cognitive overload	Environmental Restructuring, Enablement	Restructuring the physical environment, prompts/cues	“ . . . it's a change in practice for the entire team to suddenly be thinking about this additional score that they're going to look at like every visit with the patient and it is a good idea to try to limit it to the people that are in alert phase or if there's something concerning in the scores. If they're acceptable, then it's almost like I don't need to know about it.” (C3)
Provide data on PROMs uptake, evidence of effectiveness, and what to do next	•Provide clinicians with analysis of PROMs uptake •Provide clinicians with data demonstrating benefits of using PROMs in practice •Provide clinicians with strategies of what to do if a patient has a low PROMs score	Education, Persuasion	Credible source, information about consequences, problem solving	“ . . . the biggest action is going to come when we have some data . . . if we look at some of the data for uptake assignment, tying into outcomes, doesn't lead to (sic) changes in prescription or referrals I think once we have a data-driven analysis, then I think people will be very interested.” (C4) “ . . . spell it out as much as possible would be the best strategy. You could link the score with a table indicating recommended next steps or strategies to manage that low score. We have lots of scores that we look up on the Internet and it stratifies patients in terms of the score and makes a recommendation as to what you should do . . . My general recommendation is to make it as simple as possible with the least steps as possible, because in a busy clinic the docs give up very quickly on something.” (C6)
Provide time to practice assigning, interpreting, and incorporating PROMs	Practice session where clinicians can try to use PROMs in Epic on a test patient so that they get used to PROMs	Education, Training	Demonstration of the behaviour, instruction on how to perform the behaviour, behavioural practice and rehearsal	“Practicing directly so it could be one of your actual patients (sic) or in a test environment, seeing how to click on the icons, which icon leads to which next step. And maybe just having a case scenario where the patient is enrolled and then looking at how the results are populated and where to find those results on a test patient . . . Because they gave us a tip sheet to follow. But maybe doing it in practice might be easier than reading through a document.” (C5)
Clarify PROMs roles for clinicians and lead by example	•Clarify roles and responsibilities with all clinicians (ie, whose job is it to assign, interpret, and incorporate PROMs) •Ensure staff leaders are modelling behaviour and encouraging others to engage in PROMs activities	Persuasion, Modelling	Demonstration of the behaviour, credible source	“If the bosses, sometimes it's leading by example, right or just or them (sic) sort of championing it. Yeah, some like leadership role modelling. I mean the more that you make it mandatory people don’t like it (sic).” (C3)

C2–C6, clinicians 2–6; Epic, University Health Network (Toronto, Ontario, Canada) electronic health record system; QBP, quality based procedure.

Discussion

This study used implementation science frameworks to assess the implementation of PROMs in an electronic health record system in urban outpatient HF clinics.¹⁴ Overall, an increasing number of patients were assigned PROMs over time, a greater percentage of patients completed PROMs over time, and generally, patients had high KCCQ-12 scores. Interviews demonstrated that healthcare providers may lack the knowledge, skills, and time to assign, interpret, and incorporate PROMs into their clinical practice. Providers also experience barriers related to memory, attention, and decision processes, environmental context and resources, and social and/or professional role and identity. This study demonstrates the benefit of using implementation science frameworks to systematically analyze the implementation of PROMs in clinical practice and to provide actionable recommendations to health systems.

Reach and effectiveness

The level of uptake of PROMs was lower at the TWH site than at the TGH site. This difference may be in part a result of the fact that TWH has fewer cardiologists than does TGH. However, as stated in the interviews, some clinicians (particularly at TWH) lack resources (eg, nurses, staff) to support them during patient visits. Having fewer resources makes the process of assigning, interpreting, and integrating PROMs into practice more challenging. Greater knowledge translation efforts are required to implement PROMs at the TWH site.

More patients were assigned PROMs by healthcare providers over time, and the rate of assignment increased from approximately 5% to 10%, suggesting increased implementation of PROMs. The rate of completion by eligible patients increased from 38% to over 72%. In comparison, in clinical breast care settings, only 55% of eligible patients completed PROMs, which reported on 700 surveys from over 200 patients.²² Therefore, patients in these clinics completed PROMs at higher rates, compared to those in other PROMs studies. Patients also had high KCCQ-12 scores, with average scores indicating “fair-to-good” health status across quarters. Good patient health may contribute to the high completion rates, as less-healthy patients may feel burdened and have less time and energy to complete PROMs. Further efforts should be made to support PROM completion among less-healthy patients or those who experience barriers to completing PROMs in the “MyChart” online area.

Adoption

Although previous studies have examined patient use of PROMs in research settings, this study is one of the first to examine provider PROMs-related behaviour in routine clinical practice. Only 7 providers (11%) regularly assign PROMs, and 13 (21%) occasionally assign PROMs, of a total of 62 providers. These numbers are low considering that PROMs were integrated into the electronic health record, as recommended repeatedly by previous research.^11,22 This finding demonstrates that integrating PROMs into an electronic health record is not enough, in and of itself, to encourage providers to assign PROMs. An important goal is to continue to evaluate and support PROMs implementation and further probe how both prompts and results can be better presented in electronic health records to promote their utilization in clinical decision-making.

Implementation and maintenance

Important factors related to PROMs uptake among providers included the following: knowledge; skills; social and/or professional role and identity; memory, attention, and decision processes; and environmental context and resources. These factors mapped onto resources to sustain implementation using the Behaviour Change Wheel.²¹ To begin, 3 of the providers interviewed were unaware that PROMs existed in the electronic health record, and all 6 providers interviewed stated that they could not interpret or integrate PROMs into care. Training and resources for providers included presenting at cardiology rounds, providing 2 “lunch and learn” sessions, a tip sheet in Epic, and engagement of clinical leadership in the Epic– PROMs integration. Lack of awareness is a common barrier to clinician use of PROMs,^11,23 and resources should include consistent efforts to provide education and training.²¹ Training should allow time for providers to practice assigning, interpreting, and incorporating PROMs. This training involves “behavioural practice and rehearsal,” a useful behaviour- change technique when healthcare providers lack needed skills.²⁴ Therefore, providing education and training that goes beyond “demonstrating the behaviour” of assigning PROMs may be one way to support change among providers.

A unique finding in this project was that 5 of 6 providers interviewed did not view the adoption of PROMs to be part of their job. Physicians viewed PROMs as a task for nurses, and nurses did not know that they had any role in using PROMs. PROMs education and training sessions should involve physician and nurse leaders who champion use of PROMs and regularly model behaviour for other staff. The strategy known as “modelling” (according to Michie et al., 2014) has been recommended by previous research in surgical and pediatric settings.^11,23 Using champions to deliver education and training sessions involves having a “credible source,” an effective behaviour-change technique to improve eHealth competency among healthcare providers.²⁵ These strategies and techniques may help to improve staff perceptions of their roles in using PROMs in the future.

During interviews, clinicians expressed that they experience cognitive overload when using an electronic health record, making it difficult to pay attention to PROMs when caring for patients. Even though the PROMs score is summarized in Epic with emojis, and trends are accessible in the patient record, providers still face issues in viewing these data. Changes in how clinicians are notified about PROMs can facilitate better uptake. For instance, clinicians could be notified about only low PROMs scores or only when changes indicate a minimal clinically important differences in patient PROMs (eg, scores from 55 to 75). Additionally, providers indicated that they would like patient PROM scores to be automatically integrated into their notes, helping prompt them to look at the scores. Therefore, “restructuring the physical (digital) environment” may be an important intervention function, and “prompts/cues” may be a valuable behaviour-change technique to support providers’ memory, attention, and decision processes in using PROMs.

Lastly, providers said that they face barriers related to their environmental context and resources (namely, time and language). HF physicians and nurses are busy, and they expressed concern with adding another metric to discuss with patients—a common perceived barrier for physicians using PROMs^11,22 To encourage providers to believe that discussing PROMs with patients is worth their (providers’) time, participants recommended that education and training sessions use the intervention function “persuasion” by telling providers about the benefits of PROMs. Indeed, research shows that PROMs reflect what patients perceive and that they improve the accuracy of clinicians’ assessments of patients’ New York Heart Association (NYHA) classification, when available to the assessing physician.²⁶ KCCQ-12 scores also correlate with changes in functional capacity, as measured by the volume of oxygen consumption, the 6-minute walk test, and the cardiopulmonary exercise test.²⁷ This work and other research on the effectiveness of PROMs should be incorporated into education and training sessions.

Limitations

This project is not without limitations. First, only provider experiences with PROMs were explored, and no patients were interviewed. Therefore, the information is limited on how patients viewed PROMs and whether they faced barriers to completing PROMs. Future research should examine patient and caregiver experiences, especially of those who are older, do not speak English, and have more-advanced HF. Second, 6 interviews were completed, and none of these interviews were conducted with providers who occasionally assign PROMs. Only 2 interviews were conducted with providers who use PROMs regularly. Future research should explore why some providers do vs do not assign PROMs. Additionally, whether other providers who assign PROMs also interpret and integrate PROMs is unknown. Third, this study did not track KCCQ-12 scores over time according to each patient, so analyses on changes over time cannot be conducted. Finally, this research was conducted in 2 Toronto hospitals within one network, so the findings may not be generalizable to other centres.

Conclusion

PROMs uptake improved over time for both patients and providers, but only a small proportion of prioritized providers in the heart function program regularly assign PROMs. Even when providers assign PROMs in the HF clinic, the qualitative results show that few (if any) integrate PROMs into patient care. When implementing PROMs, consistent training sessions with specific behaviour-change techniques need to be provided to clinicians. Electronic health records need to be optimized to reduce cognitive overload and increase visibility and clinical utilization of PROM scores. Implementation science frameworks should continue to be employed to evaluate the integration of PROMs, with an emphasis on understanding equity concerns for patients (eg, issues related to language, age, and stage of HF).

Availability of Data and Materials

Data are provided within the article or in the supplementary files.