Qualitative evaluation of two web‐based tools to improve accessibility of evidence reports

Abstract

Introduction

The Agency for Healthcare Research and Quality (AHRQ) Evidence‐based Practice Center (EPC) Program produces evidence reports to assist health systems in improving patient care. To make EPC evidence reports more accessible and usable, AHRQ convened a panel of learning health system (LHS) senior leaders to develop two web‐based tools—a table‐based tool presenting high‐level results, and a graphical display that presented detailed data—to disseminate the reports and test the tools. Researchers examined (1) the context under which interview participants reviewed and used the evidence reported by the tools, (2) their experiences using the tools, (3) the tools' influence on clinical practice, and (4) how the tools could be improved.

Methods

Researchers collected and inductively analyzed qualitative data from tool implementation meetings with six LHSs and interviews with 27 LHS leaders and clinical staff who used the tools. Researchers used website utilization metrics to augment qualitative results.

Results

The tools were efficient, complementary, and useful sources of summarized evidence to promote system change, educate trainees and clinicians, inform research, and support shared decision making with patients and families. Clinical leaders appreciated the evidence review thoroughness and quality and viewed AHRQ as a trusted source of information. However, many felt the tools were not practical for bedside use because of their complex content. Participants also noted the reports had limited evidence strength and robustness. They suggested optimizing the tools for mobile device use to facilitate tool uptake and developing training resources about tool navigation and statistical content interpretation.

Conclusions

LHSs found the tools to be useful resources for making the EPC Program reports more accessible to and usable for health system leaders. The tools have the potential to meet some, but not all, LHS evidence needs. Their value depends on reports' usefulness, which ultimately depends on the evidence quality.

1. INTRODUCTION

1.1. Background

Systematic evidence reviews—unbiased summaries of a body of research on a topic—can serve as an information source for developing clinical guidelines and pathways, decision aids, and systemwide policies. The format and content of evidence review reports can dissuade health systems from routinely relying on them when making decisions about systemwide improvements to care. Evidence review reports can be lengthy, dense, and laden with scientific terms and pages of tables, making them time consuming to review and difficult to interpret. ¹ , ² The reports may lack the contextual information that enables health systems to apply the synthesized findings to the populations they serve. Furthermore, the research questions that guide reviews may not be relevant to clinical practice or, when relevant, the questions may not elicit clear messages to guide healthcare delivery because the resulting evidence from the systematic reviews might be inconclusive or insufficient. ³ , ⁴

Since 1997, the Agency for Healthcare Research and Quality (AHRQ) Evidence‐based Practice Center (EPC) Program has created hundreds of evidence reports and other resources promoting evidence‐based practice. ⁵ Currently, nine EPCs review the scientific literature on topics relevant to clinical decision making or healthcare delivery issues and create detailed evidence syntheses to inform practice, policy, and research. The AHRQ EPC Program wants to help learning health systems (LHSs) use evidence from its evidence reports to improve healthcare delivery.

In 2018, the AHRQ EPC Program started a project to enhance LHSs' adoption of evidence to improve patient care. AHRQ contracted with the American Institutes for Research (AIR) and its partners to convene a panel of senior leaders from 11 LHSs to guide the development of two tools to make the findings from EPC Program reports more useful and actionable to health systems. The 3‐year project was divided into a tool development, learning loop, and tool revision phase and an implementation (pilot testing) phase, followed by an evaluation. The development and implementation phases are described in a companion paper. ⁶

1.2. Web‐based tools

Two interactive, web‐based tools were created to meet the need for more summarized information that also allow users to drill down for more detailed information (Table 1). A Summary of Findings tool, developed by AIR, presents high‐level results from the evidence reports with links to granular summary data. A Visual Dashboard tool, developed by the Scientific Resource Center for the EPC Program, visually presents detailed evidence syntheses data, including information about specific studies. Filters enable users to find information on interventions and populations of interest. Hover pop‐ups display descriptive data on a group of studies and analytic data on the findings with links to PubMed abstracts. Data are displayed in tables and forest plots.

TABLE 1.

Description of electronic tools tested in learning health systems

Tool	Purpose	Key features
Summary of findings	Determine whether the full evidence report is relevant	Presents high‐level results from the evidence reports and links to more granular data in the reports and the Visual Dashboard Users can view outcome data by evidence report question (eg, benefits and harms of a set of similar treatment or prevention interventions) and for a specific population (eg, adolescents with dysthymia, adolescents with major depression) Once an outcome is selected, a table unfolds showing the critical outcome (eg, suicidality, remission); the intervention and comparator; the number of studies; the findings; and the strength of the evidence Tabs link to pages describing the evidence review questions, related review findings and clinical guidelines, clinical and policy implications, and caveats and limitations of the review and the applicable population
Visual dashboard	Quickly grasp the overall magnitude, direction of effect, and strength of the evidence	Presents data from the evidence report and individual studies in a dynamic, interactive, web‐based tool Tabs enable users to view the data three different ways, based on parameters of interest: (1) aggregate data combining data from studies with the same intervention, comparator, population, and outcome; (2) individual study data; and (3) type of outcome and intervention (eg, nonpharmacological therapies, pharmacological therapies) For reports with quantitative data appropriate for meta‐analysis, forest plots show the magnitude and direction of effect, confidence intervals, and the strength of the evidence for studies with the same intervention (eg, drug, psychotherapy); comparator (eg, different drug, placebo); and outcome (eg, functional status, symptoms, serious adverse outcome)

To conduct a pilot test of the tools, AHRQ asked the LHSs to select an evidence report topic to populate the tools. The LHSs voted for topics from a list of recently released EPC reports. The panel collectively chose two reports—the Antipsychotics for the Prevention and Treatment of Delirium ⁷ and Treatment of Depression in Children and Adolescents ⁸ EPC reports. The tools were populated with data from each EPC report and integrated into a website with content structured into the following tabs: (1) the Summary of Findings tool with a list of the high‐level findings, (2) the Visual Dashboard tool, (3) the evidence report's structured abstract, (4) the evidence report summary, and (5) a link to the full report.

1.3. LHS implementation of tools

In 2021, six LHSs implemented the tools in clinical settings, selecting tool(s) and topic(s) in accordance with their systems' priorities. Implementation was defined as reviewing and considering the tools and their content as part of the health system's usual process to support decision making. The LHSs selected implementation settings where delirium or childhood and adolescent depression were relevant to practice (Table 2). AIR coaches worked with a LHS clinical champion and an implementation team to plan their system's implementation approach and to offer guidance and encouragement. ⁶ Five LHSs implemented the tools in clinical settings where clinicians were introduced to the tools and asked to review them and provide feedback. One system chose to implement the tool as part of an ad hoc multidisciplinary workgroup. The workgroup was formed to review evidence related to childhood and adolescent depression. Workgroup members focused on how the tools might be used in practice and whether the evidence within the tools differed from their system's current approach to treating childhood and adolescent depression.

TABLE 2.

Implementation elements and associated CFIR domains and definitions

Implementation element	CFIR domains	Definition
Usability	Advantage Acceptability Compatibility Feasibility	How easily LHS staff could use tool features, such as how easy the tool was to navigate, efficiency of locating information, ease of use after returning to the tool later, and how well they like the tool. Note: The evaluation focused on ease of navigation; earlier cognitive interviews focused on the other elements of usability.
Use	Appropriateness Fidelity	Extent to which LHS staff were actively engaged in reviewing the tool and considering how the content applied to the LHS patient population and current practices.
Tool adoption	Adoption Reach	The tool's intended use compared with its actual use. Extent to which the LHS leaders chose to share the tool(s) within their system for others to review and with whom.
Evidence adoption	Actionability Effectiveness	Whether systematic changes have been implemented to support the adoption of evidence, such as policy changes or changes in the electronic health record system.

Abbreviations: CFIR, consolidated framework for implementation research; LHS, learning health system.

Because of the LHSs' need to focus resources on their COVID‐19 pandemic response and AHRQ's desire to retain LHS participation in the project, AHRQ was flexible about how LHSs approached implementing the tools. In some cases, the implementation teams consisted of clinicians rather than decision makers, as originally planned. This meant that the tools were implemented in clinical environments as part of direct patient care rather than with groups of clinical, operational, and quality leaders who would review and discuss the reports to inform changes or improvements in healthcare delivery (Table 3).

TABLE 3.

Implementation settings for the summary of findings and visual dashboard tools

Antipsychotics for the prevention and treatment of delirium tools	Treatment of depression in children and adolescents tools
Hospital medicine departments (three LHSs) Family medicine department (one LHS)	Outpatient behavioral health departments (four LHSs) Pediatric primary care clinics (three LHSs) Family health center (one LHS) School‐based health centers (one LHS) Workgroup formed to review the evidence related to childhood depression (one LHS)

Note: Three LHSs implemented the delirium tools and five LHSs implemented the childhood depression tools.

1.4. The evaluation

The evaluation of the tool implementation assessed LHS leaders' and clinicians' experiences with the tools, their impressions of the tools, and how well the tools fulfilled their need for accessing evidence. The evaluation team also learned how well the tools overcame barriers to LHS use of the EPC reports and to applying the evidence from the EPC reports.

2. QUESTIONS OF INTEREST

The evaluation focused on the following questions:

• To what extent did clinical leaders and clinicians engage with the tools?

• Under what context did clinical leaders and clinicians review the evidence reported by the tools?

• What were clinical leaders' and clinicians' overall impressions of the tools and their content?

• Did the evidence summarized in the AHRQ EPC reports and presented in the tools influence clinical practice and, if so, how?

• Could the tools be improved and, if so, how?

3. METHODS

The evaluation team interviewed LHS clinical leaders and clinicians who used the tools and analyzed website utilization data from the implementation period. Figure 1 presents the steps the evaluators took in gathering and analyzing the data.

FIGURE 1.

Project phases, data collection points, and timeline

3.1. Implementation data collection

The evaluators gathered information shared during virtual coaching meetings with implementation teams, the meetings were held to plan the implementation, demonstrate the tools, and discuss progress and modifications to the implementation approach. Researchers documented information on (1) each LHS's evidence review process, (2) contextual factors related to implementation, (3) goals for the use of the tools, (4) implementation strategies, (5) user experiences with and impressions of the tools, and (6) implementation successes and challenges. The data collected during the implementation period supplemented the interview data.

3.2. Postimplementation interviews

After the implementation period, researchers held one‐on‐one interviews with 27 health system leaders and clinical staff who used the tools. The researchers had originally planned to conduct pre‐ and post‐implementation interviews. However, because the impact of the COVID‐19 pandemic resulted in a significantly shortened implementation period, researchers conducted a single set of interviews that included questions about both the pre‐implementation and implementation periods.

The implementation champions identified three to five leaders and clinicians to participate in the interviews which were held by teleconference. Before each interview, the three researchers who served as interviewers reviewed background data collected during the implementation phase. The interviews were recorded and transcribed, and assistants took notes.

Prior to the interview, the researcher obtained the interviewee's informed consent then followed a semi‐structured interview guide, approved by AIR's Institutional Review Board. The interview questions were designed to separate feedback about the tools' format and features from feedback about the tool content. The interviewers collected information on (1) the participants' engagement with the tools, (2) how the tools were used and by whom, (3) the participants' impressions of the tools and tool content, (4) benefits and relevance of the tools and content to the LHSs and to the participants' work activities, (5) whether the LHSs or clinicians made any changes in clinical practice after reviewing the evidence in the tools, and (6) suggestions for improving the tools.

Researchers met periodically to discuss the interviews and document the themes that emerged. Using NVivo qualitative analysis software and a priori and inductive methods, the team coded the transcripts to synthesize the data into themes. The researchers developed a list of codes, guided by a modified Consolidated Framework for Implementation Research (Table 2). ⁹ , ¹⁰ The coders ensured consistency in the way the codes were applied by agreeing on operational definitions of the revised code list. Two team members coded one transcript, making note of places where additions, revisions, or deletions to the codes were needed to better fit the data. To measure interrater agreement, the team used Cohen's Kappa (until an agreement level of 0.80 was achieved between the two coders). The coders resolved any discrepancies in application of the codes by consensus.

The researchers analyzed the coded data by tool type, evidence topic, and purpose for seeking out evidence, using the constant comparative method and considering the participant's job role.

3.3. Website Utilization Metrics

The evaluators collected website metrics to provide a general sense of how many people accessed and interacted with the tools. The evaluators used Google Analytics to analyze metrics on LHSs' use of the pilot website during the implementation period. These metrics included: (1) user, defined as a person who interacted with the pilot website; (2) sessions, defined as the overall number of website visits; (3) sessions per user, defined as the average number of times someone came to the pilot website; (4) page, defined as the number of pages visited on the website; (5) pages per session, defined as the average number of pages someone viewed during a session; and (6) average pageview duration, defined as the average length of time someone spent per page.

4. RESULTS

4.1. Interview participants

The evaluation team interviewed 27 clinical leaders and clinicians who were affiliated with critical care, behavioral health, pediatrics, and primary care settings at the six LHSs (Figure 2). About three quarters (21) of the participants held one or more administrative positions, such as vice president, medical or department director, quality director, training or residency program director, or clinical manager. The remaining six participants were clinical staff, composed of four hospitalists, one hospitalist/physician scientist, and one clinical pharmacist. Most (21) of the 27 participants were asked by the implementation team to review and use the tools. Five participants were implementation champions at their health system, and one served on the LHS panel and as an implementation champion.

FIGURE 2.

Participants by clinical specialty/license and primary role

4.2. Findings

4.2.1. Health system clinical leaders' and clinicians' engagement with the tools

Data on tool use came from both the website utilization metrics and the interviews. The website utilization metrics provide a general sense of how many people (LHS leaders and staff, AIR project team and AHRQ staff) accessed the tools. Nearly 150 unique users accessed the tools from late March to the end of July 2021. Users made 311 visits to the website and averaged two returns per user, spending 1 min 14 s on average on the site. They viewed almost seven pages per session. Users bookmarked the website and shared direct page links to the tools with their colleagues.

Interviewees reported spending wide‐ranging amounts of time using the tools. The exposure of a few participants (from one LHS) was limited to observing tool demonstrations at a kickoff meeting and then navigating the tools during the interview. Other participants (from three LHSs) accessed the tools multiple times, seeking answers to specific questions. One participant reported exploring the tools for at least an hour, while another reported accessing the tool for 10 to 15 min. Others (from five LHSs) reported spending 30 to 45 min to learn how to navigate the tools and interpret the content.

“Initially … it's [the Summary of Findings, which contains] … a lot of information and I was kind of stuck on how to go about using it. … So I just … kept playing with it until I … got a little more familiar with it and then I was able to find it [a] little bit easier. It took … two or three times before I was able to get used to it."

—Hospitalist

4.2.2. Reasons for health system leaders and clinicians to review the evidence reported in the tools

Interview participants thought the tools would be helpful in a variety of situations where reviewing the evidence is beneficial. Participants most consistently supported use of the tools for informing system change and clinician training and, more generally, as a refresher on the evidence. Several participants (from four LHSs) thought the tools could help in the development of clinical care pathways and decision support.

“[The tools] show in a very quick way [that] this is the evidence for why we're recommending this, which also means it's going to be a helpful tool to create those care process models."

—Clinical director, psychiatrist

Several participants (from four LHSs) believed the tools could be used to train students, residents, and fellows and prepare for lectures. Others saw the tools as a resource for primary care providers, particularly when specialists are in short supply or unavailable. Some participants discussed using the tool as a point‐of‐care reference, although most found the content not directive enough for making treatment decisions and too time consuming to review during a patient encounter. A few participants (from two LHSs) thought the tools would make it easier to make treatment decisions; one primary care provider noted that she would use the tool to influence colleagues whose antipsychotic prescribing practices were inconsistent with the evidence.

A few participants (from two LHS) thought the childhood and adolescent depression tools could help them inform parents about the effectiveness of antidepressants and cognitive behavioral therapy as part of a discussion of treatment options. Others thought that reviewing tool content would increase their confidence when discussing treatment recommendations with families.

4.2.3. Health system clinical leaders' and clinicians' overall impressions about the tools and their content

Clinical leaders and clinicians found the tools to be efficient and useful sources of summarized evidence. Over half of interviewees (from all six LHSs) liked how the tools synthesized the evidence clearly and concisely. Participants thought the tools improved on the literature databases and search engines—their usual evidence information source. Participants appreciated how the tools enabled them to view the overall results at a glance, and then apply filters to review more detailed aggregate or study specific data. Clinical leaders, many of whom were unfamiliar with the EPC Program products, appreciated the thoroughness and quality of the evidence reviews. They saw AHRQ as a trusted information source. Participants were hopeful that AHRQ would incorporate additional EPC Program reports into the tools.

“The tool is fabulous. It's so much better than having to wade through 1,145 pages. And it will be very useful when there is evidence that is valuable to guide practice… it's a very sophisticated, very useful, slick set of tools.”

—Vice president and quality officer, pediatrician

Participants found both tools to be valuable and complementary, although some favored one tool over the other. Participants' preferences were driven by their preference for how information is displayed, and the amount of detail provided. Nearly half of participants (from five LHSs) found the Summary of Findings tool to be concise and easy to scan and understand. Participants appreciated that the filters enabled them to quickly search for drug and treatment approaches. Similarly, many of the participants (from all six LHSs) commented on how they could review and grasp the aggregated study findings quickly.

“[The Summary of Findings tool provided] quick access to the information, glancing over, not spending too much time, but walking away with some information that you retain.”

—Hospitalist

“The ability to hover over, click into, highlight, and emphasize, get to the actual articles, all of that ability to move quickly between sections or screens I think is very well done [in the Visual Dashboard]. The graphics, the whole presentation was nice.”

—Vice president and quality officer, pediatrician

Although they found the tools useful, participants also noted several challenges to tool use. Most participants thought that the Summary of Findings and Visual Dashboard tools were not practical for bedside use because of the complexity of the content and the time it would take to find information relevant to decision making. Some participants (from four LHSs) thought that the Summary of Findings tool—and specifically the number of subpopulation categories (eg, 18 pediatric depression categories in the childhood and adolescent depression tool)—included an excessive amount of information that could be a barrier to primary care provider use. Both LHS representatives attending a June 2021 LHS implementation meeting and interviewees expressed concern that clinicians might have difficulty interpreting the forest plots displayed in the Visual Dashboard tool because of the use of statistical terms and large amount of information presented. Some clinicians struggled with interpreting the statistical data during the interviews.

“I don't think that many of my primary care colleagues or even the people who we work with in the hospital … they are not going to always appreciate adolescents with the dysthymia versus major depression or some sort of combination, so I really worry that the minute that they see these distinctions, they're going to basically say, ‘Call the child psychiatrist, call the child psychologist, let them figure this out.’”

—Clinical director, psychologist

“The clinicians found the statistical information to be confusing and have told me that they don't really understand how that would be useful for them in their patient care. It doesn't seem to them like information that they understand.”

—Clinical director and training director, psychologist.

4.2.4. Influence of evidence on clinical practice

Participants spoke about the tool content in the context of making system‐level decisions about treatment and as part of their own approach to treatment. Some participants (from four LHSs) were disappointed by the insufficient evidence or inconclusive evidence on treatment of delirium and childhood depression. One department director saw the limited evidence as humbling because, before reviewing the tool, she thought the available evidence was more robust.

“[Y]ou're basing your clinical decision making on more than a tool because, again, the tools [are limited by] the lack of robust evidence.”

—Vice president and clinical director, psychiatrist.

Still, some participants said that the evidence and the key findings listed in the tools were consistent with their clinical practice or their system's approach to treatment. Although participants saw the evidence reported by the tools as unlikely to change practice, several participants (from three LHSs) thought the tools would help them to stay current and believed tool use demonstrates their system's commitment to practicing evidence‐based care.

4.2.5. Suggestions for improving the tools

Participant's suggestions for improving the tools concerned minor adjustments to formatting, text and graphics and ways to improve tool access and enhance tool usability and uptake. To augment bedside use, participants suggested adding resources to help clinicians implement the evidence. The following are examples of suggested improvements.

• Presentation of tool content and data: To find and interpret information that is most relevant to users' needs, add summary text to convey the big picture and key takeaways, add links from the key points for practice to the evidence, and make several formatting enhancements.

• Tool access: Increase visibility and uptake of the tools by optimizing the tools for use on mobile devices, adding links to the tools from the electronic health record system and apply search engine optimization techniques.

• Tool usability and uptake: Enhance tool functionality by providing more detailed and summary information, and training resources on tool navigation, how to interpret statistical data and terms, and use of the tool features to answer clinical questions.

• Additional resources for use at the point of care: Facilitate provider behavior changes and hardwire evidence‐based practices by linking to clinician‐friendly resources about treatments, such as clinical guidelines, and developing plain language resources for patients and families.

5. DISCUSSION

This study evaluated the implementation of two web‐based reporting tools that were designed to overcome the challenges associated with uptake of lengthy and technical evidence reviews. By layering and summarizing key results and data, the Summary of Findings tool and a Visual Dashboard tool aimed to enable users to access report topics of interest quickly and at the level of detail desired. In interviews, LHS leaders and staff reported that the tools made evidence report content more accessible and efficient to access. LHS leaders and staff found the tools to be useful for informing system change, clinician training, and, more generally, as a refresher on the evidence.

Although the tools were successful in overcoming barriers associated with reporting, there are opportunities to improve access to the tools and LHS leader and staff comprehension of the content. In keeping with clinicians' and leaders' reliance on smart phones as a readily accessible information source, LHSs suggested optimizing the tools for mobile use to improve uptake. Difficulty interpreting statistical data and the time required to become comfortable using the tools during the pilot test pointed to the need for brief tutorials and other types of training resources that enable busy clinical leaders and clinicians to quickly grasp statistical content and become familiar with tool features and navigation.

The evaluation also highlighted barriers associated with the robustness and actionability of the evidence. The LHSs collectively selected report topics for the implementation that aligned with organizational priorities. Unfortunately, the scientific literature on the topics selected is largely inconclusive. The lack of robust and conclusive evidence frustrated participants who were hoping to learn about recent research supporting the use of specific treatments to improve practice. The evaluation underscores the importance of accelerating the growth of practice‐based research and program evaluation that produce timely evidence of sufficient quality and quantity to guide practice. This finding also highlights the value of a continued AHRQ and LHS partnership to identify EPC report topics relevant to LHSs' information needs.

5.1. Limitations

The evaluation of the tools was limited by a shortened implementation period and the implementation settings. Because of the demands placed on LHSs during the COVID‐19 pandemic, the implementation period was reduced from 12 to 3 months. During this time, LHS leaders and clinicians were coping with the challenges of treating patients with known or suspected COVID‐19, keeping staff safe, and managing facility capacity and other concerns. The tools were implemented in direct patient care settings rather than as a resource to inform systemwide changes or improvements in health care delivery, for which the tools were designed.

6. CONCLUSIONS

The evaluation indicates that web‐based tools are an efficient and useful approach to improve the accessibility of lengthy and complex evidence report findings to health systems. Designing tools that are mobile optimized and that accommodate varied abilities to interpret graphics and technical content, and preferences for more or less detail are important for tool uptake. Developing an evidence review reporting tool that meets all potential health system user needs for evidence summaries may not be practical or feasible. LHS feedback highlights the need for an additional tool—an evidence‐based point of care tool that concisely summarizes treatment recommendations with links to the supporting evidence. Adding resources to the existing tools to help providers implement the evidence included in the reports could also bridge the gap between research to practice, potentially facilitating behavior change and hardwiring evidence‐based practices. Such resources include linking to clinician‐friendly resources about treatments, such as clinical guidelines, and developing plain language resources for patients and families. Physicians and clinical leaders might be prompted to adopt and use the tools if the tools are certified as continuing medical education so that clinicians can earn credits for time spent reviewing the tools and EPC Program reports. Provision of training resources could also help physicians and clinical leaders learn how to use the tools and interpret the data and other content.

CONFLICT OF INTEREST

None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report.

DISCLAIMER

The findings and conclusions in this paper are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

Paez KA, Shapiro R, Thompson L. Qualitative evaluation of two web‐based tools to improve accessibility of evidence reports. Learn Health Sys. 2023;7(2):e10341. doi: 10.1002/lrh2.10341