Abstract
Background: Given the reemergence of pneumoconiosis in the United States, there is a tremendous need to train rural professionals in its multidisciplinary management. The Miners’ Wellness TeleECHO (Telementoring Extension for Community Health Outcomes) Program in New Mexico, United States, provides longitudinal multidisciplinary telementoring to professionals taking care of miners. The impact of this approach has not been previously evaluated.
Objective: To examine the change in self-efficacy of professionals taking care of miners and participating in the TeleECHO Program.
Methods: This is a 12-month longitudinal study involving clinical and nonclinical professionals caring for miners. The study outcome was the change in self-efficacy scores, using a customized instrument of 14 measures grouped into three domains: clinical, medicolegal, and soft skills. The primary outcome used a retrospective pre–post design that collects “pretest” data at the postintervention timeframe.
Results: Participants reported significant improvements in 10 of 14 items (P < 0.05) and a significant decline in 1 of 14 items (with respect to their ability to interpret pulmonary function test results, P < 0.001) since their start dates in the program. Subjects also reported significant improvement with respect to their scores for all three domains and for the 14-item total score (P ⩽ 0.01). Existing participants and clinical professional groups demonstrated greater improvement in selected items than fresh participants and nonclinical professional groups, respectively.
Conclusion: This study is the first in a stepwise approach to determine the benefit of participating in a multidisciplinary telementoring intervention by improving participant self-efficacy in caring for miners with complex mining-related diseases. Our study finding represents a potential solution to a growing access-to-care gap for miners with pneumoconiosis.
Keywords: pneumoconiosis, community of practice, telementoring, self-efficacy, retrospective pretest
The recent reemergence of pneumoconiosis (i.e., dust-related lung diseases), particularly among coal miners (i.e., black lung or coal workers’ pneumoconiosis [1]), in the United States has created a challenge for rural mining communities. Although the number of miners with pneumoconiosis requiring complex multidisciplinary care has increased, the expertise required to provide the care has decreased in rural United States (2). There is a tremendous need to train rural professionals caring for miners across the multidisciplinary aspects of the management of complex mining-related diseases, particularly in the pneumoconiosis-mortality hot-spot regions of Appalachia and the Mountain West (3). These multidisciplinary skills include clinical, medicolegal, and “soft” skills, with the latter category including interpersonal skills needed to navigate highly collaborative work in the care of miners. The existing inadequacy of expertise demands innovative solutions to support quality multidisciplinary teams to improve the health and well-being of miners in rural and remote communities.
The Miners’ Wellness TeleECHO (Telementoring Extension for Community Health Outcomes) Program, set up in July 2016 in New Mexico, United States, provides longitudinal multidisciplinary telementoring to various groups of professionals taking care of miners (4). Employing virtual techniques of engagement, the program uses a “community-of-practice” approach, defined in the literature as “groups of people who share a concern or a passion for something they do and learn how to do it better as they interact regularly” (5). Our group has recently published a preliminary cross-sectional evaluation of our community-of-practice approach, which demonstrated that the community was regarded highly by participants in relation to trust, willingness to help, and being closely knit (4). Participants reported low baseline self-efficacy (defined as the belief that one can actually perform a behavior or attain specific outcome [6]) with respect to diagnosing miners’ conditions, interpreting chest radiographs using the International Classification of Radiographs of Pneumoconiosis (i.e., B reads), and determining eligibility for compensation. The longitudinal impact of the virtual community-of-practice approach on the change in self-efficacy of participants has, however, not been previously evaluated, constituting a critical gap in knowledge. Because lack of clinician self-efficacy is a barrier to patient care (7), improvement in self-efficacy of TeleECHO participants may lead to improved care of miners. Addressing this gap in knowledge may allow for new evidence-based rural interventions for providing complex multidisciplinary care for miners. The objective of this longitudinal study was to examine the change in self-efficacy of participants, using a virtual community-of-practice approach in telementoring rural professionals taking care of miners in the United States. The outcome corresponds to level 3 of the Moore expanded framework for continuing medical education (CME) activities (8). If effective, it provides an exciting opportunity for greater investment in interprofessional telementoring to foster collaborative healthcare practices in rural and remote mining communities as an approach toward countering the resurgence of pneumoconiosis.
Methods
Study Design
This is a 12-month longitudinal study of participants involved in the TeleECHO Program, a novel community–university partnership between the University of New Mexico School of Medicine, Albuquerque, New Mexico, and its rural partner, the Miners’ Colfax Medical Center, Raton, New Mexico. Together, these New Mexico sites constitute the hub site of experts. The spoke partner sites, located across the pneumoconiosis-mortality hot-spot regions of the United States, include clinicians, respiratory therapists, home health professionals, benefits counselors, lawyers or attorneys, and others, including policy-makers, administrators, and mine-safety officers (4). The hub and the spoke partners together form a virtual community of practice.
Program Description
As published previously (4), the TeleECHO sessions are scheduled at the same time twice every month for 75 minutes, adhering to a standard format. After an initial 10 minutes of introduction and announcements, an invited expert delivers a 15-minute didactic presentation followed by a 20-minute, facilitated question–answer session and a 30-minute, interactive, facilitated case discussion. This format, employing adult-learning principles, focuses more on active learning through discussion than on didactic training. Participants can avail CME credits without charge upon completing the CME survey. A multidisciplinary curriculum committee frequently reviews the feedback provided to formulate a structured curriculum.
Irrespective of whether they present the case or not, participants can view the didactic and case discussions, provide information and insight from their own experience, and receive mentoring from the expert panel and their peers. Outside the program sessions, participants have access to experts or peers at the hub or other spoke sites, by telephone or e-mail, for urgent consultation requests. Over time with iterative practice and feedback, participants gain additional expertise and become more independent. Recorded and archived sessions are available from the program’s website without charge.
Program Development
Since July 2016, the program has used the ECHO model to provide structured long-term telementoring in the care of miners. This approach contrasts with telemedicine, in which experts provide short-term care to individual patients. This approach also differs from traditional lectures or webinars because it provides a real-time, interactive discussion of cases with expert panels. By contextualizing discussions, the program fulfills key learning theories, such as deliberate practice (9), social cognitive theory (10), and situated learning and communities of practice (11). As published previously (4), the five key principles of the ECHO model include the following: 1) use of Internet technology to leverage resources; 2) use of a disease-management model shown to be effective in other diseases by sharing best practices (12–15); 3) use of the principle of case-based learning; 4) creation of a community of practice, which emphasizes reciprocity in knowledge transfer, acknowledging that all participants bring some unique expertise; and 5) use of an Internet-based database to monitor outcomes. For the curriculum outlining individual didactics during the study timeframe, see Table E3 in the data supplement.
Outcomes
The study outcome was the change in self-efficacy scores, using a customized instrument created by a multidisciplinary curriculum and evaluations committee, based on the review of the literature and feedback from individual stakeholder groups. Self-efficacy, or a provider’s perceived ability to recognize and manage essential components of care of complex miners’ diseases and to serve as a local expert in these diseases for other community professionals, was assessed, with assessment adapted from previously validated instruments (16). Participants rated each of the 14 self-efficacy measures on a scale of 1–7, in which “1” indicates “none” or no skill and “7” indicates “expert” or ability to teach others. The 14 measures were grouped into three domains: a six-item clinical-skills domain, a three-item medicolegal-skills domain, and a five-item soft-skills domain. The self-efficacy instrument, provided in Table E4, was not fully validated in the current study. The change in self-efficacy items, separately calculated since the subject’s start date in the program (using the retrospective pre-posttest method with both pretest and posttest data obtained at the 12-month study time point) and over a 12-month study timeframe (using the traditional pre-posttest data method with pretest and posttest data obtained at the 0- and 12-month study time points, respectively) was calculated. For our primary study outcome, we chose a priori the retrospective pre–post design that collects “pretest” data during the postintervention timeframe at 12 months for the following two reasons. 1) By encouraging participants to reflect back and rate self-efficacy before participation in the ECHO Program with the benefit of hindsight, the retrospective pretest evaluation helps reduce the bias of self-evaluation (13). This approach is useful, as ECHO evaluators have found that participants often do not realize how much they did not know until after they have participated in the program (17). 2) Several prior ECHO outcome evaluations have demonstrated that, as compared with traditional pretest scores, retrospective pretest scores have lower mean and/or standard-deviation values, yielding greater power for analysis for small-sized comparison studies (18, 19).
Data Collection
Data were collected during the 1-year timeframe of September 12, 2018, to September 18, 2019, using Research Electronic Data Capture (Vanderbilt University), a secure Web application for building and managing online surveys and databases.
Analytic Strategy
Data collected were analyzed using Statistical Analysis Software version 9.4 (SAS Institute, Inc.). Wilcoxon signed rank sum tests for paired measurements were used to analyze change in continuous outcomes. Subgroup analyses for change scores were analyzed using t tests. Comparison groups included clinical professional groups versus nonclinical professional groups and existing versus fresh participants. Clinical professional groups include clinicians, respiratory therapists, and home health professionals (mostly nurses). Nonclinical professional groups include lawyers or attorneys, benefits counselors, and others. Fresh participants were defined as those who first attended the community of practice in or after the summer of 2018 (defined as May 9, 2018, and onward) versus existing participants (defined as those who had first attended any time between July 1, 2016, and May 8, 2018). Existing participants thus had greater cumulative participation and experience with the TeleECHO Program than fresh participants (11.4 ± 9.8 vs. 4.6 ± 4.6 total sessions before and/or during the study timeframe; P = 0.03). The cut-point date was chosen on the basis of the date of funding by the sponsor, which allowed the frequency of the TeleECHO Program to be raised from monthly to twice a month. A two-tailed P value < 0.05 was considered statistically significant without adjusting for multiple comparisons.
Ethics Approval
Approval was obtained from the institutional review board, Human Research Protections Office, at the University of New Mexico Health Sciences Center (18-386).
Results
As shown in Figure 1, 71 subjects completed the baseline survey (which collects information for the traditional pretest evaluation), and 37 of these completed the 12-month survey (which collects information for the retrospective pretest and posttest evaluation). Another 10 subjects joined the 12-month survey but did not complete the baseline survey. A total of 47 participants were therefore available for analyses. As shown in Table 1, most respondents reported caring for miners for 10 years or fewer (55.3%) and caring for at least a 40% proportion of miners living in rural areas (59.6%); 53.2% of subjects were existing participants. Clinical professional groups constituted the largest group of subjects (74.4%), with the largest subgroup being clinicians. Despite a high level of reported professional satisfaction, a significant minority described professional isolation.
Figure 1.
A flow diagram showing an overview of participation by study subjects.
Table 1.
Characteristics of surveyed participants at 12-month study time point (n = 47)
| Participant Characteristics | n | % |
|---|---|---|
| Duration of miner care, yr | ||
| 1 | 7 | 14.9 |
| 2–5 | 11 | 23.4 |
| 6–10 | 8 | 17.0 |
| 11–15 | 3 | 6.4 |
| 16–20 | 2 | 4.3 |
| 21+ | 6 | 12.8 |
| Not reported | 10 | 21.3 |
| Gender identity | ||
| Male | 13 | 27.7 |
| Female | 24 | 51.1 |
| Not reported | 10 | 21.3 |
| Race | ||
| White | 32 | 68.1 |
| Asian | 2 | 4.3 |
| Two or more races | 2 | 4.3 |
| Other | 1 | 2.1 |
| Not reported | 10 | 21.3 |
| Ethnicity | ||
| Hispanic | 4 | 8.5 |
| Non-Hispanic | 31 | 66.0 |
| Not reported | 12 | 25.4 |
| Age, yr | ||
| ≤30 | 3 | 6.4 |
| 31–40 | 8 | 17.0 |
| 41–50 | 6 | 12.8 |
| 51–60 | 9 | 19.1 |
| >60 | 9 | 19.1 |
| Not reported | 10 | 21.3 |
| Proportion of rural miners served, % | ||
| 81–100 | 10 | 21.3 |
| 61–80 | 10 | 21.3 |
| 41–60 | 8 | 17.0 |
| 21–40 | 2 | 4.3 |
| 0–20 | 7 | 14.9 |
| Not reported | 10 | 21.3 |
| Existing vs. fresh participants | ||
| Fresh | 22 | 46.8 |
| Existing | 25 | 53.2 |
| Individual stakeholder groups | ||
| Clinical professional groups | ||
| Clinician | 17 | 36.2 |
| Respiratory therapist | 9 | 19.1 |
| Home health professional | 9 | 19.1 |
| Nonclinical professional groups | ||
| Lawyer/attorney | 2 | 4.3 |
| Benefits counselor | 6 | 12.8 |
| Other | 4 | 8.5 |
| Satisfaction in professional practice | ||
| Strongly agree to agree | 44 | 93.6 |
| Neutral to strongly disagree | 3 | 6.4 |
| Lack of professional isolation | ||
| Strongly agree to agree | 34 | 72.3 |
| Neutral to strongly disagree | 13 | 27.7 |
Using the retrospective pre-posttest method, with both pretest and posttest data obtained at the 12-month study time point, subjects reported significant improvements in 10 of 14 items (P < 0.05, positive average change scores) and a significant decline in 1 of 14 items (with respect to their ability to interpret pulmonary function test results; P < 0.001, negative average change scores) since their start dates in the program (Table 2 and Figure 2). Subjects also reported significant improvement with respect to their scores for each of the three domains of clinical skills (P < 0.001), medicolegal skills (P = 0.04), and soft skills (P = 0.01) and for the 14-item total score (P = 0.002). As compared with fresh participants, existing participants demonstrated significantly greater improvement in their self-reported ability to diagnose health conditions (i.e., clinical-skill item 1; estimate, 0.78; 95% confidence interval [95% CI], 0.29–1.26; P = 0.003), in their self-reported ability to help manage common health conditions (clinical-skill item 2; estimate, 0.59; 95% CI, 0.01–1.17; P = 0.046), and in their clinical-skills-domain score (estimate, 0.33; 95% CI, 0.06–0.59; P = 0.02). As compared with nonclinical professional groups, clinical professional groups demonstrated significantly greater improvement in their self-reported ability to assess the quality of the pulmonary function test (i.e., clinical-skill item 4; estimate, 0.56, 95% CI, 0.08–1.04; P = 0.03; Table 3).
Table 2.
Change in self-efficacy items, separately calculated since the subject start date in the program and over a 12-month study timeframe
| All Subjects | Score at 12-mo Study Endpoint |
Change in Score since the Subject Start Date in the Program, Using the Retrospective Pre-Posttest Method |
Change in Score since the Study Start Date Using the Traditional Pre-Posttest Method |
|||||
|---|---|---|---|---|---|---|---|---|
| n | Mean ± SD of 12-mo Score | n | Mean ± SD Change in Score | P Value | n | Mean Change in Score SD of Change | P Value | |
| Clinical skills | ||||||||
| Ability to diagnose common health conditions in miners | 43 | 4.35 ± 1.63 | 43 | 0.47 ± 0.91 | <0.001 | 30 | 0.47 ± 1.07 | 0.02 |
| Ability to help manage common health conditions in miners | 44 | 4.66 ± 1.58 | 44 | 0.61 ± 1.04 | <0.001 | 34 | 0.56 ± 1.13 | 0.01 |
| Ability to interpret pulmonary-function test results | 46 | 4.20 ± 1.98 | 45 | −0.38 ± 0.72 | 0.001 | 36 | −0.31 ± 0.86 | 0.04 |
| Ability to assess the quality of the pulmonary-function test | 46 | 4.43 ± 2.04 | 46 | 0.41 ± 0.78 | 0.001 | 36 | 0.14 ± 1.02 | 0.36 |
| Ability to interpret arterial blood-gas test results | 47 | 4.55 ± 2.01 | 46 | 0.22 ± 0.63 | 0.04 | 37 | 0.03 ± 0.90 | 0.88 |
| Ability to interpret B-read reports of chest radiographs | 47 | 3.62 ± 1.84 | 47 | 0.30 ± 1.02 | 0.04 | 36 | 0.33 ± 1.59 | 0.17 |
| Average 6-item clinical-skills-domain score | 47 | 4.31 ± 1.59 | 47 | 0.27 ± 0.50 | <0.001 | 37 | 0.20 ± 0.68 | 0.07 |
| Medicolegal skills | ||||||||
| Ability to determine eligibility for compensation under specific miners’ compensation programs | 46 | 4.43 ± 1.70 | 44 | 0.39 ± 1.24 | 0.049 | 35 | 0.63 ± 1.63 | 0.01 |
| Ability to advocate for your patient/client to help them navigate the compensation process | 45 | 4.53 ± 1.60 | 45 | 0.40 ± 1.30 | 0.043 | 33 | 0.24 ± 1.32 | 0.27 |
| Ability to collect information required under the miners' compensation programs | 46 | 4.52 ± 1.67 | 44 | 0.30 ± 1.30 | 0.16 | 36 | −0.14 ± 1.53 | 0.84 |
| Average 3-item medicolegal-skills-domain score | 47 | 4.49 ± 1.58 | 47 | 0.37 ± 1.17 | 0.02 | 36 | 0.24 ± 1.33 | 0.14 |
| Soft skills | ||||||||
| Ability to demonstrate empathy toward miners with work-related diseases | 47 | 5.53 ± 1.28 | 45 | 0.16 ± 1.11 | 0.35 | 37 | −0.35 ± 1.18 | 0.11 |
| Ability to identify social, linguistic, cultural, economic, and educational barriers to care for miners | 47 | 4.89 ± 1.20 | 47 | 0.43 ± 1.04 | 0.01 | 36 | −0.44 ± 1.50 | 0.09 |
| Ability to serve as the miners’ expert in your community/region | 46 | 4.48 ± 1.71 | 46 | 0.48 ± 1.30 | 0.01 | 34 | 0.38 ± 1.44 | 0.15 |
| Ability to collaborate with and educate other team members about miners’ diseases | 47 | 4.87 ± 1.56 | 47 | 0.45 ± 1.08 | 0.001 | 37 | 0.30 ± 1.41 | 0.14 |
| Ability to refer patients with diseases to appropriate experts when you do not possess the relevant expertise | 47 | 4.87 ± 1.45 | 47 | 0.23 ± 1.25 | 0.29 | 35 | −0.20 ± 1.37 | 0.45 |
| Average 5-item soft-skills-domain score | 47 | 4.94 ± 1.24 | 47 | 0.34 ± 0.89 | 0.001 | 37 | −0.07 ± 0.96 | 0.40 |
| Average 14-item total score | 47 | 4.59 ± 1.29 | 47 | 0.31 ± 0.64 | <0.001 | 37 | 0.10 ± 0.73 | 0.27 |
Definition of abbreviation: SD = standard deviation.
The change in self-efficacy items, separately calculated since the subject start date in the program, was determined using the retrospective pre-posttest method with both pretest and posttest data obtained at the 12-month study time point. The change in self-efficacy items over a 12-month study timeframe was calculated using the traditional pre-posttest method, with pretest and posttest data obtained at the 0- and 12-month study time points, respectively. Improved item scores are indicated by positive change scores, and negative change scores indicate a decline in item scores. Absolute values for effect sizes for statistically significant differences ranged from 0.31 to 0.59, indicating only small-to-medium clinically meaningful changes (34). The data for the change in scores for those participating for both types of testing strategies, limited to 37 subjects who had both 0- and 12-month data measured, showed similar results, as provided in Table E5.
Figure 2.
Self-efficacy items at baseline (pretest), estimated retrospectively (retrospective pretest), and at 12 months (posttest). Error bars are 95% confidence intervals. The symbols inside the left graph margin indicate results from nonparametric Wilcoxon signed rank tests for change measured retrospectively (retrospective pre-posttest; n = 47) and from baseline (traditional pre-posttest; n = 37) using paired measurements (-P ≥ 0.05, *P < 0.05, **P < 0.01, and ***P < 0.001). Raw data for this figure are provided in Table 2. “Diagnose” indicates the ability to diagnose common health conditions in miners; “manage” indicates the ability to help manage common health conditions in miners; “interpret PF tests” indicates the ability to interpret pulmonary function (PF) test results; “assess PF test quality” indicates the ability to assess the quality of the PF test; “interpret BG tests” indicates the ability to interpret arterial blood-gas (BG) test results; “interpret X-rays” indicates the ability to interpret chest radiograph reports using the International Classification of Radiographs of Pneumoconiosis (i.e., B reads); “determine eligibility” indicates the ability to determine eligibility for compensation under specific miners’ compensation programs; “advocate/navigate” indicates the ability to advocate for your patient/client to help them navigate the compensation process; “collect information” indicates the ability to collect information required under the miners’ compensation programs; “demonstrate empathy” indicates the ability to demonstrate empathy toward miners with work-related diseases; “identify barriers” indicates the ability to identify social, linguistic, cultural, economic, and educational barriers to care for miners; “community/regional expert” indicates the ability to serve as the miners’ expert in your community/region; “collaborate” indicates the ability to collaborate with and educate other team members about miners’ diseases; “make referrals” indicates the ability to refer patients with diseases to appropriate experts when you do not possess the relevant expertise; “Avg clinical skills” indicates the average (Avg) 6-item clinical-skills-domain score (diagnose, manage, interpret PF tests, interpret BG tests, and interpret X-rays); “Avg medicolegal skills” indicates the Avg 3-item medicolegal-skills-domain score (determine eligibility, advocate/navigate, and collect information); “Avg soft skills” indicates the Avg 5-item soft-skills-domain score (demonstrate empathy, identify barriers, community/regional expert, collaborate, and make referrals); and “Avg total score” indicates the Avg 14-item total score.
Table 3.
Subgroup analysis of change in self-efficacy items since the subject start date in the program (using the retrospective pre-posttest method with both pretest and posttest data obtained at 12-mo study time point)
| Change in (Item, Domain, or Total) Score | Fresh Participants |
Existing Participants |
Clinical Professional Groups |
Nonclinical Professional Groups |
||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| n | Mean ± SD | P Value | n | Mean ± SD | P Value | n | Mean ± SD | P Value | n | Mean ± SD | P Value | |
| Clinical skills | ||||||||||||
| Ability to diagnose common health conditions in miners | 20 | 0.05 ± 0.69 | 0.75 | 23 | 0.83 ± 0.94 | <0.001 | 33 | 0.52 ± 0.94 | 0.004 | 10 | 0.30 ± 0.82 | 0.28 |
| Ability to help manage common health conditions in miners | 22 | 0.32 ± 0.72 | 0.05 | 22 | 0.91 ± 1.23 | 0.002 | 34 | 0.65 ± 1.01 | <0.001 | 10 | 0.50 ± 1.18 | 0.21 |
| Ability to interpret pulmonary-function test results | 22 | −0.32 ± 0.78 | 0.07 | 23 | −0.43 ± 0.66 | 0.01 | 33 | −0.48 ± 0.76 | <0.001 | 12 | −0.08 ± 0.51 | 0.59 |
| Ability to assess the quality of the pulmonary-function test | 22 | 0.41 ± 0.67 | 0.01 | 24 | 0.42 ± 0.88 | 0.03 | 34 | 0.56 ± 0.79 | <0.001 | 12 | 0.00 ± 0.60 | >0.99 |
| Ability to interpret arterial blood-gas test results | 22 | 0.09 ± 0.68 | 0.54 | 24 | 0.33 ± 0.56 | 0.01 | 34 | 0.29 ± 0.72 | 0.02 | 12 | 0.00 ± 0.00 | >0.99 |
| Ability to interpret B-read reports of chest radiographs | 22 | 0.00 ± 1.02 | >0.99 | 25 | 0.56 ± 0.96 | 0.01 | 35 | 0.31 ± 1.08 | 0.09 | 12 | 0.25 ± 0.87 | 0.34 |
| Average 6-item clinical-skills-domain score | 22 | 0.09 ± 0.33 | 0.20 | 25 | 0.42 ± 0.57 | 0.001 | 35 | 0.31 ± 0.53 | 0.002 | 12 | 0.13 ± 0.35 | 0.22 |
| Medicolegal skills | ||||||||||||
| Ability to determine eligibility for compensation under specific miners' compensation programs | 21 | 0.14 ± 1.11 | 0.56 | 23 | 0.61 ± 1.34 | 0.04 | 33 | 0.36 ± 1.27 | 0.11 | 11 | 0.45 ± 1.21 | 0.24 |
| Ability to advocate for your patient/client to help them navigate the compensation process | 21 | 0.24 ± 1.04 | 0.31 | 24 | 0.54 ± 1.50 | 0.09 | 33 | 0.36 ± 1.39 | 0.14 | 12 | 0.50 ± 1.09 | 0.14 |
| Ability to collect information required under the miners’ compensation programs | 21 | 0.33 ± 1.24 | 0.23 | 23 | 0.26 ± 1.39 | 0.38 | 32 | 0.28 ± 1.40 | 0.26 | 12 | 0.33 ± 1.07 | 0.31 |
| Average 3-item medicolegal-skills-domain score | 22 | 0.23 ± 1.05 | 0.32 | 25 | 0.49 ± 1.27 | 0.06 | 35 | 0.35 ± 1.22 | 0.1 | 12 | 0.42 ± 1.06 | 0.20 |
| Soft skills | ||||||||||||
| Ability to demonstrate empathy toward miners with work-related diseases | 21 | 0.19 ± 0.51 | 0.10 | 24 | 0.13 ± 1.45 | 0.68 | 34 | 0.15 ± 1.16 | 0.46 | 11 | 0.18 ± 0.98 | 0.55 |
| Ability to identify social, linguistic, cultural, economic, and educational barriers to care for miners | 22 | 0.41 ± 0.85 | 0.04 | 25 | 0.44 ± 1.19 | 0.08 | 35 | 0.31 ± 1.05 | 0.09 | 12 | 0.75 ± 0.97 | 0.02 |
| Ability to serve as the miners’ expert in your community/region | 22 | 0.50 ± 1.01 | 0.03 | 24 | 0.46 ± 1.53 | 0.16 | 34 | 0.41 ± 1.44 | 0.1 | 12 | 0.67 ± 0.78 | 0.01 |
| Ability to collaborate with and educate other team members about miners’ diseases | 22 | 0.41 ± 1.05 | 0.08 | 25 | 0.48 ± 1.12 | 0.04 | 35 | 0.43 ± 1.20 | 0.04 | 12 | 0.50 ± 0.67 | 0.03 |
| Ability to refer patients with diseases to appropriate experts when you do not possess the relevant expertise | 22 | −0.05 ± 0.90 | 0.82 | 25 | 0.48 ± 1.48 | 0.12 | 35 | 0.11 ± 1.35 | 0.62 | 12 | 0.58 ± 0.90 | 0.046 |
| Average 5-item soft-skills-domain score | 22 | 0.29 ± 0.67 | 0.05 | 25 | 0.38 ± 1.07 | 0.09 | 35 | 0.27 ± 0.98 | 0.11 | 12 | 0.53 ± 0.55 | 0.01 |
| Average 14-item total score | 22 | 0.19 ± 0.56 | 0.12 | 25 | 0.41 ± 0.70 | 0.01 | 35 | 0.30 ± 0.69 | 0.01 | 12 | 0.34 ± 0.51 | 0.04 |
For definition of abbreviation, see Table 2.
Positive change scores indicate improved item scores, and negative change scores indicate a decline in item scores.
In the alternate analysis of 37 subjects, using the traditional pre-posttest method, with pretest and posttest data obtained at the 0- and 12-month study time points, respectively, a significant improvement in 3 of 14 items (P < 0.05) and a decline in 1 item (with respect to their ability to interpret pulmonary function test results; P = 0.04) was noted over the 12-month study timeframe (Table 2). Subjects showed no significant improvement in any of the three domain scores or their total score over this timeframe. As compared with the nonclinical professional groups, clinical professional groups demonstrated significantly greater improvement in their self-reported ability to determine eligibility for compensation (i.e., medicolegal-skill item 1; estimate, 1.67; 95% CI, 0.25–3.08; P = 0.03), their self-reported ability to advocate for the patient/client to help them navigate the compensation process (i.e., medicolegal-skill item 2; estimate, 1.23; 95% CI, 0.06–2.40; P = 0.045), their medicolegal-skill domain score (estimate, 1.35; 95% CI, 0.29–2.41; P = 0.02), and their total score (estimate, 0.62; 95% CI, 0.06–1.18; P = 0.04; Table 4).
Table 4.
Subgroup analysis of change in self-efficacy items over a 12-month study timeframe (using the traditional pre-posttest data method with pretest and posttest data obtained at 0- and 12-mo study time points, respectively)
| Change in Item | Fresh Participants |
Existing Participants |
Clinical Professional Groups |
Nonclinical Professional Groups |
||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| n | Mean ± SD | P Value | n | Mean ± SD | P Value | n | Mean ± SD | P Value | n | Mean ± SD | P Value | |
| Clinical skills | ||||||||||||
| Ability to diagnose common health conditions in miners | 13 | 0.23 ± 1.24 | 0.51 | 17 | 0.65 ± 0.93 | 0.01 | 27 | 0.44 ± 1.12 | 0.05 | 3 | 0.67 ± 0.58 | 0.18 |
| Ability to help manage common health conditions in miners | 17 | 0.53 ± 1.18 | 0.08 | 17 | 0.59 ± 1.12 | 0.046 | 29 | 0.69 ± 1.07 | 0.002 | 5 | −0.20 ± 1.30 | 0.75 |
| Ability to interpret pulmonary-function test results | 17 | −0.41 ± 0.94 | 0.09 | 19 | −0.21 ± 0.79 | 0.26 | 29 | −0.31 ± 0.93 | 0.08 | 7 | −0.29 ± 0.49 | 0.17 |
| Ability to assess the quality of the pulmonary-function test | 17 | 0.12 ± 1.05 | 0.65 | 19 | 0.16 ± 1.01 | 0.51 | 29 | 0.14 ± 0.99 | 0.46 | 7 | 0.14 ± 1.21 | 0.77 |
| Ability to interpret arterial-blood-gas test results | 17 | 0.00 ± 0.87 | >0.99 | 20 | 0.05 ± 0.94 | 0.82 | 30 | 0.13 ± 0.90 | 0.42 | 7 | −0.43 ± 0.79 | 0.20 |
| Ability to interpret B-read reports of chest radiographs | 16 | −0.06 ± 1.77 | 0.89 | 20 | 0.65 ± 1.39 | 0.05 | 29 | 0.48 ± 1.70 | 0.14 | 7 | −0.29 ± 0.76 | 0.36 |
| Average 6-item clinical-skills-domain score | 17 | 0.07 ± 0.63 | 0.65 | 20 | 0.31 ± 0.71 | 0.07 | 30 | 0.28 ± 0.69 | 0.03 | 7 | −0.16 ± 0.55 | 0.48 |
| Medicolegal skills | ||||||||||||
| Ability to determine eligibility for compensation under specific miners’ compensation programs | 15 | 0.93 ± 1.39 | 0.02 | 20 | 0.40 ± 1.79 | 0.33 | 30 | 0.87 ± 1.36 | 0.002 | 5 | −0.80 ± 2.49 | 0.51 |
| Ability to advocate for your patient/client to help them navigate the compensation process | 15 | 0.47 ± 1.46 | 0.24 | 18 | 0.06 ± 1.21 | 0.85 | 28 | 0.43 ± 1.26 | 0.08 | 5 | −0.80 ± 1.30 | 0.24 |
| Ability to collect information required under the miners' compensation programs | 16 | 0.00 ± 1.55 | >0.99 | 20 | −0.25 ± 1.55 | 0.48 | 30 | 0.07 ± 1.34 | 0.79 | 6 | −1.17 ± 2.14 | 0.24 |
| Average 3-item medicolegal-skills-domain score | 16 | 0.45 ± 1.29 | 0.19 | 20 | 0.07 ± 1.36 | 0.83 | 30 | 0.46 ± 1.12 | 0.03 | 6 | −0.89 ± 1.80 | 0.28 |
| Soft skills | ||||||||||||
| Ability to demonstrate empathy toward miners with work-related diseases | 17 | −0.06 ± 1.34 | 0.86 | 20 | −0.60 ± 0.99 | 0.01 | 30 | −0.20 ± 0.92 | 0.25 | 7 | −1.00 ± 1.91 | 0.22 |
| Ability to identify social, linguistic, cultural, economic, and educational barriers to care for miners | 17 | −0.59 ± 1.73 | 0.18 | 19 | −0.32 ± 1.29 | 0.30 | 30 | −0.27 ± 1.41 | 0.31 | 6 | −1.33 ± 1.75 | 0.12 |
| Ability to serve as the miners' expert in your community/region | 16 | 0.75 ± 1.53 | 0.07 | 18 | 0.06 ± 1.30 | 0.86 | 29 | 0.55 ± 1.38 | 0.040 | 5 | −0.60 ± 1.52 | 0.43 |
| Ability to collaborate with and educate other team members about miners’ diseases | 17 | 0.41 ± 1.23 | 0.19 | 20 | 0.20 ± 1.58 | 0.58 | 30 | 0.27 ± 1.34 | 0.28 | 7 | 0.43 ± 1.81 | 0.56 |
| Ability to refer patients with diseases to appropriate experts when you do not possess the relevant expertise | 16 | −0.44 ± 1.59 | 0.29 | 19 | 0.00 ± 1.15 | >0.99 | 30 | −0.20 ± 1.45 | 0.46 | 5 | −0.20 ± 0.84 | 0.62 |
| Average 5-item soft-skills-domain score | 17 | 0.00 ± 1.14 | 0.99 | 20 | −0.12 ± 0.80 | 0.51 | 30 | 0.02 ± 0.89 | 0.89 | 7 | −0.45 ± 1.22 | 0.36 |
| Average 14-item total score | 17 | 0.12 ± 0.76 | 0.52 | 20 | 0.08 ± 0.73 | 0.61 | 30 | 0.22 ± 0.65 | 0.08 | 7 | −0.40 ± 0.89 | 0.28 |
For definition of abbreviation, see Table 2.
Positive change scores indicate improved item scores, and negative change scores indicate a decline in item scores.
Discussion
Telementoring multidisciplinary professional groups in the care of miners improved participants’ self-efficacy with respect to clinical, medicolegal, and soft skills. The retrospective pre-posttest method used since the participant start date with the program demonstrated improvement in a greater number of self-efficacy items and aggregate scores than the traditional pretest and posttest method over a 12-month study period. Existing participants and clinical professional groups demonstrated greater improvement in selected items than fresh participants and nonclinical professional groups, respectively. Our study demonstrates that the ECHO model can be successfully applied to professionals providing complex multidisciplinary care to miners.
Mining is a key economic activity in the rural United States, with potential for health hazards. With the United States having 28% of the world’s recoverable coal reserves, mining employment in the United States is likely to remain significant for many years, despite recent mine closures (20). It is anticipated that pneumoconiosis will continue to be seen in U.S. miners for several decades because of the long latency period of the disease. Miners constitute an underserved, isolated, medically vulnerable, and often underinsured rural population (2). In the southwestern United States, miners are predominantly Hispanic or American Indian high-school dropouts (2). The key challenges for caring for miners are the lack of adequately trained rural providers, perceived isolation among existing rural providers, and the complexity of the miners’ compensation systems (based on our preprogram launch needs assessment, A. Sood and colleagues, unpublished results). Addressing these challenges is a first step in fighting the emerging epidemic of pneumoconiosis but requires access to multidisciplinary expertise and training. The ECHO model leverages the power of information technology and case-based learning to provide state-of-the-art training and mentorship for such professionals in underserved areas.
The Miners’ Wellness TeleECHO Program has successfully recruited stakeholder groups from mining communities in pneumoconiosis-mortality hot-spot regions of rural Appalachia and the Mountain West. The target audience for the TeleECHO Program includes both clinical and nonclinical professionals who diagnose, treat, manage, and assist miners with pneumoconiosis. Evidence from our needs assessment demonstrates that treating pneumoconiosis is a complex process involving a difficult-to-navigate miners’ compensation system in the United States. Our multidisciplinary telementoring approach is essential to enable both clinical and nonclinical professionals to collaboratively develop comprehensive plans for providing the best possible care to miners with pneumoconiosis.
We found different results using our two analytic strategies: retrospective pre-posttest assessment since participant start date with the program versus traditional pretest and posttest assessment over a 12-month study period. The effect sizes tended to be greater, and changes were more likely to be statistically significantly different, using the former versus the latter approach, as also described by prior studies of ECHO outcomes (18, 19). By presenting the pre- and postassessments simultaneously in the retrospective pre–post design, respondents are forced to rate their status from the same frame of reference, which can help control for “response-shift bias” (21). This bias occurs when respondents use one frame of reference at the pretest but use a different frame of reference at the posttest, rendering it difficult to compare responses and resulting in erroneous inferences about program effectiveness. On the other hand, because the retrospective pre–post design queries participants “after” they have received the intervention, the participants may intentionally misrepresent the degree of impact they received from the intervention. Furthermore, given the possibility of recall bias in a retrospective pre–post design, this study design may replace one set of biases (vis-à-vis the traditional pre-posttest study design) with another (22).
Despite improvement in multiple self-efficacy parameters, our analysis indicates a decline in self-efficacy with respect to interpreting pulmonary function tests. This may relate to the confusion among rural professionals from multiple pulmonary-function-test updates by professional organizations since 2017 (3, 23–26). This also implies a need for greater attention in the future to this competency gap identified by our telementoring program. The fact that existing participants show greater improvement than fresh participants indicates that there may be a dose effect of telementoring on change in self-efficacy. This conclusion is supported by the finding that additional adjustment for participation frequency reduced the magnitude of differences between fresh and existing participants (see Figure E1 and Table E6). In addition, greater improvement in outcomes among participants belonging to the clinical versus nonclinical professional groups may reflect the disproportionate emphasis of clinical didactics and case discussions in the program (see Table E3). This implies a need for greater attention to nonclinical topics by our multidisciplinary program.
A strength of the current study is that our ECHO intervention is located in a rural and poor but mining-intense state, designated by the U.S. Health Resources and Services Administration as a medically underserved area. We believe that this increases the generalizability of the study findings to other rural mining regions. In addition, we tested the feasibility of applying a model of healthcare delivery that is already well established in other diseases, such as hepatitis C, by reducing variation in processes of care and sharing best practices (12–15). It is likely that this model of healthcare delivery may be similarly useful in the management of other lung diseases requiring multidisciplinary discussion, such as interstitial lung diseases (27). Because the ECHO model has been adopted nationally and globally to improve access to care in a number of disciplines, there is substantial infrastructure already in place to allow for rapid scaling of the Miners’ Wellness TeleECHO model. This telementoring approach was particularly beneficial during the coronavirus disease (COVID-19) pandemic, during which multiple face-to-face professional educational programs were canceled, resulting in its recognition as a rural COVID-19 innovation by the U.S. Health Resources and Services Administration’s Rural Health Information Hub. The American Thoracic Society also recognized the program as a 2019 innovation in fellowship education.
There are also limitations to the current study. We are unable to correlate improvement in professional self-efficacy with patient outcomes. The literature, however, indicates that professional self-efficacy influences behavior or performance and improves patient outcomes (28). For example, self-efficacy is a good predictor of asthma-guideline adherence among primary care clinicians (29) and of clinical performance of student physician assistants (30). Investigators report positive findings using randomized controlled designs to test the efficacy of electronic-learning interventions on novice surgeons’ surgical performance of laparoscopic cholecystectomy (31) and nurses’ ability to detect child abuse in an emergency department (32), respectively. A listing of qualitative changes that our ECHO participants reported they were going to make in their practice, obtained as part of a CME survey requested at the end of each TeleECHO session, are provided in Table E2 (with other quantitative outcomes provided in Table E1). A small sample size and multiple comparisons in the current study raise the possibility of type I error. High-risk individuals who drop out of the program or do not volunteer to participate in the study do not provide information in the estimation of the program effects, thus introducing an element of potential participation bias. The self-efficacy instrument was not fully validated in the current study. Standardized Cronbach α-coefficient values for the instrument, computed from the baseline measurement, however, showed high levels of internal consistency (values ≥ 0.85 for all items and each of the three domains).
This study is the first in a stepwise approach to determine the benefit of participating in a multidisciplinary telementoring intervention by improving participant self-efficacy in delivering high-quality care to miners with complex mining-related diseases. The acknowledgment of provider self-efficacy as a common barrier to optimal care of complex diseases (33) and as a predictor for provider behavior or performance and for patient outcomes (28) in the literature underscores the relevance of employing the TeleECHO intervention in professions caring for miners in medically underserved regions. Although future studies are required to determine any improvement in participant behavior or performance or miner outcomes, our study finding represents a potential solution to a growing access-to-care gap for miners with pneumoconiosis. This provides the rationale for using systems that are already in place for rapid scaling of the Miners’ Wellness TeleECHO Program at other institutions nationally and globally.
Supplementary Material
Footnotes
Supported by Alpha Foundation grant AFC719-14.
Author Contributions: A.S., C.P., S.K., N.A., K.L., B.S., and O.M. conceived of and designed the study. A.S., C.P., S.K., N.A., K.L., R.K., B.S., and O.M. performed the data collection and management. A.S., C.P., S.K., N.A., K.L., R.K., B.S., and O.M. conducted the data analyses and tabulated the results. A.S., C.P., S.K., N.A., K.L., R.K., B.S., and O.M. interpreted the results and drafted the manuscript. A.S., C.P., S.K., N.A., K.L., R.K., B.S., and O.M. critically edited the manuscript. All authors have read the manuscript and approved the final version to be published.
This article has a data supplement, which is accessible from this issue’s table of contents at www.atsjournals.org.
Author disclosures are available with the text of this article at www.atsjournals.org.
References
- 1.Laney AS, Blackley DJ, Halldin CN. Radiographic disease progression in contemporary US coal miners with progressive massive fibrosis. Occup Environ Med. 2017;74:517–520. doi: 10.1136/oemed-2016-104249. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Evans K, Lerch S, Boyce TW, Myers OB, Kocher E, Cook LS, et al. An innovative approach to enhancing access to medical screening for miners using a mobile clinic with telemedicine capability. J Health Care Poor Underserved. 2016;27:62–72. doi: 10.1353/hpu.2016.0182. [DOI] [PubMed] [Google Scholar]
- 3.Dwyer-Lindgren L, Bertozzi-Villa A, Stubbs RW, Morozoff C, Shirude S, Naghavi M, et al. Trends and patterns of differences in chronic respiratory disease mortality among US counties, 1980-2014. JAMA. 2017;318:1136–1149. doi: 10.1001/jama.2017.11747. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Sood A, Assad N, Jarrell W, Kalishman S, Le Suer K, Murillo S, et al. A virtual community-of-practice approach by rural stakeholders in managing pneumoconiosis in the USA: a cross-sectional analysis. Rural Remote Health. 2020;20:5784. doi: 10.22605/RRH5784. [DOI] [PubMed] [Google Scholar]
- 5.Wenger E. Communities of practice: learning, meaning and identity. Cambridge, UK: Cambridge University Press; 1998. [Google Scholar]
- 6.Bandura A. Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev. 1977;84:191–215. doi: 10.1037//0033-295x.84.2.191. [DOI] [PubMed] [Google Scholar]
- 7.Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, et al. Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282:1458–1465. doi: 10.1001/jama.282.15.1458. [DOI] [PubMed] [Google Scholar]
- 8.Moore DE, Jr, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29:1–15. doi: 10.1002/chp.20001. [DOI] [PubMed] [Google Scholar]
- 9.Ericsson KA. Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med. 2008;15:988–994. doi: 10.1111/j.1553-2712.2008.00227.x. [DOI] [PubMed] [Google Scholar]
- 10.Bandura A. Social cognitive theory: an agentic perspective. Annu Rev Psychol. 2001;52:1–26. doi: 10.1146/annurev.psych.52.1.1. [DOI] [PubMed] [Google Scholar]
- 11.Wenger E. How we learn: communities of practice—the social fabric of a learning organization. Healthc Forum J. 1996;39:20–26. [PubMed] [Google Scholar]
- 12.Arora S, Kalishman S, Thornton K, Dion D, Murata G, Deming P, et al. Expanding access to hepatitis C virus treatment: Extension for Community Healthcare Outcomes (ECHO) project: disruptive innovation in specialty care. Hepatology. 2010;52:1124–1133. doi: 10.1002/hep.23802. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Bouchonville MF, Hager BW, Kirk JB, Qualls CR, Arora S. Endo echo improves primary care provider and community health worker self-efficacy in complex diabetes management in medically underserved communities. Endocr Pract. 2018;24:40–46. doi: 10.4158/EP-2017-0079. [DOI] [PubMed] [Google Scholar]
- 14.Deming P, Arora S. Taribavirin in the treatment of hepatitis C. Expert Opin Investig Drugs. 2011;20:1435–1443. doi: 10.1517/13543784.2011.606214. [DOI] [PubMed] [Google Scholar]
- 15.Arora S, Thornton K, Murata G, Deming P, Kalishman S, Dion D, et al. Outcomes of treatment for hepatitis C virus infection by primary care providers. N Engl J Med. 2011;364:2199–2207. doi: 10.1056/NEJMoa1009370. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Chiu CY, Motl RW. Further validation of the multiple sclerosis self-efficacy scale. Disabil Rehabil. 2015;37:2429–2438. doi: 10.3109/09638288.2015.1021022. [DOI] [PubMed] [Google Scholar]
- 17.Project ECHO evaluation 101: a practical guide for evaluating your program New York, NY: New York State Health Foundation; 2017[accessed 2020 Sep 8]. Available from: https://nyshealthfoundation.org/wp-content/uploads/2017/12/project-echo-evaluation-guide.pdf [Google Scholar]
- 18.White C, McIlfatrick S, Dunwoody L, Watson M. Supporting and improving community health services-a prospective evaluation of ECHO technology in community palliative care nursing teams. BMJ Support Palliat Care. 2019;9:202–208. doi: 10.1136/bmjspcare-2015-000935. [DOI] [PubMed] [Google Scholar]
- 19.White C, McVeigh C, Watson M, Dunwoody L.Evaluation of Project ECHO (Extension for Community Healthcare Outcomes) Northern Ireland programme 2015–6 Belfast, Northern Ireland: Report for Health and Social Care Board, European Commission; 2016[accessed 2020 Sep 8]. Available from: https://ec.europa.eu/eip/ageing/sites/eipaha/files/practices/echo_ni_evaluation_report_2015-6_final_pdf.pdf [Google Scholar]
- 20.Petsonk EL, Rose C, Cohen R. Coal mine dust lung disease: new lessons from old exposure. Am J Respir Crit Care Med. 2013;187:1178–1185. doi: 10.1164/rccm.201301-0042CI. [DOI] [PubMed] [Google Scholar]
- 21.Thomas EV, Wells R, Baumann SD, Graybill E, Roach A, Truscott SD, et al. Comparing traditional versus retrospective pre-/post-assessment in an interdisciplinary leadership training program. Matern Child Health J. 2019;23:191–200. doi: 10.1007/s10995-018-2615-x. [DOI] [PubMed] [Google Scholar]
- 22.Taminiau-Bloem EF, Schwartz CE, van Zuuren FJ, Koeneman MA, Visser MR, Tishelman C, et al. Using a retrospective pretest instead of a conventional pretest is replacing biases: a qualitative study of cognitive processes underlying responses to thentest items. Qual Life Res. 2016;25:1327–1337. doi: 10.1007/s11136-015-1175-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Culver BH, Graham BL, Coates AL, Wanger J, Berry CE, Clarke PK, et al. ATS Committee on Proficiency Standards for Pulmonary Function Laboratories. Recommendations for a standardized pulmonary function report: an official American Thoracic Society technical statement. Am J Respir Crit Care Med. 2017;196:1463–1472. doi: 10.1164/rccm.201710-1981ST. [DOI] [PubMed] [Google Scholar]
- 24.Graham BL, Brusasco V, Burgos F, Cooper BG, Jensen R, Kendrick A, et al. 2017 ERS/ATS standards for single-breath carbon monoxide uptake in the lung. Eur Respir J. 2017;49:1600016. doi: 10.1183/13993003.00016-2016. [DOI] [PubMed] [Google Scholar]
- 25.Graham BL, Brusasco V, Burgos F, Cooper BG, Jensen R, Kendrick A, et al. Executive summary: 2017 ERS/ATS standards for single-breath carbon monoxide uptake in the lung. Eur Respir J. 2017;49:16E0016. doi: 10.1183/13993003.00016-2016. [DOI] [PubMed] [Google Scholar]
- 26.Graham BL, Steenbruggen I, Miller MR, Barjaktarevic IZ, Cooper BG, Hall GL, et al. Standardization of spirometry 2019 update: an official American Thoracic Society and European Respiratory Society technical statement. Am J Respir Crit Care Med. 2019;200:e70–e88. doi: 10.1164/rccm.201908-1590ST. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Grewal JS, Morisset J, Fisher JH, Churg AM, Bilawich AM, Ellis J, et al. Role of a regional multidisciplinary conference in the diagnosis of interstitial lung disease. Ann Am Thorac Soc. 2019;16:455–462. doi: 10.1513/AnnalsATS.201811-794OC. [DOI] [PubMed] [Google Scholar]
- 28.Sinclair P, Kable A, Levett-Jones T. The effectiveness of internet-based e-learning on clinician behavior and patient outcomes: a systematic review protocol. JBI Database Syst Rev Implement Reports. 2015;13:52–64. doi: 10.11124/jbisrir-2015-1919. [DOI] [PubMed] [Google Scholar]
- 29.Cloutier MM, Salo PM, Akinbami LJ, Cohn RD, Wilkerson JC, Diette GB, et al. Clinician agreement, self-efficacy, and adherence with the guidelines for the diagnosis and management of asthma. J Allergy Clin Immunol Pract. 2018;6:886–894, e4. doi: 10.1016/j.jaip.2018.01.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Opacic DA. The relationship between self-efficacy and student physician assistant clinical performance. J Allied Health. 2003;32:158–166. [PubMed] [Google Scholar]
- 31.Pape-Koehler C, Immenroth M, Sauerland S, Lefering R, Lindlohr C, Toaspern J, et al. Multimedia-based training on Internet platforms improves surgical performance: a randomized controlled trial. Surg Endosc. 2013;27:1737–1747. doi: 10.1007/s00464-012-2672-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Smeekens AE, Broekhuijsen-van Henten DM, Sittig JS, Russel IM, ten Cate OT, Turner NM, et al. Successful e-learning programme on the detection of child abuse in emergency departments: a randomised controlled trial. Arch Dis Child. 2011;96:330–334. doi: 10.1136/adc.2010.190801. [DOI] [PubMed] [Google Scholar]
- 33.Ozer EM, Adams SH, Gardner LR, Mailloux DE, Wibbelsman CJ, Irwin CE., Jr Provider self-efficacy and the screening of adolescents for risky health behaviors. J Adolesc Health. 2004;35:101–107. doi: 10.1016/j.jadohealth.2003.09.016. [DOI] [PubMed] [Google Scholar]
- 34.Cohen J. Statistical power analysis for the behavioral sciences. England, UK: Routledge; 1977. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.