Using a Modified Delphi Methodology to Identify Essential Telemedicine Skills for Pediatric Residents

Abstract

Objective

Telemedicine use in pediatrics increased during the coronavirus disease-2019 (COVID-19) pandemic. Despite rapid uptake by pediatric residency programs, consensus on essential telemedicine skills for pediatric residents is lacking. We used a modified Delphi methodology to identify essential telemedicine skills and behaviors for pediatric residents.

Methods

A focused literature search was performed to identify items for review by pediatric telemedicine experts. A modified Delphi methodology consisting of iterative rounds of anonymous surveys was conducted until consensus for each item was reached. Consensus was defined as >80% of experts identifying a topic as “very important.” All items were mapped to one of the Accreditation Council for Graduate Medical Education (ACGME) core competencies.

Results

Seventeen pediatric telemedicine skills and behaviors achieved a consensus of “very important.” Most items mapped to the ACGME core competency domains of interpersonal and communication skills and professionalism.

Conclusions

There was a high degree of agreement among pediatric telemedicine experts on the importance of 17 telemedicine skills and behaviors for pediatric trainees. These skills can inform pediatric telemedicine curricula and provide validity evidence for pediatric telemedicine assessment tools.

Rates of telemedicine use in residency training programs increased during the COVID-19 pandemic, most notably after the Accreditation Council for Graduate Medical Education (ACGME) released a provision permitting resident use for clinical rotations.1, 2, 3, 4 Telemedicine, as used in this paper, is defined as the delivery of health care through remote synchronous video visits.⁵ An October 2021 study reported a 468% increase in telemedicine use across pediatric residency programs, most often in the longitudinal outpatient and subspeciality clinic settings.⁶ Despite the existence of pediatric telemedicine operating procedures, there is little information regarding best practices specifically for pediatric trainees.⁷

There have been a number of published telemedicine curricula, standardized patient encounters, and proposed competencies for trainees, with most focusing on principles of adult care.^{4 ,} 8, 9, 10, 11 A limiting factor for these resources is that the content was derived from experiential learning, not evidence-based methods, which is appropriate given the novelty of telemedicine in graduate medical education (GME). In October 2021, Hart et al¹² took an evidence-based approach and used a modified Delphi methodology to identify telemedicine curriculum competencies for GME, producing a list of competencies, but not specifically for pediatrics.

There remains a gap in our understanding of essential telemedicine skills for pediatric trainees, as several aspects of telemedicine practice are unique to pediatrics, including examination techniques, adolescent confidentiality, and parent education.¹³ In September 2021, the American Board of Pediatrics began to address this gap and released eight telemedicine Entrustable Professional Activities (EPAs).^{14 , 15} To our knowledge, other than these pediatric EPAs, there is no literature describing essential telemedicine practices for pediatric residents. Identification and validation of these skills and behaviors is crucial for building trainee competency in telemedicine, ultimately improving the quality of medical care provided to patients.¹⁶ The purpose of this study was to use a modified Delphi methodology to obtain consensus on essential telemedicine skills and behaviors for pediatric residents.

Methods

Study Design

The modified Delphi method is a group consensus technique that uses literature review, input from stakeholders, and the judgment of experts within a field to reach agreement. It is useful when evidence is limited, as it relies on the “collective intelligence” of group members to produce better results than any one group member could produce individually, increasing content validity. Our methodology consisted of 1) performing a focused literature review to identify telemedicine skills and behaviors, 2) refining the preliminary list with national stakeholders, and 3) conducting a modified Delphi panel to establish consensus on the importance of each item.

Focused Literature Review

A medical librarian assisted in a focused literature review to identify existing telemedicine skills and behaviors. We searched PubMed and Web of Science databases using MeSH terms for three categories: “resident,” “skill,” and “telemedicine.” The “telemedicine” MeSH term includes the terms “mobile health,” “mHealth,” “telehealth,” and “eHealth.” The term “pediatric” was not included because it limited the search results substantially. The search included English-language publications from January 2015 to July 2020. Title and abstracts were screened by the medical librarian and primary author. Full articles were reviewed by the primary author.

List Refinement

We reviewed candidate items from these searches and eliminated redundant and “not observable” items, defined as “skills or behaviors that could not be detected remotely by a third party during a telemedicine encounter." The items were reviewed by key stakeholders from a national working group developing a national telemedicine curriculum for pediatric residents.⁶ Stakeholders commented on clarity and observability, and could suggest new items that may not have been identified by the literature review. The goal of this stage was to use a variety of perspectives to refine the list of candidate items.

Modified Delphi Panel

After initial review by stakeholders, a second group of 19 national pediatric telemedicine experts were identified based on their contributions to pediatric telemedicine at the national or regional level, the scope of their clinical telemedicine use, and their level of involvement with pediatric residents (Table 2). Twelve experts participated in the panel; 66% of experts were involved in national telemedicine work, 83% practiced in academic settings, and 66% practiced in primary care or emergency medicine settings, which are common settings for pediatric telemedicine. All experts worked with residents regularly. Participation implied consent, and experts could withdraw at any time. To solicit input from a diverse group, experts were recruited using purposive sampling.

Table 2.

Characteristics of Telemedicine Experts on the Delphi Panel

Characteristic	Number of Experts (%)
Involvement in telemedicine projects or research
National	8 (66%)
Local/Regional	4 (33%)
Regular interaction and supervision of pediatric residents	12 (100%)
Location of practice
Northeast	5 (42%)
Southeast	2 (16%)
Midwest	5 (42%)
Northwest	0 (0%)
Southwest	0 (0%)
Primary clinical practice setting
Academic	10 (83%)
Private	2 (17%)
Specialty
Primary care general pediatrics	2 (17%)
Children with Medical Complexity (CMC) general pediatrics	1 (8%)
Pediatric emergency medicine	4 (33%)
Pediatric critical care	1 (8%)
Pediatric gastroenterology	2 (17%)
Neonatal-perinatal medicine	2 (17%)

Iterative rounds of surveying were conducted from April to August 2021 to achieve consensus on essential telemedicine skills and behaviors for pediatric residents. Consensus was defined a priori as 80% agreement among experts.¹⁷ Surveys were distributed to participants through REDCap, a secure web-based survey management application.¹⁸ We preserved anonymity throughout each round of the Delphi panel, except for the live video conference. In each round, experts were asked to answer the question, “how important is this skill or behavior to a pediatric resident's use of telemedicine?” and instructed to rank each item as, “not important,” “somewhat important,” or “very important” (Supplemental Data 2 and 3). Experts were also able to provide free-text comments, including grammar or formatting suggestions.

Survey results were scored after each round using a three-point scale (0 for “not important,” 1 for “somewhat important,” and 2 for “very important”) and a fourth option of “not observable.” A three-point scale was used to enable participants to express stronger or weaker agreement on importance, allowing for a better understanding of expert opinion. Items that achieved a consensus rating of “very important” were deemed essential and removed from subsequent rounds to be included in the final list. Items that achieved a consensus rating of “not important” were removed from subsequent rounds and not reviewed further. Items that achieved a consensus of “somewhat important” or items that did not achieve consensus were modified based on expert commentary and re-distributed for ranking in subsequent rounds. Items that were deemed “not observable” by >25% of the experts were removed.

Experts were given the results from each previous round before being asked to re-rank items.¹⁹ We continued with iterative surveys until there was consensus on each item or lack of further discussion or progression for the item despite not achieving consensus (e.g., no consensus and no further clarifying comments or revisions for the item). The primary author mapped items from the final list to one of the 6 ACGME core competencies, taking into consideration the goals and definitions of each competency, with final review by the team.²⁰

Study data were collected and managed using REDCap electronic data capture tools hosted at Weill Cornell Medicine. The Columbia University Medical Center Institutional Review Board deemed this study exempt.

Results

Focused Literature Review

A total of 144 articles were identified and screened. Twenty-five articles identified telemedicine-specific skills and behaviors and were used to compile the initial list of 150 items. After elimination of redundant and “not observable” behaviors by the authors, 25 items remained.

List Refinement

Items were modified based on stakeholder feedback, and a final list of 28 skills and behaviors, defined as “actions that are observable and measurable to someone watching the encounter (e.g., a preceptor), was distributed to the Delphi panel experts. Items were grouped into 3 categories: visit setup, conducting a virtual visit, and clinical telemedicine skills.

Modified Delphi Panel

Eleven out of 12 (92%) experts participated in the first survey, 12 (100%) in the video conference, 11 (92%) in the second survey, and 9 (75%) in the third.

No edits were made after the initial survey; these results were used as a starting point for discussion during the video conference (Figure ). The original 28 items were again ranked after the video conference; 14 achieved a consensus of “very important” and were removed from subsequent rounds for inclusion in the final list. One item achieved a consensus of “not important” and was removed; 2 items achieved a consensus of “somewhat important” and were removed based on expert commentary during the video conference. Eleven items did not achieve consensus, underwent significant modifications based on expert feedback, and were revised into 6 new items that were included in the final survey round. After the last survey, three items reached consensus and 3 items did not. No behaviors were deemed “not observable” by more than 25% of the group.

Figure.

Delphi panel flowchart for participants and items.

A total of 17 items achieved consensus of “very important” for pediatric resident use of telemedicine (Table 1 ). The final list comprised 6 items related to visit setup, 6 related to conducting a virtual visit, and 5 related to clinical telemedicine skills. Each item was mapped by the study team to one of the 6 core competencies for graduate medical education as follows: interpersonal and communication skills (ICS), 7; professionalism (PROF), 5; patient care, 3; systems-based practice (SBP), 2; medical knowledge (MK), 0; practice-based learning and improvement (PBLI), 0.

Table 1.

List of Essential Telemedicine Skills and Behaviors, as Identified by Delph Panelists

Skill or Behavior	ACGME Core Competency	Survey 1, % Consensus (Score)	Survey 2 % Consensus (Score)	Survey 3 % Consensus (Score)
Visit setup
1. Confirms the patient is in the most optimal environment available to them and that limitations of that environment are known and mitigated as best possible.	Professionalism	91 (2)	100 (2)	n/a*
2. Establishes rapport at the start of the visit. -Introduces self, explains if attending will join -Briefly asks if the patient/caregiver has questions regarding telemedicine before starting the visit	Professionalism	100 (2)	100 (2)	n/a
3. Ensures audio/visual connectivity. -Asks the patient/caregiver if he can hear and see the physician	Professionalism	100 (2)	91 (2)	n/a
4. Identifies the need for an interpreter and demonstrates the ability to connect to an interpreter, if necessary.	Interpersonal and communication skills	82 (2)	82 (2)	n/a
5. Identifies and engages everyone present on the visit, as appropriate (patient, caregiver, home health care worker, team members).	Interpersonal and communication skills	73 (2)	91 (2)	n/a
6. Optimizes physician presence on video, if possible. -Remains centered on screen -Creates “eye contact” by looking into camera lens -Optimizes lighting: light source behind screen/camera, reduces shadows on the face -Avoids windows, doors, bright colors, stripes, or bold patterns -Wears professional attire, including a white coat and ID badge, if appropriate	Professionalism	n/a	n/a	89 (2)
Conducting a virtual visit
7. Manages virtual communication issues, if necessary. -Uses a telephone to verbally communicate if the computer audio is not working well -Uses the platform's chat box function when necessary	Interpersonal and communication skills	73 (2)	82 (2)	n/a
8. Troubleshoots minor technology related issues. - Troubleshoots situations including potential patient/caregiver related difficulties: logging into platform, poor lighting or camera angle, difficulty using camera or microphone, needing to change browsers, in putting information into chat or EMR	Systems-based practice	64 (7)	82 (2)	n/a
9. Demonstrates empathy through a virtual space. -Acknowledges frustrations when patient/caregiver has technology problems, without blaming technology -Narrates behaviors to minimize misinterpretation -Verbalizes understanding instead of using nonverbal cues (eg, nodding) -Makes eye contact by looking directly into camera -Remains aware of any lag in conversation due to technology -Makes necessary movements or gestures in frame of camera	Patient care	73 (8)	100 (2)	n/a
10. Demonstrates awareness of privacy and confidentiality concerns unique to telemedicine, especially for adolescent patients. -Requests adolescent patients review who is in the room before discussing private matters -Asks permission from adolescent patients to have additional team members present on the visit -Asks permission from caregivers to have additional team members present on the visit	Professionalism	n/a	64 (2)	100 (2)
11. Demonstrates awareness of pediatric and age-specific challenges and benefits of telemedicine. -Encourages caregiver to place the child in the frame of the camera, including moving or walking around with the camera to capture the child -Attempts to interact with younger children despite being on camera -Attempts to speak to adolescent patients privately at some point during the visit, if necessary	Interpersonal and communication skills	100 (2)	91 (2)	n/a
12. Demonstrates awareness of legal regulations specific to telemedicine, including data security, state license limitations, and jurisdiction limitations.	Systems-based practice	n/a	n/a	89 (2)
Clinical telemedicine skills
13. Guides patient/caregiver through self-administered physical exam, as appropriate. -Attempts to have patient/caregiver obtain vital signs -Walks patient/caregiver through basic exam steps, encourages a description of his/her findings, and asks clear, targeted follow up questions -Works with the patient/caregiver to obtain information when necessary (eg, adjust/zoom the camera to visualize area of complaint, walks patient through using equipment they have at home)	Patient care	64 (2)	82 (2)	n/a
14. Determines the urgency and appropriateness of telemedicine visit and manages the situation appropriately. - Potential reason to switch to in-person visits or refer to a higher level of care, including medical emergencies, complex or sensitive physical exam needs, inability to acquire complete or accurate information to make a medical decision, mental health emergencies -Prepared for emergencies (eg, able to call 911, provides patient location, refers to closest emergency department)	Patient care	73 (2)	100 (2)	n/a
15. Communicates an appropriate and mutually acceptable treatment plan. -Provides informational resources despite not being in person such as referral to websites or virtual handouts - Incorporates patient location into the plan of care (rural location, travel distance to nearest in-person provider, access to healthy nutrition options, etc.)	Interpersonal and communication skills	82 (2)	100 (2)	n/a
16. Provides a clear follow-up plan. -Gives a specific window and modality for follow up -Provides clear instructions for how follow up will be scheduled -Provides follow up contact information for the patient/caregiver -Provides guidance on what to do if no improvement as anticipated, how to recognize deterioration (describes warning signs and next steps)	Interpersonal and communication skills	100 (2)	100 (2)	n/a
17. Ends the visit by asking for questions or concerns.	Interpersonal and communication skills	92 (2)	100 (2)	n/a
Items that did not reach consensus or were otherwise modified for redistribution.
Ensures patient identity (name and date of birth) has been confirmed.		55 (1)	73 (1)	R†
Sets expectations for a pediatric telemedicine encounter.		36 (1,2)	45 (1,2)	R
Obtains information from multiple sources: observation of home environment, collateral sources, and chart review, as appropriate.		55 (2)	55 (1)	R
Allows the patient to disconnect first.		55 (0)	82 (0)	R
Demonstrates awareness of medico-legal issues, privacy, confidentiality, safety in special circumstances.		72 (2)	91 (2)	M‡
Demonstrates awareness of and sensitivity to local cultures, resources, and views of technology.		45 (1, 2)	55 (1)	R
Maintains a professional and engaged virtual presence.		91 (2)	100 (2)	M
Uses the telemedicine platform to review data with patient/caregiver.		82 (1)	91 (1)	R
Identifies the patient's location for billing and emergency purposes.		64 (2)	45 (1,2)	M
Obtains consent from caregivers and/or adolescent patients.		73 (2)	65 (2)	M
Optimizes lighting setup.		55 (1)	55 (1)	M
Optimizes camera position.		55 (1)	55 (1)	M
Uses a neutral background.		82 (1)	62 (1)	M
Narrates behavior to minimize misinterpretation.		64 (1)	55 (1)	M

^⁎n/a = item was not ranked during this round; it was either removed in a previous round or added in a future round after it was revised.

^†R = removed after previous rounds due to lack of consensus.

^‡M = Modified for redistribution.

Of the items that were removed or heavily modified, the most common reasons for not ranking them as “very important” were a perceived lack of necessity for pediatric trainees, no supporting evidence in the literature, personal experiences with telemedicine, and no apparent benefit to the patient or resident.

Discussion

Using a modified Delphi methodology, we generated national expert consensus on a list of 17 essential telemedicine skills and behaviors for pediatric residents.

Each item could be mapped to one or more of the pediatric core competencies, the foundation of our current training system for GME; this ensures that these specific telemedicine skills and behaviors can be incorporated into our competency-based training programs.²¹

Many of the skills and behaviors identified through our Delphi process overlapped with Hart et al's¹² telemedicine curriculum competencies for GME. Our results identified relatively more items in the domains of ICS and PROF. This suggests the need for greater emphasis on these skills for pediatric residents, who may face more challenges in these domains when caring for pediatric patients through telemedicine. For example, in pediatric visits there is often a unique triad of patient/caregiver/clinician, where there is a need to balance interests and concerns of both the parent and the child, as well as rely on parental assistance for key elements of the history or physical exam. When compared to the pediatric telemedicine EPAs, released after the completion of our study, the results from our Delphi panel had considerable overlap, adding evidence for the validity of our methodology and results.

Three recurrent concepts emerged regarding telemedicine expectations for pediatric residents. First, there are practices unique to pediatric telemedicine. Experts identified unique aspects of caring for pediatric patients, specifically an emphasis on ICS with patients, parents, and caregivers, highlighting a need for frequent reliance on caregivers for aspects of the visit (e.g., physical exam) and ensuring privacy for adolescents. Second, variability exists in telemedicine workflow and platforms. During the in-person video conference portion of the Delphi panel, experts identified the variability across telemedicine platforms and workflows as a challenge for standardizing certain expectations for trainees. For example, some institutions have residents perform the intake and establish patient identification prior to a visit, while other institutions have administration and nursing do this before the physician begins the encounter. Third, many telemedicine practices require the physician to “make the implicit explicit.”²² Trainees must recognize how certain clinical practices, often routine or implied during an in-person encounter, must be made more intentional during a telemedicine encounter, such as ensuring audio/visual connectivity and expressing empathy through a virtual space. These emerging ideas may be further examined with a formal thematic analysis in future studies.

As pediatric telemedicine competencies are identified, we must develop methods to assess trainee competency that are based on the results of our study, as well as other frameworks.²³ Assessment is a crucial step in the educational process; it conveys what educators value as important, motivates learning and, ultimately, is necessary for evaluation of competency.^{16 , 24} Development of a validated telemedicine assessment tool is a critical step in ensuring trainees obtain the proper skills to provide excellent patient care. This list of skills and behaviors may serve as the basis for development of an assessment tool for telemedicine in pediatrics.

Our study has several limitations. The modified Delphi method is a consensus technique designed to be used when empirical evidence is unavailable, lending itself to biases despite the panelists’ expertise. All revisions and modifications were based strictly on feedback from experts, and there was no limit on the number of comments or edits an individual could contribute. We mitigated these biases by including a variety of clinical, educational, and research expertise on the panel, but could be further addressed by increasing the size of the panel or variety of expertise. Additionally, the third and final round of surveying only had a 75% response rate, compared to 92% from the first two rounds. Finally, the results from this study have yet to be observed in a simulated or real-time work environment.

Conclusions

A list of 17 essential telemedicine skills and behaviors for pediatric residents was created through a modified Delphi method. This list highlights the unique aspects of pediatric telemedicine and may serve as the basis for development of competency-based curricula and assessment of pediatric resident use of telemedicine.