Perspectives on Conducting Behavioral Intervention With Adolescents Using a Virtual Platform—A Thematic Analysis From the Viewpoint of Study Coordinators

ABSTRACT

Background

The COVID‐19 pandemic led to widespread adoption of virtual communication platforms. Virtual study visits were implemented in the pilot cluster randomized trial (CRT) stage of Teen Adherence in KidnEy transplant Improving Tracking To Optimize Outcomes (TAKE‐IT TOO). The present study aimed to understand study coordinators' perspectives on conducting a behavioral intervention with adolescent kidney transplant recipients using virtual conferencing platforms.

Methods

Study coordinator participants (N = 6) completed questionnaires and participated in a semi‐structured interview that probed comfort with digital technology, issues encountered, and overall perspectives on conducting virtual study visits. Qualitative thematic analysis was used to identify themes and subthemes.

Results

Participants expressed confidence with technology and ability to handle the complexities of the virtual conferencing. Some expressed that virtual study visits led to a change in work habits and higher workload due to increased technology complexity. Qualitative analyses of participant interviews revealed four themes: adaptability, accessibility, logistics (including subthemes scheduling and fluidity), and communication (including subthemes clarity, engagement, and rapport). Convenience for coordinators and the perceived comfort for adolescents were noted advantages for virtual visits. Technical issues, periodic adolescent distractions, and challenges with instructional teaching through virtual conferencing were identified as potential limitations of virtual study visits.

Conclusions

Overall, virtual study visits were appreciated and endorsed by study coordinators. Researchers should consider the feasibility of completing study‐related tasks virtually, including accessibility of visual materials on all type of electronic devices, and ensure adequate training of study personnel when deciding to implement virtual platform in CRTs.

Convenience for coordinators and the perceived comfort for adolescents were noted advantages for virtual study visits. Technical issues, periodic adolescent distractions, and challenges with instructional teaching were identified as potential limitations. Researchers should consider the feasibility of completing study‐related tasks and ensure adequate training of personnel when implementing virtual visits.

Abbreviations

CRT

cluster randomized trial

TAKE‐IT TOO

Teen Adherence in Kidney Transplant Improving Tracking to Optimize Outcomes

1. Introduction

The rapid adoption and integration of virtual clinical visits during the COVID‐19 pandemic was critical in allowing for continuous health care access despite rigid public health and governmental mandates. There is now a substantial body of literature on the advantages and disadvantages of virtual platforms for clinical care [1, 2, 3, 4]. Notable advantages include convenience and ease of access for patients [1, 2, 3, 4]. In parallel to the virtual shift observed in clinical care, remote processes were simultaneously integrated in clinical research. While COVID‐19 accelerated the use of virtual platforms for study visits, the perspectives of study personnel on this approach have not been described [5].

Like many clinical research studies, the pilot cluster randomized trial (CRT) stage of the Teen Adherence in KidnEy transplant Improving Tracking To Optimize Outcomes (TAKE‐IT TOO) study was forced to convert study visits originally planned to occur in person to a virtual platform. TAKE‐IT TOO aimed to pilot a novel electronic‐monitoring pillbox (e‐pillbox), designed specifically for adolescent kidney transplant recipients (12–21 years), together with an adherence‐promoting behavioral intervention, in a 7‐site CRT conducted in Canada and the United States. Adolescents in centers randomized to the adherence‐promoting intervention worked with a study coordinator to identify their personal barriers to medication adherence and develop concrete action plans to address these barriers. Adolescents in centers randomized to the control condition—a healthy living intervention—worked with the study coordinator to develop action plans to meet their healthy living goals related to a topic of their choice (sleep, social media use, or stress management). Adherence to immunosuppressive medications was monitored electronically.

The study was designed to include three in‐person study visits (at baseline, 4 weeks, and 14 weeks), and between‐visit check‐ins by text message (at 1 week) and by phone or video (at 6 weeks and 10 weeks). At the baseline visit, adolescents met with the study coordinator to complete a series of questionnaires and to learn how to set up and use the e‐pillbox. The intervention started at the 4‐week visit and included completion of questionnaires, review of the electronic adherence data, and action‐focused problem‐solving to address adherence barriers identified previously [6]. At the 14‐week exit visit, adolescents also completed a questionnaire on their experience with the e‐pillbox. Due to pandemic restrictions, we opted to adjust the visits at baseline, 4 weeks, and 14 weeks to occur remotely, though adolescents were permitted to complete visits in‐person if desired.

The aim of the present mixed methods study was to understand the benefits and limitations of virtual study visits with adolescents from the perspective of the study coordinators conducting the visits. Our findings may provide helpful insights for researchers considering the use of virtual platforms with adolescents in future clinical research studies. Comprehensive reporting of important study details adhere to COREQ guidelines [7].

2. Methods

2.1. Study Design, Setting, and Participants

We conducted a mixed methods study in which study coordinators completed a single one‐on‐one interview via Zoom conducted by a member of the clinical coordinating center's study team (McGill University Health Centre) and responded to two questionnaires. The interviews were conducted after all data collection for the trial had concluded. The interviewer (AG; male) served as the study coordinator for the main site, project manager for TAKE‐IT TOO, and had prior rapport with eligible participants.

All study coordinators from six TAKE‐IT TOO sites (CHU Ste‐Justine, British Columbia Children's Hospital, University of Toronto Hospital for Sick Children, St. Louis Children's Hospital, Seattle Children's Hospital and University of Pittsburgh Medical Center) were eligible. Study coordinators were the participants of the present study so are referred to as participants. Participants were invited by email to participate after the last adolescent at their site had exited the study. Participation was voluntary, and compensation was provided ($50). Participants provided written informed consent. The study was approved by the Research Ethics Board of the McGill University Health Centre, and interviews were completed in February and March 2023.

2.2. Measures

Participants provided basic socio‐demographic and professional experience information and completed two self‐report questionnaires. First, the 12‐item Computer Self‐Efficacy measure was used to assess participant level of comfort using computer technology prior to the TAKE‐IT TOO study [8]. Second, techno‐overload and techno‐complexity in remote working were assessed using items from the adapted technostress scale to assess digital platform‐induced stress in a remote working context [9]. Several constructs from the technostress scale were not relevant to participant experience conducting study visits remotely and were therefore excluded from our questionnaire. All items were measured using a seven‐point Likert scale ranging from strongly disagree to strongly agree.

2.3. Interviews

We designed semi‐structured interview questions to guide the interviews (see Supporting Information). Participants did not see the interview guide prior to participation. Topics included (1) familiarity and comfort with virtual platforms, (2) issues related to scheduling visits, (3) issues encountered during virtual visits (i.e., rapport‐building, adolescent engagement, and technical problems), and (4) overall perspectives on virtual visits (strengths and limitations). A 30‐ to 60‐min interview was conducted individually with each participant; no repeat interviews were carried out. Interviews were audio recorded; transcripts were not returned to participants for comment or correction. Participants did not provide feedback on the findings.

2.4. Analysis

All analyses were descriptive. Interview recordings were transcribed using Otter.ai, a speech to text transcription application. Interview transcripts were edited for clarity, punctuation, and grammatical errors before transfer into NVivo12 software for analysis. We applied qualitative content analysis techniques as outlined by Braun and Clark [10]. Analysis began by familiarization with the data. Initial coding was inductive and iterative until descriptive redundancy was reached by one coder. The penultimate set of codes was provided to a second research team member who condensed the set of codes independently. The final set of codes was established upon consensus between the two team members and subsequently analyzed for higher level categories. Categories were presented to the principal investigator and reviewed before agreement on the final set of themes and subthemes.

3. Results

3.1. Participant Characteristics

Six of seven eligible study coordinators participated. One declined participation, concerned that too much time had passed between their work on TAKE‐IT TOO and the interview. All participants were female and 50% were < 30 years old; one third were ≥ 60 years old. Five of the participants (83.3%) identified their race as White, and one participant identified their race as Black; none of the participants identified as Hispanic or Latino. Two thirds had undergraduate degrees, and the other third had completed graduate school. There was a wide range of experience with clinical research (range 1–17 years); two thirds had < 5 years of experience in research. The majority (n = 5) had no previous experience with virtual study visits prior to the pandemic and even the participant with some experience estimated that < 2% of their prior study visits were virtual. However, experience with virtual visits since the start of the pandemic was variable (5%–90% visits) but half indicated that 20% or fewer of all the study visits they conducted were virtual. Four of the six participants (66.6%) completed the interview 55 to 80 days after their final TAKE‐IT TOO study visit. The two remaining participants completed the interview 35 and 190 days after their final study visit.

As shown in Table 1, participants expressed a fair degree of confidence with interacting with computers prior to the TAKE‐IT TOO study based on responses to the 12‐item Computer Self‐Efficacy measure. In general, participants felt they could handle the complexity of the technology, and none felt that they had to work faster, adhere to tighter timelines, or that they had more work than they could handle (Table 2). When asked to think back to their experience working remotely and completing virtual study visits during TAKE‐IT TOO, half of the participants felt that they had to change their work habits, while two felt that they may have had a larger workload than when not working remotely.

TABLE 1.

Participant responses to Computer Self‐Efficacy Measure.

Items	Strongly disagree	Disagree	Slightly disagree	Neither disagree or agree	Slightly agree	Agree	Strongly agree
1. I can always manage to solve difficult computer problems if I try hard enough.	0 (0%)	1 (16.7%)	0 (0%)	0 (0%)	1 (16.7%)	3 (50.0%)	1 (16.7%)
2. If my computer is “acting‐up,” I can find a way to get what I want.	0 (0%)	1 (16.7%)	0 (0%)	0 (0%)	2 (33.3%)	3 (50.0%)	0 (0%)
3. It is easy for me to accomplish my computer goals.	0 (0%)	0 (0%)	1 (16.7%)	0 (0%)	0 (0%)	4 (66.7%)	1 (16.7%)
4. I am confident that I could deal efficiently with unexpected computer events.	0 (0%)	1 (16.7%)	0 (0%)	1 (16.7%)	1 (16.7%)	3 (50.0%)	0 (0%)
5. I can figure out most computer applications if I invest the necessary effort.	0 (0%)	1 (16.7%)	0 (0%)	0 (0%)	2 (33.3%)	2 (33.3%)	1 (16.7%)
6. I can remain calm when facing computer difficulties because I can rely on my abilities.	0 (0%)	0 (0%)	1 (16.7%)	0 (0%)	3 (50.0%)	2 (33.3%)	0 (0%)
7. When I am confronted with a computer problem, I can usually find several solutions.	0 (0%)	1 (16.7%)	0 (0%)	0 (0%)	3 (50.0%)	2 (33.3%)	0 (0%)
8. I can usually handle whatever computer problem comes my way.	0 (0%)	1 (16.7%)	0 (0%)	1 (16.7%)	2 (33.3%)	2 (33.3%)	0 (0%)
9. Failing to do something on the computer makes me try harder.	0 (0%)	1 (16.7%)	1 (16.7%)	1 (16.7%)	0 (0%)	2 (33.3%)	1 (16.7%)
10. I am a self‐reliant person when it comes to doing things on a computer.	0 (0%)	1 (16.7%)	0 (0%)	0 (0%)	1 (16.7%)	3 (50.0%)	1 (16.7%)
11. There are few things that I cannot do on a computer.	0 (0%)	0 (0%)	2 (33.3%)	0 (0%)	2 (33.3%)	1 (16.7%)	1 (16.7%)
12. I can persist and complete most any computer‐related task.	0 (0%)	0 (0%)	1 (16.7%)	0 (0%)	2 (33.3%)	2 (33.3%)	1 (16.7%)

Note: Participants were asked to think back to before they began Stage 3 of TAKE‐IT TOO when answering the Computer Self‐Efficacy Measure.

TABLE 2.

Participant responses to remote work and virtual visit experience questionnaire.

Items	Strongly disagree	Disagree	Slightly disagree	Neither disagree or agree	Slightly agree	Agree	Strongly agree
1. I did not have enough knowledge about the new technologies to handle my job satisfactorily.	2 (33.3%)	3 (50.0%)	1 (16.7%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
2. I did not find enough time to study and upgrade my technology skills.	2 (33.3%)	4 (66.7%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
3. I often found it too complex for me to understand and use new technologies.	4 (66.7%)	2 (33.3%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
4. I was forced to work much faster with the new technology.	3 (50.0%)	3 (50.0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
5. The technology I used for work forced me to do more work than I can handle.	4 (66.7%)	2 (33.3%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
6. The technology I used for work forced me to work with very tight time schedules.	2 (33.3%)	4 (66.7%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
7. I was forced to change my work habits to adapt to new technologies.	0 (0%)	3 (50.0%)	0 (0%)	0 (0%)	1 (16.7%)	1 (16.7%)	1 (16.7%)
8. I had a higher workload because of increased technology complexity.	1 (16.7%)	3 (50.0%)	0 (0%)	0 (0%)	2 (33.3%)	0 (0%)	0 (0%)

3.2. Virtual Study Visits

The majority of baseline TAKE‐IT TOO visits were conducted using online videoconferencing, but modality (in‐person, phone, virtual) for the baseline visit was not captured in the database. Modality for the other visits was recorded. Of the seventy 4‐ and 14‐week visits originally planned to be in‐person, 58 were completed virtually. Three participants each conducted 5 to 7 visits virtually, two participants each completed 12 visits virtually, and one participant completed 16 visits virtually. There were 64 follow‐up visits (at 6 and 10 weeks); 43 of these were completed virtually.

3.3. Thematic Analysis

Four main themes summarizing participant experience with virtual study visits emerged: (1) adaptability, (2) accessibility, (3) logistics (including subthemes scheduling and fluidity), and (4) communication (including subthemes clarity, engagement, and rapport). Table 3 summarizes the themes, subthemes, and supporting statements.

TABLE 3.

Identified themes, subthemes, and supporting statement examples.

Theme	Subtheme	Supporting statements
Adaptability	N/A	“I feel like we had done quite a bit of Zoom prior to the study that the virtual visits were quite regular.” [Participant #2]
		“Pretty straightforward. It's what we usually do, right? We're used to using Teams or cell phones and all that, so it wasn't an issue for me.” [Participant #4]
		“…since I was working during COVID previously, it wasn't via Zoom. […] I was still in‐person all the time. So I didn't actually know how to use Zoom at all, and it wasn't too hard to get used to.” [Participant #5]
		“I was kind of nervous to meet the patient for the first time through the video call.” [Participant #6]
		“But being the leader, the person with control and sharing the screen and all of that… I was a little nervous about that. And, therefore, I was much more comfortable with in‐person meetings.” [Participant #3]
		“So I say in the beginning, the learning curve was pretty steep in terms of getting the technology on board and being confident in myself that I could manage the technology during the visit.” [Participant #1]
		“And then, after a couple I'm like, “Okay, I know how to do this now.” Same thing with the first few visits.” [Participant #3]
		“And then as I got used to them, I really enjoyed them, I didn't find them too difficult.” [Participant #1]
		“And so over the course, it obviously became a lot smoother in terms of making sure I knew what I needed.” [Participant #2]
Accessibility	N/A	“Again, it depend[s] on the studies but if a study doesn't need any physical exam or anything, I prefer to do it virtual.” [Participant #6]
		“In doing this study, I preferred the virtual visits.” [Participant #1]
		“Easy part is definitely the scheduling. There's so much flexibility in the scheduling like you can take half an hour of your day, evening or weekend from home as opposed to having to go in.” [Participant #1]
		“I think for the age group that we're studying, they're very comfortable using video conferencing.” [Participant #1]
		“I liked it because, especially since this study was about teenagers, I felt like it was smart that we did it through using technology just because I think that makes sense to them.” [Participant #5]
		“But the preference I think for my patients was by Zoom. It just worked into their lifestyle and schedule and what they're used to, quite frankly.” [Participant #3]
		“But as a whole I do think it made it more doable for people.” [Participant #5]
		“And as I'm talking, I realized like they were at home, they were… they didn't have to travel for this, pay for parking and do all that type of thing.” [Participant #2]
		“I'm sure the parents would not have agreed to sign up because they would have wanted to bring their kids back and forth, if we hadn't had the option of the virtual visits.” [Participant #1]
		“I don't know how many kids go back to school after their clinic visits anyways because it depends on how close they live and stuff. But I think, you're definitely taking out of their regular day.” [Participant #2]
		“…when they're coming in somewhere, they're much more closed off, but I felt like virtually, once we got over that initial weirdness of being virtual, then I felt like there was more of a comfort level, just with them being at home…” [Participant #5]
		“…sometimes I do the visit in‐person where [they're] in the hospital bed. Or after their biopsy or something. It wasn't the same as when they are doing really well in their house.” [Participant #6]
Logistics	Scheduling	“I personally thought the scheduling, the logistics of actually emailing the family, finding out a time, setting up the Zoom visit, having the link there… once I figured out like booking a room for me to do it with a laptop, that part of it became really easy and comfortable and great.” [Participant #2]
		“…it wasn't an issue for me because I always text them and remind them and sometimes they're just like, ‘oh, sorry, we didn't get home yet’, or ‘sorry, we had something coming up’. I mean, I was fine with that.” [Participant #4]
		“I mean, I didn't love it. I'm kind of a person that protects my evenings and I don't know, I just feel like the that's a boundary that I usually have. But it was… it didn't happen that often. So it was okay.” [Participant #5]
		“So I was accommodating, but the participants as well were accommodating. So it kind of helped.” [Participant #4]
		“That was difficult… mostly because their schedules were so just jam packed. And some of that was the kids [having] so many scheduled things after school.” [Participant #5]
		“Certainly I've tried to remind them about their visit, send them an email ahead of time to remind them.” [Participant #1]
	Fluidity	“I'd say I did a lot of prep before those first couple of visits just to make sure that I knew what I was doing and could make it make it flow okay.” [Participant #1]
		“I did not do everything 100% online. I still printed out a paper form. And then… I'd say for the majority of the visits, I did that. And then I would go back in and enter the data.” [Participant #3]
		“No, it seemed easy because like as I said, I had one [e‐pillbox] on my end and the camera and [I'd] say ‘go to this button’ and they [did exactly] what I'm doing. I [didn't] have any difficulty with this.” [Participant #6]
		“I did not find it difficult at all. I found it really easy, honestly. Yeah, show them the portal by Zoom, just shared my screen and click, click, click and they got it.” [Participant #3]
		“I don't think any of the participants, none of the participants had any issues seeing the questions and answering so that was great.” [Participant #2]
		“I think some things probably would have been easier in‐person. Like explaining certain, actual physical things with the pillbox definitely would have been easier.” [Participant #5]
		“I had a couple of participants that could not come up with an idea for an action plan on their own. So then like giving them a few options, it would have been nice for me to visually, like write that out for a little bit. And I know there's like a way to do that in Zoom, but I'm not super savvy or that would have complicated things for me.” [Participant #5]
		“…maybe some of the more difficult teaching pieces like the box may have been easier to accomplish in‐person but I think it was doable online.” [Participant #1]
		“There were a couple of times I had an issue, connecting with the patient. So, with the Zoom link, I couldn't sometimes get onto the hospital network that I needed to get on to.” [Participant #1]
		“I think she was on her cell phone, it was difficult to screen share and then the Internet was lagging a bit. So the screen sharing part for her didn't work.” [Participant #4]
		“So you always try to find a solution. And if you cannot screenshare, I read the questions. So, you find solutions.” [Participant #4]
		“I'm probably sure that happened, and to the point where I had to log out and log back in, something like that, but it didn't cause me to just cancel the whole meeting.” [Participant #3]
Communication	Clarity	“And I believe that the fact that we're talking by Teams, the message is still the same. The words are still the same; the definitions are still the same.” [Participant #4]
		“I don't think it was an issue in communicating from myself. I would be curious if someone said that they had a harder time communicating through Zoom.” [Participant #2]
Communication
	Engagement	“The other patients, most of them, they were really nice and [gave] me all their time and we were able to do everything fine.” [Participant #6]
		“For the most part, patients and parents were really engaged throughout the whole study. And through all the visits.” [Participant #2]
		“As I said, it varies from patient to patient. So the most difficult [is] to get patient focusing, but it depends on the patient.” [Participant #6]
		“It's interesting; some of the young ones were totally focused and engaged.” [Participant #2]
		“I had a few participants that were very interested in the study and knew all the things that have gone on with their pillbox in the last two weeks themselves. And then I had other participants that were kind of giving me the one word answer.” [Participant #5]
		“…maybe for one participant you can feel that she wanted the study to be over, maybe by the end…” [Participant #4]
		“…with that participant, it was more difficult. And I definitely felt like distraction was an issue, among not actually being able to hear me.” [Participant #5]
		“At the very last visit, I could tell that they were trying to do homework on their laptop at the same time. And I didn't know how to just call them out on it. I just gave more patience in terms of re‐asking the question if needed, if I could tell they weren't focused when I asked the question.” [Participant #2]
		“So sometimes at home, mom is screaming something, [that kind] of stuff. And then she's answering another text in between.” [Participant #4]
		“So, it's more of getting their attention back, right? So sometimes by saying “okay, I don't have as [many] questions. There [are a few] questions until the end of [this] questionnaire” or “it's the same questionnaire that you already answered, so you're used to it,” just to [get] their attention back.” [Participant #4]
	Rapport	“But I think on video conferencing, there's still a connection with a person so as long as you're on video, I don't think it's a problem.” [Participant #1]
		“I don't know that I feel like there's a big gap between building rapport with the in‐person versus a virtual.” [Participant #3]
		“I think you can still do that virtually and you actually get a peek into their lives and not just what they present as at clinic which is kind of fun.” [Participant #2]
		[with in‐person visits] “We'd walk upstairs. We're chatting along the way. So in some ways, there's maybe more time for chit chat and things.” [Participant #3]
		“I think the physical presence of people is different than just a screen. Even though people are very used to doing virtual and can still connect while and still engage, I think there's something about being physically present with people.” [Participant #2]
“And, in those moments, I wish I could have been in‐person because then I could have had more body language of caring about that instead of just sitting here staring at them.” [Participant #5]

3.3.1. Theme 1: Adaptability

Participants described adaptability in using virtual conferencing, overcoming anxieties, and managing technical difficulties. Most participants reported having prior experience with virtual conferencing platforms and were familiar with videoconferencing functionalities, such as joining a video call or sharing their screen. One participant did not have prior experience with virtual conferencing platforms.

Two participants noted feeling anxious prior to their first virtual baseline visit, specifically related to using virtual conferencing technology. However, both were able to manage technical issues that arose, such as screen sharing difficulties, and described feeling more comfortable with virtual visits after completing the first few and eventually enjoying them.

And then as I got used to them, I really enjoyed them, I didn't find them too difficult. (Participant #1)

3.3.2. Theme 2: Accessibility

Virtual visits were viewed positively by most participants and four acknowledged a preference for conducting study visits virtually. Five expressed that they enjoyed the flexibility of scheduling virtual study visits, which allowed them to work in their own environment and fit study visits conveniently in their day. Participants appreciated the ability to work from home instead of going into the office.

Virtual conferencing was perceived to increase adolescent accessibility to study participation. Participants noted that most adolescents were familiar with virtual platforms, particularly following the shift toward remote learning during the pandemic. It was felt that the time and expense related to traveling to the clinic could discourage study participation, especially for those living long distances from hospital centers. Participants were of the unanimous opinion that virtual study visits worked well for adolescents. Terms used to describe the impact of virtual study visits included: “beneficial,” “smart to do,” and “critical.” Many highlighted the convenience offered by virtual conferencing, which allowed adolescents to schedule study visits after school or work.

For adolescents, from the moment the [adolescent] is in high school, I think it's really smart. You can do that after school when they have [time] off; they just login and then, you do what you have to do. (Participant #4)

Five participants expressed that adolescents may feel more comfortable doing study visits in their home environment. Some noted that while adolescents may be closed‐off or less talkative during an in‐person study visit, these challenges are mitigated by “being in the comfort of their own home.”

I actually think maybe they felt more comfortable too because they were in their home.

(Participant #2)

Overall, the accessibility of virtual study visits was viewed to be advantageous for adolescents. Two participants even perceived that virtual study visits may have increased enrollment.

3.3.3. Theme 3: Logistics

Comments related to the logistics of scheduling and conducting study visits virtually included two subthemes: (1) Scheduling and (2) Fluidity.

3.3.3.1. Scheduling

Participants reported few challenges with scheduling virtual study visits. Virtual study visits were scheduled via e‐mail or text messages with adolescents or their family members. Most participants agreed that adolescents had more availability when scheduling a virtual study visit than they did when scheduling in‐person visits and this led to easier scheduling. One participant felt that adolescents had busy schedules and reported more challenges with scheduling adolescents.

I find that scheduling the virtual visits was much easier than [when] I had to schedule in‐person visits. Her appointments changed, and there was much less flexibility with the in‐person appointments because they're spread out so far. (Participant #1)

That was difficult… I mean their schedules were so just jam packed. And some of that was [because] the kids had so many scheduled things after school. (Participant #5)

All participants indicated flexibility in their own schedules to accommodate adolescents' schedules. Most virtual study visits occurred in the afternoons, evenings, and occasionally on weekends to suit the availabilities of adolescents. However, three participants set firm boundaries on their schedules, as they preferred not to host virtual study visits in the late evenings or weekends.

It was reported that adolescents would show up late or miss virtual study visits entirely. The number of late or missed virtual study visits varied, ranging from “fewer than 5 times” to “10% of the time” and up to “40% of the time.” Participants were able to manage when adolescents were late to virtual study visits by following up by e‐mail or text message, but some expressed frustration with rescheduling.

I think the biggest struggle with scheduling was people just not showing up and also not telling me that they weren't going to show up. (Participant #5)

Conversely, other participants experienced few instances of late or missed virtual study visits and were generally more tolerant. Reminder emails or texts were noted by four participants to be an important element of scheduling virtual visits; two participants acknowledged they could have incorporated reminders better to avoid missed or late visits.

3.3.3.2. Fluidity

Participants discussed the process of preparing for a virtual study visit to ensure that it went smoothly. Some preferred to print out scripts, questionnaires, and other study documents to have on hand during virtual study visits. Others arranged relevant documents and webpages on their computers ahead of the visit.

Participants discussed the fluidity of using virtual conferencing for the following elements of the study: e‐pillbox and web‐portal teaching, action plan formation, completing questionnaires and troubleshooting Internet and virtual conferencing issues.

All participants felt that e‐pillbox and web‐portal teaching using virtual conferencing was manageable. Most noted having a demo e‐pillbox with them was helpful in guiding adolescents in set‐up and instruction of e‐pillbox functionalities. This allowed participants to demonstrate specific actions using their demo e‐pillbox so that adolescents could mirror the actions on their own e‐pillbox.

“So when I put the pillbox down, you know, I'm looking at my screen to see what they're showing me. So you can tell them “okay, no, it's not the right button” or “oh, yes, that's good.”” (Participant #4)

Two participants noted that helping patients set up and manage their e‐pillbox virtually encouraged and facilitated e‐pillbox use. The fact that adolescents were at home with their e‐pillbox and all their medications made it easier to advise them on filling up the e‐pillbox and placing the e‐pillbox in an accessible location in the home. Similarly, one participant noted that video conferencing was practical for troubleshooting an adolescent's e‐pillbox after the baseline visit.

…it was more meeting them where they were at, in their home with their stuff, like their pills, in their organizational system. And I do think that was helpful, because it's one less step that the participant had to do on their own. (Participant #5)

However, most participants felt that teaching adolescents how to use the e‐pillbox might have been easier to do in‐person. As participants reported feeling able to demonstrate and demo effectively, a notable challenge was to ensure adolescents were repeating the actions correctly on their e‐pillbox. One participant felt that teaching through a smaller screen was challenging.

Maybe a little more clarity in complex teaching, especially if you have a visual aid in front of you, so maybe a little more clarity, a little bit more hands on with the box in terms of teaching. (Participant #1)

One participant mentioned demonstrating how to use the web‐portal might have been easier in‐person, but also did not feel that it was particularly difficult through virtual conferencing. Two participants noted some challenges with developing an action plan through virtual conferencing, such as not being able to see what the adolescent was writing down.

Most participants reported ease with screen sharing and completing questionnaires with adolescents. However, screen sharing was noted to be problematic when there were Internet lags or when the adolescent was using a cellphone, as the text was too small for some to read. In these situations, participants read the questionnaires out to the adolescent. Two participants felt that reading out questionnaires to adolescents through videoconferencing was ineffective and time‐consuming.

…like to read all the question[s] and the survey[s], but then I started sharing the screen with them, but most of them were using their phone. So it was really hard for them to read it. So I felt like this is very time consuming. (Participant #6)

The majority of participants reported experiencing and troubleshooting Internet and technical issues during virtual study visits but were able to implement solutions in these instances with relative ease. In some instances, participants were not able to troubleshoot the technical issue; thus, workarounds were implemented. Participants employed the following strategies when facing a technical difficulty: signing out of and back into the videoconferencing meeting, switching to a phone call if video or audio was not functioning, and asking adolescents to relocate to another room.

3.3.4. Theme 4: Communication

The Communication theme was separated into the following subthemes: (i) clarity, (ii) engagement, and (iii) rapport.

3.3.4.1. Clarity

Three participants reported that adolescents understood the concepts presented to them during virtual study visits. Participants felt that the virtual conferencing platform did not influence adolescents' comprehension when they were asking questions, teaching, or action planning.

And I believe that the fact that we're talking, you know, by [virtual conferencing], the message is still the same. The words are still the same; the definitions are still the same. (Participant #4)

Furthermore, four participants mentioned they did not have trouble expressing themselves during virtual study visits. One participant reported that facial expressions and body language help with communication during virtual conferencing.

3.3.4.2. Engagement

Participants reported that adolescent engagement was generally strong throughout virtual study visits. As expected, engagement levels varied depending on the age group and personality of adolescents, and this was not viewed as related to virtual conferencing. Occasionally, adolescents were distracted during virtual visits. Interruptions and background noise were common during virtual visits. In addition, participants recounted visits where adolescents would multi‐task, such as preparing food in the kitchen or completing their homework. Generally, participants noted that distractions were not extreme enough to interfere with completion of the study visit and that adolescents could generally be re‐focused.

So then sometimes, maybe the participant is doing something else while answering some of your questions, so that's one of the main issue[s] but it wasn't complicated, it was okay. (Participant #4)

3.3.4.3. Rapport

All participants felt as if they were able to build rapport virtually during study visits and did not find virtual conferencing to be a major barrier to rapport‐building. Many participants noted they were able to connect with the adolescents and helped them feel comfortable during virtual study visits.

I guess I'm skilled [in] talking to the patient and just making [them] more comfortable and more confident answering the question[s] and getting to know them more via Zoom. (Participant #6)

Two participants noted that virtual study visits gave fewer opportunities to engage in small talk with adolescents and their families. They felt that in‐person study visits allowed for more personal interactions before and after the visit. When asked about what a participant may gain from completing in‐person study visits as opposed to virtual study visits, most participants noted the importance of in‐person contact. Most appreciated the value of in‐person contact in connecting with patients on a personal level. As a result, one participant preferred in‐person visits to virtual as they felt they could better foster patient connection and handle sensitive discussion topics, such as personal struggles, with adolescents.

…it's probably just that face to face, more human in a way. You know, there is an element of perhaps being able to connect quicker perhaps or in a deeper way if you're there personally… (Participant #3)

4. Discussion

Conducting study visits virtually was a new experience for many study teams facing pandemic restrictions. The present study aimed to understand the advantages and disadvantages of conducting virtual study visits with adolescents from the perspective of the research staff of the TAKE‐IT TOO pilot CRT. TAKE‐IT TOO was a complex study: virtual study visits included administering questionnaires, instructing the use of an e‐pillbox and companion medication‐tracking web‐portal, and delivering a behavioral intervention.

Overall, study coordinator participants viewed virtual study visits favorably and endorsed the adoption of virtual study visits in future studies. Participants demonstrated adaptability in navigating uncertainties and overcoming anxieties associated with virtual conferencing for virtual study visits. Additionally, participants reported that communication with adolescents through virtual conferencing was strong despite periodic distractions. Similarly, most felt that using a virtual platform facilitated scheduling study visits with adolescents; virtual visits were viewed as more convenient than in‐person visits, more flexible in terms of timing, and afforded savings in both time and costs for patients and families. Furthermore, participants reported that they were able to accomplish all necessary tasks and activities through virtual conferencing. However, technical and connectivity issues and challenges with complex teaching were identified as potential limitations for virtual study visits. There were also some concerns about engagement and human connection on a virtual platform.

The pandemic resulted in a dramatic increase in the use of virtual platforms for research activities [5, 11]. Previous literature emphasized the increased accessibility afforded by virtual visits citing flexible scheduling, cost savings, and reduced travel times [5, 11]. Institutional infrastructure needs to ensure appropriate privacy and workflow with virtual platforms [11]. In the present study, we identified themes important to researchers considering virtual visits for future clinical research: adaptability, accessibility, logistics, and communication.

4.1. Adaptability

While study personnel expressed some anxiety about shifting to a virtual platform, they displayed adaptability and were able to manage the virtual environment with relative ease. Adolescents were also perceived as eager to adapt to the virtual platform.

4.2. Accessibility

There was general consensus that virtual study visits provide greater accessibility and convenience for participants. Requirement of absence from school or work and traveling long distances to attend in‐person study visits remain significant deterrents to research participation [12, 13]. The flexibility of virtual visits may be attractive to adolescents whose needs and lifestyles do not align with in‐person visits. Some study coordinators felt adolescents may not have participated in the study if traveling to the hospital center was a requirement. Virtual visits have previously been noted to have the potential to increase adolescents' participation in research [12, 14]. Clinical trials in dermatology using virtual visits had higher recruitment and lower drop‐out rates than trials using in‐person visits [15]. Adoption of virtual visits may also allow for ease of access to clinical trials in populations where the barriers for participation are highest, such as in underserved communities and patients living in low‐income areas [11]. Study coordinators also benefitted themselves from the convenience and flexibility of virtual study visits.

Some of the disadvantages of virtual visits noted were frequent missed and late visits as well as frequent scheduling outside of standard working hours—which was viewed by some as an unrealistic expectation. When deciding whether to implement virtual study visits in future clinical studies including adolescents, it will be important to consider how virtual study visits may affect participant accessibility, recruitment, and retention, how study personnel will manage missed and late visits, and the acceptability of flexible scheduling by study staff.

4.3. Logistics

Technical issues with video conferencing systems, which may hamper effectiveness, were commonly reported. Frequent technical problems with virtual technology may lead to loss of motivation for some individuals. In addition, inadequate training of study personnel on virtual platforms may lead to inefficient or abandoned virtual visits. Appropriate training of study personnel is important to reduce potential technical problems, promote motivation, and minimize the burden of technology management. The accessibility of study participants to an adequate Internet connection must also be considered; people living in communities with no or poor Internet connectivity may be excluded from participating in studies using virtual visits. It is also important to ensure that participants joining virtual visits are able to view all study materials properly regardless of the device used, including a small cellphone screen.

The practicality of conducting some activities during virtual visits, such as e‐pillbox and web‐portal instruction, developing action plans, and completing questionnaires were reported to be manageable. However, some felt that instruction and education on the use of a physical device may have been easier in‐person. The complexity of the activities to be undertaken during study visits should be considered when deciding on whether to use virtual conferencing.

4.4. Communication

In the present study, communication during virtual study visits, including clarity of expression and comprehension, were perceived to be of high‐quality. Participants generally felt that they were able to build rapport with adolescents and that the adolescents were engaged during virtually visits. There was also a perception that participants may have felt more comfortable during virtual visits because they were in their home environment. While this enhanced comfort may promote rapport, some concerns were also raised about rapport‐building and adolescent engagement. Depending on the nature of the study, in‐person visits may be more effective in building rapport. For example, for discussion of sensitive topics, it may be difficult to duplicate the intimacy of an in‐person visit on a virtual platform. Some study coordinators noted instances of distracted adolescents simultaneously pursuing other tasks and interruptions by background noise. The level of focus and concentration required by study participants and the skill of study personnel in engaging adolescents must be considered when choosing virtual versus in‐person study visits. Overall, researchers should consider how the virtual environment will influence communication between participants and study personnel.

4.5. Limitations

While this is the first study to our knowledge to enhance understanding of the benefits and limitations of virtual study visits with adolescents from the perspective of the study coordinators, it does have limitations. The sample size was small because only study coordinators in TAKE‐IT TOO could participate. The generalizability of the findings may be limited by the homogeneity participants, who were all female, predominantly White, and who all had college education. Furthermore, participants had prior rapport with the interviewer, which may have introduced social desirability bias. Participant experience with virtual study visits will also depend on the nature of the study procedures; experience in TAKE‐IT TOO may differ from that in other studies. Finally, accuracy or completeness of participant recollections of study visits may have been influenced by the time elapsed between the end of TAKE‐IT TOO and the semi‐structured interview.

5. Conclusion

Since the onset of the COVID‐19 pandemic, virtual visits for clinical care increased exponentially. Similarly, virtual visits have been embraced for many clinical research studies. Study coordinators for the TAKE‐IT TOO CRT identified both advantages and potential disadvantages to virtual study visits with adolescent participants. Greater accessibility and flexibility in scheduling virtual than in‐person visits and lower cost and time burden were noted advantages to virtual visits. These advantages may help improve recruitment and retention of adolescents in research studies. Limited accessibility to people living in areas with poor Internet coverage, logistical challenges, and potential barriers to rapport‐building and participant focus were noted as possible disadvantages. Advantages and disadvantages must be weighed in the context of the study population, specific objectives, and activities of each study in which virtual visits are considered. Researchers employing virtual visits must be careful to design their studies to optimize functionality on a virtual platform and ensure appropriate training of study personnel.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Appendix S1.

Data Availability Statement

The authors have nothing to report.