Abstract
We conducted a pair of studies to test the validity, reliability, feasibility, and acceptability of using video chat technology as a novel method to quantify dietary and pill-taking (i.e., supplement and medication) adherence. In the first study, we investigated whether video chat technology can accurately quantify adherence to dietary and pill-taking interventions. Mock study participants ate food items and swallowed pills while performing randomized scripted “cheating” behaviors design to mimic non-adherence. Monitoring was conducted in a crossover design, with two monitors watching in-person and two watching remotely by Skype on a smartphone. For the second study, a 22-question online survey was sent to an email listserv with more than 20,000 unique email addresses of past and present study participants to assess the feasibility and acceptability of the technology. For the dietary adherence tests, monitors detected 86% of non-adherent events (sensitivity) in-person versus 78% of events via video chat monitoring (p=0.12), with comparable inter-rater agreement (0.88 vs. 0.85; p=0.62). However, for pill-taking, non-adherence trended towards being more easily detected in-person than by video chat (77% vs. 60%; p=0.08), with non-significantly higher inter-rater agreement (0.85 vs. 0.69; p=0.21). Survey results from the second study (N=1,076 respondents; at least a 5% response rate) indicated that 86.4% of study participants had video chatting hardware, 73.3% were comfortable using the technology; and 79.8% were willing to use it for clinical research. Given the capability of video chat technology to reduce participant burden and to outperform other adherence monitoring methods such as dietary self-report and pill counts, video chatting is a novel and highly promising platform to quantify dietary and pill-taking adherence.
Keywords: dietary adherence, compliance, diet monitoring, supplement adherence, medication adherence, video chat technology, mHealth, Telehealth, Skype, controlled feeding
INTRODUCTION
Since the first generation of digital communications, electronic communication devices have become more affordable and increasingly sophisticated, leading to nearly ubiquitous use of portable internet-connected devices in society(1). Consequently, these technologies have increasingly been integrated into healthcare at a number of levels with positive results(2). Hospitals and medical schools now commonly use remote audio- and video-based technologies for instruction and training(3; 4; 5), and many physicians provide video consultations through real-time online consultation platforms such as MDLiveCare and SwiftMD(6). Health care providers are even using digital real-time video technology platforms to deliver health care in rural and/or resource-limited areas(7; 8; 9; 10; 11; 12) and to remotely diagnose, monitor, or treat medical conditions ranging from orthopedic trauma to neurologic disorders(1; 10; 13; 14; 15; 16; 17; 18; 19; 20; 21).
Despite the rapidly growing use of telehealth in clinical care, very few dietary, lifestyle, or pharmaceutical research studies use visual digital communication technologies as a clinical research tool. A small number of research studies have used video chatting platforms such as Skype and Facetime to conduct interviews, thereby supplanting telephone or in-person interviews(22; 23; 24). Only a few clinical research studies have used visual electronic technologies to capture novel information, such as to collect health behavior data in real-world settings or to more accurately quantify intervention adherence(25), which is the ability of participants to follow prescribed medication or lifestyle changes. An example of the novel application of visual communication technology to dietary research is food “photography” methods, such as the Remote Food Photography Method© (RFPM) and SmartIntake™ smartphone application(12; 26; 27; 28). Using a smartphone app, study participants take still images of the food they eat and the smartphone images are relayed back to researchers to estimate the quantity and quality of food consumed. This technique has been found to accurately measure the energy and nutrient intake of adults(12; 26; 27; 28), and it can be used both for real-world studies of health behaviors and for monitoring of adherence to controlled feeding interventions.
For the latter purpose—to monitor dietary adherence—RFPM is frequently thought to be better than data collection and monitoring methods such as self-report, which is widely known to be of limited accuracy(29; 30; 31; 32). For monitoring adherence to other health behaviors—such as supplement or medication taking—visual monitoring technology has not yet been implemented, though novel technologies such as the Medication Event Monitoring System (MEMS; AARDEX Ltd., Zug, Switzerland) can accurately quantify when pill bottles are opened via a computer chip that is built into a bottle cap. MEMS is marginally better than standard pill counts and even unannounced telephone-based pill counts(33), yet none of these methods can detect if pills were removed from the container and discarded/not consumed.
By contrast, video chat technology has the potential to solve the problem of monitoring dietary and pill-taking (i.e., supplement and medication) adherence by providing video of the behavior from start to finish, minimizing the ability of participants to “cheat” or engage in non-adherent behaviors. While there are a multitude of studies investigating the feasibility and usability of video chat technology to assess health behaviors, currently a small number of clinical research studies have used the technology to monitor or enhance adherence to a clinical intervention(29; 34; 35; 36; 37; 38; 39; 40; 41; 42; 43; 44; 45; 46; 47), and none have tested its validity and reliability for this purpose. We therefore designed a pilot study to empirically evaluate the validity and reliability of video chat technology to quantify adherence to dietary and pill-taking interventions remotely compared to the gold standard of in-person monitoring. We hypothesized that diet and pill adherence could be quantified as reliably and accurately by webcam, as in-person. In parallel, we also administered a survey to determine the acceptability and feasibility of using video chat technology to participate in clinical research. To our knowledge, this is the first pair of studies to rigorously investigate the issues of the validity, reliability, acceptability, and accessibility of video chat technology in the context of clinical intervention adherence.
METHODS
Both studies were conducted at Pennington Biomedical Research Center (Baton Rouge, LA), approved by the institution’s Institutional Review Board, and registered in the clinicaltrials.gov registry (www.clinicaltrials.gov; NCT # 02204540). The studies were conducted in accord with the Declaration of Helsinki, and no compensation was provided for either study.
Validity and Reliability Study
The aim of the first study was to assess the accuracy and reliability of detecting compliance to dietary and pill-taking interventions by video chat, in comparison to in-person monitoring. To test validity, seven research center staff members were recruited (employee status was the only inclusion/exclusion criterion), and all seven provided written informed consent. Four of the volunteers served as monitors, and three of the volunteers served as mock study participants while: (1) eating meals, as if they were participating in a dietary clinical study, and (2) swallowing pills, mimicking participants in a supplement or pharmaceutical trial. The participants ate the meals and swallowed pills (Biotin vitamin capsules were used for this trial), while following behavioral scripts that outlined instructions to perform non-adherent behaviors (called “cheats”).
Cheating Behaviors
Non-adherent behaviors include deviant actions such as not eating all of the food by spitting food into a napkin, placing a pill under one’s tongue and discarding it, etc. We developed a list of common cheating behaviors by convening a group of 6 registered dieticians and staff members from Pennington Biomedical Research Center who professionally monitor study participants for compliance in clinical studies. For reference, the focus group estimated that less than 5–10% of study participants blatantly cheat, and that the most common reason for cheating is study fatigue. The cheating behaviors identified as common during dietary interventions included behaviors hiding food in a napkin and not eating it; stacking containers to hide unfinished food; and removing some of the food from a container before the meal monitoring starts.
The mock study participants were given standardized instructions for eating and swallowing the pills (Supplementary Materials 1). These instructions were designed to be identical to those that would be given to real study participants to make it easier for monitors to accurately assess compliance. The instructions included showing the pill to the camera; showing the participant’s empty hands to the camera after swallowing the pill; showing each empty food container to the camera after eating; and keeping their hands and head in the field-of-view of the webcam at all times. In addition, when scripted to cheat, participants were instructed to deliberately avoid being caught cheating.
The three mock study participants together ate a combined total of 30 meals (192 food items), during which 60 cheating events were scripted to occur, and they swallowed a total of 60 pills with 30 scripted cheating events. For simplicity, the pill swallowing tests were performed in conjunction with the dietary adherence tests: one pill was swallowed before each meal and a second pill was swallowed after each meal. (Due to unforeseen changes in circumstances, one mock study participant performed half of the cheating behaviors for both meals and pills, and the other half of cheating behaviors were unevenly divided among the other two mock participants.) The cheating events were scripted as follows. Each meal was randomized to contain an average of 2 cheating behaviors (range 0–4), and each instance of pill swallowing was randomized to have 0 or 1 cheating behaviors. Which specific cheating behavior was performed in each instance was also randomized. The monitors were blinded as to the total number of cheats. On rare occasions, participants accidentally forgot to cheat as instructed or performed additional cheats; thus, the results are expressed as a percentage of all behaviors that actually occurred.
Monitoring
During data collection, two monitors watched each meal or pill swallow in-person and two additional evaluators rated each event remotely by watching the video-chat recorded videos. The monitoring was conducted in a crossover design, so that each pair of monitors observed both in-person and by video chat in a balanced order. The monitors were employees of the research center but had no prior training in monitoring compliance for dietary or pill-taking intervention trials; monitors were not trained research dieticians so as to better generalize our results to trials conducted at research facilities without dieticians and so as to avoid any bias since our research dieticians were heavily involved in determining the cheating behaviors we tested. Similar to actual study procedures, monitors were given images of the actual meals to be consumed and the pill to be swallowed about 5–10 minutes ahead of time, and they documented any cheating behaviors on a standardized form. In-person monitors were seated 1.5–1.8 meters away from the participant. A large divider was placed between the two observers, and each wore earplugs, so that they could not see or hear each other. To video record each instance of eating a meal or swallowing a pill, a smartphone was placed about 1–1.5 meters away from the participant. All recordings were performed using Skype and CallNote premium on either a Samsung smartphone or an iPhone, paired with both a tripod and wide-angle lens. We describe in detail how we identified this combination of software and hardware as being optimal in Supplementary Material 2, and we strongly encourage clinicians and researchers who want to implement video chat technology to read this section for advice. The video recordings were made by smartphone since more people have smartphones than other devices with webcams (see Results), but our methods were designed to generalize to video chat technology in general. Lastly, an exit survey was given to monitors to gather feedback on their experiences with monitoring compliance by video chat technology, to determine what they liked and did not like about monitoring by video chat and to determine what technical aspects of the video quality made it easier or harder to detect cheating
Statistical Analysis
The accuracy (or validity) of detecting cheating by video chat in comparison to in-person was quantified by sensitivity (percent of cheats that were detected) and specificity (1 – false positive rate, where each pill or food item consumed without cheating was counted as 1 event). The motivation for this manuscript was the dietary adherence sensitivity testing, which was powered at the 80% level to detect a 15% absolute difference between in-person and video chat monitoring, assuming 95% of cheats were detected in-person (one-sided test). The pill-taking testing was added later as an exploratory pilot analysis, since we could find no reasonable published or anecdotal data to formulate an estimate for the percentage of cheats that could be detected in-person. Reliability was operationally defined as inter-rater agreement adjusted for agreement by random chance, which was quantified primarily by Cohen’s kappa; Cronbach’s alpha was also calculated as a second measure of inter-rater agreement. Statistical differences between in-person and video monitoring of compliance were assessed using Fisher’s Exact Test. The Type I error rate (α) was set at 0.05 for all analyses.
Feasibility and Acceptability Study
To investigate the feasibility and acceptability of using video chat technology to participate in clinical research, a 22-item survey that takes less than 5 minutes to complete was developed (Supplementary Material 3).
Survey Respondents
The survey was administered through SurveyMonkey (www.surveymonkey.com) and was sent via a listserv that Pennington Biomedical Research Center uses to promote its clinical trials. The listserv contains more than 20,000 unique email addresses, although it is unknown how many of those are currently valid. Survey responses were collected primarily over the two-month period from December-January 2014. Because Study 2 included an anonymous online survey, a waiver of informed consent was granted. There was no testing for legitimacy of email addresses or for validity of content.
Survey Design
We designed a close-ended survey to investigate whether study participants have access to and experience with video chat technology (feasibility), and whether they are willing to use the technology in clinical research (acceptability). In addition, since people often cite scheduling conflicts and commuting time as reasons for not participating in clinical research, we also investigated whether video chat technology might reduce barriers to participating in clinical research in general. Section 1 of the questionnaire asked respondents about their past participation in clinical studies, whether scheduling difficulties prevented them from participating in studies, and whether offering study visits on evenings or weekends would help them participate in more studies. Section 2 asked respondents to indicate their comfort with and prior use of video chatting, what video chatting software they have used, their access to webcams for video chatting (home and/or work), and what hardware that they own (e.g., smartphone). Section 3 asked participants about whether they prefer to do study visits by video chat or in-person clinic visits (along with the reasons why), whether they wanted behavioral support by video chat, and whether they would be willing to use video chat technology to participate in a clinical trial. Section 4 included demographic questions.
Statistical Analysis
Survey responses are expressed as a percentage of those who answered each question. Chi-squared tests were performed to test if survey responses differed by demographic variables. Given the multitude of association tests, the Bonferroni correction was applied. All statistical tests performed were two-tailed.
RESULTS
Validity and Reliability Study
Dietary Adherence Monitoring
As shown in Table 1, inter-rater agreement by Cohen’s kappa for dietary adherence monitoring was high for monitoring both in-person and by video chat, at 0.88 and 0.85 (p=0.62), respectively. This was supported by values of 0.94 and 0.92, respectively, for Cronbach’s alpha. The sensitivity (true positive rate) for detecting cheating in-person was 86%; surprisingly, in-person monitors did not detect about 1 of 6 cheats. The sensitivity of detecting cheating remotely through video chat was 78%, which was not significantly different from in-person monitoring (p=0.12). When examining individual cheating behaviors (Table 1), there were no statistically significant differences between monitoring in-person versus by video chat (p-values>0.10). Removing food before monitoring started was the most common cheating behavior not detected by the monitors, with the behavior detected less than 20% of the time. Spitting food into a napkin was also difficult for raters to detect, with only about half of events detected by in-person and remote monitors. Monitors detected the remaining cheating behaviors most of the time. For both the in-person and remote monitoring, the false positive rate was very low (1%).
TABLE 1.
(A) Sensitivity, specificity, and inter-rater agreement for monitoring dietary and pill-taking adherence. (B) Percentages of specific cheating behaviors detected. “Pill was empty” means that the powered contents with the active ingredient were removed from the clear pill capsule, while “No pill was present” means that participant pretended to clutch a pill and to swallow it, but no pill was ever there.
A. Compliance Detection | In-person | By Video Chat |
Dietary Compliance | ||
Sensitivity | 86% | 78% |
Specificity | 99% | 99% |
Inter-rater agreement (Cohen’s kappa) | 0.88 | 0.85 |
Pill-taking Compliance | ||
Sensitivity | 77% | 60%* |
Specificity | 100% | 98% |
Inter-rater agreement (Cohen’s kappa) | 0.85 | 0.69 |
B. Individual Cheating Behaviors Detected | In-person | By Video Chat |
Dietary Compliance | ||
Stacked containers to hide unfinished food | 100% | 100% |
Food item was missing | 100% | 100% |
Wrong item substituted for correct Item | 100% | 100% |
Additional food item present | 100% | 100% |
Did not eat food item (no fancy tricks) | 100% | 88% |
Did not eat all or part of condiment/liquid | 100% | 86% |
Dropped food and did not eat it | 88% | 81% |
Spit food into napkin | 58% | 50% |
Removed food before monitoring started | 19% | 6% |
Pill-taking Compliance | ||
Pill was empty | 88% | 100% |
Hid pill in hand and did not swallow it | 67% | 83% |
Spit pill into drinking cup and did not swallow | 100% | 38%** |
Dropped pill and did not swallow it | 75% | 63% |
No pill was present | 63% | 63% |
Wrong pill (a substitute) was swallowed | 75% | 63% |
Hid pill in mouth and did not swallow it | 50% | 0%* |
P < 0.05,
P < 0.10.
Pill-taking Adherence Monitoring
While the inter-rater agreement by Cohen’s kappa for in-person monitoring of pill-taking adherence (0.85) was comparable to that for dietary adherence, inter-rater agreement was somewhat lower for monitoring remotely through video chat (0.69); however, this difference did not reach statistical significance (p=0.21). Similarly, Cronbach’s alpha was 0.93 for in-person inter-rater agreements versus 0.82 for video chat monitoring. The sensitivity of detecting cheating was 77% in-person, meaning that about one-quarter of cheating events were not detected by the gold-standard method of in-person monitoring. By comparison, the sensitivity tended to be lower at 60% for remote monitoring through video chat (p=0.08). The cheating behavior least likely to be detected was hiding the pill in the mouth, such as under the tongue, instead of swallowing it. In-person monitors detected this behavior correctly 50% of the time, while the video chat monitors did not detect it (0.0%; p=0.08). In addition, spitting the pill in a drinking glass instead of swallowing it was easy to detect in-person (100%), whereas when monitors detected this behavior significantly less frequently when they were monitoring by video chat (38%; p=0.03). Similar to monitoring of dietary adherence, the false positive rates were very low (0.0% and 2%; p=0.50) for both in-person and remote monitoring, respectively.
Acceptability by Monitors
Three of the four monitors were somewhat more confident in their ability to detect cheating in-person versus recorded videos, with one monitor finding remote monitoring by video chat easier. Most monitors reported that the video resolution and lighting were the two most important factors that impacted their ability to detect cheating behaviors in the recorded videos. In the case of lighting, glare and identifying the correct food item/pill were the most common issues. In addition, it was clear that the frame rate (number of frames per second) of the video influenced the ability to detect cheating for the pill adherence tests. Frame rate and resolution are largely determined by the internet connection speed. As it happened, these varied in the area of the building where the validity and reliability tests were conducted, allowing us to qualitatively assess their impact on sensitivity. When the frame rate and resolution were low, it was easier for participants to drop the pill or hide the pill without the monitor detecting it, thus reducing the sensitivity of detecting cheating remotely, whereas the sensitivity of monitoring dietary compliance was less affected by video quality. All monitors reported that being able to pause, rewind, and fast-forward the videos was very or extremely helpful in detecting cheating.
Survey Respondents
For the second study—which investigated the feasibility and acceptability of video chat technology—1,076 respondents completed the online survey. The respondents’ demographics are shown in Table 2. Approximately three-quarters of participants were female. Nearly 70% of respondents were Caucasian, and one-quarter were African or African-American, consistent with local demographics. Approximately half of respondents had participated in one or more clinical studies in the past, and all were interested in participating in clinical research in general. As shown in Table 3, 48.0% of respondents reported that scheduling conflicts prevented them from participating in clinical research on at least one occasion. Importantly, approximately three-quarters of respondents reported that being able to schedule study visits on evenings and/or weekends would enable them to participate in more research studies, with 32.6% of respondents preferring having the option of both evening and weekend visits. Transportation issues were less of a barrier to study participation: nearly three-quarters of respondents reported that transportation did not prevent them from participating in clinical research (Table 2). Nonetheless, one-fifth (21.3%) reported that a long commute time/bad traffic made it difficult for them to be involved in clinical research. These data demonstrate that video chat technology may also be useful to supplant in-person clinic visits and to reduce barriers to participating in clinical research in general.
TABLE 2.
Demographics of survey respondents. N=1,076.
Characteristic | Percent | |
---|---|---|
Age | ||
18–24 | 6.5% | |
25–34 | 19.7% | |
35–44 | 20.3% | |
45–54 | 20.6% | |
55–64 | 20.8% | |
65+ | 12.1% | |
Gender | ||
Male | 21.7% | |
Female | 78.3% | |
Race | ||
African or African-American | 24.0% | |
Asian | 0.7% | |
Caucasian | 69.8% | |
Hispanic | 0.7% | |
Other | 2.1% | |
Prefer not to answer | 2.8% | |
Ethnicity | ||
Hispanic | 2.1% | |
Non-Hispanic | 93.2% | |
Prefer not to answer | 4.7% | |
Prior Participation in Research | ||
No | 49.4% | |
Yes, once | 29.4% | |
Yes, multiple times | 21.2% |
TABLE 3.
(A) Scheduling Conflicts are a Barrier to Study Participation. (B) Technology use and comfort levels among survey respondents. N=1,076.
A. Study Participation Barriers | Percent |
Scheduling issues ever a barrier? | |
Multiple times | 32.6% |
Once | 15.4% |
Never | 48.8% |
Transportation issues ever a barrier?* | |
No | 72.8% |
Yes, commute is too long | 21.3% |
Yes, cost is an issue | 7.0% |
Yes, no reliable transportation was available | 5.1% |
Yes, other | 2.2% |
Would adding additional hours for clinic visits increase your participation? | |
Weekends and evenings | 36.2% |
Weekends only | 23.5% |
Evenings only | 17.2% |
No | 26.7% |
B. Technology Use and Comfort | Percent |
Do you have a computer with internet and a webcam? | |
No | 13.6% |
Yes, at home only | 40.6% |
Yes, at work only | 4.0% |
Yes, at both home and work | 41.8% |
Which mobile devices do you have?* | |
Smart Phone | 81.8% |
Portable laptop | 45.7% |
Tablet | 58.1% |
None | 8.3% |
Unsure | 1.5% |
How comfortable are you with technology in general? | |
Very comfortable | 20.8% |
Comfortable | 20.6% |
Neutral | 20.3% |
Uncomfortable | 19.7% |
Very uncomfortable | 6.5% |
Have you ever participated in live video chatting? | |
Yes | 79.8% |
No | 20.2% |
Which video chat software have you used?* | |
Skype | 68.0% |
FaceTime | 60.7% |
Google Hangouts | 13.7% |
ooVoo | 7.5% |
Other | 8.6% |
denotes a question that allowed multiple responses.
Feasibility of Video Chat Technology
Next, we investigated whether video chat technology might be a feasible tool for use in clinical research. As shown in Table 2, 86.4% of respondents reported that they have the hardware necessary for video chatting. Since 81.8% reported owning a smartphone with a webcam, a webcam-endowed smartphone was clearly the most widely accessible video chat platform. In terms of comfort with video chat technology, 73.3% of respondents are very comfortable or comfortable with using video chat technology, whereas only 7.1% are uncomfortable or very uncomfortable (Figure 1). As shown in Table 2, this figure is higher than that for comfort with technology in general, for which only about 40% of respondents reported being very comfortable or comfortable. Importantly, nearly 80% of respondents have participated in live video chatting before, with Skype (68.0%) and FaceTime (60.7%) being the two most popular video chat software programs. Google Hangouts was in distant third place at 13.7% usage.
FIGURE 1.
Most Respondents Embrace Use of Video Chatting In Research Studies. (A) 73.3% of respondents were comfortable with video chat technology, whereas only 7.1% were not. (B) Almost twice as many respondents preferred study visits to be done remotely via video chat (45.2%) versus in-person (25.3%). (C) A majority of respondents were interested in receiving behavioral support to help them adhere to study interventions via video chat. (D) Nearly 80% of respondents were willing to use video chat to participate in research studies, whereas only 5.4% were opposed. About a quarter of those willing to use video chat would agree to using the software only on the condition that the video chat session was not recorded.
Acceptability of the Technology
Finally, we queried participants to determine whether they would be willing to use video chat technology to participate in research. When offered an option to conduct a study visit via video chat or in-person, nearly half (45.2%) preferred to conduct the visit remotely by webcam, with only 25.3% preferring to have the visit in-person (Figure 1); the remaining had no preference. Of those preferring an in-person clinic visit (Table 3), the most common two reasons were having in-person contact/accountability (68.6%) and disliking being watched by video chat (39.3%); discomfort with or lack of the technology were minor contributing reasons (17.0%). Of those who instead prefer doing remote visits via video chat, most cited the commute (62.6%) and their work (59.6%) and family/social (44.2%) schedules as the reasons why. Additional reasons included liking using technology (39.5%), living or working too far away (35.0%), and preferring to save money on transportation (33.4%). Figure 1 shows that if offered an option of receiving behavioral support to adhere to the study intervention (i.e, encouragement and motivation to stick to the intervention) by video chat, 57.7% would want it and another 33.3% declared they might want it. Lastly and most importantly, nearly 80% of respondents were willing to use video chat technology to participate in a clinical trial, whereas only 5.4% were opposed. About one-quarter of those willing to use video chat technology would agree to using the software only if video chat session was not recorded (data not shown).
Demographic Associations
We tested the survey results for demographic associations. There were no associations between gender and the survey questions. There was one association with ethnicity—Hispanics were less comfortable with using technology in general (p=0.0009) —although the number of Hispanics in our sample was small (N=22), so we caution extrapolation of this one result. To test for associations with race, we limited testing to Caucasians and African-Americans because of low numbers of respondents in other racial groups. In comparison to Caucasians, African-Americans were more likely to report that scheduling conflicts prevent them from participating in research (73.5% vs. 60.2%; p=0.0001) and that being allowed to schedule study visits on weekends would help them participate in more studies (63.2% vs. 52.5%; p<0.0001). African-Americans also were more likely to want behavioral support through video chatting than Caucasians (69.3% vs. 54.6%; p=0.0002). Interestingly, African-Americans were less likely to have a computer with internet access and a webcam (p=0.0001), but fortunately just as likely as Caucasians to have a smartphone equipped for video chatting (p=0.46).
Age was associated with the responses for nearly all survey questions; however, a majority of associations were modest in effect size (i.e., <25 percentage-point difference in survey responses between the very oldest and youngest cohorts). Older respondents were less likely to report that transportation or scheduling conflicts prevented them from participating in studies (p<0.0001); slightly less likely to have a webcam-enabled device for video chatting (p<0.0001); less comfortable with technology in general and with video chatting (p<0.0001); and less inclined towards using (p<0.0001), and less willing (p=0.0003) to use video chat technology to participate in clinical trials. In particular, 90% of respondents aged 18–24 years old were willing to use video chat technology to participate in clinical trials, whereas 62% of those aged 65+ years old were willing to use the technology. Similar numbers—99% and 62% of those aged 18–24 and 65+ years old, respectively—had prior experience in video chatting. The one exception to these trends with increasing age is that respondents in the middle two age groups (35–44 and 45–55 years old) were somewhat more likely to want motivational support by video chat, than either their younger or older counterparts (p=0.0001).
DISCUSSION
Current methods of determining adherence to dietary and pill-taking interventions in free-living subjects rely on strategies such as self-report, pill counts, and returning empty food containers, which are well-known to be of limited accuracy. Conversely, the alternative of conducting controlled studies in an inpatient setting or under staff supervision results in high participant burden, excluding many would-be participants. We therefore propose using video chat technology as a novel method that has the potential to solve both these problems by remotely quantifying intervention adherence. Video chat technology could replace self-report and pill counts in pharmaceutical and supplement trials, and in dietary studies, it could replace self-report and “empty container” method of estimating adherence, such as counting yogurt lids in a probiotic yogurt study. It could also reduce participant burden in controlled feeding studies by obviating the need to commute to the research center to eat meals under supervision. For instance, in a meal timing study, participants could demonstrate that they followed the assigned eating schedule by logging onto video chat software and eating the meals at the appropriate times. Finally, video chat technology could be used to replace food diaries by capturing data on the type of food eaten, the time of day, and even an estimated amount.
However, whether the technology is effective and feasible to use in clinical research is an open question. Therefore, in this pair of studies, we investigated the validity, reliability, feasibility, and acceptability of using video chat technology to quantify dietary and pill-taking (supplement and medication) adherence The first study was a pilot study designed to test the ability of video chat technology to detect non-adherence to dietary and pill-taking interventions, in comparison to the gold standard of in-person monitoring—the first time such a study has been performed. For dietary adherence monitoring, the reliability or agreement among raters calculated using Cohen’s kappa for both in-person and video chat monitoring was excellent (0.88 vs. 0.85), with no statistical difference between the two methods. The validity testing revealed that 86% and 78% of cheats were detected by in-person and remote meal monitoring, respectively, and the methods did not differ from each other. Interestingly, we found that the mock study participants could successfully evade being caught cheating one in six times, even by the gold standard of in-person monitoring. One cheating method that was very difficult to detect using either monitoring method was the removal of food from the container prior to monitoring beginning. To address this limitation, meal monitors may need to be better trained (our volunteers were untrained staff members) or the way in which the foods are packaged may need to be modified to better indicate if the package was opened prior to the monitoring period. Additionally, spitting food into napkins was also difficult to detect by either method, though this limitation can be addressed by requiring study participants to shake out their napkin after the meal. Overall, video chat technology proved to be comparably valid and reliable to in-person monitoring of dietary adherence, and we therefore feel confident that it is ready for adoption as a widespread method for quantifying adherence in nutrition research, particularly for controlled feeding studies.
However, our exploratory testing of pill adherence found that monitoring remotely by video chat trended towards being inferior. Inter-rater agreement by Cohen’s kappa was higher for in-person monitoring (0.85) than video chat (0.69) monitoring. Unlike meal monitoring, detecting non-adherence when taking pills tended to be more difficult via video chat (60% of cheats detected) versus in-person monitoring (77%), yet only two cheating behaviors were or tended to be significantly harder to detect by video chat (hiding the pill in the mouth and spitting the pill into the cup without swallowing it). The fact that these differences trended towards significance is likely due to the fact that our statistical power was more limited. Although our pilot study for pill adherence was not statistically powered, the detection of only 77% of cheats in-person would have weakened post-hoc statistical power by almost a factor of two if the study has been powered identically to the dietary adherence testing; thus, it is appears that about double the number of non-adherent events as in our dietary adherence testing is needed for future full-scale validity and reliability testing of pill-taking monitoring.
Additionally, detection of some cheating behaviors remained difficult for either method, such as hiding pills under the tongue. Closer examination of the recorded videos revealed that natural fluctuations in internet speed and latency lighting sometimes reduced the resolution and frame rate of the video, making it easier for the participants to avoid getting caught cheating during pill taking. This provides some insight into ways in which the ability to detect cheating during pill taking via video chat can be improved. A faster Internet connection, good lighting, and higher resolution video may increase the sensitivity of monitoring pill adherence by video chat, and such technical resources are expected to be more widely available over time.
Based on our pilot study of pill adherence monitoring, further optimization of the technical set-up and a larger sample size follow-up study are needed to determine whether video monitoring of pill-taking compliance is truly inferior to in-person monitoring. Unlike for monitoring of dietary adherence, we suspect that monitoring pill-taking adherence by video chat technology will likely prove to be inferior to in-person monitoring, although this needs to be confirmed in a larger study. Despite these limitations, the platform is clearly superior to no monitoring at all and to self-reported adherence, pill counts, and inspecting empty food containers. These methods allow participants to discard pills and foods surreptitiously, while falsely pretending to have been compliant. Furthermore, even if the validity and reliability are slightly inferior to in-person monitoring for pill-taking adherence, it is important to remember that in-person monitoring is rarely feasible because of the high burden (e.g., scheduling constraints and commute time) it imposes on participants. Moreover, several controlled studies have demonstrated that simply monitoring pill-taking adherence remotely by video can increase adherence rates(34; 36; 39; 40; 45; 46; 48; 49), relative to no form of visual monitoring or self-report, and it boosts adherence rates to levels similar to those achieved with in-person monitoring(15; 35). Moreover, when no video is received from a participant by a certain time each day, the participant’s non-adherence can be detected in real-time, and s/he can then be reminded to take the pill or follow his/her prescribed dietary intervention(47). Therefore, this study provides evidence that video chat technology provides a valid and reliable platform for remotely quantifying diet and pill adherence and likely also for encouraging better adherence.
The second study investigated the feasibility and acceptability of using video chat technology to participate in clinical research—the largest study to test the feasibility and acceptability of using webcam or video chat technology either in clinical research or in patient care(>1,000 participants)—with the important finding that it is a feasible and acceptable method for the overwhelming majority of potential study participants. A majority (86.4%) of respondents had the hardware necessary to video chat, with nearly the same proportion having a smartphone configured for video chatting to occur at any location with either a WiFi Internet connection or a cellular signal. Also, approximately three-fourths of respondents were familiar with video chat and nearly half (45.2%) preferred to conduct study visits via webcam or video chat. Finally, nearly 80% of participants are willing to use video chat technology to participate in clinical research. This concords with several clinical care studies conducted with small numbers of patients that have similarly demonstrated very high satisfaction (range 65–93%) and good feasibility with remote video monitoring(10; 13; 15; 44; 50; 51). Collectively, these data indicate that few technological barriers exist to conducting study visits via video chat, participants are familiar with and accepting of the technology, and that in fact more respondents preferred to conduct study visits via video chat versus in-person.
Although not the focus our investigation, the survey also revealed that approximately one-half of former and potential study participants encountered scheduling difficulties that prevented them from participating in one or more research studies—a problem that may also be mitigated by video chat technology. Importantly, the timing of clinic visits within people’s busy schedules, not transportation, was the primarily issue for most individuals. Video chat technology offers a novel alternative that can reduce these barriers. Conducting study visits remotely by video chat reduces the inconvenience and expense of visiting the clinic. More specifically, it can allow appointments to be scheduled in between commitments that are otherwise too close together to commute to the clinic, to be scheduled outside of business hours, or even to happen spontaneously at the participant’s convenience. The method particularly has value for controlled feeding studies, where the high participant burden has made such studies increasingly difficult to do and has resulted in fewer controlled feeding studies. A related but unanticipated finding is that video chat technology may particularly help reduce barriers to research participation experienced by African-Americans, a group that is frequently underrepresented in research studies(52). African-Americans were more likely than Caucasians to report that scheduling conflicts prevented them from engaging in research and to want behavioral support through video chatting.
Our feasibility and acceptability study is not without limitations. About three-quarters of our sample were women and all respondents had to have email addresses. As a result, our study sample was likely enriched in technology-savvy female users, which somewhat compromises generalizability. Reassuringly, though, we found no gender differences in responses to any of the 22 questions.
Furthermore, there are notable limitations to using video chat technology to monitor adherence or conduct visits remotely that were not addressed by the survey. Most obviously, study visits involving blood draws for bioactive compounds cannot be conducted remotely. In addition, study participants must have a data plan or requisite internet access and must be in proximity of their webcam-enabled devices if their study appointment is scheduled at a particular time.
In summary, the two studies reported herein were, to our knowledge, the first of their kind to determine if video chat technology is an acceptable and feasible method to participate in dietary or pharmaceutical/supplement clinical research, and to empirically evaluate the ability of video chat technology to remotely quantify adherence. Validity and reliability by video chat were excellent for dietary adherence and decent but less good for pill adherence. About 80% of participants have the technology and a similar percentage were willing to use the technology to participate in clinical research. It is therefore expected that video chat technology will be increasingly used to monitor dietary adherence, to collect study data, and even to conduct study visits remotely in order to reduce participant burden and barriers to participating in research. Video chat technology is therefore a very promising tool that is ripe for integration into clinical research methods.
Supplementary Material
TABLE 4.
Respondent reasons for preferring in-person study visit versus remote study visits via video chat. Both questions permitted multiple responses.
Respondent Reasons | Percent |
---|---|
Reasons for preferring in-person study visits (N=318) | |
Prefer in-person contact | 68.6% |
Do not like being watched by video chat | 39.3% |
Not comfortable with video chat technology | 8.5% |
Do not have the technology | 8.5% |
Other reason | 6.3% |
Reasons for preferring remote visits via video chat (N=555) | |
Prefer not to commute | 62.6% |
Work schedule | 59.6% |
Family/social schedule | 44.2% |
Like using technology | 39.5% |
Live or work too far away | 35.0% |
Prefer to save gas money, bus fare, etc. | 33.4% |
Too busy to make time otherwise | 28.0% |
Prefer electronic contact | 6.1% |
No reliable transportation | 4.2% |
Other | 3.2% |
Acknowledgments
We are grateful to our study participants, without whom this research would not be possible. We also thank Jonathan Goldman and Dr. Josh Boehm for their advice on video chat technology.
FUNDING SUPPORT
This work was supported by a NORC Center Grant P30 DK072476 entitled “Nutritional Programming: Environmental and Molecular Interactions”; by 1 U54 GM104940 (to CMP and JWA) from the National Institute of General Medical Sciences of the National Institutes of Health, which funds the Louisiana Clinical and Translation Science Center; and by the Albert Noyce Summer Internship Program (to CW). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Footnotes
Clinical Trial Registration Number: NCT02204540
CONFLICTS OF INTEREST
None.
AUTHORSHIP
CMP conceived of the study and the use of video chat technology to quantify adherence, and CMP, JWA, CW, and CKM all contributed to the study design. CMP, JWA, and CW collected the data and were involved in data analysis. CMP and CKM drafted the manuscript, and all authors edited and approved the final manuscript.
REFERENCES
- 1.Armstrong DG, Giovinco N, Mills JL, et al. FaceTime for Physicians: Using Real Time Mobile Phone-Based Videoconferencing to Augment Diagnosis and Care in Telemedicine. Eplasty. 2011;11:e23. [PMC free article] [PubMed] [Google Scholar]
- 2.Free C, Phillips G, Felix L, et al. The effectiveness of M-health technologies for improving health and health services: a systematic review protocol. BMC Res Notes. 2010;3:250. doi: 10.1186/1756-0500-3-250. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Hayden EM, Navedo DD, Gordon JA. Web-conferenced simulation sessions: a satisfaction survey of clinical simulation encounters via remote supervision. Telemed J E Health. 2012;18:525–529. doi: 10.1089/tmj.2011.0217. [DOI] [PubMed] [Google Scholar]
- 4.Langenau E, Kachur E, Horber D. Web-based objective structured clinical examination with remote standardized patients and Skype: resident experience. Patient Educ Couns. 2014;96:55–62.s. doi: 10.1016/j.pec.2014.04.016. [DOI] [PubMed] [Google Scholar]
- 5.Burckett-St Laurent DA, Cunningham MS, Abbas S, et al. Teaching ultrasound-guided regional anesthesia remotely: a feasibility study. Acta Anaesthesiol Scand. 2016 doi: 10.1111/aas.12695. [DOI] [PubMed] [Google Scholar]
- 6.Glauser W. The Skype solution. CMAJ. 2011;183:E798. doi: 10.1503/cmaj.109-3928. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Miyashita T, Iketani Y, Nagamine Y, et al. FaceTime((R)) for teaching ultrasound-guided anesthetic procedures in remote place. J Clin Monit Comput. 2014;28:211–215. doi: 10.1007/s10877-013-9514-x. [DOI] [PubMed] [Google Scholar]
- 8.Schulz TR, Richards M, Gasko H, et al. Telehealth: experience of the first 120 consultations delivered from a new refugee telehealth clinic. Intern Med J. 2014;44:981–985. doi: 10.1111/imj.12537. [DOI] [PubMed] [Google Scholar]
- 9.Johnson KA, Meyer J, Yazar S, et al. Real-time teleophthalmology in rural Western Australia. Aust J Rural Health. 2015;23:142–149. doi: 10.1111/ajr.12150. [DOI] [PubMed] [Google Scholar]
- 10.Armfield NR, Bradford M, Bradford NK. The clinical use of Skype--For which patients, with which problems and in which settings? A snapshot review of the literature. Int J Med Inform. 2015;84:737–742. doi: 10.1016/j.ijmedinf.2015.06.006. [DOI] [PubMed] [Google Scholar]
- 11.Chai PR, Babu KM, Boyer EW. The Feasibility and Acceptability of Google Glass for Teletoxicology Consults. J Med Toxicol. 2015;11:283–287. doi: 10.1007/s13181-015-0495-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Daugherty BL, Schap TE, Ettienne-Gittens R, et al. Novel technologies for assessing dietary intake: evaluating the usability of a mobile telephone food record among adults and adolescents. J Med Internet Res. 2012;14:e58. doi: 10.2196/jmir.1967. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Good DW, Lui DF, Leonard M, et al. Skype: a tool for functional assessment in orthopaedic research. J Telemed Telecare. 2012;18:94–98. doi: 10.1258/jtt.2011.110814. [DOI] [PubMed] [Google Scholar]
- 14.Freeman KA, Duke DC, Harris MA. Behavioral health care for adolescents with poorly controlled diabetes via Skype: does working alliance remain intact? J Diabetes Sci Technol. 2013;7:727–735. doi: 10.1177/193229681300700318. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Johnston B, Wheeler L, Deuser J, et al. Outcomes of the Kaiser Permanente Tele-Home Health Research Project. Arch Fam Med. 2000;9:40–45. doi: 10.1001/archfami.9.1.40. [DOI] [PubMed] [Google Scholar]
- 16.Hsieh PF, Chang CH, Lien CS, et al. Remote monitoring of videourodynamics using smart phone and free instant messaging software. Neurourol Urodyn. 2013;32:1064–1067. doi: 10.1002/nau.22387. [DOI] [PubMed] [Google Scholar]
- 17.Dorsey ER, Venkataraman V, Grana MJ, et al. Randomized controlled clinical trial of "virtual house calls" for Parkinson disease. JAMA Neurol. 2013;70:565–570. doi: 10.1001/jamaneurol.2013.123. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Armfield NR, Gray LC, Smith AC. Clinical use of Skype: a review of the evidence base. J Telemed Telecare. 2012;18:125–127. doi: 10.1258/jtt.2012.SFT101. [DOI] [PubMed] [Google Scholar]
- 19.Drake TM, Ritchie JE. The Surgeon Will Skype You Now: Advancements in E-clinic. Ann Surg. 2016;263:636–637. doi: 10.1097/SLA.0000000000001505. [DOI] [PubMed] [Google Scholar]
- 20.Brunett PH, DiPiero A, Flores C, et al. Use of a voice and video internet technology as an alternative to in-person urgent care clinic visits. J Telemed Telecare. 2015;21:219–226. doi: 10.1177/1357633X15571649. [DOI] [PubMed] [Google Scholar]
- 21.Boots RJ, Singh S, Terblanche M, et al. Remote care by telemedicine in the ICU: many models of care can be effective. Curr Opin Crit Care. 2011;17:634–640. doi: 10.1097/MCC.0b013e32834a789a. [DOI] [PubMed] [Google Scholar]
- 22.Weinmann T, Thomas S, Brilmayer S, et al. Testing Skype as an interview method in epidemiologic research: response and feasibility. Int J Public Health. 2012;57:959–961. doi: 10.1007/s00038-012-0404-7. [DOI] [PubMed] [Google Scholar]
- 23.Hamilton RJ. Using skype to conduct interviews for psychosocial research. Comput Inform Nurs. 2014;32:353–358. doi: 10.1097/CIN.0000000000000095. [DOI] [PubMed] [Google Scholar]
- 24.Janghorban R, Latifnejad Roudsari R, Taghipour A. Skype interviewing: the new generation of online synchronous interview in qualitative research. Int J Qual Stud Health Well-being. 2014;9:24152. doi: 10.3402/qhw.v9.24152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Park LG, Howie-Esquivel J, Dracup K. Electronic measurement of medication adherence. West J Nurs Res. 2015;37:28–49. doi: 10.1177/0193945914524492. [DOI] [PubMed] [Google Scholar]
- 26.Martin CK, Correa JB, Han H, et al. Validity of the Remote Food Photography Method (RFPM) for estimating energy and nutrient intake in near real-time. Obesity. 2012;20:891–899. doi: 10.1038/oby.2011.344. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Martin CK, Han H, Coulon SM, et al. A novel method to remotely measure food intake of free-living individuals in real time: the remote food photography method. The British journal of nutrition. 2009;101:446–456. doi: 10.1017/S0007114508027438. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Ptomey LT, Willis EA, Honas JJ, et al. Validity of energy intake estimated by digital photography plus recall in overweight and obese young adults. J Acad Nutr Diet. 2015;115:1392–1399. doi: 10.1016/j.jand.2015.05.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Beaton GH, Burema J, Ritenbaugh C. Errors in the interpretation of dietary assessments. Am J Clin Nutr. 1997;65:1100S–1107S. doi: 10.1093/ajcn/65.4.1100S. [DOI] [PubMed] [Google Scholar]
- 30.Tran KM, Johnson RK, Soultanakis RP, et al. In-person vs telephone-administered multiple-pass 24-hour recalls in women: validation with doubly labeled water. J Am Diet Assoc. 2000;100:777–783. doi: 10.1016/S0002-8223(00)00227-3. [DOI] [PubMed] [Google Scholar]
- 31.Schoeller DA, Bandini LG, Dietz WH. Inaccuracies in self-reported intake identified by comparison with the doubly labelled water method. Can J Physiol Pharmacol. 1990;68:941–949. doi: 10.1139/y90-143. [DOI] [PubMed] [Google Scholar]
- 32.Bandini LG, Schoeller DA, Cyr HN, et al. Validity of reported energy intake in obese and nonobese adolescents. Am J Clin Nutr. 1990;52:421–425. doi: 10.1093/ajcn/52.3.421. [DOI] [PubMed] [Google Scholar]
- 33.Fredericksen R, Feldman BJ, Brown T, et al. Unannounced telephone-based pill counts: a valid and feasible method for monitoring adherence. AIDS Behav. 2014;18:2265–2273. doi: 10.1007/s10461-014-0916-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Noda Y, Sakata Y, Kubota M, et al. Supervised administration of Alzheimer's patients using information communication technology. Gan To Kagaku Ryoho. 2014;41(Suppl 1):30–32. [PubMed] [Google Scholar]
- 35.Harris MA, Freeman KA, Duke DC. Seeing Is Believing: Using Skype to Improve Diabetes Outcomes in Youth. Diabetes Care. 2015;38:1427–1434. doi: 10.2337/dc14-2469. [DOI] [PubMed] [Google Scholar]
- 36.Hommel KA, Hente E, Herzer M, et al. Telehealth behavioral treatment for medication nonadherence: a pilot and feasibility study. Eur J Gastroenterol Hepatol. 2013;25:469–473. doi: 10.1097/MEG.0b013e32835c2a1b. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Richter KP, Shireman TI, Ellerbeck EF, et al. Comparative and cost effectiveness of telemedicine versus telephone counseling for smoking cessation. J Med Internet Res. 2015;17:e113. doi: 10.2196/jmir.3975. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38.Faria CD, Teixeira-Salmela LF, Nadeau S. Clinical testing of an innovative tool for the assessment of biomechanical strategies: the Timed "Up and Go" Assessment of Biomechanical Strategies (TUG-ABS) for individuals with stroke. J Rehabil Med. 2013;45:241–247. doi: 10.2340/16501977-1106. [DOI] [PubMed] [Google Scholar]
- 39.Hommel KA, Gray WN, Hente E, et al. The Telehealth Enhancement of Adherence to Medication (TEAM) in pediatric IBD trial: Design and methodology. Contemp Clin Trials. 2015;43:105–113. doi: 10.1016/j.cct.2015.05.013. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Smith GE, Lunde AM, Hathaway JC, et al. Telehealth home monitoring of solitary persons with mild dementia. Am J Alzheimers Dis Other Demen. 2007;22:20–26. doi: 10.1177/1533317506295888. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Eaton AM, Gordon GM, Konowal A, et al. A novel eye drop application monitor to assess patient compliance with a prescribed regimen: a pilot study. Eye (Lond) 2015;29:1383–1391. doi: 10.1038/eye.2015.155. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Nourse SE, Olson I, Popat RA, et al. Live Video Diet and Exercise Intervention in Overweight and Obese Youth: Adherence and Cardiovascular Health. J Pediatr. 2015;167:533–539. e531. doi: 10.1016/j.jpeds.2015.06.015. [DOI] [PubMed] [Google Scholar]
- 43.Garfein RS, Collins K, Munoz F, et al. Feasibility of tuberculosis treatment monitoring by video directly observed therapy: a binational pilot study. Int J Tuberc Lung Dis. 2015;19:1057–1064. doi: 10.5588/ijtld.14.0923. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44.Chan DS, Callahan CW, Sheets SJ, et al. An Internet-based store-and-forward video home telehealth system for improving asthma outcomes in children. Am J Health Syst Pharm. 2003;60:1976–1981. doi: 10.1093/ajhp/60.19.1976. [DOI] [PubMed] [Google Scholar]
- 45.Carroll CC, Trappe TA. Personal digital video: a method to monitor drug regimen adherence during human clinical investigations. Clin Exp Pharmacol Physiol. 2006;33:1125–1127. doi: 10.1111/j.1440-1681.2006.04503.x. [DOI] [PubMed] [Google Scholar]
- 46.Skrajner MJ, Camp CJ, Haberman JL, et al. Use of Videophone Technology to Address Medication Adherence Issues in Persons with HIV. HIV AIDS (Auckl) 2009;1:23–30. doi: 10.2147/HIV.S6325. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Sakata Y, Kubota M, Uemura K, et al. Support for the medication monitoring of patients with alzheimer's dementia by the interactive care system using IT. Gan To Kagaku Ryoho. 2012;39(Suppl 1):45–47. [PubMed] [Google Scholar]
- 48.Haimoff EH, Rudin DE. Videos enhance patient compliance. Dent Econ. 1993;83:29–30. 32. [PubMed] [Google Scholar]
- 49.Saberi P, Yuan P, John M, et al. A pilot study to engage and counsel HIV-positive African American youth via telehealth technology. AIDS Patient Care STDS. 2013;27:529–532. doi: 10.1089/apc.2013.0185. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50.Robinson MD, Branham AR, Locklear A, et al. Measuring Satisfaction and Usability of FaceTime for Virtual Visits in Patients with Uncontrolled Diabetes. Telemed J E Health. 2015 doi: 10.1089/tmj.2014.0238. [DOI] [PubMed] [Google Scholar]
- 51.Antonini TN, Raj SP, Oberjohn KS, et al. An online positive parenting skills programme for paediatric traumatic brain injury: feasibility and parental satisfaction. J Telemed Telecare. 2012;18:333–338. doi: 10.1258/jtt.2012.120404. [DOI] [PubMed] [Google Scholar]
- 52.Shavers-Hornaday VL, Lynch CF, Burmeister LF, et al. Why are African Americans under-represented in medical research studies? Impediments to participation. Ethn Health. 1997;2:31–45. doi: 10.1080/13557858.1997.9961813. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.