Skip to main content

Some NLM-NCBI services and products are experiencing heavy traffic, which may affect performance and availability. We apologize for the inconvenience and appreciate your patience. For assistance, please contact our Help Desk at info@ncbi.nlm.nih.gov.

NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Sep 1.
Published in final edited form as: Curr Opin HIV AIDS. 2017 Sep;12(5):442–446. doi: 10.1097/COH.0000000000000396

Addressing Engagement in Technology-Based Behavioural HIV Interventions through Paradata Metrics

José A Bauermeister 1, J M Golinkoff 1, Kathryn E Muessig 2, Keith J Horvath 3, Lisa B Hightow-Weidman 2
PMCID: PMC5637536  NIHMSID: NIHMS910270  PMID: 28617711

Abstract

Purpose of review

The goal of this review was to examine how often researchers report participants’ online engagement using paradata (i.e. intervention usage metrics) when describing the outcomes of online behavioural HIV prevention and care interventions. We also highlight the utility of paradata collection and analysis in future technology-based trials.

Recent findings

We focused on studies indexed on PubMed and published between January 1, 2016 and March 31, 2017 that reported the development and testing of online behavioural interventions for HIV prevention and/or care. Of the 705 extracted citations, six met study criteria.

Summary

Only one study reported paradata reflecting participants’ engagement with a technology-based intervention. Researchers should systematically collect and analyze paradata to strengthen the evidence-base for technology-based interventions (do they work?), advance the use of behaviour change theory across modalities and platforms (how/why do they work?), and inform reach and scale-up efforts (for whom do they work?). Researchers may also rely on paradata to examine dose-response relationships due to user engagement, to identify replicable core components linked to behaviour change outcomes, to allocate resources judiciously and drive down development costs, and to pool these metrics for use in future meta-analyses.

Keywords: Paradata, usage, logs, eHealth, metrics

Introduction

Individuals increasingly rely on online information as learning tools that aid in their decision-making. As a result, HIV prevention and care researchers have made efforts for more than a decade to develop HIV prevention and care interventions in these digital mediums. Several reviews show an increase in the number of technology-based HIV prevention and care interventions developed and evaluated, with mounting evidence suggesting that these behavioural interventions are acceptable and efficacious in promoting behaviour change [14]. Although intervention outcomes have primarily relied on analyses by group assignment, this approach has limitations. Most notably, the evaluation of technology-based interventions often assume that participants’ engagement is equitable within the study arms and stable for the duration of the trial.

Compared to in-person and computer-based interventions delivered in supervised settings (e.g., clinics, schools), researchers have less control of participants’ engagement with technology-based interventions that provide flexibility for when and where they are accessed. Technology-based interventions typically consist of one or more components (e.g., personalized content, participation in forums, interactive or gamified activities, videos, geospatial links to local resources) that are presented to users using various platforms and modes of delivery (e.g., SMS, WebApps, social media, and/or native apps) across interventions. Data on where and how users access and move through an intervention may be automatically collected as users log on to the system (e.g., user’s operating system, browser, and type of device) and use its different components (e.g., timestamps in system logs to quantify time spent using the intervention) to assess online engagement [56]. These metrics, referred by Couper et al. [5] as paradata, can be used as “auxiliary data that capture details about the process of the interaction with the online intervention.” (p. 2). Paradata have been used to gauge the proportion of participants who accessed the intervention and remained on protocol for the online study (i.e., adherence), the frequency (i.e., exposure) and amount of time (i.e., engagement) spent on an intervention and its components, and the types of components used by participants (i.e., usage). Crutzen et al. [7], for example, noted the value of using Google Analytics as a tool that can quantify user behaviour within a sexual health website and characterize traffic to their website.

Within the larger public health literature on online engagement, paradata metrics have allowed researchers to show that increased intervention engagement is associated with stronger treatment effects on study outcomes and increased participant satisfaction and retention [812]. Within the HIV literature, however, efforts to understand how participants’ engagement with Technology-based interventions is linked to HIV prevention and treatment outcomes remain underdeveloped. To this end, the goal of this review was to examine how often researchers reported metrics that describe participants’ engagement, to examine whether these metrics were associated with trial outcomes, and to highlight the utility of paradata collection and analysis in future online behavioural interventions for HIV prevention and care.

Methods

We searched PubMed from 1/1/2016 to 3/31/2017, using the following keywords and MeSH terms in combination: sex/intercourse/coitus/unprotected/condomless anal intercourse; Pre-Exposure Prophylaxis or PrEP; Post-Exposure Prophylaxis or PEP; test/testing; prevention; engagement; antiretroviral virus/antiretroviral agents/antiretroviral therapy/highly active antiretroviral therapy/HAART/ART; medication adherence. Among articles and abstracts identified through these searches, we focused on studies that reported the pilot test or large scale efficacy trial of a behavioural intervention for HIV prevention and/or care that utilized internet and/or smartphone-based technology as a primary component of the intervention. We excluded publications that: 1) did not include a clear intervention, or 2) did not include an HIV-specific outcome (e.g. focused exclusively on pregnancy or other STIs). We also excluded studies that relied solely on text-based technologies (e.g., mobile phone text messaging; SMS) as those interventions would have no online interface from which to derive meaningful paradata. Of the 705 articles extracted, six met criteria for inclusion in our review [1318].

The use of paradata in technology-based HIV behavioural trials

Only one study, published by co-authors of this review, reported paradata [13]. Baltierra et al. used data from a one-month pilot trial (healthMpowerment) focused on 15 young (ages 20–30) Black men who have sex with men and transgender women, 9 of whom were living with HIV. The mobile-optimized intervention was designed to reduce sexual risk behaviours by including theoretically-informed risk reduction content, encouraging online community interaction between users, and promoting engagement through gamification features (e.g., quizzes, rewards, self-assessments). Baltierra et al. [13] measured participants’ engagement in the pilot trial using Google Analytics to record the number of visits to the intervention, overall time spent on the intervention, time spent on specific sections of the intervention, and number of gamification points for completing intervention activities (e.g., submitting health questions, completing quizzes, contributing to a message board conversation).

Google Analytics data indicated that the site had 544 visits over the four weeks of the trial, with a mean number of 20 pages viewed per participant per session. Overall time spent on the intervention over the four weeks of the trial varied: 2 participants never used the intervention (0 seconds), 2 participants logged on for less than an hour, 4 participants interacted between 1–5 hours, and 5 participants engaged for more than 5 hours (range 5–13 hours). Additional analysis of paradata metrics used time spent on specific sections of the intervention to identify the most popular activities: the forum (used by 11 of 15 participants) and quizzes (used by 9 of 15 participants). Participants’ self-reported intervention satisfaction was associated with the number of completed quizzes (r =.83) and total time spent on site (r=.65).

A large-scale RCT of the healthMpowerment intervention is now underway. Although the evaluation of intervention effects using traditional statistical methods will examine the overall efficacy of the intervention based on treatment assignment, secondary analyses of healthMpowerment’s paradata will help advance the field when examining whether differential efficacy on study outcomes arose based on participants’ time spent on the site (i.e., dose-response effect) and/or engagement with different intervention components.

How can paradata analyses advance the field?

Paradata can help scholars identify what intervention components should be kept, removed, or redesigned between versions of their technology-based intervention [1920] or prior to scale-up and dissemination. For instance, similar to Baltierra et al. [13], examining paradata from a pilot trial may indicate what components are most engaging, popular, and associated with satisfaction with the intervention. Paradata may also afford researchers the opportunity to evaluate whether core activities central to the intervention need to be revisited in design and/or content to ensure fidelity to the behavioural theory guiding the intervention. For example, do participants spend most of their time in activities focused on shifting social norms (e.g., forums, message boards) without engaging in self-appraisal (e.g., journaling, quizzes) or skill-building activities (e.g., games and challenges)? In sum, paradata can support researchers decisions about how to best invest study resources, promote fidelity with the theoretical framework guiding the intervention, and achieve maximum participant engagement and satisfaction to increase the intervention’s efficacy.

Examination of paradata can also facilitate understanding of participant use patterns (i.e. participant behaviours within the intervention and technology used to access the intervention) [9, 12]. For example, do active contributors show greater intervention effects than “lurkers” (those who view others’ posts but do not contribute original material themselves [21])? If so, researchers can work with designers and developers to refine the intervention to include persuasive cues that encourage participants’ active engagement with the intervention. Conversely, if “lurkers” show similar intervention effects [2223], then design resources may be moved elsewhere.

Another promising use for paradata in online health interventions could be modelled after a commonly-used technique on ecommerce websites: store users’ browsing histories to recommend other relevant products or services. Similar principles could be used to route participants to unexplored and/or relevant content within a technology-based HIV intervention. Given the nascent use of machine learning to promote user engagement in technology-based HIV interventions, future research examining whether paradata can be used in real-time to deliver tailored content and inform more effective navigation are warranted.

The potential of paradata as an innovation catalyst in our field raises important conceptual questions regarding the measurement of participants’ exposure to a technology-based intervention and offers opportunities to reconsider how to examine interventions’ effects on outcomes of interest. Compared to in-person interventions where behavioural activities are typically sequential (i.e., all participants are exposed to the same order of intervention content and activities), many technology-based interventions are designed to promote maximum engagement by allowing nonsequential content navigation. Building an intervention with a non-directive sequence might contribute to its vitality and popularity, especially if it is designed to feature user-generated content (e.g., forums, message threads, sharing of images), to embed social media elements (e.g., Twitter feeds), or incorporate other interactive online tools (e.g., online test locators or other widgets). If trial participants choose when, how, and for how long they engage with a technology-based intervention, can we truly say that all participants within the same treatment arm are systematically exposed to the same intervention? Could participants require varying amounts of temporal engagement and/or personalized content to reach similar behavioural effects from an intervention? These nuances are not currently considered when evaluating the effects of technology-based HIV prevention and treatment interventions in the traditional randomized controlled trial framework, where the primary analysis often focuses on comparing outcomes in the intervention arm to those in the control condition. Future research examining how scholars can use paradata to address these differential exposures is warranted. Undoubtedly, data science methods and statistics, often used to analyse big data in bioinformatics, could help address these issues and accelerate our understanding regarding who benefits most from technology-based behavioural interventions.

Timelines for the design and implementation of randomized controlled trials for technology-based HIV interventions often do not align with real-world technology advancement [24]. Undoubtedly, the deployment of market-driven technologies outpaces researchers’ ability to integrate and test these innovations in clinical trials due to timeline, regulatory, and funding considerations. As a result, technology-based HIV prevention and treatment interventions lag behind participants’ standards for web- or app-based functionality and technology. To appeal to and sustain the attention of research participants, researchers should consider whether real-time paradata can inform when to introduce new intervention activities and/or content (e.g., version update) to a study. Mohr and colleagues [24], for example, have proposed the Continuous Evaluation of Evolving Behavioral Evaluation Intervention Technologies (CEEBIT) as a method to reduce delays in making data-driven decisions regarding an intervention’s efficacy, and accelerating researchers’ ability to innovate in their design. Within the CEEBIT framework, researchers can introduce updates (e.g., new content or new activity) to an intervention over time and determine whether a new version (e.g., version 2.0 vs 1.0) is superior by examining paradata and real-time tracking of outcome data. Participants in the inferior version (e.g., version 1.0) would then be re-allocated to an upgraded condition (e.g., version 3.0). This comparative process between versions would continue until none of the active versions of the intervention meets inferiority criteria. Although beyond the scope of this review, future research examining whether designs like CEEBIT could address some of the methodological concerns faced by researchers deploying technology-based HIV prevention and treatment interventions is warranted.

Conclusion

Paradata analyses have promise to accelerate technology-based HIV interventions in several important areas. We encourage researchers to develop aims and hypotheses regarding the relationship between participant engagement and intervention outcomes as they develop their research protocols. During the design phase of their interventions, researchers should work with their developers to map out what paradata they wish to collect as part of their trial. Moreover, given that paradata is often generated through process logs (e.g., users’ timestamped actions within a site), careful consideration on how these data should be structured to be meaningful and useful for future analysis is vital. For example, researchers and developers should critically evaluate whether their intervention will have a timeout feature whereby the participant is automatically logged out after a period of inactivity. Without a timeout feature, usage data pertaining to temporal engagement may become skewed and unusable.

As an emergent field, paradata analyses will require multidisciplinary collaborations between experts in intervention development, bioinformatics and biostatistics. Building on lessons learned from big data analytics in e-commerce and social media [2526], these partnerships will be critical to determining what and how paradata will be collected and analyzed. Finally, researchers are strongly encouraged to report usage metrics as part of their study protocols and primary findings paper for their online trial. Availability of these data may allow for pooling of metrics across online trials, inform which user engagement metrics should be measured across technology-based interventions, and inform meta-analyses in this area.

Technology-based interventions are designed to reach large numbers of participants across diverse regions, increase access to HIV prevention and treatment tools, and promote iterative learning. Beyond traditional outcome analysis (e.g., differences in outcomes between treatment conditions) used in clinical trial designs, researchers should systematically collect and analyze paradata to examine whether online engagement creates dose-response relationships within their trials. Paradata analyses also may help identify how use of specific intervention components may lead to behaviour change, offer insights into what components to include across modes of delivery, and provide opportunities to compare the efficacy of specific components across trials. These data could advance the field of behaviour change theory for online interventions. This would have cost-saving implications by helping researchers more effectively allocate financial and technological resources when designing new interventions, and reduce churning components shown to be ineffective across studies. Future research in this area is encouraged.

Key points.

  • Paradata have been associated with differential treatment outcomes in online health behavior interventions, yet they remain under examined and underreported in technology-based HIV prevention and care interventions.

  • Researchers should work with programmers and statisticians to identify and quantify usage metrics during the intervention design phase, and propose analyses to examine these analytics as indicators of intervention feasibility, exposure and usage.

  • Paradata analyses might help identify how components lead to behaviour change, offer insights into what components to include across modes of delivery, and provide opportunities to compare the efficacy of specific components across trials.

Acknowledgments

We thank Dr. Sara LeGrand and Duke University’s Library Service for their assistance during the literature review.

Financial support and sponsorship

This work was supported by a NIH award to Drs. Muessig and Bauermeister (R21MH105292), a NIH grant to Dr. Bauermeister (R34MH101997), and a NIH grant to Dr. Hightow-Weidman (1U19HD089881). The content is solely the responsibility of the authors and does not represent the official views of the funding agencies.

Footnotes

Conflicts of interest

None.

References and recommended reading

  • 1.Hightow-Weidman L, Muessig K, Bauermeister JA, Zhang C, LeGrand S. Youth, technology and HIV: Recent advances and future directions. Current HIV/AIDS Reports. 2015;12(4):500–515. doi: 10.1007/s11904-015-0280-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Muessig K, Nekkanti M, Bauermeister JA, Bull S, Hightow-Weidman L. A systematic review of recent smartphone, internet and web 2.0 interventions to address the HIV continuum of care. Current HIV/AIDS Reports. 2015;12(1):173–190. doi: 10.1007/s11904-014-0239-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Conserve DF, Jennings L, Aguilar C, Shin G, Handler L, Maman S. Systematic review of mobile health behavioural interventions to improve uptake of HIV testing for vulnerable and key populations. Journal of Telemedicine and Telecare. 2017;23(2):347–359. doi: 10.1177/1357633X16639186. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Garett R, Smith J, Young SD. A review of social media technologies across the global HIV care continuum. Current Opinion in Psychology. 2016;9:56–66. doi: 10.1016/j.copsyc.2015.10.024. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5••.Couper MP, Alexander GL, Zhang N, Little RJ, Maddy N, Nowak M, McClure JB, Calvi JJ, Rolnick SK, Stopponi MA, Johnson CC. Engagement and retention: Measuring breadth and depth of participant use of an online intervention. Journal of Medical Internet Research. 2010;12(4):e52. doi: 10.2196/jmir.1430. This seminal study examined how paradata can be used to examine engagement, study retention, and key outcomes as part of a randomized controlled trial. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Sowan AK, Jenkins LS. Paradata: A new data source from web-administered measures. Computers, Informatics, Nursing. 2010;28(6):333–342. doi: 10.1097/NCN.0b013e3181f698fd. [DOI] [PubMed] [Google Scholar]
  • 7.Crutzen R, Roosjen JL, Poelman J. Using Google Analytics as a process evaluation method for Internet-delivered interventions: an example on sexual health. Health Promotion International. 2012;28(1):36–42. doi: 10.1093/heapro/das008. [DOI] [PubMed] [Google Scholar]
  • 8.Van Gemert-Pijnen JE, Kelders SM, Bohlmeijer ET. Understanding the usage of content in a mental health intervention for depression: An analysis of log data. Journal of Medical Internet Research. 2014;16(1):e27. doi: 10.2196/jmir.2991. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Kelders SM, Bohlmeijer ET, Van Gemert-Pijnen JE. Participants, usage, and use patterns of a web-based intervention for the prevention of depression within a randomized controlled trial. Journal of Medical Internet Research. 2013;15(8):e172. doi: 10.2196/jmir.2258. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 10.Graham ML, Strawderman MS, Demment M, Olson CM. Does usage of an eHealth intervention reduce the risk of excessive gestational weight gain? Secondary analysis from a randomized controlled trial. Journal of Medical Internet Research. 2017;19(1):e6. doi: 10.2196/jmir.6644. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Cantrell J, Ilakkuvan V, Graham AL, Richardson A, Xiao H, Mermelstein RJ, Curry SJ, Sporer AK, Vallone DM. Young adult utilization of a smoking cessation website: An observational study comparing young and older adult patterns of use. Journal of Medical Internet Research Protocols. 2016;5(3):e142. doi: 10.2196/resprot.4881. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Donkin L, Hickie IB, Christensen H, Naismith SL, Neal B, Cockayne NL, Glozier N. Rethinking the dose-response relationship between usage and outcome in an online intervention for depression: Randomized Controlled Trial. Journal of Medical Internet Research. 2013;15(10):e231. doi: 10.2196/jmir.2771. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13••.Baltierra NB, Muessig KE, Pike E, LeGrand S, Bull SS, Hightow-Weidman LB. More than just tracking time: Complex measures of user engagement with an internet-based health promotion information. Journal of Biomedical Informatics. 2016;59:299–301. doi: 10.1016/j.jbi.2015.12.015. This pilot trial of a HIV technology-based intervention (HealthMPowerment) is an ideal case study for identifying and measuring user exposure and engagement in a theoretically-informed behavioural progarm. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Castillo-Arcos L, Benavides-Torres R, Lopez Rosales F, Onofre-Rodriguez D, Valdez-Montero C, Maas-Gongora The effect of an Internet-based intervention designed to reduce HIV/AIDS sexual risk among Mexican adolescents. AIDS Care. 2016;28(2):191–196. doi: 10.1080/09540121.2015.1073663. [DOI] [PubMed] [Google Scholar]
  • 15.Fernandez MI, Hosek SG, Hotton A, Gaylord SE, Hernandez N, Alfonso SV, Joseph H. A randomized controlled trial of POWER: An internet-based HIV prevention intervention for Black bisexual men. AIDS & Behavior. 2016;20:1951–1960. doi: 10.1007/s10461-016-1403-0. [DOI] [PubMed] [Google Scholar]
  • 16.Garett R, Menacho L, Young SD. Ethical issues in using social media to deliver an HIV prevention intervention: Results from the HOPE Peru Study. Prevention Science. 2017;18(2):225–232. doi: 10.1007/s11121-016-0739-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17.Garofalo R, Kuhns LM, Hotton A, Johnson A, Muldoon A, Rice D. A randomized controlled trial of personalized text message reminders to promote medication adherence among HIV-positive adolescents and young adults. AIDS & Behavior. 2016;26:1049–1059. doi: 10.1007/s10461-015-1192-x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Greene GJ, Madkins K, Andrews K, Dispenza J, Mustanski B. Implementation and evaluation of the Keep It Up! Online HIV Prevention Intervention in a community-based setting. AIDS Education and Prevention. 2016;28:231–245. doi: 10.1521/aeap.2016.28.3.231. [DOI] [PubMed] [Google Scholar]
  • 19.Morrison C, Doherty G. Analyzing engagement in a web-based intervention platform through visualizing log-data. Journal of Medical Internet Research. 2014;16(110):e252. doi: 10.2196/jmir.3575. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries N, Brug J, Oenema A. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. Journal of Medical Internet Research. 2011;13(1):e2. doi: 10.2196/jmir.1639. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Han JY, Hou J, Kim E, Gustafson DH. Lurking as an active participation process: a longitudinal investigation of engagement with an online cancer support group. Health Communication. 2014;29(9):911–23. doi: 10.1080/10410236.2013.816911. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.van Uden-Kraan CF, Drossaert CH, Taal E, Seydel ER, van de Laar MA. Self-reported differences in empowerment between lurkers and posters in online patient support groups. Journal of Medical Internet Research. 2008;10(2):e18. doi: 10.2196/jmir.992. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Malik SH, Coulson NS. A comparison of lurkers and posters within infertility online support groups. Computers, Informatics, Nursing. 29(10):564–73. doi: 10.1097/NCN.0b013e318224b31d. 201. [DOI] [PubMed] [Google Scholar]
  • 24••.Mohr DC, Cheung K, Schueller SM, Brown H, Duan N. Continuous evaluation of evolving behavioral intervention technologies. American Journal of Preventive Medicine. 2013;45(4):514–523. doi: 10.1016/j.amepre.2013.06.006. This methodological paper provides an alternative framework to the randomized controlled trial for evaluating technology-based interventions. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.O’Neil C, Schutt R. Doing data science. Sebastopol, California: O’Reilly Media; 2014. [Google Scholar]
  • 26.Provost F, Fawcett T. Data science for business. Sebastopol, California: O’Reilly Media; 2013. [Google Scholar]

RESOURCES