Abstract
Digital health technologies are ubiquitous in the healthcare landscape. Older adults represent an important user group who may benefit from improved monitoring of physical and cognitive health and in-home access to care, but there remain many barriers to widespread use of digital health technologies in gerontology and geriatric medicine. The National Institute on Aging Research Centers Collaborative Network convened a workshop wherein geriatricians and gerontological researchers with expertise related to mHealth and digital health applications shared opportunities and challenges in the application of digital health technologies in aging. Discussion broadly centered on 2 themes: promises and challenges in (i) the use of ecological momentary assessment methodologies in gerontology and geriatric medicine, and (ii) the development of health promotion programs delivered via digital health technologies. Herein, we summarize this discussion and outline several promising areas for future research.
Keywords: Behavioral intervention, Digital health, Ecological momentary assessment, eHealth, mHealth
Translational Significance: Digital health technologies present opportunities and challenges to the care of older adults. These tools can extend care into the home and guide decision making based on data collected in the context in which people live. Conversely, how best to design these tools to meet the needs and preferences of older adults and those who care for them, and how to leverage data produced via digital health technologies to guide care remain key challenges. This article summarizes discussion arising from a workshop on digital health approaches to research in aging and provides recommendations for clinical care and future research.
Digital health is ubiquitous within the healthcare landscape, a phenomenon accelerated by the coronavirus disease 2019 (COVID-19) pandemic (1). Digital health can be understood as the field of knowledge and practice associated with the development and use of digital technologies to improve health (2). This definition encompasses the terms eHealth and mHealth, where eHealth describes the provision of health services via information and communication technology, and often the term is used to describe internet- or computer-delivered services. mHealth refers specifically to the use of mobile technologies, and is commonly used to describe health-oriented smartphone or tablet applications (3). Digital health further describes other uses of digital technology to support health, using technologies as diverse as machine learning and big data analytics to the “internet of things.” These tools offer daily or even within-day insight into important psychological, social, and physical determinants of health, and they extend the reach of traditionally center-based behavior change programming into the home.
Older adults are a particularly important digital health user group. Digital health technologies promise the ability to reach those with limitations in their ability to access in-person care (eg, due to lack of access to transport or geographic or social isolation). Moreover, these tools can provide continuous, autonomous monitoring of threats to health and safety such as fall detection via accelerometry, restricted life space via motion sensors, or indications of cognitive decline (eg, leaving a stove on), thereby assisting older adults in aging in place. While the field of digital health is rapidly advancing, in many ways, the research and clinical application of these technologies is in its nascency, with many barriers to widespread use remaining. In November 2022, the National Institute on Aging Research Centers Collaborative Network (RCCN) convened a workshop (4) on the application of digital health technologies in aging research. Workshop attendees and presenters included geriatricians and gerontological researchers with expertise in clinical, behavioral, measurement, informatic, and statistical considerations related to mHealth and digital health applications. Two key areas of interest arose naturally both in presentation and in discussion, including promises and challenges in (i) the use of ecological momentary assessment (EMA) methodologies (5) in gerontology and geriatric medicine, and (ii) the development of health promotion programs via eHealth and mHealth technologies. What follows is a summary of key points of discussion related to these 2 themes.
Considerations in the Measurement of Patient-Reported Outcomes in Daily Life and in Real Time
Widespread access to internet and mobile technologies has provided researchers and clinicians with unprecedented opportunities to measure health-related phenomena at increasingly fine resolution and over longer timescales. This is especially true of the ability to measure psychosocial phenomena—which are often highly dynamic and capture subjective states—in the real world and in real time. A major theme of the workshop centered on the well-established advantages of EMA methodologies in the context of aging research, and the emerging research gaps that widespread use of these strategies presents.
What Is EMA?
Contemporary conceptualizations of successful aging consider a biopsychosocial approach that recognizes that poor health is driven by interacting physical, social, and psychological determinants (6). In traditional clinical and research contexts, valid and reliable objective (eg, cognitive performance, functional performance) and self-reported (eg, affect, stress) measures are either not ascertained unless prompted for a clinical diagnosis, or are collected at infrequent intervals, often with the aim of characterizing a given phenomenon over a span of time (7). Such an approach is subject to important limitations, impeding the understanding of the causes and consequences of health and health behaviors. For instance, researchers and clinicians are often interested in the dynamics of a psychosocial health determinant (eg, a momentary peak in pain or stress) relative to one’s “average” experience of the phenomenon. EMA strategies aim to capture one’s experiences in the real world and in real time (5). EMA protocols often entail repeated daily assessments of patient-reported data, allowing researchers to investigate dynamic within-person relationships between psychosocial states and health outcomes of interest. For instance, Phillips and colleagues (8) investigated the relationships between EMA-reported breast cancer treatment-related symptoms and objective physical activity behaviors in a sample of 67 women undergoing chemotherapy. The authors demonstrated that daily fluctuations around one’s average pain, fatigue, and daily functioning ratings were associated with physical activity behaviors on that day as well as on the following day.
Infrequent self-report summary assessments are also subject to biases that affect the extent to which one’s rating reflects their lived experience. For instance, individual personality traits powerfully affect the accuracy of one’s recall, with traits like neuroticism or extraversion contributing to exaggerated ratings of distress and positive affect, respectively, relative to diary reports (9). The duration over which a person is asked to reflect also affects the memories an individual calls upon to answer a given item. Winkielman et al. (10) demonstrated that when asked about how often one felt angry over the previous year, participants called on rare but intense periods of anger to produce their response. By contrast, when queried about anger in the previous week, participants referenced more common and less intense experiences in forming their response. Interestingly, the dynamics of a health determinant, such as within-day or day-to-day variation in a construct, also affect the accuracy of ratings. Stone and colleagues demonstrated that those with consistent daily pain experiences more accurately reported their pain on a retrospective survey than did those who experienced more heterogeneous pain symptoms (11). Similar observations have been made of objectively assessed phenomena, including cognitive and physical functioning, such that assessments collected in distraction-free clinical environments may poorly generalize to function in the context in which a person lives, and that variation in function may capture unique information in daily function and health. This is explored further in “EMAs of Cognition” below.
Recognizing the value of more frequent ecologically valid assessment of dynamic health determinants, research use of EMA methodologies has increased rapidly. Indeed, more than 1 700 publications have employed these methods to date (7). Notably, clinical use of EMA strategies remains limited, though many EMA protocols are similar to common self-monitoring strategies employed by cognitive-behavioral clinicians (12). We next provide a general overview of the reliability and validity of EMA techniques while highlighting advantages over traditional methodologies.
EMA and Traditional “Gold Standard” Assessments
While EMA use is on the rise, many unanswered questions and opportunities specific to aging research remain. Smyth et al. recently published a review of pressing issues in EMA research (13), and many of the opportunities and challenges the authors outlined were echoed in presentation and conversation within the workshop. A recurring theme in the workshop was the desire to contrast EMA measures with contemporary infrequent measures of cognition, stress, affect, and other key causes and consequences of health in aging. As Stone et al. note, the very structure of many EMA protocols (eg, the collection of frequent assessments of phenomena over short timescales and in the real world) means the method captures information that is necessarily unique from information captured via summary measures, even when EMA measures are summarized at the individual level. This discrepancy was illustrated by Moore and colleagues (14), who conducted a randomized clinical trial of Mindfulness-Based Stress Reduction among emotionally distressed older adults (N = 67). Participants completed paper-and-pencil measures of mindfulness, depression, and anxiety pre- and postintervention. They also completed EMA surveys of the same items 3 times daily for 10 days before and following the intervention, with the exception being that EMA items queried the individuals’ current state, whereas paper-and-pencil measures asked about these constructs over the prior week. The researchers found that outcomes differed based on measurement method, such that significant changes in symptoms were only found when using the EMA data.
As underscored by Stone and colleagues (7) there is considerable room for focused research on the ways in which older adults interpret specific EMA items and draw upon information to answer a question, as well as the optimal methods for training participants to accurately and consistently respond to EMAs, and for reporting training protocols, as this is often lacking in EMA publications. From a clinical perspective, research is warranted investigating the utility of repeated versus infrequent assessment for providing better and more timely care. Put differently, it remains to be established whether the added complexity and burden of EMA meaningfully enhances quality of care. What is clear, however, is that the practice of contrasting EMA-collected data against traditional gold-standard summary self-reported measures is not likely a useful endeavor and instead researchers and clinicians would benefit by clarifying in what ways these methods capture unique and valuable information.
Designing With Burden in Mind
An especially important consideration in the design of EMA research is participant burden. An EMA protocol must match the sampling frequency to the expected rate of fluctuation in the phenomenon of interest. For instance, capturing rapid changes in affect or stress states may require much more frequent sampling than measures of sleep quality. This must be held in balance with participant burden (15). Assessing too many constructs, utilization of lengthy questionnaires, or prompting participants too frequently risks high levels of attrition or missingness (13). This poses a second key challenge to the use of conventional “gold standard” measures of psychosocial determinants of health. Carefully elucidating what information is lost by gathering more frequent brief assessments of key phenomena remains a fruitful area for additional research, acknowledging that some constructs may be more amenable to brief report than others. A related and growing area of investigation concerns the extent to which sensor-equipped wearable devices may provide complimentary data to EMA (eg, estimates of sleep duration alongside EMA-reported perceived sleep quality) or may more accurately capture items historically captured by self-report (eg, physical activity volume). It is also notable that burdensome EMA schemes run the risk of becoming their own interventions, contributing to stress and reactivity, which may influence responses (16). Promisingly, well-designed EMA protocols consistently result in high completion rates across populations. Indeed, persons with schizophrenia and older adults with and without cognitive impairment average completion rates of ≥80% (17,18). In a newly published paper by the International Society of Clinical Trials Methodology Negative Symptoms Working Group, active and passive digital remote assessment technologies were evaluated for their readiness for implementation in pharmacological clinical trials targeting complex negative symptom domains. The Working Group concluded that EMA methods were ready for implementation as clinical endpoints in trials targeting negative symptoms, though the group emphasized the need to define EMA-specific gold standards and indeed to define processes for establishing these standards (19).
EMAs of Cognition
A relatively recent area of EMA research concerns the collection of many repeated assessments of cognitive performance in the real world. Moreover, emerging research suggests that intraindividual variability in cognition may be a particularly sensitive marker of future cognitive decline (20–22) and fluctuations over time are hallmark features of some age-related neurodegenerative diseases (eg, sundowning, fluctuating attention in dementia with Lewy bodies). Schmitter-Edgecombe and colleagues (23) found that older adults’ intraindividual variability on a tablet-based n-back test (a measure of working memory) taken 4 times per day over 1 week was more strongly associated with self-reported functional status than an individual’s average performance on the task, or that same person’s lab-based global neurocognitive functioning. Aschenbrenner and colleagues (24) found that greater intraindividual variability on mobile cognitive tests taken 28 times over 1 week was associated with preclinical AD risk. A number of toolsets have emerged in recent years to facilitate the collection of cognitive performance measures repeatedly and in the field. For instance, NeuroUX (25), a platform to collect EMA, mobile cognitive testing, and passive sensor data, is now available in multiple languages and is being used at institutions across the United States as well as internationally. As another example, the Mobile Toolbox (26) arose from a collaboration between researchers at 8 institutions and offers a point-and-click system allowing for collection of mobile cognitive data on research participants.
Just as with the use of EMA methods to assess self-reported and subjective phenomena, the collection of cognitive performance within an EMA framework is a relatively new field of study, and there are several exciting and important opportunities for additional research in aging. First, there is considerable ongoing debate regarding the extent to which mobile collection of cognitive data should be expected to correlate with measures taken infrequently in a controlled environment. Studies directly comparing mobile versus controlled assessments of cognitive performance report correlations ranging from r = 0.48 to 0.53 (27,28). This is somewhat unsurprising: as noted previously, the repeated collection of data in the environments in which people live may capture unique information relative to infrequent assessment in controlled environments. A unique aspect of mobile cognitive assessment is the need to account for the effect of a high degree of practice on performance (29). However, the assessment of practice effects themselves may be valuable, as some researchers have found it to be a marker of cognitive decline (30). Second, as with the use of EMA to assess self-reported phenomena, developing and disseminating training paradigms that can efficiently prepare older adult participants of varying socioeconomic backgrounds and cognitive abilities to complete remote and repeated cognitive tests will be required for widespread clinical and research uptake of these procedures. Third, researchers should continue to work to disentangle the added information gleaned from repeated field assessments of cognition from the added noise of completing these tasks in potentially distracting contexts. Finally, working to develop easily understood and clinically valuable metrics derived from common cognitive assessments will be vital for establishing value for both clinicians and older adults around the use of EMA broadly, and EMA-measured cognition specifically.
Confidentiality in the Clinical Use of EMA Techniques
An especially important consideration in deploying EMA methodologies in clinical care and research is data privacy and confidentiality. Indeed, this emerged as a consistent point of conversation across the workshop. Relative to in-clinic assessment, the remote collection and internet transmission of potentially sensitive psychophysiological data clearly increases the risk of loss of confidentiality. It is notable that at the present time, there are no standard technologies that are widely used for the collection of clinical EMA data. As such, risks to confidentiality may be heightened depending on factors such as which technologies are selected to collect psychophysiological EMA data, and technology literacy on the part of the researcher/clinician and patient. For instance, short messaging service (ie, SMS; text messaging) is widely used for prompting and patient data collection, though SMS data are not end-to-end encrypted and are stored in transmission with the wireless carrier. Similarly, integration of third-party technology providers (eg, wireless carriers, social media platforms, wearable devices) without transparent data-sharing policies increases the likelihood of patient data being shared or commercialized without their knowledge. To that end, it is important that researchers and clinicians interested in the use of EMA methodologies—or indeed the use of digital health technologies more broadly—collaborate with institutional privacy officers, information technology professionals, and legal professionals to select tools that meet institution privacy requirements and establish contracts with technology vendors to ensure data are handled securely.
Considerations in Delivering Digital Technology-Supported Health Promotion Interventions to Older Adults
The same features that make smartphones, tablets, computers, and wearable devices attractive for data capture also make these digital health technologies valuable for the delivery of health-enhancing interventions. These technologies provide unprecedented access to individuals regardless of location or transport ability, the opportunity to dynamically tailor content to a person’s preferences and context, and the ability to offer much more frequent exposure to theory-based intervention tools in daily life. To this end, the second major theme of the workshop centered on challenges and opportunities in the design and implementation of technology-supported health behavior programming for older adults, and key design considerations that may help to avoid common pitfalls. First, speakers emphasized that while technologies may enhance access to healthcare, those technologies must be designed to meet the needs of the older adult, they must be perceived as useful and usable by the individual, and systems must be in place to distribute technologies and provide self-efficacy-oriented training and technical support designed to the diverse needs of older adults. Second, the downstream impacts of digital health technologies on related health determinants such as loneliness and isolation were highlighted, with speakers providing examples of socially supportive programming facilitated by technology. These 2 key topics are detailed further in the following sections.
The Importance of Patient-Centered Design
Digital health technologies offer tremendous potential in terms of enhancing the health, well-being, and independence of aging adults, but this potential can only be realized if aging adults adopt and subsequently successfully engage with a given tool. Generally, data indicate that aging adults are receptive to using new technologies, are able to learn to interact with new technology systems and devices, and—most importantly—are more likely to adopt a broad array of new technologies if they perceive a technological tool as useful and contributing to quality of life (31). Further, although the uptake of technology has generally increased among older persons (typically defined as those 65 years of age and older), an age-related digital divide remains, especially among those with disabilities, of lower socioeconomic status (SES), or who live in rural locations. According to recent data from the Pew Internet Research Group (32), 75% of those aged 65 and older use the internet (as compared to 59% in 2014), and 64% indicate that they have home broadband. With respect to mobile devices, 61% of people ages 65 and older own a smartphone as compared to 83% of those ages 50–64 and 95% of those ages 30–49. Currently, only about 44% of adults aged 65+ report owning a tablet computer. In addition, digital literacy (ie, possessing the skills required to successfully interact with digital technology such as a mobile device) is lower among older adults, especially for minority populations and those with lower education and SES (33,34). Older adults also tend to have lower self-efficacy in their ability to interact with technology and less comfort using technology than do younger people (35). There is also an important rural–urban divide in technology usage wherein urban-dwelling older adults are more likely to use the internet, modern technological devices, and health technologies than are those living in rural communities (36–39).
Taken together, reduced access and greater prevalence of poor digital literacy and self-efficacy threaten to exaggerate health disparities among older adults. For example, data from the National Health and Aging Trend Study (40) demonstrated that although the use of telehealth services among older adults increased during the COVID-19 pandemic, older age and lower income were negatively associated with telehealth use. Importantly, however, in models adjusted for technology-enabling factors such as access to a digital device, online experiences, and learning a new technology skill during the pandemic, age and income were no longer significant predictors of telehealth usage. These data underscore the importance of providing older adults with meaningful access to digital technologies (ie, both access and the training required for skill and self-efficacy development). Common barriers to adoption of technology by aging adults include lack of access to and awareness of technology, lack of available training and technical support, cost, continual changes in technology, and usability issues. Importantly, meaningful access to digital health technologies requires understanding the functionality, limitations, and maintenance requirements of technology, having the requisite skills and training, and having access to technical support.
Discussion between workshop attendees highlighted the importance of centering older adults in the design and refinement process of the digital health tools they are expected to use. During tool development, it is important to incorporate as design partners older adults who are representative of the target user in terms of knowledge, skill, and self-efficacy for technology usage. This is especially important as the needs of the target user—including preferences, motor and perceptive needs, and understanding of user interface and experience norms—are likely to be quite different from members of the research or clinical development team. Moreover, factors such as one’s socioeconomic and cultural background may drive differences in preferences and needs within the older adult population. Next, as noted previously, purposeful training protocols designed to develop self-efficacy toward the specific tool should be developed with feedback from target users, and these protocols should be published to help to establish norms among geriatricians and researchers implementing digital health technologies. Notably, clinical or research staff that interact with users of the digital health tool must also be trained in how to communicate about the technology (eg, on the importance of positively framing the tool to avoid reinforcing potentially negative implicit associations with technology, on discussing privacy concerns). Finally, easily accessed technology support structures must be in place (2) to ensure that users receive several opportunities for low-stakes exposure to the technology and can have technical or user challenges addressed rapidly. Broadly, key considerations such as how will technologies be designed and adapted to the users, how will users be trained, and how will users be supported should be explicitly defined in funding applications, research protocols, and publications to establish norms in the research and clinical communities around the development of digital health technologies to support aging.
Sustaining Social Support in Remotely Delivered Interventions
Another important consideration in the deployment of digital health tools is the extent to which they support or hamper other key inputs to well-being such as social connectivity. In the interest of cost-efficient scaling of health-enhancing programming, it is tempting to create fully automated (sometimes termed self-guided or unguided) eHealth and mHealth programs for older adults (41). There is a natural tension, however, between such an approach and the broader study of motivated behavior change, which places meaningful interpersonal connection at the hub of both well-being and behavior change and maintenance. As outlined by biopsychosocial models of healthy aging (6) and all widely used behavioral theories (eg, self-determination and social cognitive theories) (42,43), access to meaningful social connections is a core human need that underlies one’s ability to successful adopt and sustain health behaviors such as physical activity or healthy eating.
Physical activity behavior offers a useful example for understanding the interplay between social connection and motivated behavior change. Activity and social health are deeply connected constructs. Socially isolated older adults have higher levels of sedentary behavior (44), while enhancing an older adult’s social network is associated with improved functional and psychological health (45). Social support specific to physical activity (46), larger social networks, and engaging in social activities is associated with higher levels of physical activity in older adults (47). For example, a longitudinal study of close to 17 000 adults indicated that high social support during the COVID-19 pandemic was associated with 64% increased odds of sustaining physical activity (48).
In the context of activity promotion, mHealth technologies can provide objective real-time feedback on activity and sedentary behaviors, which is highly valuable as accurate self-monitoring is a requirement for effective behavioral self-regulation (49). With individuals tending to overestimate time spent in moderate-to-vigorous activity and underestimate time spent in sedentary behaviors (50,51), this feedback not only promotes self-awareness but also provides multiple opportunities to make necessary changes in behavior. Technologies can also cue goal successes in real time, which is vital for the development of self-efficacy (43,52), and can facilitate synchronous or asynchronous social connections at any time. Evidence suggests that access to social connections may underpin ongoing health technology usage among older adults (53). These connections provide myriad inputs to behavioral motivation, including (but not limited to): supportive interpersonal communication and feedback on activity progress; a sense of accountability; and an enhanced social environment and associated sense of security and relatedness (42,43,54,55). In a study of older adults living with Parkinson disease, there was a 31% increase in objectively measured physical activity after engagement in a socially supported physical activity intervention (56). Participants indicated that the addition of social engagement was more motivating than simply using an activity tracker alone to stimulate physical activity. A similar pattern was found in a study of older adults that compared the use of a mobile application (app) alone to the use of the app plus social interaction focused on physical activity (57). Those who had the combination of social interaction and the app significantly increased their mean steps/week, whereas those that only used the app did not. The COVID-19 pandemic provided a naturalistic environment to explore the relationship between physical activity, social connection, and digital health technology. A large cross-sectional study found that among adults over 55 years of age, those who usually exercised with friends and who did not engage with technology for exercise had a reduction in exercise levels during the pandemic (58).
Digital health technologies are purported to be a means to address health disparities and overcome common barriers to healthcare in underserved adult populations (59,60); however, social support may be even more integral to engage these groups. Adults from underrepresented groups identify social support and camaraderie as facilitators of enhanced physical activity (61,62). A study in adult Black women found that some participants needed added social support to maintain physical activity (63). Similarly, Black men reported that activity trackers with social components provided a means to extend their network of physically active individuals and sustain motivation for a healthy lifestyle (64). Within Hispanic populations health behavior may be defined by family needs and the influence of familismo, the cultural value of the family being of primary importance and influence (65). It is notable that many of these same principles may be of value across cultural groups. For instance, diabetes self-management interventions that have had a family-centered approach have been effective in enhancing physical activity and disease self-management (66,67). Digital health technologies should be culturally tailored to effectively engage these large segments of the population and help to bring our society closer to health equity.
Finally, it is notable that digital health technologies stand to affect—both positively and negatively—that patient–provider relationship, another key social connection affecting the health of older adults. Trust in the patient–provider relationship is associated with greater participation in, and satisfaction with, care (68–70). Easier access to telehealth care, especially for those with transport limitations or who are geographically isolated, may provide the opportunity for more frequent interaction and the development of more robust relationships with care providers. Likewise, access to objective information about a patient’s lived experience and health behaviors may contribute to more tailored and informed care. By contrast, access to more data may add burden to the patient–provider interaction (eg, discussing heart rate notifications given by a wearable device), and greater technology-related distraction or reduced face-to-face contact may hamper the development of trusting relationships. A review of 79 reviews describing the impact of digital health technologies on relationships and trust with providers (68) reported positive effects when these technologies supported personalized and collaborative care, and when they support and supplement rather than replace face-to-face interactions. Supporting recommendations we made above, the authors underscored the importance of user-centered design to ensure these technologies meet the preferences and needs of diverse patient populations and the implementation of training on effective technology use for both patients and providers.
Future Directions and Research Opportunities
The emergence and rapid ubiquity of the internet and smartphone devices, and the downstream technologies facilitated by these platforms, represent very recent, socially and culturally transformative technological advancements. There is no doubt that these technologies will continue to change gerontology and geriatric medicine, and steps taken now can help to maximize benefit while avoiding worsening inequity and isolation. Speakers and attendees of the 2022 RCCN workshop on mHealth and digital health in aging highlighted several important future clinical and research directions:
1. Refining objective and self-reported measures collected via EMA methodologies. It is clear that the use of digital health technologies to assess self-reported and objective phenomena in daily life and in real time will become increasingly commonplace. To this end, it will be critical for clinical and research communities to identify reliable and valid EMA questions, prompting schemes, training methodologies, and measurement protocols. Moreover, given these complexities, it will be important to develop open and centralized platforms for researchers and clinicians to share their training and measurement protocols.
2. Cultivating cultural and infrastructural support for successful uptake and maintenance of digital health technologies. Ensuring uptake of digital health tools across experience levels and preferences requires that tools are perceived as useful and usable by the target user and are therefore designed in collaboration with representative target users. Those users must have access to training designed to impart the skills and efficacy required to adopt a new technology, and they must have access to technology support staff who are trained to address the needs of the target user in a way that sustains confidence and value toward the technologies. Indeed, all clinical or research staff interacting with individuals using digital health technologies should be trained on how to communicate about the technology in a proactive and positive manner to sustain value and self-efficacy. Integrating digital navigators, who are trained to provide technical support in healthcare, may be one fruitful avenue for minimizing clinical burden during deployment of new digital health technologies. This was underscored in a digital health action plan published by the World Health Organization (2), which emphasized that institutions must cultivate the cultural and infrastructural support needed to drive diverse uptake of new digital health tools (71).
As important is establishing utility, value, and support at the level of the health system. Presently, it is estimated that a primary care physician would require 26 hours per day to meet guidelines for providing preventive, acute, and chronic disease care and associated documentation and inbox management (72). Issuing digital health technologies, providing orientation and technology support, and providing necessary care and feedback therefore requires a team-based approach with funding, training, and clinical norms supporting this role on the clinical care team. Similarly, given the centrality of electronic health record (EHR) systems in clinical care, incorporation of digital health technologies into the EHR in ways that are low-burden and highly valued by the care team will also be a requirement for widespread clinical integration (73).
3. Sustaining interdisciplinary collaboration. It is important to recognize that the development, deployment, and maintenance of useful digital health technologies is a multidisciplinary endeavor. Clearly collaborative computer scientists, human factors engineers, and designers who are well-versed in clinical and behavioral needs will continue to play a critical role in the development of new and more effective digital health technologies. Clinicians, implementation scientists, and health systems professionals are required to understand how to integrate EMA and health-promoting technologies into current and emerging care models. Statistical, data science, and bioinformatic experts are required to develop and share techniques for deriving meaningful results from multiple high-resolution data streams capturing unique self-reported and objective phenomena (74). This expertise is especially important for navigating the benefits and pitfalls associated with increasing integration of artificial intelligence and machine learning models in digital health. Additionally, given that technologies should be developed and refined to meet the needs of the local population, it should be expected that tools and measurement protocols may differ by location. Therefore, methods for harmonizing data from unique measures collected using unique protocols and in unique environments are required. Finally, experts in the behavioral sciences play an important role in considering the myriad indirect effects of supplying novel technologies on overall health and well-being (eg, by providing access to new social connections vs incentivizing isolation via automated delivery of care).
Conclusion
Digital health technologies will play an increasingly important role in gerontology and geriatric medicine, especially for understanding the dynamic changes in health status with aging. In many ways, the application of these tools is in its early days, and workshop attendees identified several key challenges and research opportunities, including (i) the need to identify valid and reliable measures that leverage EMA methodologies, recognizing such an approach captures unique information relative to the traditional use of infrequent summary questionnaires; (ii) the need to develop infrastructure to train and support both older adult participants/patients and staff in the use of these measurement and intervention technologies; (iii) the importance of assembling interdisciplinary teams dedicated to advancing the use of digital health tools in research and clinical practice; and (iv) the importance of following a user-centered design philosophy in the creation, evaluation, and implementation of digital health technologies.
Acknowledgments
We would like to acknowledge and thank workshop organizers, presenters, and attendees.
Contributor Information
Jason Fanning, Department of Health and Exercise Science, Wake Forest University, Winston-Salem, North Carolina, USA.
Tina E Brinkley, Department of Internal Medicine, Gerontology and Geriatric Medicine, Wake Forest University School of Medicine, Winston-Salem, North Carolina, USA.
Laura M Campbell, Department of Psychiatry, University of California San Diego School of Medicine, San Diego, California, USA.
Cristina Colon-Semenza, Department of Kinesiology, University of Connecticut, Storrs, Connecticut, USA.
Sara J Czaja, Division of Geriatrics and Palliative Medicine, Weill Cornell Medicine, New York City, New York, USA.
Raeanne C Moore, Department of Psychiatry, University of California San Diego School of Medicine, San Diego, California, USA.
Nicholas M Pajewski, Department of Biostatistics and Data Science, Wake Forest University School of Medicine, Winston-Salem, North Carolina, USA.
Stephen Kritchevsky, Department of Internal Medicine, Gerontology and Geriatric Medicine, Wake Forest University School of Medicine, Winston-Salem, North Carolina, USA.
Funding
This work was supported by the National Institute on Aging (NIA; U24AG058556). J.F., T.E.B., N.M.P., and S.B.K. are supported by the Wake Forest Claude D. Pepper Older Americans Independence Center (P30AG21332), and N.M.P. is also supported by the National Coordinating Center for the Pepper Centers (U24AG059624). Salary support to R.C.M. provided by NIA (R01AG070956, R01AG062387, R01AG076756), and NIEHS (R01ES025792).
Conflict of Interest
R.C.M. is a cofounder of NeuroUX Inc. and KeyWise, Inc. The terms of these arrangements have been reviewed and approved by the University of California San Diego in accordance with its conflict of interest policies.
Data Availability
This article does not report data and therefore the preregistration and data availability requirements are not applicable.
References
- 1. Lee P, Abernethy A, Shaywitz D, et al. Digital health COVID-19 impact assessment: Lessons learned and compelling needs. NAM Perspect. 2022;1–34. 10.31478/202201C [DOI] [PMC free article] [PubMed]
- 2. World Health Organization. Global strategy on digital health 2020–2025. 2021. https://apps.who.int/iris/bitstream/handle/10665/344249/9789240027633-chi.pdf. Accessed April 12, 2023.
- 3. Moss RJ, Süle A, Kohl S.. eHealth and mHealth. Eur J Hosp Pharm. 2019;26(1):57–58. 10.1136/ejhpharm-2018-001819 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Research Centers Collaborative Network. mHealth and digital health approaches to research in aging. https://www.rccn-aging.org/m-health-digital-health-rccn-workshop. Accessed April 9, 2023.
- 5. Shiffman S, Stone AA, Hufford MR.. Ecological momentary assessment. Annu Rev Clin Psychol. 2008;4(1):1–32. 10.1146/annurev.clinpsy.3.022806.091415 [DOI] [PubMed] [Google Scholar]
- 6. Martin P, Kelly N, Kahana B, et al. Defining successful aging: A tangible or elusive concept? Gerontologist. 2015;55(1):14–25. 10.1093/geront/gnu044 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Stone AA, Schneider S, Smyth JM.. Evaluation of pressing issues in ecological momentary assessment. Annu Rev Clin Psychol. 2022;19(1):107–131. 10.1146/annurev-clinpsy-080921-083128 [DOI] [PubMed] [Google Scholar]
- 8. Phillips SM, Welch WA, Fanning J, et al. Daily physical activity and symptom reporting in breast cancer patients undergoing chemotherapy: An intensive longitudinal examination. Cancer Epidemiol Biomarkers Prev. 2020;29(12):2608–2616. 10.1158/1055-9965.EPI-20-0659 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Van Den Brink M, Bandell-Hoekstra ENG, Huijer Abu-Saad H.. The occurrence of recall bias in pediatric headache: A comparison of questionnaire and diary data. Headache. 2001;41(1):11–20. 10.1046/J.1526-4610.2001.111006011.X [DOI] [PubMed] [Google Scholar]
- 10. Winkielman P, Knäuper B, Schwarz N.. Looking back at anger: Reference periods change the interpretation of emotion frequency questions. J Pers Soc Psychol. 1998;75(3):719–728. 10.1037//0022-3514.75.3.719 [DOI] [PubMed] [Google Scholar]
- 11. Stone A, Schwartz J, Broderick JE, Shiffman SS.. Variability of momentary pain predicts recall of weekly pain: A consequence of the peak (or salience) memory heuristic. Pers Soc Psychol Bull. 2005;31(10):1340–1346. 10.1177/0146167205275615 [DOI] [PubMed] [Google Scholar]
- 12. McDevitt-Murphy ME, Luciano MT, Zakarian RJ.. Use of ecological momentary assessment and intervention in treatment with adults. Focus. 2018;16(4):370–375. 10.1176/appi.focus.20180017 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Smyth JM, Jones DR, Wen CKF, Materia FT, Schneider S, Stone A.. Influence of ecological momentary assessment study design features on reported willingness to participate and perceptions of potential research studies: An experimental study. BMJ Open. 2021;11(7):49154. 10.1136/bmjopen-2021-049154 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Moore RC, Depp CA, Wetherell JL, Lenze EJ.. Ecological momentary assessment versus standard assessment instruments for measuring mindfulness, depressed mood, and anxiety among older adults. J Psychiatr Res. 2016;75:116–123. 10.1016/j.jpsychires.2016.01.011 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Wrzus C, Neubauer AB.. Ecological momentary assessment: A meta-analysis on designs, samples, and compliance across research fields. Assessment. 2023;30(3):825–846. 10.1177/10731911211067538 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Fanning J, Silfer JL, Liu H, et al. Relationships between respiratory sinus arrhythmia and stress in college students. J Behav Med. 2020;43(20):308–317. 10.1007/s10865-019-00103-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Harvey PD, Miller ML, Moore RC, Depp CA, Parrish EM, Pinkham AE.. Capturing clinical symptoms with ecological momentary assessment: Convergence of momentary reports of psychotic and mood symptoms with diagnoses and standard clinical assessments. Innov Clin Neurosci. 2021;18(1–3):24–30. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8195558 [PMC free article] [PubMed] [Google Scholar]
- 18. Cain AE, Depp CA, Jeste DV.. Ecological momentary assessment in aging research: A critical review. J Psychiatr Res. 2009;43(11):987–996. 10.1016/j.jpsychires.2009.01.014 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Daniel DG, Cohen AS, Velligan D, et al. Remote assessment of negative symptoms of schizophrenia. Schizophr Bull Open. 2023;4(1):1–9. 10.1093/schizbullopen/sgad001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Holmqvist SL, Thomas KR, Brenner EK, et al. Longitudinal intraindividual cognitive variability is associated with reduction in regional cerebral blood flow among Alzheimer’s disease biomarker-positive older adults. Front Aging Neurosci. 2022;14:859873. 10.3389/fnagi.2022.859873 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Lövdén M, Li SC, Shing YL, Lindenberger U.. Within-person trial-to-trial variability precedes and predicts cognitive decline in old and very old age: Longitudinal data from the Berlin Aging Study. Neuropsychologia. 2007;45(12):2827–2838. 10.1016/j.neuropsychologia.2007.05.005 [DOI] [PubMed] [Google Scholar]
- 22. MacDonald SWS, Li SC, Bäckman L.. Neural underpinnings of within-person variability in cognitive functioning. Psychol Aging. 2009;24(4):792–808. 10.1037/a0017798 [DOI] [PubMed] [Google Scholar]
- 23. Schmitter-Edgecombe M, Sumida C, Cook DJ.. Bridging the gap between performance-based assessment and self-reported everyday functioning: An ecological momentary assessment approach. Clin Neuropsychol. 2020;34(4):678–699. 10.1080/13854046.2020.1733097 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. Aschenbrenner AJ, Hassenstab J, Morris JC, Cruchaga C, Jackson JJ.. Relationships between hourly cognitive variability and risk of Alzheimer’s disease revealed with mixed-effects location scale models. Neuropsychology. 2024;38(1):69–80. 10.1037/neu0000905 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. NeuroUX. NeuroUX Inc. https://www.getneuroux.com/. Accessed July 17, 2023.
- 26. Mobile Toolbox. Toolbox Assessments, Inc. https://www.mobiletoolbox.org/. Accessed April 25, 2023.
- 27. Nicosia J, Aschenbrenner AJ, Balota DA, et al. Unsupervised high-frequency smartphone-based cognitive assessments are reliable, valid, and feasible in older adults at risk for Alzheimer’s disease. J Int Neuropsychol Soc. 2023;29(5):459–471. 10.1017/S135561772200042X [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Moore RC, Parrish EM, Van Patten R, et al. Initial psychometric properties of 7 neuroux remote ecological momentary cognitive tests among people with bipolar disorder: Validation study. J Med Internet Res. 2022;24(7):e36665. 10.2196/36665 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Öhman F, Berron D, Papp KV, et al. Unsupervised mobile app-based cognitive testing in a population-based study of older adults born 1944. Front Digit Health. 2022;4:227. 10.3389/fdgth.2022.933265/bibtex [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Jutten RJ, Grandoit E, Foldi NS, et al. Lower practice effects as a marker of cognitive performance and dementia risk: A literature review. Alzheimers Dement. 2020;12(1):e12055. 10.1002/dad2.12055 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Moxley J, Sharit J, Czaja SJ.. The factors influencing older adults’ decisions surrounding adoption of technology: Quantitative experimental study. JMIR Aging. 2022;5(4):e39890. 10.2196/39890 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Faverio M. Share of tech users among Americans 65 and older grew in past decade. Pew Research Center; 2022. https://www.pewresearch.org/fact-tank/2022/01/13/share-of-those-65-and-older-who-are-tech-users-has-grown-in-the-past-decade/. Accessed April 3, 2023. [Google Scholar]
- 33. Mamedova S, Pawlowski E.. A description of U.S. adults who are not digitally literate. National Center for Education Statistics; 2018. https://nces.ed.gov. Accessed April 3, 2023. [Google Scholar]
- 34. Haynes N, Ezekwesili A, Nunes K, Gumbs E, Haynes M, Swain JB.. “Can you see my screen?” addressing racial and ethnic disparities in telehealth. Curr Cardiovasc Risk Rep. 2021;15(12):1–9. 10.1007/S12170-021-00685-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Lee CC, Czaja SJ, Moxley JH, et al. Attitudes toward computers across adulthood from 1994 to 2013. Gerontologist. 2019;59(1):22–33. 10.1093/geront/gny081 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36. Hale TM, Cotten SR, Drentea P, Goldner M.. Rural–urban differences in general and health-related internet use. Am Behav Sci. 2010;53(9):1304–1325. 10.1177/0002764210361685 [DOI] [Google Scholar]
- 37. Haggstrom DA, Lee JL, Dickinson SL, et al. Rural and urban differences in the adoption of new health information and medical technologies. J Rural Health. 2019;35(2):144–154. 10.1111/jrh.12358 [DOI] [PubMed] [Google Scholar]
- 38. Calvert JF, Kaye J, Leahy M, Hexem K, Carlson N.. Technology use by rural and urban oldest old. Technol Health Care. 2009;17(1):1–11. 10.3233/THC-2009-0527 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39. Greenberg AJ, Haney D, Blake KD, Moser RP, Hesse BW.. Differences in access to and use of electronic personal health information between rural and urban residents in the United States. J Rural Health. 2018;34:s30–s38. 10.1111/jrh.12228 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Choi NG, DiNitto DM, Marti CN, Choi BY.. Telehealth use among older adults during COVID-19: Associations with sociodemographic and health characteristics, technology device ownership, and technology learning. J Appl Gerontol. 2022;41(3):600–609. 10.1177/07334648211047347 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41. Lattie EG, Stiles-Shields C, Graham AK.. An overview of and recommendations for more accessible digital mental health services. Nat Rev Psychol. 2022;1(2):87–100. 10.1038/s44159-021-00003-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Deci EL, Ryan RM.. Self-determination theory: A macrotheory of human motivation, development, and health. Can Psychol. 2008;49:182–185. 10.1037/a0012801 [DOI] [Google Scholar]
- 43. Bandura A. Self-Efficacy: The Exercise of Control. W. H. Freeman and Company; 1997. [Google Scholar]
- 44. Schrempft S, Jackowska M, Hamer M, Steptoe A.. Associations between social isolation, loneliness, and objective physical activity in older men and women. BMC Public Health. 2019;19(1):74. 10.1186/s12889-019-6424-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45. Cornwell B, Laumann EO.. The health benefits of network growth: New evidence from a national survey of older adults. Soc Sci Med. 2015;125:94–106. 10.1016/j.socscimed.2013.09.011 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. Lindsay Smith G, Banting L, Eime R, O’Sullivan G, van Uffelen JGZ.. The association between social support and physical activity in older adults: A systematic review. Int J Behav Nutr Phys Act. 2017;14(1):56. 10.1186/s12966-017-0509-8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47. Ho EC, Hawkley L, Dale W, Waite L, Huisingh-Scheetz M.. Social capital predicts accelerometry-measured physical activity among older adults in the U.S.: A cross-sectional study in the National Social Life, Health, and Aging Project. BMC Public Health. 2018;18(1):804. 10.1186/s12889-018-5664-6 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48. Hailey V, Fisher A, Hamer M, Fancourt D.. Perceived social support and sustained physical activity during the COVID-19 pandemic. Int J Behav Med. 2022;30(5):1. 10.1007/S12529-022-10125-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49. Bandura A. Social cognitive theory of self-regulation. Organ Behav Hum Decis Process. 1991;50(2):248–287. 10.1016/0749-5978(91)90022-l [DOI] [Google Scholar]
- 50. Heseltine R, Skelton DA, Kendrick D, et al. “Keeping moving”: Factors associated with sedentary behaviour among older people recruited to an exercise promotion trial in general practice. BMC Fam Pract. 2015;16(1):1–9. 10.1186/S12875-015-0284-Z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 51. Schaller A, Rudolf K, Dejonghe L, Grieben C, Froboese I.. Influencing factors on the overestimation of self-reported physical activity: A cross-sectional analysis of low back pain patients and healthy controls. Biomed Res Int. 2016;2016:1497213. 10.1155/2016/1497213 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 52. Fanning J, Roberts S, Hillman CH, Mullen SP, Ritterband L, McAuley E.. A smartphone “app”-delivered randomized factorial trial targeting physical activity in adults. J Behav Med. 2017;40(5):712–729. 10.1007/s10865-017-9838-y [DOI] [PubMed] [Google Scholar]
- 53. Project Catalyst H. Building a better tracker: Older consumers weigh in on activity and sleep monitoring devices. 2016. https://www.aarp.org/pri/topics/health/prevention-wellness/activity-sleep-trackers-older-consumers.html
- 54. Porges SW. The Polyvagal Theory: Neurophysiological Foundations of Emotions, Attachment, Communication, and Self-Regulation. Vol. 21. W.W. Norton & Company; 2011. [Google Scholar]
- 55. Brawley LR, Flora PK, Locke SR, Gierc MSH.. Efficacy of the group-mediated cognitive behavioral intervention: A decade of physical activity research. In: Eys M, Beauchamp M, eds. Group Dynamics in Exercise and Sport Psychology. 2nd ed. Routledge/Psychology Press; 2014:183–202. [Google Scholar]
- 56. Colón-Semenza C, Latham NK, Quintiliani LM, Ellis TD.. Peer coaching through mHealth targeting physical activity in people with Parkinson disease: Feasibility Study. JMIR Mhealth Uhealth. 2018;6(2):e42. 10.2196/mhealth.8074 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 57. Liu Y, Lachman ME.. A group-based walking study to enhance physical activity among older adults: The role of social engagement. Res Aging. 2021;43(9–10):368–377. 10.1177/0164027520963613 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 58. Constandt B, Thibaut E, De Bosscher V, Scheerder J, Ricour M, Willem A.. Exercising in times of lockdown: An analysis of the impact of COVID-19 on levels and patterns of exercise among adults in Belgium. Int J Environ Res Public Health. 2020;17(11):4144–4110. 10.3390/ijerph17114144 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 59. Anderson-Lewis C, Darville G, Mercado RE, Howell S, Di Maggio S.. mHealth technology use and implications in historically underserved and minority populations in the United States: Systematic literature review. JMIR Mental Health. 2018;6(6):e128. 10.2196/mhealth.8383 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 60. Rivers BM, Bernhardt JM, Fleisher L, Green BL.. Opportunities and challenges of using technology to address health disparities. Future Oncol. 2014;10(4):519–522. 10.2217/fon.14.17 [DOI] [PubMed] [Google Scholar]
- 61. Hooker SP, Wilcox S, Rheaume CE, Burroughs EL, Friedman DB.. Factors related to physical activity and recommended intervention strategies as told by midlife and older African American men. Ethn Dis. 2011;21(3):261–267. https://europepmc.org/article/med/21942156. Accessed April 3, 2023. [PubMed] [Google Scholar]
- 62. Griffith DM, Bergner EM, Cornish EK, McQueen CM.. Physical activity interventions with African American or Latino men: A systematic review. Am J Mens Health. 2018;12(4):1102–1117. 10.1177/1557988318763647 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 63. Affuso O, Kinsey AW, Whitt-Glover MC, Segar M, Bowen P.. Social environments and physical activity among active Black women. Am J Health Promot. 2022;36(8):1275–1283. 10.1177/08901171221102139 [DOI] [PubMed] [Google Scholar]
- 64. Dean DAL, Griffith DM, McKissic SA, Cornish EK, Johnson-Lawrence V.. Men on the move-Nashville: Feasibility and acceptability of a technology-enhanced physical activity pilot intervention for overweight and obese middle and older age African American men. Am J Mens Health. 2018;12(4):798–811. 10.1177/1557988316644174 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 65. Davila YR, Reifsnider E, Pecina I.. Familismo: Influence on Hispanic health behaviors. Appl Nurs Res. 2011;24(4):e67–e72. 10.1016/j.apnr.2009.12.003 [DOI] [PubMed] [Google Scholar]
- 66. Hu J, Amirehsani KA, Wallace DC, McCoy TP, Silva Z.. A family-based, culturally tailored diabetes intervention for Hispanics and their family members. Diabetes Educ. 2016;42(3):299–314. 10.1177/0145721716636961 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 67. McEwen MM, Pasvogel A, Murdaugh C, Hepworth J.. Effects of a family-based diabetes intervention on behavioral and biological outcomes for Mexican American adults. Diabetes Educ. 2017;43(3):272–285. 10.1177/0145721717706031 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 68. Ramachandran M, Brinton C, Wiljer D, Upshur R, Gray CS.. The impact of eHealth on relationships and trust in primary care: A review of reviews. BMC Prim Care. 2023;24(1):228. 10.1186/s12875-023-02176-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 69. Chen W, Feng Y, Fang J, et al. Effect of trust in primary care physicians on patient satisfaction: A cross-sectional study among patients with hypertension in rural China. BMC Fam Pract. 2020;21(1):1–13. 10.1186/S12875-020-01268-W [DOI] [PMC free article] [PubMed] [Google Scholar]
- 70. Chipidza FE, Wallwork RS, Stern TA.. Impact of the doctor–patient relationship. Prim Care Companion CNS Disord. 2015;17(5):27354. 10.4088/PCC.15F01840 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 71. Torous J, Bucci S, Bell IH, et al. The growing field of digital psychiatry: Current evidence and the future of apps, social media, chatbots, and virtual reality. World Psychiatry. 2021;20(3):318–335. 10.1002/wps.20883 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 72. Porter J, Boyd C, Skandari MR, Laiteerapong N.. Revisiting the time needed to provide adult primary care. J Gen Intern Med. 2023;38(1):147–155. 10.1007/s11606-022-07707-x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 73. Shapiro M, Renly SR, Maiorano AR, et al. Digital health at enterprise scale: Evaluation framework for selecting patient-facing software in a digital-first health system. JMIR Form Res. 2023;7:e43009. 10.2196/43009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 74. Bent B, Wang K, Grzesiak E, et al. The digital biomarker discovery pipeline: An open-source software platform for the development of digital biomarkers using mHealth and wearables data. J Clin Transl Sci. 2021;5(1):e19. 10.1017/CTS.2020.511 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
This article does not report data and therefore the preregistration and data availability requirements are not applicable.
