Abstract
Collaborations between scientists, care providers, and technology industry professionals are becoming more relevant for developing, testing, and implementing behavioral health technologies. As the need for such partnerships increases, it is important to understand stakeholders’ attitudes about their role in partnering for developing such technologies and how much do they expect technology to impact behavioral research and care. The aim of this study was to investigate how much technology disruption do stakeholders expect in healthcare, as well as their perceived contribution in partnering for developing behavioral health technologies. Stakeholders (N = 74) responded to an online convenience sampling survey. Over 89% of participants reported expecting that technology will bring at least a moderate amount of disruption in the current models of behavioral healthcare, with respondents with the most experience in digital health expecting the most disruption. As for their perception of each other’s role in partnering for developing behavioral health technologies, one group’s weakness was considered to be complemented by another group’s strength. Academics were perceived as having more theoretical and research expertise but being less technology-savvy, while industry professionals were considered to excel at technological and marketing activities. Providers were considered to have the most clinical and real-world healthcare industry expertise. Our results indicate that technology is expected to disrupt current healthcare models, while also highlighting the need for collaboration, as no single group was considered to have sufficient expertise and resources to develop successful, effective behavioral health technologies on its own. These results may contribute to a better understanding of how technology disruption is affecting behavioral healthcare from the standpoint of its key players, which may lead to better collaborative models of research and care delivery.
Keywords: Behavioral health technologies, Healthcare, Stakeholders, Partnerships
INTRODUCTION
The rapid expansion of technology is transforming behavioral science, bringing disruptive innovation in healthcare delivery and research [1–5]. Integrating technology-based tools in healthcare has a tremendous potential to foster behavioral change across a spectrum of health and wellness goals, from smoking cessation to chronic disease management [6, 7]. Indeed, a growing body of research demonstrates the acceptability, effectiveness, and cost-effectiveness of behavioral intervention technologies (BITs) [4], indicating that embracing technology in behavioral healthcare can enhance the quality of care and minimize implementation barriers that traditional behavioral interventions encounter (e.g., lack of time, lack of social support, few incentives for behavioral modifications) [4–10].
The emerging role of technology in behavioral healthcare is also changing how behavioral health scientists conduct their research, as BITs change and evolve over time at a faster rate than the usual academic research process [3, 11–13]. More so, the expansion and adoption of technology use in behavioral healthcare is also disrupting the traditional setting of research. If previously most behavioral health research has been done exclusively in academic or medical settings, at a pace dictated by government research funding, we are now witnessing the emergence of new research models and platforms (e.g., Open mHealth, LifeGuide, Purple) [14–16] that can accommodate both the rapid expansion of technology and the need for evidence-based interventions.
Exchanges and collaborations between scientists and technology industry partners are becoming indispensable for developing, testing, and implementing BITs. Such collaborations may lead to faster development and adoption of new technologies, and they are currently encouraged by NSF and NIH in order to establish a better connection between fundamental science, clinical practice, and technology development, deployment, and use [17, 18]. As the need for such partnerships in developing BITs is becoming more and more recognized, it is important to know how stakeholders (i.e., academics, providers, and industry professionals) feel about their role in partnering for developing BITs and how much do they expect technology to impact behavioral research and care. The goal of the present study was to explore: (a) how much technology disruption do stakeholders’ expect in healthcare and in their own careers, (b) stakeholders’ perceived role and contribution in partnering for developing BITs, (c) stakeholder’s attitudes on the challenges of integrating technology in healthcare delivery, and (d) stakeholders’ opinions on academia-healthcare-industry partnerships. Investigating these variables may contribute to a better understanding of how technology disruption is affecting the behavioral healthcare from the standpoint of its key players, which in turn may lead to better collaborative models of research and care delivery.
METHOD
Participants
Participants were academic researchers, healthcare providers, and industry professionals, recruited through convenience sampling including emails sent to professional associations special groups (e.g., SBM, Digital Health Council), posting on social media and professional sites (e.g., LinkedIn digital health groups) and relevant meet-up groups (e.g., Digital Health Research). Eligibility criteria included being professionally affiliated with the medical or behavioral healthcare field (either in an academic, healthcare, or industry setting), ability to read English, and having Internet access. Out of 105 participants who opened our survey, 74 completed it (70% participation). The survey was discontinued when repeated posting did not yield new participants.
The survey was completed during March, 2015. All responses were anonymous. We collected limited demographic information aside from professional roles. Of those who completed the survey, 44.6% identified themselves as academics, 45.9% as having overlapping hybrid roles within academia and industry, and 9.5% as pure industry professionals. Approximately 70% self-described has having moderate or more expertise, reported in technology-based behavioral tools, while 60% had participated in at least one digital health conference. The majority (93.2%) have assisted to the development of at least one BIT product. Work setting information is presented in Table 1.
Table 1.
Professional information (n = 74)
| Professional role/setting | Percentagea |
|---|---|
| Academic research | 70.2% |
| Academic teaching | 29.8% |
| Mental health provider | 17.3% |
| Medical provider | 11.2% |
| Professional working in a start-up | 11.2% |
| Professional working in a company | 10.6% |
| Professional working in a nonprofit/governmental institution | 13.5% |
aCategories are not mutually exclusive
Procedure
A description of the study was posted through the online resources described above, along with a link to the web-based survey. The recruitment message read “We are conducting a survey on how academics, providers and those in industry feel about their roles and contribution in building tech tools for behavioral health. We are trying to understand how we can build successful partnerships. Please complete this brief survey to help us understand how we can best do this and understand the motivations of all stakeholders.” Interested participants could follow a link to an external, web-based survey hosted by Survey Monkey, which has been used as a survey host in numerous research studies. Because we did not compensate participants and the survey was anonymous, we did not add in any specific safeguards such as reverse coded responding. We included IP blocking so participants could only complete the survey once and disabled the back button so participants could not change their answers based on triggers from subsequent questions. No personal identification or health information was collected. Participant responses were stored in a password-protected online database, which was created for the present study and then transferred to an institutional secure server.
Survey
The survey was developed for the purpose of this study and included (1) professional information items (i.e., participants’ professional setting, previous partnerships experience, previously attended digital health conferences, and their expertise with behavioral health technology, (2) the expected disruption of current behavioral healthcare delivery models due to technology (i.e., “How much do you think technology will disrupt/change our current models of behavioral healthcare delivery?”, Likert item, 0 = No disruption, 10 = complete disruption), (3) expected percentage of automated healthcare delivery in the next 10 years (i.e., “What percentage of current person-to-person behavioral healthcare delivery do you think will be automated in 10 years?”, Likert item, 0% = No automation, 100% = Complete automation), (4) expected job disruption due to technology (i.e., “How much do you think technology will disrupt/change your job and role in its current form”, Likert item, 0 = No disruption, 10 = Complete disruption), (5) anxiety about job security (i.e., “How anxious/concerned are you about your job security in the next 10 years”, Likert item, 0 = Not at all concerned, 10 = Extremely concerned), (6) perceived strengths and weaknesses of each professional group in BITs development (i.e., “When it comes to building behavioral health technology products, what do you think are the greatest strength and weakness, respectively, of academic behavioral health professionals (e.g., academic researchers)/behavioral health providers (e.g., clinicians)/industry professionals (e.g., industry technology behavioral health companies experts)”, 3 open ended items), (7) perceived challenges of integrating BITs in health care delivery (i.e., “Overall, what do you believe are the greatest challenges we face when integrating technology based tools into behavioral healthcare?” open-ended item), and (8) ensuring effective partnerships (i.e., “What do you believe is needed to create the most effective partnerships between all stakeholders in behavioral health technology development?”, open-ended item).
Statistical analysis
Quantitative data were analyzed with descriptive and inferential statistics (descriptive statistics bivariate correlations, regressions, and t tests) using SPSS, version 22. Qualitative data was analyzed by using a hybrid process of inductive and deductive thematic analysis to (i.e., identification of themes within the data) [19]. The following steps were undertaken (adapted from Fereday and Muir-Cochrane, 2006 [19]). In the first step, the authors familiarized themselves with the data by reading the open-ended answers. The second step involved generating the initial codes (i.e., themes derived from participants’ responses) and discussing them within the research team. The third step involved a code manual, which involved the following steps: (1) establishing the code title or name, (2) a definition of what the themes concerns, and (3) a description of how to know when the theme occurs (e.g., examples and quotations). The fourth step involved testing the reliability of codes by independently identifying them in the data and analyzing the inter-rated reliability. Two independent coders performed this analysis, obtaining an inter-rated reliability of 91.0%. For the final step, all disagreements were discussed with a third author with reference to the original text (participant’s response) until consensus was reached and a frequency count was performed.
RESULTS
Expected disruption of current behavioral healthcare delivery models due to technology
Over 89% of participants reported expecting that technology will bring at least a moderate amount of disruption in the current models of behavioral healthcare delivery. When looking at responses by professional setting, 58% of academics reported expecting that technology will bring an almost complete disruption of current behavioral healthcare delivery models, compared with 89% of industry professionals and 50% of professionals working in hybrid settings (see Table 2 for a more detailed analysis).
Table 2.
Attitudes about technology disruption by stakeholders’ professional setting
| Professional setting (n = 74) | Expected disruption of behavioral healthcare models due to technology | Expected automation of person-to-person behavioral healthcare delivery due to technology | Expected change job/professional role due to technology | Anxiety about job security in the next 10 years | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Almost none | Moderate | Almost complete | Less than 30% | About 50% | Over 70% | Almost none | Moderate | Almost complete | Not concerned at all | Moderate | Extremely concerned | |
| Academia (n = 33) | 6% (2) | 36% (12) | 58% (19) | 27% (9) | 52% (17) | 21% (7) | 39% (13) | 52% (17) | 9% (3) | 55% (18) | 42% (14) | 3% (1) |
| Hybrid (n = 32) | 12% (4) | 38% (12) | 50% (16) | 12% (4) | 66% (21) | 22% (7) | 25% (Rogler et al.) | 53% (17) | 22% (7) | 78% (25) | 19% (6) | 3% (1) |
| Industry (n = 9) | 0% | 11% (1) | 89% (Rogler et al.) | 0% | 56% (5) | 44% (4) | 22% (2) | 56% (5) | 22% (2) | 67% (6) | 33% (3) | 0% |
Expected disruption due to technology was associated with perceived expertise in technology-based behavioral intervention and assessment, r (74) = .23, p = .04, such that participants with higher expertise predicted more disruption due to technology. It was also associated with the number of attended digital conferences, r (74) = .23, p = .02, such that those attending more conferences predicted more disruption. Professional variables did not affect participants’ expectations for technology disruption, all ps > .05. When perceived expertise and number of attended digital conferences were entered into a linear regression model, only perceived expertise remained significantly associated with expected disruption, (β = .43, p < .001). The overall model fit was R 2 = .24, F (2.70) = 11.84, p < .001. No other professional variables affected participants’ expectations for technology disruption, all ps > .05.
Expected percentage of automated healthcare delivery in the next 10 years
Over 82% of participants reported expecting that at least half of the person-to-person healthcare delivery will become automated in the next 10 years. Twenty-one percent of academics reported expecting that almost the entire current person-to-person behavioral healthcare delivery will be automated in the next 10 years, compared with 44% of industry professionals and 22% of professionals working in hybrid settings (see Table 2 for a more detailed analysis).
Expected percentage of automated healthcare delivery was associated with perceived expertise in technology-based behavioral intervention and assessment, r (74) = .36, p < .001, and with the number of attended digital conferences, r (74) = .32, p < .001, such that participants with higher expertise who had attended more conferences predicted more automation in the next 10 years. Although when these variables were entered into a linear regression model, none of individual variables remained significant (although a trend was observed for perceived expertise, β = .23, p = .08), the overall model fit was R 2 = .16, F (2.70) = 7.01, p = .002. No other professional variables affected participants’ expectations for behavioral healthcare automation, all ps > .05.
Expected job disruption due to technology
Over 52% of participants reported expecting at least a moderate disruption in their current job/professional role. Almost complete disruption in their role due to technology was predicted by 9% of academics, compared with 22% of industry professionals and 22% of professionals working in hybrid settings (see Table 2 for a more detailed analysis).
Expected job disruption was also associated with the number of attended digital conferences, r (74) = .32. Expected disruption was not equally distributed among professional work settings, with providers being less likely to expect disruption than academics or industry professionals, X 2 (N = 74) = 41.97, p = .03. Although when number of attended digital conferences and professional role were entered into a linear regression model, none of individual variables remained significant (ps > 05), the overall model fit was R 2 = .08, F (2.70) = 3.43, p = .038.
Anxiety about job security
Over 31% of the participants reported feeling at least a moderate level of anxiety about their job security in the next 10 years (see Table 2 for a more detailed analysis). Anxiety about job security was associated with expected disruption of job in its current form, r (74) = .23, p = .04. No other professional variables affected participants’ anxiety about job security, p > .05.
Perceived strengths and weaknesses of each professional group in BITs development
When asked about the strengths of academics in developing BITs, 19% of the participants reported that academics use evidence-based, scientific approaches (e.g., “academics make decisions based on data, research and evidence-based information”), 20% reported that academics have expertise in behavior change theories, 21% reported that academics have expertise in research design and analysis, and 15% praised academics’ unbiased curiosity and motivation. When asked about the weaknesses of academics in developing BITs, 17% of participants pointed to that research and development can be slow in academia (“takes too long to get things done”), 16% considered that researchers in academia have a limited familiarity with technology (“not savvy on the tech aspect”), and 18% pointed to academics’ limited resources to develop BITs (“often don’t have the infrastructure”).
Sixteen percent of the participants reported that providers understand patients’ needs (“they are tuned in to patients’ needs”), and 14% pointed to their practical knowledge of healthcare setting (“clinicians understand realities of clinical practice”). When asked about the weaknesses of providers when developing BITs, 15% of the participants pointed to the fact that providers may have a limited familiarity with technology, with the same percentage of participants also indicating that providers might lack time, financial resources, and interest to try new technologies (“time to engage with new technologies can be limited”; “unwilling to use technology”).
Twenty-seven percent of the participants reported that industry professionals have expertise with technology, 16% reported that industry professionals have business and marketing skills (“they know what sells”), and 7% reported that industry professionals have the resources to invest in technology development and research (“resources [people and money] to develop new products”). Fifteen percent of the respondents also reported that industry professionals are more nimble in terms of research and development (“can move quickly and get product to the market fast”). With regard to weakness, 26% of the participants pointed to the fact that industry professionals might be favoring a business approach rather than an evidence-based one (“more concerned with marketability than with efficacy”), 23% considered that industry professionals have a limited understanding of healthcare settings (“lack of knowledge of behavior change theory/techniques”; “lack of understanding of healthcare systems and providers and how they provide care”), and 10% considered that they also have a limited clinical expertise (“lack of clinical knowledge”).
Perceived challenges of integrating BITs in health care delivery
The most frequently (23%) reported challenges were providers’ resistance (“resistance to change from current providers”) and the lack of appropriate models for implementing technical innovations (22%, “lack of appropriate models of care that can absorb these innovations”).
Ensuring effective partnerships
When asked about their opinions on how partnerships can be improved, 50% of respondents indicated a need to make sure there is a clear agreement on the terms of the partnership when approaching BITs projects together (“agreed upon values, motivation, timelines and commercialization plans”).
DISCUSSION
Our results indicate that the majority of stakeholders expect at least a moderate disruption of current healthcare models due to technology expansion, predicting that at least 50% of person-to-person behavioral healthcare delivery will become automated in the next 10 years. As the healthcare system is under constant pressure from politics and public to increase service levels and decrease escalating costs and as new technologies and communication media are becoming more ubiquitous, it is perhaps of no surprise that professionals expect technology to permeate a considerable amount of healthcare delivery [20–22]. Professionals in the behavioral healthcare space are expecting disruption and see the shift in models despite few tangible changes in care delivery. Not surprisingly, those having more experience (either in the form of attending digital health conferences or in the more applied form of actually participating in the development of BITs) reported expecting more disruption and believed that a greater percentage of healthcare would be automated in the next 10 years. Interestingly, while there were trends indicating that the professional role might impact participants’ predictions on technology disruptions, with industry professionals expecting a slightly higher technology disruption, level of expertise was the only significant predictor of expected technology disruption. This prediction may be a result of knowledge about technology, but also it might stem from the fact that those with greater expertise have more to gain from greater disruption, and they might make more optimistic predictions.
As for expected job disruption, over half of the sample expected at least a moderate disruption in their current professional role due to technology. Of all professional work settings, providers in clinical settings were significantly less likely to expect that their job would be disrupted. These findings could be explained by the fact that innovation is exceedingly slow in day-to-day practice. It is possible that since technology has not yet infiltrated these settings, providers are not seeing actual changes despite the hype around health technology. At the same time, since previous research indicates that the success of technology as a tool to support the delivery of healthcare is tributary to its adoption by healthcare providers [23, 24], we can speculate that the expected disruption could be influenced by providers’ willingness to adopt new technologies in their day-to-day practice. Another variable associated with participants’ predictions for job disruption due to technology was attending digital health conferences, possibly because participants endorsing those predictions were more familiar and perhaps more invested in this topic. However, as none of these variables was significant predictors of job disruption, future research is needed to investigate a wider array of relevant variables.
The great majority of professionals reported feeling secure about their careers. Fifty-five percent of academics, 78% of the professionals in hybrid settings, and 67% of industry professionals reported having no concerns at all about their job security. As such, anxiety about job security was fairly low in this sample, with only 31% reporting moderate anxiety. The association between job anxiety and expected job disruption due to technology highlights that a subgroup may have anxiety due to the uncertainty of one’s role in an increasingly digital world. Because those in academic settings were more likely to have at least some concerns about career disruption, more research is needed to understand the concerns of this subgroup. Overall, results suggest that individuals are expecting disruption, but are preparing to adapt to an increasingly digital world.
The strengths and weaknesses of each professional category strongly highlighted the need for partnerships between academics, providers, and industry professionals. Academics were perceived as experts in research methods and behavior change theory, providers on clinical workflow and care, and industry professionals on the actual technology and implementation. At the same time, one group’s weakness was complemented by another group’s strength. Academics were perceived as less technology-savvy, while industry professionals were considered to excel at that. Industry professionals might lack clinical skills and knowledge, but providers can compensate with their clinical expertise. These results further indicate the need for collaboration, as no single group was considered to have sufficient expertise and resources to develop successful, effective BITs on its own. This has important ramifications for the current models of care and research, which at the moment assume that these categories of professionals work independently in siloed settings to produce, investigate, and implement such BITs.
The perceived challenges of integrating BITs in healthcare further spoke for the need to develop better partnership models. Participants considered providers to be reluctant to use technology innovations, a result that needs to be further confirmed by future research. Such reluctance, if proven, could be justified by the fact that organization change is slow and most healthcare settings comply with traditional reimbursement models which can be at odds with behavioral health technologies. Data security within traditional settings also represents a significant barrier to adoption in traditional healthcare settings. Another frequently noted challenge reported by our participants was that currently there are no research frameworks that can investigate and absorb these innovations in a timely manner. The gold standard of research, randomized clinical trials, are notoriously long—the average duration of 5.5 years from enrollment to publication, which, considering the speed of technology development, risks of rendering an intervention obsolete by the time its proven to be effective [25]. To respond to this challenge, new frameworks were already proposed and are currently beginning to be more widely used for investigating technology in healthcare. For example, Mohr and colleagues proposed the Continuous Evaluation of Evolving Behavioral Intervention Technologies (CEEBIT) framework as an alternative to RCTs [12], a methodology that can be used for testing health apps, as it is statistically powered to continuously evaluate their efficacy throughout trial duration and accounts for changing app versions through a sophisticated elimination process [26]. Similarly, newer health technology evaluation frameworks are being developed to include therapeutic persuasiveness and other nontraditional metrics to help understand this new medium beyond the clinical trial. In addition to alternative research frameworks, dissemination models with multiple stakeholders are also necessary. Newer models of dissemination such as collaborative consortiums with different stakeholders (e.g., Junto Health) are attempting to increase dissemination, but integration will be slow until alternative models of research and dissemination are validated.
As healthcare technology expands, the key stakeholders need to understand how to work with each other, from obtaining funding to adopting new behavioral health technologies into practice. The most frequent answer when asked about ensuring an effective academia-industry partnership was agreeing on values, motivation, timelines, and commercialization plans. Moreover, groups need to start with a common language that supports collaborations. These results might inform universities and companies about the need to offer collaboration training and collaboration platforms in order to ensure successful partnerships that can lead to more effective BITs.
LIMITATIONS AND FUTURE RESEARCH
This study had several limitations that affect the reliability and validity of the findings. First, the study had a small, convenience sample that was obtained by targeting professional groups focused on digital health interventions and was therefore skewed towards individuals with interest and expertise in the development of behavioral intervention technologies. More so, the sample did not equally represent the three professional categories included in the study (e.g., academics, care providers, and industry professionals), as the majority of our respondents described themselves as academics and hybrid academic-industry professionals. As such, caution needs to be taken when generalizing these results to other professional groups. Similarly, because there was no compensation, only a small fraction of the target population completed the survey which may have resulted in additional skewing of the population. While these are limitations, those with the most digital health experience are also those with the most knowledge about the strengths and weakness of different professions on digital health development and implementation. Second, the survey was administered and completed online anonymously. Thus, all respondents had at least a minimal level of Internet skill and computer ability. It is possible that a paper-and-pencil survey approach might have reached a broader sample of respondents, including those who are less comfortable with computer/Internet use. Third, due to the fact that there was no validated preexistent measure for stakeholders’ attitudes on technology disruption and academia-industry partnerships, we developed and used a new, face-valid measure that has not yet been validated. Fourth, the survey was designed to keep participants’ burden to a minimum by specifically focusing on a limited array of professional variables. More so, institutional rules and regulations (e.g., licensing, intellectual property) about partnerships may affect professionals’ attitudes. Additional information could potentially improve our understanding of stakeholders’ attitudes towards forming partnerships and towards technology disruption in general. These limitations may pave the way for future studies, which could explore predictors of attitudes more broadly.
IMPLICATIONS
This study addresses and fills a gap in knowledge by providing a timely insight into professionals’ attitudes towards the use of technology in behavioral healthcare and towards the partnerships needed in the process. Developing effective behavioral health technologies depends on strong partnerships among academia, healthcare, and industry, as no single group has sufficient expertise and resources to develop successful, effective behavioral health technologies on its own. Understanding stakeholders’ attitudes contributes to a better understanding of how technology disruption is affecting the behavioral healthcare from the standpoint of its key players, which in turn may lead to better collaborative models.
Footnotes
Implications
Policy: Our findings have implications for research policy, shedding light on a topic that is becoming more and more relevant in the behavioral medicine field and catalyzing conversation on developing better collaborative models of research and care delivery.
Research: We believe our study has also has research implications, as understanding stakeholders’ attitudes towards technology, and the need for academia-industry partnerships when employing it in healthcare will contribute to a better understanding of how technology disruption is affecting the behavioral healthcare from the standpoint of its key players.
Practice: The study also has implications for practice, as it will spark debate on practice frameworks that can accommodate and benefit from technology disruption.
References
- 1.Nilsen W, Kumar S, Shar A, et al. Advancing the science of mHealth. Journal of Health Communication. 2012;17(Suppl 1):5–10. doi: 10.1080/10810730.2012.677394. [DOI] [PubMed] [Google Scholar]
- 2.Marsch LA, Gustafson DH. The role of technology in health care innovation: a commentary. Journal of Dual Diagnosis. 2013;9:101–103. doi: 10.1080/15504263.2012.750105. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Mohr D, Schueller S, Riley W, Brown CH, Cuijpers P, Duan N, et al. Trials of intervention principles: evaluation methods for evolving behavioral intervention technologies. Journal of Medical Internet Research. 2015;17(7):e166. doi: 10.2196/jmir.4391. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Mohr DC, Burns MN, Schueller SM, Clarke G, Klinkman M. Behavioral intervention technologies: evidence review and recommendations for future research in mental health. General Hospital Psychiatry. 2013;35(4):332–338. doi: 10.1016/j.genhosppsych.2013.03.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Oh H, Rizo C, Enkin M, Jadad A. What is eHealth (3): a systematic review of published definitions. Journal of Medical Internet Research. 2005;7(1):e1. doi: 10.2196/jmir.7.1.e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Eckard C, Asbury C, Bolduc B, et al. The integration of technology into treatment programs to aid in the reduction of chronic pain. Journal of Pain Management & Medicine. 2016;2(3):118. [PMC free article] [PubMed] [Google Scholar]
- 7.Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental health smartphone apps: review and evidence-based recommendations for future developments. Eysenbach G, ed. JMIR Mental Health. 2016;3(1):e7. doi:10.2196/mental.4984. [DOI] [PMC free article] [PubMed]
- 8.Teyhen DS, Aldag M, Edinborough E, et al. Leveraging technology: creating and sustaining changes for health. Telemedicine and E-Health. 2014;20:835–849. doi: 10.1089/tmj.2013.0328. [DOI] [PubMed] [Google Scholar]
- 9.Wang CJ, Huang AT. Integrating technology into health care what will it take? JAMA-Journal of the American Medical Association. 2012;307:569–570. doi: 10.1001/jama.2012.102. [DOI] [PubMed] [Google Scholar]
- 10.Schueller SM, Munoz RF, Mohr DC. Realizing the potential of behavioral intervention technologies. Current Directions in Psychological Science. 2013;22(6):478–483. doi: 10.1177/0963721413495872. [DOI] [Google Scholar]
- 11.Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, et al. Mobile health technology evaluation: the mHealth evidence workshop. American Journal of Preventive Medicine. 2013;45(2):228–236. doi: 10.1016/j.amepre.2013.03.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Mohr DC, Cheung K, Schueller SM, Hendricks BC, Duan N. Continuous evaluation of evolving behavioral intervention technologies. American Journal of Preventive Medicine. 2013;45(4):517–523. doi: 10.1016/j.amepre.2013.06.006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clinical Translational Medicine. 2013;2(1):10. doi: 10.1186/2001-1326-2-10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Estrin D, Sim I. Health care delivery. Open mHealth architecture: an engine for health care innovation. Science. 2010;330(6005):759–760. doi: 10.1126/science.1196187. [DOI] [PubMed] [Google Scholar]
- 15.Williams S, Yardley L, Wills GB. A qualitative case study of LifeGuide: users’ experiences of software for developing Internet-based behaviour change interventions. Health Informatics Journal. 2013;19(1):61–75. doi: 10.1177/1460458212458915. [DOI] [PubMed] [Google Scholar]
- 16.Schueller SM, Begale M, Penedo FJ, Mohr DC. Purple: a modular system for developing and deploying behavioral intervention technologies. Eysenbach G, ed. J Med Internet Res. 2014; 16(7): e181. [DOI] [PMC free article] [PubMed]
- 17.NSF: Smart and Connected Health (SCH) Available at https://www.nsf.gov/funding/pgm_summ.jsp?pims_id=504739. Accessibility verified on September 9 2016.
- 18.NIH: CRADAs. Available at https://www.ott.nih.gov/cradas. Accessibility verified on September 9 2016.
- 19.Fereday J, Muir-Cochrane E. Demonstrating rigor using thematic analysis: a hybrid approach of inductive and deductive coding and theme development. International Journal of Qualitative Methods. 2006;5(1):80–92. doi: 10.1177/160940690600500107. [DOI] [Google Scholar]
- 20.Booth RG. Informatics and nursing in a post-nursing informatics world: future directions for nurses in an automated, artificially intelligent. Social-Networked Healthcare Environment. Nursing Leadership. 2016;28(4):61–69. doi: 10.12927/cjnl.2016.24563. [DOI] [PubMed] [Google Scholar]
- 21.Chowdhury B, D’Souza C, Sultana N. The use of emerging technology to improve the performance of health service delivery. Tencon 2009–2009 Ieee Region 10 Conference, Vols 1–4, 134−+.
- 22.Posadzki P, Mastellos N, Ryan R, Gunn LH, Felix LM. Automated telephone communication systems for preventive healthcare and management of long-term conditions. Cochrane Database of Systematic Reviews(12). 2006; Artn Cd009921 doi: 10.1002/14651858.Cd009921.Pub2 [DOI] [PMC free article] [PubMed]
- 23.Gagnon MP, Desmartis M, Labrecque M, Car J, Pagliari C. Systematic review of factors influencing the adoption of information and communication technologies by healthcare professionals. Journal of Medical Systems. 2012;36(1):241–277. doi: 10.1007/s10916-010-9473-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Gagnon MP, Ngangue P, Payne-Gagnon J, Desmartis M. m-Health adoption by healthcare professionals: a systematic review. Journal of American Medical Informatics Association. 2016;23(1):212–220. doi: 10.1093/jamia/ocv052. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Ioannidis JP. Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. Journal of the American Medical Association. 1998;279(4):281–286. doi: 10.1001/jama.279.4.281. [DOI] [PubMed] [Google Scholar]
- 26.Pham Q, Wiljer D, Cafazzo JA. Beyond the randomized controlled trial: a review of alternatives in mHealth clinical trial methods. JMIR Mhealth Uhealth. 2016;4(3):e107. doi: 10.2196/mhealth.5720. [DOI] [PMC free article] [PubMed] [Google Scholar]