Methodology and Reporting of Mobile Heath and Smartphone Application Studies for Schizophrenia

Abstract

The increasing prevalence of mobile devices among patients of all demographic groups has the potential to transform the ways we diagnose, monitor, treat, and study mental illness. As new tools and technologies emerge, clinicians and researchers are confronted with an increasing array of options for clinical assessment, through digital capture of the essential behavioral elements of a condition, and intervention, through formalized treatments, coaching, and other technology-assisted means of patient communication. And yet, as with any new set of tools for assessment or treatment of a medical condition, establishing and adhering to reporting guidelines – i.e., what works and under what conditions – is an essential component to the translational research process. Here, we review the methodological strengths and weaknesses in the existing literature on schizophrenia smartphone and wearables utilizing the recently published World Health Organization mHealth Evaluation, Reporting and Assessment (mERA) guidelines for evaluating mobile health applications. While growing evidence supports the feasibility of using several mobile tools in severe mental illness, most studies to date failed to adequately report accessibility, interoperability, costs, scalability, replicability, data security, usability testing, or compliance with national guidelines or regulatory statutes. Future research efforts addressing these specific gaps in the literature will help advance our understanding and realize the clinical potential of these new tools of psychiatry.

Introduction

Few would dispute the unmet need in mental health services for individuals with severe mental illness, both in terms of access to services and ability to provide high quality services at the point of care¹. While research into the genetic and neural mechanisms of schizophrenia has yielded important breakthroughs, these discoveries have not yet transformed care for patients. Duration of untreated psychosis ranges from many months to many years³, reflecting the formidable challenge faced in screening for serious mental illness. Once treatment is initiated, patients' most pressing illness concerns, such as getting back to work or school (i.e. function), and gaining an understanding of how their symptoms are likely to evolve over time with and without intervention (i.e. prognosis) are often not fully addressed. Antipsychotic medications, once viewed as a panacea for schizophrenia when first developed in the 1950s, are largely ineffective for many of its symptoms including cognitive and negative symptoms² and are often accompanied by adverse effects that can carry significant lifelong morbidity. Patients and providers are understandably eager for innovative approaches to address any of these challenges that might improve access or outcomes for individuals with severe mental illness.

Mobile and connected technologies are one such prominent example, offering tremendous opportunities to address some of the real-world difficulties in working with individuals in schizophrenia and other forms of severe mental illness. As detailed below, increased ownership of smartphones coupled with increasingly sophisticated sensor and communication technologies has created new opportunities for innovation. And yet, as the field seeks to understand and validate these tools, considerable challenges remain in both the implementation and potential uses (and misuses) that must be thoroughly vetted as with any new proposed tool or treatment modality. Here we aim to present a balanced view of both the opportunities and challenges in the field of mobile health and smartphone application studies for schizophrenia, through the lens of the strengths and weaknesses in currently published accounts of mobile applications. While the scope of the present review was limited to the mobile health application literature in schizophrenia, we anticipate the insights gleaned from our focused study will extend to other areas of psychiatry where similar technology-based solutions are being applied and tested.

Increased Smartphone Access

While barriers still exist, smartphone use among patients with serious mental illness is rising⁵. On the one hand, socioeconomic factors such as education, income, and cultural norms likely keep modern technologies out of reach for many individuals⁶. However, the idea of a “digital divide” seems to be rapidly closing, with recent evidence suggesting that individuals with schizophrenia now own smartphones at rates closer to the general population than ever before^6,7. A National Alliance of Mental Illness (NAMI) study from 2014 also surveyed those with schizophrenia and found that in a sample of 451 patients, 54% had access to a smartphone⁸. Younger patients, especially those with first episode schizophrenia, seem especially interested in using digital technologies like smartphones as part of their care^9,10. Data from the NAMI study also suggested that patients may already be using their phones for therapeutic purposes with 42% of respondents reporting that they often or very often listen to music or audio files on their devices to help manage or block voices⁸.

Advances in Device Capabilities

The increased ownership and interest in smartphones for care in schizophrenia parallels the increasing technical capabilities of these devices to track relevant features of psychiatric syndromes. Not only can smartphones capture patients' real time symptoms via surveys and other brief interactions, but they can also be harnessed to collect more objective social and behavioral measurements. Current and in progress studies are collecting data from smartphone sensors to shed light on how patients with schizophrenia experience the illness¹¹.

Smartphones can collect what is known as ‘active data’ such as symptom surveys or voice recording samples that the patient agrees to provide. Real time symptom surveys delivered on the phone have the potential to minimize recall bias and provide more accurate reporting; and voice data has been shown to be a predictor of conversion to schizophrenia in those at risk¹² and is the subject of ongoing research projects. But smartphone platforms can also capture ‘passive data’ which is collected without the active involvement of the patient. For example, utilizing global positioning system (GPS) data from phones, it is possible to learn about the mobility traces of patients and how active they are outside of the clinic or hospital. Recording anonymized call and text logs can provide data on how social and engaged patients may be. Wearable sensors and smart watches can now also collect physiological data, such as heart rate, galvanic skin conductance, and sleep patterns. The ability to collect large quantities of personal data from patients also raises ethical concerns¹³, and it is therefore imperative that such data is used to help patients rather than to profile them. But the ability to collect such data also raises hopes that, when combined with careful analysis and appropriate statistical methods, it may reveal new markers, patterns, and views onto schizophrenia and other mental illnesses via novel ‘digital phenotypes’^11,15.

Mobile Intervention

In addition to monitoring symptoms, smartphones also have the potential to offer adjunctive therapies and treatments for patients with both mild and severe forms of mental illness. On the milder side of the illness spectrum, app-based interventions could provide self-management tools that meet the patient's need without engaging a costly healthcare infrastructure, much like “diet and exercise” have become the first-line therapy for many cardiometabolic conditions when confronted at a mild stage. Conversely, in severe illness, app-based adjunctive interventions could help patients with treatment adherence, between-visit check-ins, and other prompts designed to promote positive behavioral change.

Research has shown that patients with schizophrenia find text message reminders about medication adherence delivered to their phone useful and easy to use¹⁶. Therapies like cognitive behavioral therapy (CBT) are increasingly being shown effective in schizophrenia, showing high acceptability among patients and small-medium between group effects for psychotic symptoms, ¹⁷ and there is a growing research literature on the ability of smartphones to deliver CBT¹⁸. While evidence for CBT on a smartphone for schizophrenia is limited, studies have demonstrated the ability of interventions delivered via the phone to increase motivational behaviors and quality of life in patients with schizophrenia¹⁹. A systematic review of smartphones apps for schizophrenia published in 2015 noted overall high rates of feasibility and acceptability of use among patients, although a lack of efficacy data²⁰.

There is also a robust body of evidence regarding the use of computers to deliver cognitive remediation to patients with schizophrenia and ongoing research to assess how smartphones can deliver such therapy to patients²¹. In addition, the large screens and increasingly faster data connections of smartphones make them practical platforms for telepsychiatry – offering the potential of easy to access digital visits.

Evidence is lacking; studies and standards needed

Despite the potential and encouraging early study results, the impact of smartphones and apps to change psychiatric clinical care has not yet been realized. Neither psychiatric care nor schizophrenia treatment can be formulated as engineering problems that will be solved with a new technology alone. Any solutions or advancement in these areas can only come through considering the myriad of personal and social factors involved, careful clinical investigation of new interventions, and strongand reproducible science. While the number of clinical studies on smartphones and connected devices for schizophrenia care remains limited²¹, there has been no investigation to date on the methods and reporting of mobile health studies for schizophrenia.

At this early stage, examining the methods and reporting of mobile health (mHealth) studies in schizophrenia is perhaps currently of greater value than understanding app-based studies solely in terms of their individual outcomes, due to the myriad of potential clinical targets for schizophrenia along with the constant increase in the number (and capabilities) of mental health apps. Understanding the quality, completeness, and objectivity²² of studies is important when considering and evaluating their results. Furthermore, comparing existing research methodologies in a standardized manner can highlight broad gaps in study methods, which can advance research conduct for future studies. Encouraging better methodologies and thorough reporting of methods and results will in turn lead to better knowledge of how these technologies can be used for clinical care.

Just as smartphones and apps are a relatively new technology, standards for reporting the results of clinical studies on these technologies are also new. In early 2016 the World Health Organization released the mHealth Evidence and Assessment (mERA) checklist. The checklist offers 16 criteria developed by expert consensus and field trials to define the minimum information necessary to understand the content, context, technical features, and reproducibility of mHealth studies. It is important to note that “The checklist does not aim to support the design or implementation of such studies, or to evaluate the quality of the research methods used. Rather, it is intended to improve transparency in reporting, promote a critical assessment of mHealth research evidence, and help improve the rigor of future reporting of research findings.”²² While mERA checklist has not been well validated or widely used in the literature at the time this paper was written, it may offer a useful framework to guide a discussion around mobile health research through ensuring many disparate aspects of research methodology are covered. While not specific to psychiatry or schizophrenia research, the mERA framework may still serve as a tool evaluate the existing literature and study the reporting of clinical studies for smartphone interventions in a standardized manner.

Methods

The aim of this systematic review was to identify all existing studies of smartphone interventions for schizophrenia, and evaluate their methodology and reporting against pre-established criteria for app-based studies in psychiatric research. We chose to study schizophrenia as it is a clinically important topic, research in this condition is representative of the state of mobile psychiatry efforts, and the literature is well-defined because of widespread agreement regarding the definition and treatment of schizophrenia, as compared to mood disorders which are a much broader and diverse category.

We conducted a literature review searching for all studies utilizing smartphones in the care of patients with schizophrenia. We based our search off a prior 2015 review paper on smartphones and fitness trackers for schizophrenia. Details of the search criteria can be found in an earlier paper¹⁷. The search was extended by approximately one year from May 24, 2015 to July 24, 2016. The extended search identified three new studies in addition to the eight studies identified by the earlier review. A copy of the mERA checklist is shown below in Figure 1²².

Figure 1. The mERA Checklist.

For each of the eleven total studies, three of the authors (JT, JF, and NM) applied the 16-item mERA checklist and noted whether each item on the checklist was met, in a binary manner, by that study. Three authors rated studies on the mERA checklist independently and any disagreements were resolved through discussion until consensus was reached.

Results

Cohen's Kappa was used to assess inter-rater reliability, between the rating consensus reached by two of the authors, and ratings made independently by a third author. A strong agreement between raters was found (κ=.64). The discrepancy in ratings occurred mainly on two criterion: Technology Platform (6 differences in rating), and Usability/content testing (5 differences in rating). These differences are most likely due to a differing opinions of what extent of detail satisfies the criterion. The number of studies that met each criteria, after consensus was reached between raters are displayed in Figure 2.

Figure 2.

WHO mERA criteria met by the 11 published papers reporting on mobile health apps in schizophrenia as of this publication.

Eleven studies were identified though our search.^{23,24,25,26,27,28,29,30,31,32,33.} None of the papers reviewed reported information regarding a cost assessment of the technology used in the study or sufficient information. Only Macias et al³⁰ met the criteria “compliance with national guidelines or regulatory status,” by indicating that their application “supports two-way HIPAA compliant messaging,” and only Forchuk et al³³ mentioned the ways the mobile technology interacted and integrated with the current health information systems used in the population studied, meeting the criteria for “interoperability/health information systems.”

Ainsworth et al²⁵ and Naslund et al³² both met criteria for “limitations for delivery at scale.” The authors reflected on potential stumbling blocks for the adoption of the technology in real-word situations. For example Ainsworth et al²⁵ noted “As this technology makes the transition from research to real-world clinical application it will be vital to assess the feasibility and uptake of this software over longer periods of time and factors influencing non-participation and withdrawal,” acknowledging that there may be barriers to translating an intervention that is successful in a research study into an broad intervention for wider use. Ainsworth et al²⁵ lists specific potential solutions for the scalability of interventions including “machine learning in order to tailor the choice of questions,” “person -tailored sampling,” and “automated and clinician delivered feedback.”

Five studies^{25,26,27,30,31} reported data on the methods of “data security” used on their technology platforms, including the methods used to protect the privacy of the study participants. Seven studies^{23,24,25,26,27,28,32} were determined to have described the study procedure in sufficient detail to meet the “reliability” criterion. Similarly sufficient description of the “technology platform” used was assessed in nine^{23,24,25,26,27,28,29,32,33} of the studies.

Eight^{24,25,26,27,30,31,32,33} of the eleven studies mentioned using the population of interest in developing the content of the intervention. While the purpose of many of the studies reviewed was to assess the usability of the technology and content of the proposed intervention, Ben-Zeev et al²⁷ was particularly proactive in including the study population in every stage of the development of their mobile application, working with “patients and clinicians and community settings” to develop their app and then testing the app “with individuals with schizophrenia in laboratory conditions,” before beginning their pilot study of the app in a community context. Most studies that met criteria did not provide this much detail; instead they usually provided data on feedback participants had given over the course of the study.

Discussion

Our results suggest both areas of strength and weakness in the current reporting of clinical studies of smartphone apps for schizophrenia. There were substantial differences between the studies in the technologies used and the sizes and characteristics of cohorts studied. Nonetheless, several trends emerged with respect to common missing elements in the reporting of studies as outlined in the mERA checklist. These categories with scores of less than ten papers meeting the specific eERA criteria included reporting on the accessibility of technology platforms, interoperability/health information systems, cost assessment, limitations of delivery at scale, replicability, data security, usability testing, and compliance with national guidelines or regulatory statutes, each of which are explored in greater detail below. Understanding the current limitations of each of these areas is important in guiding future research on smartphones for schizophrenia. Additionally, it is important to consider that these same limitations may equally apply for technology research on many other psychiatric conditions, such as major depressive or bipolar disorders.

Technology Platforms Should Be Accessible

In order to understand the impact of a mobile health study, it is important to understand the technology behind that study. The mERA checklist notes that links to code used to support the technology should be publicly available, hardware choices described in detail, and other information provided to allow others to replicate the study. No studies reviewed made code publically available, but this may be in part because none of the studies were designed as open source and public platforms. While two of the eleven studies examined used commercial devices^28,32, all other studies utilized customized apps or devices. While it may be possible to come closer to approximating and recreating the smartphone apps used in earlier schizophrenia studies^23,25, for more recent and complex schizophrenia app studies^27,30 the technology platforms are more complex given the numerous interactive and passive data features. This raises a challenge for mobile health research moving forward in that as apps become more complex, understand the technology platform behind them and recreating similar versions by other groups wishing to reproduce studies may be nearly impossible. For research to advance, groups will need to make access to their platforms available to others and provide access to code used to create these platforms.

Technology Platforms Should Support Interoperability

Despite the potential of mobile health technologies to improve care, it is also possible they can disrupt care through fragmenting information, and/or creating unsupported care pathways which are detached from established health systems. The importance of interoperability of mobile health technologies was recently underscored by a report from the Office of the National Coordinator for Health Information Technology, which called for better integration of smartphone apps and wearables with medical record systems³⁴. Of note, only one of the papers discussed interoperability³³ although the lack of reporting on this topic in part can be explained by the pilot nature of these studies. Going forward studies should at least discuss strategies for clinical integration and what such integration might accomplish. With new data sharing standards like Substitutable Medical Applications, reusable technologies (SMART) and Fast Health Interoperability Resources (FHIR)³⁵ interoperability is becoming technically easier and future studies may have the opportunity of integrating mobile data into the medical record system, although barriers towards such still remain high at the time this paper was written.

Technology Platforms Should Report on Cost Assessment

While cost assessment are often omitted from clinical research, reporting data on the financial implications of mobile health interventions is important for future studies. The mERA framework puts forth that reporting costs of these technological interventions can help in comparing between alternatives, and is critical towards evaluating the cost-effectiveness of their implementation. Such data will also likely be critical in helping mobile health interventions gain greater support and buy-in from clinics and hospital systems if they are proven to enhance the efficacy and efficiency of existing care systems. Especially for disorders like schizophrenia, where funding is already limited, it will be important for smartphone apps to demonstrate evidence of cost effectiveness prior to widespread implementation. Although none of the eleven eligible studies reported on cost assessment, it should be considered that these were small-scale studies. Thus, cost assessment may be less valuable here, although will be informative for future larger scale studies, and indeed necessary for the translation of research findings into clinical practice.

Technology Platforms Should Discuss Their Scalability

By their nature as pilot studies, the current research base for mobile health interventions for schizophrenia focuses more on feasibility rather than scalability. However, there is a current lack of knowledge on how such interventions could be utilized on a regional or even population-level scale. Only two of the eleven papers discussed limitations for delivery of scale^25,32. Outside of schizophrenia research, one recent study examined how a mobile app intervention for alcoholism could scale beyond a single clinical study³⁶ to regional clinics. The study noted that the app faced numerous barriers in scaling up; with only 15% of clinics still using it at two years³⁷. In part, the potential of smartphone interventions to address population-level unmet needs, and to provide remote support to those who cannot physically access clinical services, has driven the interest in using this technology for schizophrenia. However, the feasibility and benefits of using these technologies to achieve this has not yet been evaluated. To reach their full potential, smartphones app need to show that they can be scalable beyond a single clinical study.

Technology Platforms Studies Should Be Replicable

The essence of any scientific advance and clinical intervention rests upon replicability. While none of the eleven studies we reviewed have been reproduced, we noted that seven of the eleven would be possible to replicate based on the mERA guidelines. The seven^{23,24,25,26,27,28,32} that seem replicable are the four earliest studies^23,24,25,26, the FOCUS study²⁷, and the two studies of a fitness tracker^28,32, and represent assessment of individual scales, modules, or ‘out-of-the-box’ devices applied towards schizophrenia. As apps for smartphones become more complex and smartphone interventions continue to utilize more sensors, features, and modules, replicability of app studies will become more complex. While the mERA does not focus on replicability of data analysis and statistical methods, this is also an important consideration given that an app platform collecting data is only as useful as the methods to process that data. Thus, there is a growing need for the establishment of standards by the field to ensure that app studies can be replicated.

Technology Platforms Should Provide Appropriate Data Security

While smartphone apps are often not thought of as devices that can cause harm, data security is one area where these tools create risk. If apps are created and implemented without adequate data security standards, then private patient health information is vulnerable to public disclosure and trust in these tools will be quickly lost. Vulnerabilities in data security were in large part responsible for the United Kingdom's National Health Services (NHS) closing their app library in October 2015³⁸, which had offered recommended a selection of smartphone apps - many directed at psychiatric conditions. Lack of appropriate data security will also make it impossible to implement the app in clinical care, as such security is often mandated by national and local laws and regulations. While it can be easy to make an app that collects patient data, making an app that meets complex and demanding security requirements such as the United States' Health Insurance Portability and Accountability Act (HIPAA) regulation presents a necessary challenge for apps aiming to improve clinical care. Thus, research articles must report on data security of their app, or explain how their results should be interpreted if such is lacking from their platform. The fact that only four of the eleven papers reported on data security suggests that this is an area where more efforts and research will be necessary.

Technology Platforms Should be in Compliance with National Guidelines and Regulatory Statutes

Any smartphone app and technology platform will be most useful if it can actually be used in clinical situations and towards patient care. Only one³⁰ of the eleven papers reviewed discussed how their technology complied with regulatory statues. Although these app regulations are complex and often change, it is still important to place clinical app research within the context within which these apps ultimately be required to function. Recently the Food and Drug Administration created a new website offering app developers an easy to use tool to learn which regulations may apply to their software. Researchers could also use this tool to report on which regulations would need to be considered to implement their app in clinical care. Without knowledge of how smartphone apps actually function under the real world conditions of various national guidelines and regulatory statutes, it will be difficult to assess how smartphone apps can actually offer benefit or harm in practice.

Technology Platforms Should Conduct Usability Testing

For mobile health interventions to be successful, they need to be valued and used by patients. Usability was a focus of the majority of the eleven papers we reviewed, and it is interesting to note that in only one²⁷ were patients highly involved in development of the mobile technology. This same study also showed ultimately high levels of acceptability and potential effectiveness. Ensuring that patients are involved early on, not just in the testing of applications but also in their development, is important from both an ethical and clinical outcomes perspective.

Additional Considerations

The WHO mERA criteria are not exhaustive in scope, although appear to offer a useful tool to evaluate mobile health studies with. Other important areas to consider, especially for schizophrenia research include incentives for user engagement, as well as clinician contact. Across numerous smartphone studies, engagement and use of the app have been shown to decrease with time^18,39,40. While there are numerous factors that may cause a user to engage less with any particular health app⁴¹, why this occurs in schizophrenia is largely unknown. A parallel consideration is knowledge of exactly how participants were compensated during smartphone clinical studies. Similar in nature to the mERA scalability factor, it is important to understand whether participants were engaged and used the app because they were paid to do so, or whether they found intrinsic value in the app. Often payment values are unclear as participants will be offered new smartphones or wearable sensors for partaking in studies, and the monetary value of such devices is often large although unreported. Finally, another measure that would be useful is study staff and clinician time spent with participants. An app may be useful because it facilities more clinical visits and encourages people to use more healthcare services, or it may be useful because it offers a valuable service apart from connecting users with direct care. Both are potentially useful services although different in their means of action. Data on how much time participants in app studies spend with clinicians and staff would thus offer useful data to understand how an app is working to achieve its stated goal. While understanding the mechanism of action of apps is more complex, this data would be helpful in beginning to learn how apps are working and what staff resources are necessary to support them if implemented in clinical care. Of course the mERA, or any other scale for that matter, can only evaluate what is reported in the literature, so it is possible that individual journal limitations on word count and page limits may bias reporting. However, researchers should consider making use of the supplementary online materials afforded to most journals in order to provide full and comprehensive reporting of study methodology.

While the mERA checklist offers a useful tool to guide a discussion about mobile health research, it is only a single scale and not yet well validated. It does not have the influence and weight of, for example, the CONSORT guidelines that have become an essential tool in the funding and publication of clinical trials. Additionally, the mERA is structured to offer equal weight to each criterion although some may be more or less relevant depending on the study in consideration. For example, if a study is not replicable, cost-effectiveness and scalability may become irrelevant. Despite these limitations, our findings that study methodologies are especially lacking with regards to compliance with national guidelines, replicability, scalability, cost assessment, and interoperability were clear and consistent, and likely would persist even if we had used a different rating tool to assess the literature.

Given that the majority of articles we examined were reported on pilot studies, it is interesting to consider if there is a role for a more specialized evaluation tool focused on early stage studies. Still, the mERA in its current form offers useful data even for pilot studies as the results of this review have suggested gaps in the literature that new pilot studies can explore. The mERA also does not take into account disease-specific considerations, such as decision-making capacity in schizophrenia, which could improve the utility of future evaluation strategies. Finally, while our focus on schizophrenia studies makes it difficult to generalize to other areas of psychiatry, we are not aware of any evidence that mobile applications targeting schizophrenia and other forms of severe mental illness should be evaluated using different criteria from tools designed for other disorders, such as depression or bipolar disorder. Thus, we anticipate that our findings and recommendations will remain largely relevant and applicable across many areas of psychiatry.

Conclusions

Technology for psychiatry and schizophrenia is rapidly changing. While smartphone apps are a forefront topic of research at the time of this writing (mid-2016), we anticipate that other technologies, such as wearables and virtual or augmented reality, will attract similar levels of attention, hope, and speculation in the years to come, as the field continues to seek out innovative solutions to the complex challenges inherent to managing mental illness. In each case, whether or not to adopt new technologies in particular treatment settings should depend on both a rigorous evaluation for clinical effectiveness and also consideration for the many other factors (e.g., data privacy, usability, cost) that would likely govern their use in real-world settings. While further refinements are to be expected, the WHO mERA guidelines provide a useful tool for this type of comprehensive assessment of mobile health research methodologies in the evolving landscape of digital health.

Finally, while the literature on schizophrenia and smartphones may still be nascent, a systematic examination using established criteria nonetheless helps to identify areas of promise in addressing the many known clinical challenges, as well as gaps where further research or better reporting is needed. The criteria reflected in the WHO mERA guidelines can help to ensure that the most promising technical solutions are developed in a manner that remains consistent with both core treatment principles and real-world constraints. By focusing our collective resources and research agendas on solutions that work both in principle and in practice, we as a field can dramatically increase the potential of these new technologies to improve both access and outcomes even in the most severe forms of mental illness.