Speech as a promising biosignal in precision psychiatry

Abstract

Health research and health care alike are presently based on infrequent assessments that provide an incomplete picture of clinical functioning. Consequently, opportunities to identify and prevent health events before they occur are missed. New health technologies are addressing these critical issues by enabling the continual monitoring of health-related processes using speech. These technologies are a great match for the healthcare environment because they make high-frequency assessments non-invasive and highly scalable. Indeed, existing tools can now extract a wide variety of health-relevant biosignals from smartphones by analyzing a person’s voice and speech. These biosignals are linked to health-relevant biological pathways and have shown promise in detecting several disorders, including depression and schizophrenia. However, more research is needed to identify the speech signals that matter most, validate these signals against ground-truth outcomes, and translate these data into biomarkers and just-in-time adaptive interventions. We discuss these issues herein by describing how assessing everyday psychological stress through speech can help both researchers and health care providers monitor the impact that stress has on a wide variety of mental and physical health outcomes, such as self-harm, suicide, substance abuse, depression, and disease recurrence. If done appropriately and securely, speech is a novel digital biosignal that could play a key role in predicting high-priority clinical outcomes and delivering tailored interventions that help people when they need it most.

Many patients experiencing mental health problems today do not receive adequate treatment (Thornicroft et al., 2017; WHO, 2021). Moreover, among those receiving treatment, the modal number of sessions attended for psychotherapy and medication treatment is one, with little follow-up thereafter (e.g., Connolly Gibbons et al., 2011). As a result, patients are not presently being followed in a way that could help detect increases in symptoms or prevent relapse. To address these issues, providers are increasingly using ecological momentary assessments, phone check-ins, and automatic chatbots to monitor patients’ symptoms and prevent health emergencies from occurring. These monitoring practices are invasive and burdensome for both patients and providers, though, and they are also subject to self-report biases caused by social desirability, unawareness, and stigma, thus limiting their precision and utility.

Passive data collection bypasses these challenges as it uses technology to monitor patient progress. These technological devices include smartphone pedometers to track physical activity and phone apps to track sleep quality, social activity and engagement, and emotional words typed. Passive data collection modalities that measure health-relevant biological activity, or biosignals, are particularly informative as they index the activity of systems that are directly relevant for health and wellbeing. Consequently, they hold great promise for helping both patients and health care providers catch mental health emergencies before they occur. Below, we describe what we believe is one of the most promising health-relevant biosignals to monitor: speech.

1. Speech contains critical psychosocial information

When people talk, their voice and language contain more information than the mere messages they are conveying. Although word selection is important, it can be limited or biased by a lack of awareness or trust, inability to access emotions, feelings of shame, and more. In contrast, from the speech signal itself, we can now get metrics of emotion and human functioning that are not subject to these biases (Slavich et al., 2019). Examples include phonetic markers due to physiological changes such as muscle tension and semantic markers due to psychological changes such as increased use of first-person singular pronouns. These markers contain valuable, health-relevant information. Moreover, the collection and analysis of these data come with unique advantages over and above self-reported data. For example, high-quality microphones are now present in all smartphones, making data collection easy, and smartphones can contain apps that make data storage, analysis, and transmission fast, immediate, automated, and secure. Consequently, assessing speech could provide a highly scalable, accessible, and non-invasive strategy for monitoring psychosocial functioning and delivering just-in-time adaptive interventions (JITAIs) that help people when they need it most (See Fig. 1).

Fig. 1.

A visual representation of the added value of speech analysis versus ecological momentary assessment (EMA) in tracking stress levels. Although peoples’ stress levels change continuously over time, tracking these levels using EMA by phone (blue line) is intermittent, burdensome, and subject to several self-report biases, thus limiting its precision and clinical utility. In contrast, tracking stress and other health-relevant psychosocial processes using speech analysis (orange line) can be done non-invasively and continuously, and can in turn be used to both detect stressful life events (black circles) and deliver just-in-time adaptive interventions (JITAIs) precisely when individuals need them most. This figure has been designed using assets from Freepik.com.

Once biosignals are translated into health-relevant indices, they have the potential to become biomarkers of stress and health that can be collected passively and analyzed automatically, and thus help predict the emergence and/or recurrence of high-priority clinical outcomes. In addition to improving human health and resilience, therefore, speech can help relieve pressure on the healthcare system. To this extent, high-frequency patient monitoring has shown to be promising in predicting self-harm, suicide, depression, bipolar disorder, schizophrenia, alcohol and substance abuse, and more (Colombo et al., 2019; Faurholt-Jepsen et al., 2018; Gee et al., 2020; Mote and Fulford, 2020; Serre et al., 2015). Moreover, JITAIs that use smartphones and wearables to provide tailored support to people at just the right time have been found to be useful in managing mental illness, alcohol use, smoking, obesity, and suicide (Nahum-Shani et al., 2018). To date, JITAIs have not used speech data despite it being readily available, but making this connection is not difficult. Ultimately, expanding high-frequency patient monitoring and risk detection to include voice recordings could help enable the delivery of life-saving interventions.

2. Stress and speech

In terms of health-relevant processes to monitor, several exist, but the highest priority is probably stress (Slavich, 2016). Stress is a strong predictor of morbidity across a variety of diseases and is associated with 9 out of 10 leading causes of death in the U.S. today (Bhushan et al., 2020; See Table 1). Measuring stress using speech overcomes several limitations in the assessment of stress, including self-reporting biases and the need for real-time assessment. Moreover, these data can prompt the delivery of evidence-based JITAIs for stress, of which there are several (Sarker et al., 2017).

Table 1.

Role of stress in the top 10 causes of death in the United States.

Disease	Role of stress
Heart disease	Stress causes cardiovascular hyperreactivity and stress-induced hyperactivation of the hypothalamic-pituitary-adrenal (HPA) axis and the sympathetic nervous system (SNS), both related to coronary heart disease as acute coronary syndromes. Stress causes inflammation, increases in cholesterol, depression, risk of smoking, and metabolic syndrome, all strongly related to heart disease (Bhushan et al., 2020; Bunker et al., 2003; Wirtz and von Känel, 2017).
Cancer	Persistent activation of the HPA axis impairs the immune response and, together with chronic inflammation, heightens the risk of cancer and promotes the spread of cancer after development (Bhushan et al., 2020; Grivennikov et al., 2010; Reiche et al., 2004).
Accidents (unintentional injuries)	Stress can impair cognitive function, reaction time, and decision-making, which can increase the risk of accidents. Moreover, chronic and early life stress leads to more risk seeking behavior, further increasing the risk of accidents.
Chronic lower respiratory diseases	Stress can weaken the immune system and make it more difficult for the body to fight off infections, which can increase the risk of respiratory diseases. The exact associations with stress need to be further studied, but are strongly present (Hughes et al., 2017; Petruccelli et al., 2019).
Stroke	Acute stress can increase the risk of blood clots, leading to a higher likelihood of heart attacks and strokes due to changes in endothelial cell function, arterial stiffness, vessel wall damage, elevated blood viscosity, and hypercoagulability. Stress also increases the risk of strokes through the metabolic syndrome (Bhushan et al., 2020)
Alzheimer’s disease or dementia	Although the reasons for neuropsychiatric disorders are multifaceted and intricate, alterations to the brain’s threat response, pain perception mechanisms, motivation and reward pathways, and impulse control are linked to toxic stress and are considered to play a part in increasing the likelihood of these disorders. Also, accelerated cellular aging as a component of toxic stress physiology may lead to higher rates of Alzheimer’s disease and other types of dementia (Bhushan et al., 2020).
Diabetes	Similarly to heart disease and stroke, diabetes is a risk factor of stress through the metabolic syndrome. Moreover, stress affects glucose regulation, insulin resistance, and insulin secretion, with effects even occurring intergenerationally (Bhushan et al., 2020; Lloyd et al., 2005).
Influenza and pneumonia	Unknown
Kidney disease	Stress increases the risk of kidney disease and is believed to be increased by other factors, such as heart disease, obesity, diabetes, and high blood pressure, which can also cause damage to the kidneys. It is suggested that dysregulation of endothelin-1, a molecule involved in regulating blood pressure and arterial stiffness, may be an underlying mechanism through which stress and other risk factors contribute to the development of both cardiovascular and kidney disease (Bhushan et al., 2020; Bruce et al., 2015).
Suicide (attempts)	In addition of stress functioning through depression, anxiety, and other mental health problems to affect risk for suicide, multiple models of suicide include stress directly. It is proposed that suicidal behavior is a result of an interaction between acutely stressful life events and a susceptibility to suicidal behavior (Bhushan et al., 2020; Van Heeringen, 2012).

Note. The association between stress and these causes of death is complex and multifactorial. Moreover, stress may not be the only or even the primary factor contributing to these causes of death. However, reducing stress through lifestyle changes, stress management techniques, and seeking support from family and friends can have numerous health benefits, including reducing the risk of these and other serious health problems.

Early research on stress and speech is promising. For example, this work has identified key acoustic features of speech (Van Puyvelde et al., 2018; see also Kappen et al., 2022b) and how their associations are modulated by stress (Kappen et al., 2022a). In turn, these biosignals have been found to reliably predict emotion, heart rate, respiration, and cortisol responses (Baird et al., 2021), and peoples’ experience of everyday work stressors (Langer et al., 2022; see also Lu et al., 2012). Likewise, a recent systematic review of 127 speech acoustic studies synthesized research describing the use of speech for detecting a variety of psychiatric disorders, including depression, schizophrenia, bipolar disorder, posttraumatic stress disorder, anxiety, anorexia, obsessive-compulsive disorder, and bulimia (Low et al., 2020). Despite promising results with regard to the identification of stress, limited research has translated this information into JITAIs for managing everyday stress, where it could have huge benefits for both patients and caretakers.

3. Real-world validation and application

Looking forward, more research is needed to validate speech against clinical outcomes, and in this context, focusing on disease recurrence should be a top priority for a few reasons. First, these patients already have a demonstrated disease risk (often accompanied by decreased stress resilience) and, therefore, a reason to be followed. Second, early detection and intervention in these patients would have substantial cost savings, help prevent complete relapse from occurring, and lead to less time-intensive treatments. Finally, these patients are already connected to clinical care, making immediate intervention easier and more likely.

4. Ethical and legal issues

Ultimately, the real-world implementation of speech data collection comes with multiple privacy and ethical concerns, and these issues must be taken seriously to maximize the potential benefits and minimize the risks associated with this emerging technology. As described by Slavich et al. (2019), risk minimization should include, at minimum: (a) telling users what devices are sampling, assessing, and/or transmitting speech, and providing examples of possible risks; (b) enabling users to digitally control the listening function of devices; (c) enabling users to physically control the listening function of devices (e.g., using the audio equivalent of a physical lens cap); (d) allowing users to manage access to their speech data; (e) permitting users to opt in to having devices in their environment; and (f) allowing users to opt out of having their speech logged or analyzed. Several high-profile breaches of speech data have occurred (see Slavich et al., 2019), and when it comes to these issues, we believe user data protection must come first.

5. Conclusion

In conclusion, speech can drive the next frontier in monitoring health and delivering JITAIs to prevent disease recurrence and foster resilience. Looking forward, more research is needed to validate speech against ground-truth outcomes, including clinical functioning, diagnoses, biomarkers, and subjective self-reported responses (Sarker et al., 2017). Focusing on stress makes a lot of sense in this context, as stress plays a crucial role in the development and recurrence of numerous mental and physical health conditions, and can be treated using existing evidence-based strategies (Slavich and Auerbach, 2018). We already have the technological devices needed to realize the promise of JITAIs in our hands. To solve some of the world’s biggest health challenges, all we need to do is empower these devices with the right diagnostic and therapeutic programs.