Stress Measurement Using Speech: Recent Advancements, Validation Issues, and Ethical and Privacy Considerations

Abstract

Life stress is a well-established risk factor for a variety ofmental and physical health problems, including anxiety disorders, depression, chronic pain, heart disease, asthma, autoimmune diseases, and neurodegenerative disorders.The purpose of this article is to describe emerging approaches for assessing stress using speech, which we do by reviewing the methodological advantages ofthese digital health tools, and thevalidation, ethical, and privacy issues raised bythese technologies.As we describe, itis now possible to assess stress via the speech signalusing smartphones and smart speakers that employ software programs and artificial intelligence to analyze several features of speech and speech acoustics, including pitch, jitter, energy, rate, and length and number of pauses.Because these digital devices are ubiquitous, we can nowassess individuals’ stress levels in real time in almost any natural environment in which people speak.These technologies thus have great potential for advancing digital health initiatives that involve continuously monitoring changes in psychosocial functioning and disease risk over time. However, speech-based indices of stress have yet to bewell-validated against stressbiomarkers (e.g., cortisol, cytokines) that predict disease risk. In addition, acquiringspeech samplesraises the possibility thatconversations intended to be private could one day bemade public; moreover, obtaining real-time psychosocial risk information prompts ethical questions regarding how these data should be used for medical, commercial, and personal purposes. Althoughassessing stress using speech has enormous potential, there are critical validation, privacy, and ethical issues that must be addressed.

Hospital rooms, bedrooms, cars, and dormrooms arelocationswe still assume are relatively private—places where we believe we can have intimate conversations thatwill not betransmittedto, or analyzed by, third parties. However, smartphones and smartspeakersare rapidly changingall of that.In this article, we discuss the quickly growing capability for thesedigital devicesto assess human stress levels and psychosocial wellbeing, and thecritical concernsthat are raised by such surveillance. Ultimately, although technologies that can assess stress using speech acousticshold enormous potential for monitoring and potentially improving human health, serious validation, privacy, and ethical issues exist that must be addressed.

Questions regardinghow health data should be acquired and used have circulated for many years. However, the recent increase in availability of low-cost microphones and sensors—and a corresponding increase in interest in digital health—have made these issues a high priority in medical ethics(Rivas&Wac, 2018). By 2020, for example, it is estimated that everyindividual will own an average of seven internet-connected devices that have the ability to transmit health-related information to distant third parties (Topol, Steinhubl, &Torkamani, 2015). Smartphones and smart speakers arerelatively unique in this context, though, as they arealready ubiquitous and can non-invasively collectand sendlarge amounts of rich information (i.e., big data) that canbe used to indicate real-time disease risk.

If you want to continuously assess aprocess that greatly affects disease risk, then focusing on stressis a great option. This is because stress is implicated in not just a few disorders, but rather is a common risk factor for a variety of different mental and physical health problems, especiallywhenit is chronic (Miller, Cohen, & Ritchey, 2002; Slavich, 2016a, 2016b).For example, greaterstress exposure is associated with increased risk foranxiety disorders, posttraumatic stress disorder, depression, chronic pain, coronary heart disease, asthma, respiratory infections, autoimmune diseases, and neurodegenerative disorders, among others (Cohen, Janicki-Deverts, & Miller, 2007; Juster, McEwen, & Lupien, 2010; Slavich & Irwin, 2014).Stress is also associated with accelerated biological aging and premature mortality (Holt-Lunstad, Robles, & Sbarra, 2017; Kelly-Irving et al., 2013; Mayer et al., 2019), making it a critical factor to assess when predicting human health (Epel et al., 2018; Malat, Jacquez, & Slavich, 2017; Toussaint, Shields,Dorn, & Slavich, 2016).

Standard Methods for Assessing Stress

Given these effects of stress on health, numerous approaches have been developed for assessing individuals’ stress levels. The current gold-standard method involves conducting life stress interviews using instruments such as the Life Events and Difficulties Schedule, UCLA Life Stress Interview, and Stress and Adversity Inventory (Monroe & Slavich, in press; Slavich, 2019). In turn, the most commonly used approach involves administering brief self-report questionnaires such as the Perceived Stress Scale (Monroe, 2008). Interview-based measures can be time consuming and costly, though, and self-report questionnaires often lack item specificity and validity (Cohen, Kessler,Underwood,&Gordon, 1997; Dohrenwend, 2006; Shields & Slavich, 2017). Moreover, both methods are retrospective in nature and subject to (often unmeasured) degrees of cognitive bias and social desirability that can influence the veracity, reliability, and validity of the resulting scores (Monroe, 2008; Monroe & Slavich, 2016).

Stress has also been assessed by measuring biological processes that are upregulated by stress and implicated in disease. Assessing stress in this way has several advantagesover and above interview-based and self-report instruments. Two ofthe most importantadvantagesare that stress-related biomarkers are(a) proximally related to biological disease processes and (b) not subject to self-report biases. The full list of biologicalindices that are known to increase in response to stress is very long and beyond the scope of the present discussion. As an example, however, these outcomes span cardiovascular, sympathetic, neuroendocrine, and immune outcomes, and include things like heart rate, systolic blood pressure, skin conductance, cortisol, adrenocorticotropic hormone (ACTH), dehydroepiandrosterone (DHEA, and its sulfated ester, DHEA‐S), epinephrine, norepinephrine, α-amylase, and the pro-inflammatory cytokines interleukin(IL)-1β, IL-2, IL-6, and tumor necrosis factor-α (Allen, Kennedy, Cryan, Dinan, &Clarke, 2014; Irwin & Slavich, 2017; Slavich, in press; Slavich& Auerbach, 2018).Thesestress signals have become easier to assess over time—for example, smart watches can now be used to continuously monitor heart rate, skin conductance, and skin temperature—but for the most part, assessing stress-related biomarkers is still relatively invasive, requiring (for example) a blood or saliva sample from the individual (Shirtcliff, Buck, Laughlin, Hart, Cole, & Slowey, 2015).

Assessing Stress Using Speech

These limitationsof interview-, self-report, and biomarker-based approachesmake assessing stress using speech very attractive, especially given that doing so is nowrelatively inexpensive and non-intrusive.When preparing to speak, an individual must decide which sequence of words will best communicatehis or her intended message.Stress can affect these decisions and change the wording, grammar, and timing of speech, which can in turn be used as vocal markers of stress (Paulmann,Furnes, Bøkenes, & Cozzolino, 2016; Scherer& Moors, 2019). However, stress induces other changes as well. In order to produce speech, for example, the bodymodulates the tension of numerous muscles in order to force air through the vocal folds and out the vocal tract to produce sound waves (Titze, 2000). Stress increases both muscle tension and respiration rate, which in turn change the mechanics of speech production and, consequently, the way that speech sounds (Sondhi, Khan, Vijay, &Salhan, 2015; Zhou,Hansen, & Kaiser, 2001).

Current approaches for assessing stress using speech take advantage of these stress-based changes in the quality and pattern of speech acousticsto quantify the amount of a stress a person iscurrently experiencing.As summarized in Figure 1, this can beachieved by assessing several features, including the fundamental frequency (i.e., pitch), jitter (i.e., changes in pitch over a short period of time), energy in different frequency bands (e,g., mel frequency cepstrum coefficients, MFCCs), speaking rate, and length and number of pauses made while speaking (Hansen & Sanjay, 2007).These features areanalyzed using machine learning to produce a real-time index of an individual’s stress level (Fernandez & Picard, 2003).The resulting continuous stress signal can, in turn, contribute to quantifyinga person’s continuous health risk—something that is not possible with interview-based or self-report instruments.

Figure 1.

Assessing stress using speech. (a) When an individual speaks in the presence of (b) an actively recording smartphone or smart speaker, (c)an audio signalis captured. (d) Various features of the audio signal (e.g., pitch, jitter, energy, speaking rate, length and number of pauses) are then extractedand used as inputs to (e)a machine learning algorithm that yields a stress score. (f) The resultingscore can then be integrated into an individual’s clinical chart as an indicator of the person’s potential disease risk.

This last step—namely, analyzing features of speech to produce a stress level output—has been accomplished in many ways. Some models are physiologically-based and directly incorporate what is known about speech production and how it changes under stress to estimate an individual’s stress level (e.g.,Mendoza&Carballo, 1998; Van Puyvelde, Neyt, McGlone, &Pattyn, 2018). Others models have usedonly simple acoustic features (e.g., MFCCs) and deep neural networks(e.g., Han, Kyunggeun,& Hong-Goo, 2018; Hansen & Womack, 1996).The interpretability of the physiological models makes them especially attractive tophysicians, who often want to knowwhy particular outputsare given.The neural network-based models, in turn, are enticingbecause theyrequire very little domain knowledge to achieve acceptable accuracies.

Regardless of the particular method used for extracting a stress signal, what is arguably most impressive is how easily this can now be done. Indeed, whereashigh-quality speech analysis was once only possible in labsequipped with specialized recording equipment and teams of signal processing experts, researchers can now assess stress in speech inexpensively and without specialized training, using portable devices that can be carried around or placed anywhere in the natural environment. For example, Lu and colleagues recently used Android smartphones to detect instances of stress in multiple environments, including indoors during a job interview and outdoors while participants were interacting with other individuals (Lu et al., 2012). The modeling strategy used wasimpressive, accurately detecting the presence of stress in 81% and 76% of cases, respectively, for indoor and outdoor environments, when evaluated against the ground truth of increased skin conductance as assessed using an electrodermal activitysensor that was calibrated for each individual.Opensource programs like openSMILEin turn make the collection and analysis ofvoice data relatively easy for those without a background in signal processing (see Eyben, Weninger, Gross, &Schuller, 2013). In sum, therefore, assessing stress using speech and speech acoustics is now widely possible and relatively inexpensive.

Validation, Privacy, and Ethical Concerns

These technical advancements have transformed our ability to monitor individuals’ stress-related disease risk. Indeed, smartphones and smart speakers like the Amazon Echo and Google Home are now commonplace, with one market analysis suggesting that nearly one million smart speakers will be integrated into hospital rooms by 2021 to help facilitate patient-physician communication (Montany, 2018). Moreover, at least one major university in the United States announced that itplaced smart speakers in everydorm roomin 2018 to help students communicate with the university and learn about campus-wide events (Montag, 2018), and two other universities were already using the devices in select environments on campus by that time (Brown, 2018).

Given the widespread introduction of these digital devices into previously private settings, the same technology that is empowering our ability to monitor and potentially help individuals under stress is also prompting numerous questions about the validation, privacy, and ethics of this approach to digital health. With respect to validation, the main concern is that the race to promote widespread adoption of this technology will take precedence over making sure that voice-based approaches for assessing stress are validated against well-established biomarkers ofstress exposure and disease risk. Thefield of digital health, and especially the much broader field of “self-help”, are replete with examples of technologies that have become widely used before being well-validated.One such example is digital “brain training” programs, which acquired more than 50 million users despite possessing little-to-no-evidence that they worked (Simons et al., 2016). Given that we are still in theearly days of being able to assess stress using speech, much more carefully conducted validation work is needed to ensure that the stress indicesbeing used have clear clinical utility.

In addition, there aremany serious questions about privacy and ethics. With respect to privacy, what if a hospital-based smart speaker discloses HIPAA-protected information to a non-authorized person? Companies that sell smartphones and smart speakers have spent substantial time assuring users that their privacy is not at risk. As summarized in Table 1, however, several recent events have shown that privacy cannot be guaranteed even with huge investments in technology. For example, even Apple, whose leadership speaks the most about privacy and has more than $200 billion in cash on top of enormous technical resources, recently admitted that it had discovered a bug in its FaceTime communication platform that allowed callers to see and hear through the camera of a person they were calling before the person answered the call (Johnson, 2019). Having a device that can listen to you, even if made by a reputable company, thus means not only that your privacy could be one day compromised, but that your stress levels or health status could be potentially revealed without your consent. Similarly, multiple cases have recently been documented in which speakers used for other purposes (e.g. Amazon Echo, which is usually used for shopping or for controlling simple household devices) have been manipulated to listen in on private conversations, save the recordings, and transmit them to a third party (Charlton, 2018). Such hacks appear to be rare at present, but the point is that the technological capability already exists for using these devices for nefarious purposes, which is quite contrary to the goal of improving human health and wellbeing by assessing stress.

Table 1.

Examples of recent data breaches with smart speakers

Year	Device	Description of Incident	Reference
2017	Google Home Mini	Audio was recorded and stored without the wake word being used	Burke, 2017
2018	Amazon Echo	Echo sent a message to an owner’s contacts without the owner knowing	Shaban, 2018
2018	Amazon Echo	Amazon mistakenly sent 1,700 audio recordings to the wrong person	Ivanova, 2018
2018	Amazon Echo	Another Amazon Echo on the same WiFinetwork as a malicious device could record and transmit the audio it detected	Charlton, 2018
2019	Apple iPhones, iPads	A FaceTime bug allowed callers to hear the receiver’saudio and see their videofeed even if theydid not answer the call	Johnson, 2019

Along similar lines, what happens if private conversations captured by stress-assessing smart devices are disclosed (accidently or on purpose) to a third party?Is an employer or boss allowed to take action if they accidentally overhear something about an employee’s health that may affect their work?If evidence exists that someone is currently under extreme stress, what responsibility does the monitoring party have to act?Do users of the technology have the right to be told, first and privately, that their speech indicates that they are increasingly stressed and may be becoming depressed?Will physicians be more guarded knowing they are also potentially being monitored? After all, their speech can be sampled not just by their own smartphone, but also by their patients’ phones.

In addition, what happens if an advertising company or business uses a voice-based stress assessment technology to take advantage of an individual’s compromised emotional state? Is it ethical to provide stressed individuals with information regarding nearby psychotherapy or anti-depressant medication options? If so, what about fast food options that are known to be strongly preferredwhen individuals are under stress (Geiker et al., 2018)?In sum, when it is appropriate to usesuch digital health information for commercial purposes and when is it not?

Finally, what if a smartphone transmits evidence of domestic violence, or if a smart speaker in a dorm room detects self-harm or suicide but a university does not intervene? Is the company that manufactured the technology or that processes the data responsible? What about the company, school, or organization that provided the technology to the user or that has partial or full access to the resulting stress information? All of these scenarios can happen with today’s technology, and the newer smarter sensing approaches will only amplify the accuracy of the information that can be gathered and the scale of the impact it can have—whether for early detection and treatments that may reduce human disease risk, or for accidental or nefarious harm.

Solutions for Minimizing Risk

To minimize the risks associated with using smartphones and smartspeakersto assess human stress levels and psychosocial wellbeing, wemust recognize and address the privacy and ethical issues that are raised by these devices with the same vigor that is directed at advancing the technologies themselves. For starters, we believe these challenges could be addressed by (a) clearly informing users what the devices are transmitting and assessing, and providing examples of the possiblerisks involved; (b) enabling users to easily turn the listening function of the devices on and off as they wish; (c) enabling users to also have the audio equivalent of a physical lens cap—a “noise jamming” or other device that insures that no audio from their speech can be detected in case the “off” button does not work as expected; (d) allowing users to easily control who can access their data and how it is used; (e) permitting users to opt into having the devices in their surrounding environment; and (f) allowing them to opt out of having their speech logged or analyzed if they must live or work in an environment that listens.

More broadly, we believeit is critical for companies that develop and use these technologies to adopt strict policies to help ensure that users are immediately notified of technological malfunctions and data breaches. In addition, they should have comprehensive plans in place to quickly provide users with adequate identity protection services and compensation after a data breech has occurred.Stories of companies withholding critical information about a recent platform malfunction or data breachare common.When it comes to users’ data, we believe that individuals have a right to immediately know when their information has been inappropriately accessed or used, and that all companies that work with such data shouldaffirm their commitment toputting users’ data privacy and safety first.

Conclusion

In conclusion, stress is apowerful riskfactor for poor health that is in dire need of better measurement (Slavich, 2019; Slavich & Shields, 2018), and speech is one process that we can now easily measure to help address this need. To maximize the benefits and minimize the risksassociated with monitoring speech, however, we will needto take very seriously the validation, privacy, and ethical issues that are prompted by these technological advancements. We will also need to do a much better job at educating users about these issues and innovating better ways to protect users’ data beyond simply having a device “off”switch.

Looking forward, there areseveral avenues that could be pursued to make these technologies better and less risky for users. First, as alluded to earlier, research isneeded to validate speech-based assessments of stress against stressbiomarkersand clinical outcomes. In addition, since much of the original work on assessing stress with speech was conducted in quiet lab settings or with vocal actors, additional research is neededto validate these technologies in a variety of contexts, given that different environments can change an individual’s vocal signature (Giddens, Barron, Byrd-Craven, Clark& Winter, 2013). This will likely require a collaborative effort between private companies and research institutions to consolidate large corpuses of speech data with high-quality stress labels. Second, artificial intelligence techniques have been applied toassess emotional and behavioral states like depression and suicidality using speech (e.g., Cummins, Scherer, Krajewki, Schnieder, Epps, & Quatieri 2015), and although the sensitivity and specificity of these assessments have not yet been shown to achieve levels required for medical diagnosis, applying artificial intelligencemay well be helpful for enhancing the detection of stress as well.

Third, future methods for assessing stress will undoubtedly benefit from combining voice and facial recognition datato enhance the detection of stress and other emotional processes (Giannakakis et al., 2017), with the addition of other biometric data in the future. Finally, we believe that more crosstalk is sorely needed between developers, privacy experts, and medical ethicists to help ensure that the information gathered by these cutting-edgetechnologies is handled properly. Digitally driven approaches for assessing stress can ultimately play a key role in the future of digital health. To realize the full potential of this approach while minimizing possible risks, though, balanced attention needs to be paid to the technological, validation, privacy, and ethical issues raised here.