An End-to-End Overview of Clinical Speech AI

Abstract

There has been a surge of interest in the use of speech as a biomarker for a wide range of health conditions, based on the premise that neurological, mental or physical impairments affecting speech production can be objectively assessed by automated analysis. Recent advances in clinical speech artificial intelligence (AI) have applied supervised learning techniques, similar to those used in general-purpose speech technology applications, to support the diagnosis and monitoring of mental health, cognitive decline, and motor disorders. Although clinical speech AI offers significant promise as a scalable and low-burden tool for health assessment, it also presents unique challenges compared to other speech technology applications. These include the need for condition-specific speech elicitation tasks, limited and heterogeneous datasets, sensitive data collection protocols, diverse speech representation and modeling strategies, and uncertainty in ground-truth clinical labels. This overview synthesizes the emerging literature that addresses these challenges in the full clinical speech AI pipeline: from speech elicitation and recording to representation, model development, and deployment. We provide a practical review of the field, including the technical pipeline for clinical AI, the design of clinical speech tasks, data collection practices, traditional and clinically oriented speech representations, predictive modeling approaches, and ethical considerations. We also discuss open technical and translational challenges and outline future research directions. The goal is to equip researchers and algorithm developers with the knowledge needed to move clinical speech AI from experimental development to real-world clinical impact.

I. Introduction

Clinical speech AI applies AI-based and traditional speech and language processing methods to extract acoustic and linguistic information from speech for health assessment. Its core promise lies in enabling objective, automated evaluations of neurological [1], [2], [3], [4], [5], mental [6], [7], [8], or physical impairments [9], [10] that impact speech production. It can support a wide range of clinical contexts [11], including diagnostic assistance, non-specific risk assessment, longitudinal tracking post-diagnosis, the use of speech as a meaningful endpoint for interpretable insights, improved clinical trial measurement, and the development of digital therapeutics [12].

Speaking is a complex process that involves selecting words to express an idea, structuring them according to linguistic rules, and coordinating over 100 muscles to produce intelligible speech. This task relies on precise neural activity and fine motor control, making it highly sensitive to disruptions in physiological or neurological systems. Because of this sensitivity, speech serves as both a marker and a mirror of health. As illustrated in Fig. 1, speech can reflect upstream indicators, such as the onset and progression of neurological changes, as well as downstream consequences, including reduced quality of life and impaired social interaction [13]. Sometimes described as the “newest vital sign” [14], speech provides a rich, accessible source of information for health assessment [15].

Fig. 1.

The dual role of speech and language in clinical applications. Changes in speech and language can serve as an indicator of neurologic change or it can reflect the downstream consequences of that change.

Clinical speech AI aims to harness this potential by extracting clinically meaningful patterns from speech, providing a complementary and underutilized data stream alongside traditional health measures. Here, we define clinically meaningfulness using three criteria. First, a measure is clinically meaningful if it captures an aspect of health that matters to patients; for speech, this may include the ability to communicate effectively, participate socially, or carry out daily activities. Second, it must reflect a change that clinicians and patients agree is beneficial or detrimental, even when the underlying biological mechanism is complex or indirect. Third, it should relate, directly or indirectly, to clinical outcomes that influence decision-making, such as disease progression, symptom burden, or functional status.

Over the past decade, there has been a surge of interest in academia and industry in developing AI models for diagnosis, prognosis, and tracking of different clinical conditions using speech. Examples include applications in mental health (e.g., [16], [17]), cognitive assessment (e.g., [18], [19], [20], [21], [22]), neuromuscular diseases (e.g., [23], [24], [25]), emotion recognition (e.g., [6]), children’s speech sound disorders (e.g., [26], [27], [28]), voice disorders (e.g., [29], [30]), respiratory disease (e.g., [31], [32], [33]), and cardiovascular disease (e.g., [34], [35]). The growing momentum has been further fueled by dedicated sessions, tutorials, and keynote presentations at recent speech science and technology conferences, highlighting the field’s expanding significance and impact.

Across all these applications, the development of clinical speech AI algorithms often relies on supervised learning, similar to other speech technologies such as automatic speech recognition (ASR), speaker verification, keyword spotting, etc. The process typically begins with the collection of labeled datasets, where speech samples are paired with clinical labels (e.g., diagnosis, prognosis, clinical standard tests). This is followed by feature extraction and model training [36], [37], [38]. Within this supervised framework, researchers iteratively refine model inputs and architectures to improve prediction accuracy. Benchmark datasets and open competitions have been established to support model development and comparison. Examples include datasets and challenges focused on emotion and breath tracking ([39]), cognitive and physical load classification ([40]), detection of Alzheimer’s disease ([21], [41]), dysarthric speech recognition ([42]), COVID-19 cough detection ([43]), etc. While this supervised learning approach, paired with public benchmarks, has driven rapid progress in technologies like ASR and speaker verification, clinical speech AI presents a unique set of constraints.

Clinical speech AI faces a distinct set of challenges that distinguishes it from other speech technology applications. One of the most significant hurdles is the limited size and variability of labeled clinical datasets, including benchmarks. For instance, studies on dementia detection, dysarthric speech assessment, and other pathological speech tasks often rely on datasets containing only tens to hundreds of minutes of speech [21], [44], [45]. This is in contrast to large-scale speech recognition datasets, which boast thousands of hours of data [46], [47], [48]. Additional challenges include the need for condition-specific speech elicitation tasks, controlled setups for data collection, diverse strategies for feature extraction and model training, and the inherent uncertainty in diagnostic labels. Together, these factors contribute to concerns about the generalizability of purely data-driven techniques in the field.

Recent evidence suggests that reported performance metrics in clinical speech AI may be overly optimistic. In particular, some studies have shown that model accuracy declines as sample size increases; this is a counterintuitive trend given expectations from learning curve theory [49], [50]. This overoptimism may result from a combination of factors, including analytical flexibility combined with publication bias (e.g., trying many modeling choices and only reporting the best outcomes) [51], [49] and information leakage between training and test sets [52]. These practices can artificially inflate reported results and ultimately undermine trust in clinical speech AI. In response to the challenges of developing robust clinical speech AI models, recent work has focused on improving every stage of the development pipeline from data collection and feature extraction to model design and validation. These efforts aim to build systems that accurately capture clinically relevant information, perform reliably across diverse real-world settings, and provide actionable insights for healthcare professionals.

Given the breadth of both technical and clinical considerations, there is a pressing need for a comprehensive overview of the clinical speech AI pipeline. This paper addresses the field’s unique challenges and traces the full development process: from data acquisition to model deployment and evaluation. The ensuing sections are organized as follows:

• Speech as a biomarker: To illustrate how speech can function as a biomarker, we draw on the U.S. Food and Drug Administration (FDA) BEST Resource framework, which provides the regulatory basis for classifying biomarkers across diverse clinical contexts and helps clarify the clinical utility of speech across different use cases.

• Clinical speech AI and the technical pipeline: We highlight the important differences between clinical applications and other applications of speech AI. We then review the FDA-informed technical pipeline for the training and deployment of clinical AI models as Software-asa-Medical Device (SaMD) tools, providing a high-level overview of the technical components specific to clinical speech AI.

• Design of speech elicitation tasks: Understanding how humans produce speech is fundamental to the development of clinical speech AI. We draw on Levelt’s model of speech production to explain the roles of Conceptualization, Formulation, and Articulation, and how different health conditions can disrupt these stages [53]. Certain types of speech elicitation tasks allow symptoms of a given clinical condition to be amplified. We review common clinical speech elicitation tasks, categorized by content structure and cognitive/motoric load imposed on individuals.

• Clinical speech data collection: The challenge of speech data collection stems from the fact that characteristics of recorded speech are dependent on additional factors such as the types of speech recording device, post-processing firmware, environmental setups, etc. This section will review the sources of noise in speech data, microphone selection, recording section setups, validation methods for ensuring data quality, and diagnostic label uncertainty.

• Traditional speech representations applied to clinical applications: Clinical speech AI often leverages features originally developed for non-clinical applications. This section reviews the signal processing–based and deep learning–based speech representations that are routinely used in clinical speech AI.

• Clinically focused speech representation: Recent efforts in clinical speech AI emphasize clinically-linked measures that capture relevant constructs of interest. The first part of this section reviews validity and reliability concepts from measurement theory [54]. The review then moves to technical approaches for generating clinically-relevant speech measures, and analytical validation to ensure their accuracy and consistency.

• Development and evaluation of clinical prediction models: From simple to complex, this section reviews traditional and deep learning models for clinical label prediction, covering explanation methods, robustness enhancement techniques, clinical validation, post-deployment monitoring, and common causes of model degradation.

• Ethical and participant considerations: Current deployment and regulation of clinical speech models present multiple challenges and risks from ethical, privacy, and security perspectives. This section reviews potential biases in data acquisition and annotation, adversarial attacks and defenses, patient data privacy, and risks of misuse in clinical AI models.

• Discussion: We discuss current challenges in technical implementations and considerations. These include usability of clinical models, overoptimism in clinical speech AI, defining an appropriate context of use, transition from speech features to speech measures, adoption of simpler and interpretable predictive models, prospective validation, opportunities for large language models and spoken language models, and the necessity of regulatory approval for clinical speech AI.

• This overview paper aims to address an important gap in the clinical speech AI literature: the lack of technically grounded resources that span both the diversity of speech-affecting conditions and the full end-to-end development pipeline. While existing reviews often focus on specific disorders or emphasize modeling techniques in isolation [55], [18], [19], [6], [36], this paper provides a systems-level perspective. It covers upstream components such as task design and data collection, as well as downstream considerations like clinical validation, deployment monitoring, robustness to bias, security risks, and misuse. These issues are frequently overlooked in technical literature, yet they raise foundational challenges in model generalization, reliability, and integration - challenges shared across many high-stakes applications of AI. As such, the paper bridges traditional modeling pipelines with emerging real-world deployment needs. Rather than prescribing a fixed research and development path, as we did in a previous tutorial [56], we equip readers to reason about where to begin and what to prioritize, depending on the application context and constraints.

Throughout the paper, we use the terms speech features, speech measures, and speech representations with distinct meanings. We define speech features as multi-dimensional representations extracted from speech that serve as inputs to AI models. These features need not be interpretable; they may reflect acoustic or linguistic properties, mathematical transformations of the signal, or higher-level abstractions. Within this broader category, speech measures refer to a validated subset that are explicitly linked to clinical constructs and are therefore interpretable by design. We use the term speech representations to refer collectively to both features and measures.

II. Speech as a Biomarker

Biomarkers represent measurable characteristics that serve as indicators of normal biological processes, pathogenic processes, or responses to exposures or interventions. Recent work proposes a hierarchical organization of speech biomarkers, from broad digital biomarkers through vocal biomarkers down to specific physiological/cognitive domains (respiratory, voice, speech/articulatory, cognitive/language), thus providing a mechanistic, subsystem-based organization [69]. Herein, we build upon the mechanistic subsystem definitions by adopting a regulatory-aligned perspective to make contact with existing regulatory pathways, clinical validation processes, and translational frameworks for biomarkers. According to the U.S. Food and Drug Administration (FDA) BEST (Biomarkers, EndpointS, and other Tools) Resource, biomarkers comprise seven distinct categories: susceptibility/risk, diagnostic, monitoring, prognostic, predictive, pharmacodynamic/response, and safety biomarkers [57]. The versatility of speech lies in its adaptability as a biomarker across these categories. Table I provides a mapping of different clinical use cases of speech as a biomarker to the corresponding BEST categories, along with rationales for their alignment.

TABLE I.

Alignment of speech-as-a-biomarker use cases with FDA BEST biomarker categories

FDA BEST Biomarker Category	Definition [57]	Corresponding Use Case(s) [12]	Rationale for Alignment
Diagnostic biomarker	Detects or confirms presence of a disease or identifies its subtype.	Diagnostic assistance	Objective measures of known speech symptoms (e.g., dysarthria patterns, coherence, tangentiality) can improve diagnostic accuracy and reproducibility [58], [59].
Monitoring biomarker	Repeatedly measured to assess status of a disease or response to treatment.	Longitudinal tracking postdiagnosis; Improving clinical trials	Repeated speech measurements can track disease progression (e.g., amyotrophic lateral sclerosis, Parkinson’s disease) or detect therapeutic response; frequent, objective endpoints improve trial efficiency [24], [60].
Pharmacodynamic / Response biomarker	Indicates that a biological response has occurred in an individual exposed to a medical product.	Speech as a clinically meaningful endpoint; Improving clinical trials	Speech-derived measures (e.g., intelligibility, communicative participation) change in response to intervention and reflect meaningful functional outcomes [61], [62], [63].
Susceptibility / Risk biomarker	Indicates increased likelihood of developing a disease in the future.	Non-Specific risk assessment	Early or progressive within-person speech changes may indicate elevated disease risk before diagnosis, although not disease-specific [64].
Predictive biomarker	Identifies individuals more likely to experience a favorable or unfavorable response to a treatment.	Development of digital therapeutics	Speech features could help personalize interventions (e.g., communication or mental health therapies) by predicting treatment response [65].
Safety biomarker	Indicates likelihood, presence, or extent of toxicity or adverse effects.	Development of digital therapeutics	In adaptive or home-based speech monitoring systems, changes in speech could flag treatment-related adverse effects (e.g., cognitive or motor decline) [66], [67].
Clinical Outcome Assessment (COA) (related but distinct from biomarkers)	Directly measures how a patient feels, functions, or survives (FDA definition of “clinically meaningful”).	Speech as a clinically meaningful endpoint	Speech measures such as intelligibility or communicative participation are direct indicators of functional status and quality of life; how patients feel and function [68].

III. Clinical Speech AI and The Technical Pipeline

Developments in speech AI have led to a wide range of new technologies, from voice assistants [70] and speech evaluations [71] to clinical tasks such as assessing neurological or mental health conditions [72], [6]. These applications differ significantly in both their stakes and the reliability of their ground-truth labels. Fig. 2 illustrates these distinctions: the x-axis represents the stakes involved, defined as the consequence or impact of speech AI outcomes, while the y-axis captures the variability in the labels used for supervised learning. For instance, transcripts for speech recognition tasks (e.g., from healthy controls) are typically reliable, whereas diagnostic labels for conditions like mild cognitive impairment or depression often exhibit considerable variability [73], [74]. Clinical speech AI is distinguished by both elevated stakes and greater label variability compared to traditional speech applications.

Fig. 2.

Traditional speech AI vs. clinical speech AI by stakes and label variability. Conventional tasks such as speech recognition are low-stakes with minimal label variability, whereas clinical tasks, such as diagnosing mild cognitive impairment (MCI), are high-stakes and exhibit great label variability due to inherent diagnostic uncertainty.

Unlike commercial ASR systems (e.g., Siri, Alexa), where misrecognitions usually have minor consequences, inaccuracies in clinical speech AI can lead to misdiagnosis or delayed treatment. Moreover, diagnostic labels themselves are often variable and costly to obtain, particularly in the early stages of diseases. Meta-analyses reveal that misdiagnosis rates for Alzheimer’s disease show sensitivities ranging from 71% to 87% and specificities from 44% to 71%. Similarly, the diagnosis of Parkinson’s disease within the first five years is reported to be inaccurate nearly half the time [75]. This stands in contrast to other applications of speech AI, such as ASR, where ground-truth labels are clear and exhibit low inter-transcriber variability [76]. These challenges stem from the progressive nature of diseases, subjective clinical interpretations, and comorbidities that complicate diagnosis. Consequently, clinical speech AI models require more rigorous validation than traditional systems. Beyond conventional accuracy metrics (e.g., word error rate for ASR, equal error rate for speaker verification), clinical models must undergo extensive testing across diverse populations, incorporate real-world deployment feedback, and satisfy regulatory requirements to ensure safety and effectiveness. Although these demands increase development complexity and cost, they are important for building trust and ensuring clinical utility in high-stakes healthcare environments.

Given these heightened stakes and the inherent variability in clinical labels, the development of clinical speech AI requires a more rigorous and structured process than conventional speech technologies. To address these demands, regulatory agencies such as the FDA have outlined best practices for the development of AI-based Software-as-a-Medical-Device (SaMD) tools. The technical pipeline for training and deploying clinical AI models, aligned with FDA guidelines, is illustrated in Fig. 3 [77]. This pipeline begins with the design of a data collection protocol, which outlines how the target clinical population is recruited, how speech elicitation tasks are crafted, and how data is collected. During data collection, participants’ speech data is captured in the form of acoustic signals by audio recording devices. Clinical labels, such as disease symptoms, clinical scores, or other clinically relevant indicators, are provided by experts and linked to the recorded speech. In AI terms, the speech signals and clinical labels become the “data” and “labels” stored in a structured database. Feature engineering plays an important role in extracting meaningful information from the speech data to enable effective clinical speech AI. This process can incorporate domain expertise from fields such as speech signal processing, speech production, and clinical speech science. Once sufficient data are available, statistical models are trained to identify relationships between the features and labels. These models are then validated using a held-out test dataset to optimize features and models prior to deployment. After deployment, the model’s real-world performance is continuously monitored. As new speech data is collected during deployment, the model can be iteratively updated and redeployed to improve its accuracy and relevance.

Fig. 3.

Technical pipeline of clinical AI model training and deployment. Components include data acquisition protocol design, training data collection, feature engineering, model training and validation, and post-deployment monitoring.

Errors at any stage of this development process can compromise the effectiveness of clinical speech AI systems. Improper data collection, such as poorly designed speech tasks or inadequate recording setups, can result in features that lack clinical relevance, undermining model training and performance. Similarly, feature extraction that overlooks clinical domain knowledge or models that are unnecessarily complex can reduce interpretability, limiting clinical adoption. Training on datasets that do not reflect real-world deployment conditions further risks degrading model performance in practice.

The following sections provide an overview of each stage of the clinical speech AI development pipeline in greater detail, beginning with protocols for speech data collection. We review current practices and identify considerations for building models that are accurate, interpretable, and clinically validated.

IV. Design of Speech Elicitation Tasks and Protocols

Selecting appropriate elicitation tasks is essential for collecting speech that is clinically informative and that meaningfully stresses the underlying production systems. Because speech arises from the coordinated interaction of respiration, phonation, articulation, prosody, and cognitive–linguistic planning, different tasks selectively load these components. Well-designed protocols allow investigators to probe specific physiological or cognitive processes and detect subtle deviations that may not appear in spontaneous speech. We first review the stages of speech production, then present a task taxonomy to guide researchers in selecting or designing tasks suited to their objectives.

A. Human Speech Production Mechanism

The process of speaking is mediated by neural circuits that activate more than 100 muscles synergistically, producing sounds we recognize as speech. To better understand and analyze this complex process, a simplified stage-based approach compartmentalizes the various components involved. This approach also allows deficits caused by diseases or clinical conditions to be classified into specific stages, aiding in diagnosis and decision-making. Levelt’s seminal work, “Speaking: From Intention to Articulation” presents a modular model of all known aspects of speech production. It provides a valuable framework for understanding the various processes at play and how things can go wrong [53]. In this section, we review a simplified adaptation of Levelt’s model, focusing on three well-researched and widely understood stages: Conceptualization, Formulation, and Articulation. These stages serve as a foundation for understanding how speech is produced and how clinical conditions can impact each phase.

1). Conceptualization:

Conceptualization occurs within the context of an individual’s perception of self, time, and place, and an appreciation for the perspective of the receiver of this message (i.e., Theory of Mind). Conceptualization requires adequate functioning of the parts of the brain responsible for judgment, reasoning, memory, emotion, and social motivation [78]. There are several conditions that interfere primarily with the Conceptualization Stage of speaking, including psychiatric disorders (schizophrenia, bipolar disorder, major depressive disorder), dementias that include personality changes and/or hallucinations (frontotemporal dementia, Lewy body dementia), and the later stages of Alzheimer’s disease. Speech characteristics of impaired conceptualization result from the presence of ‘negative symptoms’ (psychomotor retardation, apathy), ‘positive symptoms’ (mania, hallucinations), and a distorted sense of reality (disorientation, paranoia, psychosis). Negative symptoms often manifest in speech as reduced speech output, imprecise articulation, and a tendency toward monotonicity and low or monoloudness [79]. Positive symptoms are associated with rapid, pressed speech and a lack of coherence in the message being conveyed [80]. Individuals with a distorted sense of reality may exhibit erratic speech patterns, such as speaking at an unusually fast or slow pace, abruptly shifting topics, or producing disjointed and incoherent language. These speech characteristics can be identified through both acoustic analysis of the speech signal and examination of the linguistic content in speech transcripts.

2). Formulation:

In Formulation, words are selected and sequenced to best convey the conceptualized message. The sequencing must follow the rules of the spoken language. The process also involves abstract decisions about the level of specificity of words to use, emotional valence and intensity, and literary devices. These decisions pair with paralinguistic decisions on how the outflow of the speech message (prosody) will enhance the intended meaning. Health conditions that interfere with the Formulation Stage typically involve damage to the cortical and subcortical language circuits of the brain. A cerebrovascular accident (CVA, or stroke) of the left cerebral hemisphere can cause aphasia, characterized by varying patterns of difficulty finding the words one wishes to speak, difficulty sequencing the words into meaningful sentences, and difficulty understanding what others are saying. In contrast, a CVA in the right cerebral hemisphere can leave language preserved, but one may struggle with perceiving and producing paralinguistic information that signals humor, sarcasm, or emotion. Individuals with early to middle-stage Alzheimer’s disease can also struggle to find words, and the words they do find tend to be less specific [81]. As with the Conceptualization Stage, evidence of all these speech characteristics can be found in both the acoustic speech signal as well as in analysis of the transcripts of the spoken message.

3). Articulation:

Following Conceptualization and Formulation, the formulated sensorimotor commands are executed to produce speech. When one takes a deep breath, a column of air is exhaled and ascends through the trachea, larynx, and vocal tract to produce speech. As the column of air reaches the larynx, the closed vocal folds create resistance, building subglottal pressure until they open and begin to vibrate. This vibration chops the airflow into bursts of sound, generating phonation. The pitch of speech is determined by the speed of vocal fold vibration, which is controlled by laryngeal muscles. Loudness depends on subglottal air pressure and vocal fold closure. Greater pressure and tighter closure increase amplitude and loudness. Vocal quality is shaped by how symmetrically and completely the vocal folds meet, influencing clarity. As air passes through the larynx, vibrating portions produce voiced sounds (e.g., vowels, nasals, liquids), while undisturbed airflow forms unvoiced sounds (e.g., unvoiced stops and fricatives). The upper vocal tract refines these sounds by shaping airflow with the tongue, jaw, lips, and soft palate. The soft palate and throat then direct sound through the nasal or oral cavities, producing nasal or non-nasal speech [82]. Movements of the tongue, jaw, and lips shape different resonant cavities, modifying phonation and producing formants that distinguish vowel sounds. Close approximations of the lips, tongue, or hard palate create turbulence, generating fricatives like /s/, while complete closures build pressure, releasing as stops like /p/. Clear and intelligible speech relies on distinct resonant cavities, precise constrictions, and well-timed closures, all working in harmony to form spoken message.

Any condition disrupting the coordinated movement of the respiratory, phonatory, articulatory, and resonatory systems will affect the acoustic speech signal. Focal conditions have specific effects; for example, a paralyzed vocal fold causes breathy phonation without affecting other subsystems. In contrast, conditions like amyotrophic lateral sclerosis (ALS) impact multiple brain and spinal cord regions, leading to spastic and flaccid paralysis that result in slowed, imprecise speech. Speech symptoms vary based on the location and extent of nervous system damage [64]. Structural and functional impairments, such as cleft palate, laryngeal cancer, or chronic obstructive pulmonary disease, also impact speech in predictable ways [83], [84], [85].

B. Speech Elicitation Task Design

Fig. 4 shows several examples of common speech elicitation tasks plotted relative to the degrees of freedom of the spoken responses (x-axis) and the degree of computational and/or physical pressure inherent in completing the task (y-axis). For example, reading aloud a list of words has a low number of degrees of freedom, since each word has exactly one correct spoken response. Reading a word places little pressure on the cognitive-language system and does not require higher-order executive function or memory. Describing what is happening in a picture requires a synthesis of the objects and actions depicted in the picture, and inferences about their temporal and spatial relations, thereby making it a more computationally challenging speech elicitation task than word reading. While the picture constrains the content of the response, the degrees of freedom are still lower than in tasks with unconstrained topics, such as open-ended conversations.

Fig. 4.

Categorization of speech elicitation tasks by degrees of freedom and computational/physical load. Low-freedom/low-load tasks (e.g., object naming, text reading) use fixed scripts, bypassing message planning and requiring minimal effort. Low-freedom/high-load tasks (e.g., diadochokinesis (DDK), picture description) have certain constraints on content while maximizing motor and cognition demands. High-freedom/variable-load tasks (e.g. conversation, interviews), allow broad content variability within topic constraints, where cognitive and motor loads vary by context.

The framework in Fig. 4 offers a useful way to conceptualize the alignment between task properties and the symptoms targeted for assessment. The following subsections review commonly used elicitation tasks, organized by structural complexity, and discuss how task selection can be tailored to specific clinical applications.

1). Structured elicitation tasks:

Structured speech tasks, such as isolated pseudo-word tasks, sentence reading tasks, and word repetition tasks, involve predefined spoken content that is carefully selected to target specific phonetic elements. These tasks place minimal cognitive or linguistic demands on the speaker and are particularly useful for assessing speech intelligibility, phonetic contrastivity, and speech acquisition. In clinical contexts, they are commonly used with individuals who have impaired sensorimotor function but intact language and cognitive abilities, such as patients with cleft lip or oral cancer [86], [87], [88]. For children with developmental speech disorders, multisyllabic word repetition tasks help elicit target phonemes and reveal characteristic speech errors, offering insights into underlying difficulties in speech acquisition [89], [90], [91].

To increase cognitive demands, some clinical speech studies incorporate tasks such as the Stroop Color Word Test [92] or word list learning and recall [93]. The Stroop test assesses cognitive inhibition by requiring speakers to resolve conflicts between word meaning and font color, and has been used in evaluating patients with frontal lobe lesions and Parkinson’s disease [94], [95]. Word list learning tasks impose moderate computational load and are commonly used to assess memory function in individuals with conditions such as major depressive disorder or dementia [93], [96].

Certain tasks at the outer edges of the elicitation task space in Fig. 4, where speakers are pushed to their sensorimotor / physical limits in terms of speed, coordination, and vocal control, are often referred to as “maximum performance tasks”. Examples include the diadochokinetic (DDK) tasks, which evaluatesarticulatory speed and coordination by having speakers rapidly repeat syllables like /pә/, /tә/, and /kә/; the maximum phonation time task, which measures respiratory function and glottal efficiency by assessing how long a speaker can sustain a vowel on one breath; and the pitch glide task, which requires speakers to sweep their pitch range to assess phonatory control and vocal range [97], [98], [99].

These edge-case tasks are particularly informative for evaluating motor speech and voice disorders, where deficits stem from physiological impairments or reduced neuromuscular control. They are commonly used in populations with vocal fold paralysis or spasmodic dysphonia [100], [101], neurodegenerative diseases such as ALS, and respiratory conditions like chronic obstructive pulmonary disease (COPD).

2). Loosely structured / unstructured elicitation tasks:

For populations with cognitive or perceptual impairments, such as Alzheimer’s disease, aphasia, or schizophrenia, speech elicitation tasks that impose higher cognitive demands, including picture description and storytelling, can be particularly effective for revealing deficits in language production.

Common examples include the Cookie Theft picture description and the Cinderella story narrative, which are widely used to assess cognitive-linguistic function [102], [103]. These tasks require coordination of memory, attention, and discourse organization. Patient responses can reveal impairments in language proficiency, coherence, and informativeness, with deficits often manifesting as repetitive, disorganized, or underspecified speech [104], [105]. These tasks have also been used to distinguish speech patterns between individuals with schizophrenia and healthy controls [58], as well as in studies on Alzheimer’s disease [102], aphasia [106], [107], traumatic brain injury [108], and other conditions. While these tasks are considered loosely structured, they still provide a thematic framework that guides responses, distinguishing them from fully unstructured tasks.

Unstructured tasks, such as open-ended conversations or clinical interviews, offer greater freedom in topic and form and can elicit speech behaviors not observable in more constrained settings. These tasks are particularly useful when the speech production mechanism remains largely intact, but higher-level language, memory, or cognitive processes are impaired. In mood disorders, longer interview-based protocols that prompt discussion on predefined topics (e.g., anxiety or well-being) have been shown to produce speech features, such as increased pausing, slower speaking rate, and reduced response length, that correlate with depression severity [109], [110]. In schizophrenia, unstructured dialogue can reveal abnormalities in turn-taking [111] and reduced semantic coherence [112], which may be masked during more guided elicitation tasks.

The discussion above highlights a broad spectrum of elicitation tasks, ranging from simple, highly structured formats to cognitively demanding, unstructured interactions. There is no one-to-one mapping between clinical conditions and specific tasks. For instance, both recall tasks and picture descriptions are commonly used to assess cognitive impairment. As a result, many studies compare multiple tasks to determine which one reveals clinically relevant deficits the most effectively. For example, [113] found that aphasic patients produced more words and exhibited greater linguistic complexity and accuracy during storytelling tasks compared to picture description, suggesting that storytelling may better capture language deficits for this patient population. In [114], aphasic patients showed improved word retrieval and error correction during conversations compared to single-word naming tasks. These differences highlight how task design can influence the observed speech output and associated features.

Feature variability between tasks further complicates the comparisons. For instance, fundamental frequency (F0) variability can be extracted from either reading passages or sustained vowel tasks, but these tasks capture different vocal characteristics, e.g. coordinated pitch use in natural speech versus maximum pitch control in a constrained context, making direct comparison inappropriate [115], [116]. In another example, [117] demonstrated that the spatiotemporal index (STI), which is used to assess speech motor control of the tongue and jaw, varied significantly across sentences of differing length and complexity, underscoring the sensitivity of features to the structure of the elicitation task.

There is no one-size-fits-all speech elicitation task for clinical AI applications. Selecting the right speech task for clinical AI requires careful consideration of the target condition, model objectives, and patient population. The purpose of the model, whether for diagnosis, classification, or monitoring, determines the required sensitivity, specificity, and data requirements. Task selection should be driven by the most affected speech processes (e.g. Conceptualization, Formulation, or Articulation) and account for patient limitations such as cognitive ability, sensory impairments, and language or cultural factors. Additionally, the choice of speech analysis (e.g., ASR, acoustic features) must align with both the task and clinical goals. For concrete examples of task selection, readers can refer to the tutorial paper in [56] (see Tables I and II).

C. Clinical Speech Data Collection

The speech data collection protocol plays an important role in capturing clinically relevant acoustic and linguistic features, but the recorded speech can still be affected by confounding factors such as the type of recording device, post-processing firmware, and environmental conditions, all of which can impact the raw signal and the features extracted from it.

A major source of noise in data collection stems from the flexibility in choosing devices (e.g., smartphones vs. professional recorders), platforms (face-to-face vs. remote sessions), and environmental noise. For example, Ge et al. compared speech data collected by different devices, i.e. smartphones and laptops [118], where notable differences in second and higher-order formant frequencies, spectral moments, and voice quality features were found between devices. With increasing popularity in speech data collection in virtual meeting platforms, Tran et al. explored how speech compression algorithms used during data transmission could affect acoustic features in the assessment of dysarthric speech [119]. The findings revealed that articulatory-based features were significantly impacted by variations in sampling frequencies, bit rates, and coding formats. Additionally, [120] demonstrated that reverberation in background could influence paralinguistic features extracted from open-source toolkits, affecting algorithms for evaluating emotion from speech. Clinical conditions can further complicate the data collection process due to challenges in speech production. For instance, individuals with movement disorders may experience tremors, dystonic postures, cognitive issues, and fatigue, all of which can affect speech loudness and clarity [121].

To address these challenges, studies have proposed reliable protocols for clinical speech data collection. For dysarthric speakers, [121] outlined strategies to improve consistency, such as controlling the mouth-to-microphone distance, minimizing environmental noise by using head-mounted microphones, and avoiding firmware post-processing (e.g., compression, equalization, gain control, noise cancellation) that alters the original speech signals. Another comprehensive guideline for voice-related studies [122] recommends using microphones with a flat frequency response across the range of voice frequencies, ensuring noise levels are at least 15 dB quieter than the softest phonations, and selecting devices with a dynamic range that exceeds the loudest phonations. Directional microphones should be placed at their optimal distance (e.g., 30 cm or 5 cm) to avoid proximity effects and ensure accurate spectral and level measurements.

In addition to front-end protocols aimed at minimizing variability during data collection, some studies focus on verifying the integrity of recorded speech through hardware validation, one of the key steps in the V3 validation framework introduced in [123]. This approach ensures that the recording hardware and firmware accurately capture and process the speech signal. It includes both inter-sensor comparisons, which evaluate consistency across different devices, and intra-sensor comparisons, which assess the stability of recordings from the same device under different conditions.

1). Inter-sensor comparison:

Inter-sensor comparisons validate the consistency of data captured across different sensors. For example, Fahed et al. compared speech data from Huntington’s disease, patients who have involuntary movements of the body that impact their position relative to sensors. Data collected via smartphones, tablets, and professional microphones in a telepractice revealed vulnerability to these positional changes. [124]. Acoustic features such as F0, harmonic-to-noise ratio (HNR), jitter, and shimmer were analyzed from sustained vowel tasks. The results revealed significant differences in HNR and shimmer but strong agreement in F0 and jitter between devices. This highlights the need for caution when using HNR and shimmer in clinical speech models if data is collected from various mobile devices. In another study, Szabo et al. validated voice accumulators for continuous monitoring of F0, phonation time, and sound pressure level [125]. While F0 and phonation time correlated well with human-annotated ground truth, inter-device variability was high, emphasizing the importance of careful device placement and attachment for reliable features.

2). Intra-sensor comparison:

Intra-sensor comparison ensures the consistency in data collected by the same sensor over time. For instance, Printz et al. validated a dual-microphone setup for measuring voice range profiles (VRP), which are used to assess phonosurgery outcomes [126]. In a test-retest experiment, healthy participants repeated voice tasks within 6-37 days, and parameters such as F0, semitone range, sound pressure level, and VRP were analyzed. The results showed strong Pearson correlation coefficients (>0.7 on average) for most acoustic parameters, confirming the reliability of the dual-microphone setup for speech data acquisition. Similarly, Hu et al. conducted a test-retest reliability study to evaluate consumer-grade mobile devices for clinical speech data collection [127]. The study involved 51 participants completing two sessions at least 24 hours apart, performing tasks such as sustained phonation, DDK tasks, and picture description. The findings indicated that DDK tasks reliability significantly decreased on consumer-grade devices compared to professional ones, while F0 and cepstral peak prominence in sustained phonation showed consistent reliability across sessions and devices. This suggests that while consumer-grade equipment may be suitable for some analyses, it can reduce reproducibility in other tasks.

These studies highlight the importance of rigorous hardware validation in ensuring that clinical speech data remain consistent across devices and stable over time. When combined with carefully designed data collection protocols, such validation enables the creation of datasets that preserve clinically meaningful acoustic and linguistic features. In the following sections, we will discuss approaches for extracting informative speech features from clinical datasets to address relevant clinical tasks.

D. Trade-off between Data Quality and Collection Feasibility

There is often a trade-off between data quality and feasibility in speech data collection. For example, consider remote assessment of speech in patients with ALS. Strictly enforcing the recording controls described in the previous section may improve data quality but can make the protocol burdensome for patients, ultimately reducing feasibility and hindering model development.

Conversely, relaxing control introduces variability that propagates into the representations. Acoustic features such as F0, formants, jitter, and shimmer are highly sensitive to noise, reverberation, and microphone placement [128], [129], whereas linguistic features derived from ASR (e.g., word embeddings) are more robust because modern ASR systems tolerate moderate signal degradation. Unchecked variability can degrade the system’s ability to detect clinically meaningful changes and diminish its clinical utility.

To balance quality and feasibility, researchers can draw on established guidance for recording setups [130], [131] and use algorithms that automatically evaluate speech quality [132]. Feature-level analyses (e.g., repeatability measures, statistical tests) can quantify the effect of recording conditions, while external metadata (e.g., microphone type, noise levels) can be incorporated to monitor collection quality or condition models to mitigate noise effects. User surveys can further highlight practical challenges in deploying protocols within clinical workflows and inform iterative refinements [133]. Ultimately, the clinical context determines the acceptable trade-offs and mitigation strategies.

E. Uncertainty in Diagnostic Labels

Even when data collection protocols are properly implemented, label uncertainty remains inevitable in the process. Many symptoms are nonspecific and manifest similarly across different diseases [75], [134], [135]. The hierarchical framework of speech production (see Section IV-A) provides a basis for understanding the variations in reliability of clinical diagnostic labels. Diagnostic label uncertainty increases progressively as the primary deficits shift from lower-order (Articulation) to higher-order (Conceptualization) impairments. For instance, lower-order conditions (e.g. cleft palate, vocalfold paralysis, Parkinson’s disease, ALS) are typically characterized by high diagnostic specificity. These symptoms have an immediate and evident impact on speech production mechanics. In contrast, higher-order conditions (e.g. aphasia, Alzheimer’s disease, depression, schizophrenia) exhibit extensive symptom overlap. The relationship between the conditions and produced speech is indirect. This results in higher rates of misdiagnosis and label noise. For further details on how label uncertainty increases across the speech-production hierarchy, see [12].

High specificity in ground-truth diagnostic labels is essential for training supervised models capable of differentiating conditions with overlapping speech symptoms. However, the inherent noise and uncertainty associated with higher-order conditions inevitably contaminate the training labels of any diagnostic classification model built on them. Recognizing the expected level of label uncertainty for a given condition thus provides essential guidance for model design, such as choosing techniques that account for or mitigate such noise.

Having discussed how speech is elicited and collected in clinical settings, we now turn to how the resulting audio is represented for use in AI models. The choice of representation determines which aspects of the speech signal are preserved as model inputs and influences how model outputs are interpreted by end users.

V. Traditional Speech Representations

Many time-domain and frequency-domain speech representations used in clinical applications have been repurposed from other domains of speech AI, such as ASR, speaker verification, and speech synthesis. Their effectiveness in achieving good published classification accuracy results across more traditional speech technology applications has led to their gradual adoption in AI-driven clinical speech applications.

Spectral representations such as Mel-frequency cepstral coefficients (MFCCs), filter-bank (F-bank) coefficients, and linear predictive coding (LPC) coefficients capture the distribution of energy across frequencies over time and (principally) reflect characteristics of the vocal tract [136]. Their use in clinical settings is based on the assumption that if a condition affects speech production, the resulting acoustic changes and their variability across individuals are preserved within these lower-dimensional representations. For example, in [137], MFCCs were extracted from sustained vowel and rainbow passage tasks to distinguish pathological speech from healthy speech. In [138], high-order statistics of LPC coefficients were extracted from sustained phonation to assess pathological speech quality. Similarly, in [139], the amplitude and phase of spectrograms were used to estimate pathological voice quality. In these cases, spectral representations of speech were used to predict clinical labels in a straightforward, end-to-end manner, without distilling intermediate representations from the speech spectrum.

Research in clinical speech AI has also explored representations borrowed from speaker recognition and verification systems [140], [141]. These systems, originally designed for biometric verification, aim to derive fixed-dimensional representations that capture speaker identity, typically distilled from spectral representations of speech. These speaker representation vectors are designed to capture utterance-level, long-term temporal patterns, enabling the learning of each speaker’s unique traits. Common methods include i-vector [142] and deep learning based x-vectors [143], [144]. While these representations were initially shown to capture demographic information such as gender, age, and language [145], [146], researchers began applying them in clinical contexts, motivated by the potential for these representations to capture a broader spectrum of speaker-related information. For instance, i-vectors and x-vectors have been used for depression assessment [147], oral cancer speech evaluation [148], speech sound disorder detection [149], and Parkinson’s disease analysis [150]. In [147], i-vectors were paired with a binary logistic regression classifier to detect depression. In [148], x-vectors extracted from patients with oral and neck cancer were used to predict speech intelligibility through regression tasks. Similarly, in [149], i-vectors derived from probability outputs of an ASR system were used to capture pronunciation characteristics in children, with binary classifiers trained on these i-vectors to detect speech sound disorders in children.

In recent years, it has become standard practice to use deep learning to extract representations from speech data for specific clinical tasks. Notably, speech and language foundational models have gained popularity and are now widely available to the research community. These large-scale models are trained on vast amounts of speech or text data in an unsupervised manner, with the goal of capturing a wide range of phonetic, acoustic, and linguistic information. For example, transformed-based self-supervised learning speech models such as Wav2Vec2.0, HuBERT, WavLM, and Whisper [151], [152], [153], [154], [155] are increasingly used to extract speech representations for clinical applications. These self-supervised speech representations have been used for pathological speech recognition [156], voice disorder classification [157], and dysarthric speech severity classification [158], etc. Research has uncovered insights into what selfsupervised speech representations learn from speech signals. For instance, Choi et al. found that these models are better at capturing phonetic details than semantic meaning [159]. The probing experiments by Ma et al. revealed that these pretrained representations can encode a wide range of phonetic information, significantly outperforming traditional MFCCs in phonetic classifications [160]. Research has also explored which transformer layers work best for the clinical tasks of interest, with findings varying by application. [161] found that the final layers of Wav2Vec2.0 performed best for dysarthria detection, while [162] showed that middle layers worked well for hypernasality prediction. In cross-lingual depression detection [163], early layers of HuBERT and WavLM proved most effective. While no universal consensus exists, these studies suggest that selectively using specific layers could improve model performance.

In addition to designing representations that focus on short segments of speech, recent advancements have introduced representations aimed at analyzing suprasegmental characteristics, patterns that extend over longer stretches of speech. For example, conformer-based paralinguistic speech models have been developed to process speech signals using longer time windows (e.g., 2 seconds) [164], [165], [166]. Evaluations on various non-semantic speech tasks [167] demonstrated that these models outperformed those using shorter segmental analysis windows (e.g., 20 ms), such as Wav2Vec2.0. These suprasegmental representations have been successfully applied to predict the progression of ALS [168], assess the severity of depression [169], and evaluate speech production skills in children [170]. In these approaches, high-dimensional speech representations are extracted to encapsulate the clinical information embedded in the speech signal, with the expectation that they will enhance the performance of downstream clinical prediction and classification tasks.

While these representations are primarily selected to optimize model accuracy, where their success suggests they capture valuable clinical information, the resultant features still lack interpretability. To bridge this gap, efforts have been made to interpret these representations relative to well-understood clinical constructs. For instance, correlational analyses were conducted between MFCCs, often considered clinically uninterpretable, and other clinically interpretable acoustic features. The results revealed that the second MFCC coefficient can be interpreted as a feature which was previously linked to disease-related voice changes [171]. Similarly, in [172], spectrograms of dysarthric speech were analyzed using principal component analysis (PCA). The findings showed that the first principal component closely aligned with the long-term average spectrum (LTAS), while the second component was associated with formant frequencies. To interpret speaker representations or other deep learning based acoustic features derived from disordered speech, researchers often use low-dimensional visualizations to demonstrate separability between clinical groups [173], [174].

Recent advances have shifted from repurposing traditional representations toward extracting clinically meaningful and interpretable biomarkers of speech directly from audio. While traditional features can be retrospectively interpreted, clinically focused representations are better aligned with clinical use. The next section reviews approaches for deriving and validating such representations.

VI. Designing and Validating Clinically Focused Speech Representations

While many speech representations in clinical speech AI are adapted from general-purpose applications, a growing body of work focuses on clinically interpretable representations that map directly to physiological, neurological, or cognitive constructs. We refer to these representations as speech measures - a term drawn from measurement theory that emphasizes their alignment with defined clinical constructs and their potential for both analytical and clinical validation.

Links between health conditions and specific speech behaviors are well established across speech, neuroscience, and clinical research. For example, individuals with Parkinson’s disease often exhibit reduced pitch variability, perceived as monotonicity [175], [176]; individuals with dysarthria may show reduced vowel space area (VSA), reflecting limited articulatory movement and reduced intelligibility [177], [178], [179]; and patients with Alzheimer’s disease frequently present with reduced fluency, word-finding difficulties, and increased grammatical errors [180].

These clinical patterns have motivated the development of speech measures that quantify relevant constructs directly. For instance, intelligibility can be estimated using word error rate from ASR systems [181], [182]; VSA can be automatically extracted to assess articulatory precision in dysarthria [183], [184], [185]; and pitch variability can be captured through F0 contour analysis to assess monotonicity in Parkinson’s disease [175].

Grounded in clinical theory and interpretable at the level of specific constructs, these measures offer greater potential for integration into clinical workflows and support scalable, explainable use in AI systems. Importantly, this interpretability enables formal analytical and clinical validation by assessing how well a given measure reflects the construct it is intended to quantify.

In the following subsections, we review the concepts of validity and reliability as they apply to speech measures, drawing on principles from measurement theory [54]. We then examine approaches for generating these measures and outline methods for evaluating their analytical validity.

A. Reliability and Validity of a Speech Measure

A valid speech measure must meet two criteria in clinical research: reliability and validity. These concepts are illustrated in Fig. 5, where the green target represents a clinical construct that we aim to measure (such as speech intelligibility). Each star denotes an individual measurement, and its position reflects how closely that measurement aligns with the true construct (validity) and how consistently it can be reproduced (reliability).

Fig. 5.

Conceptual relationship between reliability and validity of a measure. Scattered distribution of data points indicates a low reliability and high variability of measures. The radius of the circle refers to the degree of validity. A small radius indicates high validity of the measure. Measures falling in the green zone are ideal as they have both good reliability and validity.

In the top-left quadrant, the measure is neither reliable nor valid: repeated attempts to assess intelligibility result in scattered, inaccurate values. The top-right quadrant shows a reliable but not valid measure, for example, a signal-derived measure that yields consistent scores across trials, but fails to reflect actual intelligibility as perceived by listeners or quantified through word error rates. In the bottom-left, the measure is valid but not reliable: it captures the true construct on average but with high variability, making it less suitable for tracking change or comparing across contexts. The ideal scenario is shown in the bottom-right, where the measure is both reliable and valid, consistently capturing intelligibility in a way that aligns closely with clinical reality.

Establishing reliability and validity ensures that a measure is both aligned with the construct it aims to assess and capable of producing consistent results across studies and use cases. Validity and reliability are best understood as evolving properties, strengthened over time as evidence accumulates. The process often begins with demonstrating that a measure consistently reflects the targeted clinical construct. As additional data are collected, statistical analyses reveal stronger and more significant associations with established, validated measures. Reliability is reinforced when the measure performs consistently across different contexts, populations, and conditions. Real-world validation, through testing in practical clinical settings, adds another layer of confidence. Ultimately, widespread acceptance follows when the accumulated evidence is sufficient to inspire trust among clinicians and researchers.

1). Reliability:

Reliability refers to the consistency of a measure [54]. In clinical research, two types of reliability are commonly considered: test-retest reliability and internal consistency.

Test-retest reliability examines whether a construct is measured consistently over time. For example, acoustic, aerodynamic, and perceptual measures in voice assessments were validated using test-retest reliability [186]. Measures collected over five consecutive days showed moderate to high intraclass correlation coefficients (ICC = 0.64–0.99), while subglottal pressure and perceptual voice evaluation had low ICCs, indicating poor reliability. On the contrary, in [187], the repeatability of speech and language features extracted from open-source toolkits like openSMILE [188] and Praat [189] was tested on disordered speech. The ICC and within-subjects coefficient of variation (WSCV) revealed that most features failed to meet acceptable thresholds for clinical decision-making [190], [191].

Internal consistency, on the other hand, evaluates whether different speech elicitation tasks produce consistent measures. A common approach is the split-halves method, where the set of items is divided into two parts, and the correlation between the two sets’ measures reflects consistency. For instance, in [192], a clinical assessment tool for dysarthric speech was validated by comparing human-rated intelligibility scores from sentence-reading and single-word tasks, using Cronbach’s alpha to indicate internal consistency. Similarly, in [9], an objective hypernasality measure was evaluated by computing measures from two sets of elicitation materials. A Pearson correlation of approximately 0.9 demonstrated strong consistency.

2). Validity:

Validity refers to how well a speech measure captures the clinical construct it is intended to assess. In clinical speech AI, establishing validity is important for evaluating model performance and interpretability.

One commonly assessed form of validity is convergent validity, which evaluates whether a new measure correlates with existing, validated instruments targeting the same construct. For example, Rowe et al. [193] validated several acoustic speech measures for ALS, such as syllable duration ratios, variability in timing, and F2 slope, by comparing them to expert ratings of speech coordination, consistency, and speed. High correlation coefficients supported the use of these acoustic measures in profiling motor speech deficits. In contrast, Yawer et al. [194] examined a speech-based stress detection tool by comparing its outputs to the Perceived Stress Scale [195], a widely accepted psychometric instrument. Weak correlations indicated insufficient validity for clinical application. Another example comes from Iter et al. [196], who proposed speech incoherence and tangentiality measures for assessing schizophrenia. Though initially promising, subsequent analysis by Hitczenko et al. [197] showed the tangentiality metric failed to generalize and was more strongly correlated with sentence length and demographic variables than with clinical symptoms.

These examples highlight the need for rigorous validation protocols, including diverse speech tasks, repeated measures, and multiple clinical labels. They also underscore a broader challenge: the quality of the reference standard itself. Many clinical assessments rely on perceptual ratings, which are inherently subjective and can vary by rater experience, task difficulty, and environmental factors [198]. Such variability introduces noise into the “ground truth” labels used for validation. Improving the reliability of these reference standards through standardization, improved training, or consensus methods can strengthen the foundation for validating automated measures.

B. Knowledge-driven Measure Design

For the clinical condition of interest, many valid speech measures identified in existing clinical studies can be extracted using signal-processing techniques or text analysis, which do not require use of AI or machine learning. A review by Voleti et al. summarized a range of measures derived from the acoustic and language aspects of speech to assess cognitive and thought disorders [62] (See Table II). These acoustic-based measures are not only clinically meaningful but also widely applicable across various conditions such as autism spectrum disorder in children [199], Parkinson’s disease [200], [201], [202], and Huntington’s disease [203], [204]. These are further categorized into: Prosodic measures, such as durations of pairwise voiced segments, duration of silent segments, loudness, periodicity, and non-verbal cues such as interruptions, interjections, natural turns, reflect irregularities in rhythm and timing of speech [205], [72]; Articulatory measures such as formant frequencies, vowel space area, and formant trajectories over time describe atypical movements of speech articulators [206]; and Vocal quality measures, such as jitter, shimmer, and HNR, describe the vibratory characteristics of the vocal folds. In the extraction of acoustic measures, opensource toolkits such as Torchaudio [207], Parselmouth [208], Praat [209], SpeechBrain [210], and openSMILE [188] are widely available. However, it is important to note that these tools may differ in their methodologies for extracting the same type of measure. Such differences can introduce variability in the results, making it challenging to compare across papers using different toolkits.

Many conditions affecting Conceptualization and Formulation (see Section IV) are also associated with atypical language use. As a result, speech measures can be derived from text transcriptions using natural language processing (NLP). From a language perspective, lexical diversity, which reflects unique vocabulary usage, can be calculated by analyzing the number of unique words relative to the total word count in the speech sample [211]. Additionally, the lexical and syntactic complexity of spoken sentences can be analyzed using parse trees to derive grammatical structure, with language complexity measured by statistics related to the branches of these trees [212], [213]. These language-based measures have been shown to be useful in assessing conditions related to language and cognition.

Knowledge-driven measures, particularly those validated in clinical studies, offer an initial baseline for clinical speech AI models. They are especially valuable when dealing with new clinical speech applications or rare conditions with limited data. In such cases, starting with measures that are already known to change with the condition provides a reliable baseline. These measures can be directly incorporated into speech models, offering a level of robustness and interpretability. One tradeoff is that knowledge-driven measures focus on preestablished patterns and may not uncover new or previously-unknown relationships between speech characteristics and the disease.

C. Measures from Acoustic Models

Clinical speech samples are limited across many conditions [48], [47], [214], [215]. This has motivated the use of general population data to train models that capture representations of healthy speech, which can be applied to assess various aspects of clinical speech. An example of this approach is acoustic modeling, a core component of conventional ASR systems. These acoustic models are typically trained on speech data from the general population to recognize linguistic units from short segments of speech. With a proper training setup, acoustic models can capture important articulatory information since it is well known that clinical speech conditions often manifest at the segmental level, such as distorted sounds, persistent mispronunciations, and misarticulations, cf. [216], [217].

Ideally, a poorly produced phoneme would be assigned a low probability by the acoustic model. As a result, the model’s probability outputs for linguistic units can reflect the quality of speech production. There has been broad application of probability outputs (e.g. posterior features) in clinical speech AI. In [29], given voice-disordered speech, frame-level posterior features were obtained from the ASR outputs. Statistics such as mean and standard deviation were applied to the features to derive a utterance-level representation to indicate voice quality. Experimental results showed the representation could differentiate between various severity levels of voice disorder. In [218], it was hypothesized that the variation in speech characteristics can be described by concatenated posterior outputs from multiple acoustic models from different languages. The multilingual posterior features were shown effective in distinguishing between pathological and well-produced healthy speech. In [219], speech from aphasic speakers was analyzed by comparing sequences of posterior features extracted from both a strong and a weak ASR system using a Siamese neural network. This comparison aimed to identify differences in the posterior features caused by speech impairment. The output of the Siamese network, which reflects the similarity between the two posterior sequences, demonstrated the ability to distinguish between speakers with high severity and those with low severity of aphasia. In [220], [221], phonological features that categorize phonemes based on phonology were derived from acoustic model output and used to assess intelligibility or voice characteristics in pathological speech.

With proper validation, these probabilistic outputs can serve as valuable measures for describing the target clinical construct and assessing clinical speech. The general pipeline for utilizing acoustic model outputs to derive speech measures is illustrated in Fig. 6. In this approach, a pre-trained acoustic model trained on speech from the general population is used to evaluate clinical speech samples. The outputs of these models are validated against existing clinical constructs, making it possible to reason about the relationships between model predictions and clinical conditions.

Fig. 6.

Acoustic models are trained to predict speech units (e.g. phones, phonological features, oral/nasal sounds). These output posteriors enable computation of speech measures, such as goodness of pronunciation (GoP), articulatory precision, hypernasality, etc. Once validated, they offer reliable, interpretable inputs for clinical speech AI models.

The goodness of pronunciation (GOP) score, first introduced in [222], is a notable measure of articulation based on probability outputs from acoustic models. The GOP uses the posterior probability of an acoustic model to quantify the discrepancy between a speech sound’s actual and expected realization. This measure has been widely used in computer-assisted pronunciation training (CAPT) [222], [223]. The concept has been extended to clinical speech research, where the GOP measures computed from pathological speech have been shown to correlate significantly with clinical assessments of neurological and anatomical disorders [224].

The approach in Fig. 6 can be generalized beyond GOP. For example, there are several conditions where it may be useful to assess hypernasality in speech, precision of consonantvowel transitions, place and manner of articulation in children’s speech, etc. By using domain-specific linguistic units as labels in acoustic modeling, posterior probability outputs can offer different clinical interpretations, capturing the quality of speech production from various perspectives. In [225], a recurrent neural network (RNN) was trained using MFCCs and phonological features to classify speech into 16 articulation categories based on place and manner. A distance metric was then computed from the classifier outputs to compare these classifications with those of healthy speakers, enabling an evaluation of articulation abilities in individuals with dysarthria. The derived measures were compared to perceptual ratings, revealing significant correlations for half of the classes, confirming their reliability in capturing clinical constructs. In [9], the authors introduced an algorithm to assess hypernasality in cleft speech using a model trained solely on healthy speech. They categorized English consonants and vowels into nasal and oral groups, trained an acoustic model on MFCCs, and used posterior probabilities to derive an objective hypernasality measure (OHM). The OHM demonstrated strong internal reliability, validated through the split-half method, and demonstrated significant external validity, as supported by strong inter-rater reliability on an unseen dataset.

A posterior-based measure was also designed to evaluate the fidelity of consonant-vowel transitions, which are sensitive to imprecise articulation [226], [227]. In [228], a convolutional neural network (CNN) was trained on mel-spectrogram of speech from Librispeech to classify the 20 English consonants based on the consonant-vowel segments. Once trained, an objective articulation measure (OAM) was derived from the posterior probabilities outputs. The resulting OAM showed strong correlations (0.83 for dysarthric speech, 0.66 for cleft speech, see Fig. 7) with perceptual ratings of articulation precision and speech acceptability. Saliency maps further confirmed the OAM’s focus on relevant speech regions, and it outperformed traditional measures like GOP in reliability and consistency. These validation results have demonstrated the potential of OAM being used as acoustic correlate of articulatory precision across a variety of clinical applications.

Fig. 7.

Analytical validation of a proposed speech measure. The scatter plots illustrate speaker-level OAM vs. a) perceptual ratings for dysarthric speech; b) acceptability rating of cleft palate speech. Figure reprint from [14]

Beyond using the final posterior outputs from acoustic models to design speech measures, the activation patterns of thousands of neurons in the intermediate layers of deep neural network during phone recognition can also be leveraged to create new speech measures. The study in [229] explored this approach to assess speech disorders caused by head and neck cancer. The activation values of the neurons in a trained deep neural network (DNN) were analyzed to identify those most sensitive to specific phonological features. For each feature, an Artificial Neuron-based Phonological Similarity (ANPS) score was calculated based on the activation values of these selected neurons, providing a measure of speech production quality tailored to the phonological feature in question. Visualization of ANPS score for each phonological feature allows straightforward interpretation about deficits in production of specific speech sounds (see Fig. 8). Validation revealed that most ANPS scores significantly correlated with perceptual measures of both consonant and vowel production quality, demonstrating their potential as reliable indicators of speech impairment.

Fig. 8.

Visualization of ANPS scores per consonant phonological feature (y-axis) and disordered speakers with head and neck cancer, sorted by severity (x-axis). Figure reprint from [229].

These examples highlight the broad applicability and potential of using acoustic models to derive speech measures for assessing a wide range of speech-related conditions. However, it is important to note that systems based on acoustic modeling are not equally reliable across all languages. For analyses conducted in languages other than English [230], [231], acoustic errors can occur even with flawless pronunciation, adding complexity to the analysis. This occurs due to a lack of data and necessitate caution when interpreting results in languages where the data used to train the acoustic models is sparse.

D. Modeling Perceptual and Self-reported Speech Scores

Assessment in many clinical speech applications has traditionally relied on perceptual judgments made by experienced clinicians or self-reports by patients or caregivers, focusing on constructs such as articulation precision in dysarthria [233], social communication skills in schizophrenia or bipolar disorder [234], [235], and vocal quality in dysphonia [236]. Properly trained and validated AI models hold significant potential to automate these assessments by predicting existing speechbased perceptual scales directly from speech and language data. This approach seeks to develop objective proxies for these well-validated clinical assessments, and it is also widely adopted in domains such as speech quality assessment [237]. Focusing on clinically important outcomes inherently linked to the speech signal makes the problem more tractable and enhances the likelihood of success.

1). Prediction Models Based on Speech Acoustics:

For several conditions, perceptual assessment of patient-produced speech is clinically valuable. For instance, in the assessment of dysarthria, perceptual ratings such as articulation precision and speech intelligibility, provided by trained clinicians, are important for tracking patient progress [238], [239]. Given their importance, these perceptual measures offer valuable targets for AI models. The goal of these models is to replicate how professional listeners assess clinically-relevant acoustic properties of speech directly from acoustic data. As illustrated in Fig. 9, once clinical speech data and corresponding perceptual ratings are collected, a supervised learning model can be trained to predict these ratings from the input speech. With sufficient validation, these predicted ratings can serve as reliable and objective speech measures for clinical assessment of conditions like dysarthria.

Fig. 9.

Speech recordings are rated by human listeners for clinical dimensions such as speech intelligibility, voice quality, hypernasality, articulatory precision. In parallel, objective validated measures can be computed. Pairing these ratings and/or measures with the recordings allows training a regression model to predict the scores directly from new speech inputs. This bypasses manual rating and multi-step computational pipelines.

Tu et al. [232] demonstrated this approach by utilizing a DNN to predict perceptual ratings for the objective assessment of dysarthric speech. An intermediate layer in the DNN served as an information bottleneck, which was trained to predict multiple perceptual dimensions such as nasality, vocal quality, articulatory precision, and prosody. This interpretable layer acted as a regularizer, constraining the solution space of the neural network and preventing overfitting, especially when training data is limited. Using the predicted perceptual ratings from the bottleneck layer, a shallow neural network was further trained to predict the severity of dysarthria as the diagnostic output. The predicted ratings showed strong correlations with manual perceptual ratings, achieving Pearson correlations between 0.7 and 0.8. The severity scores predicted using these predicted measures also correlated well with ground truth severity levels, with a Pearson correlation of 0.8, supporting their clinical validity. This approach was further extended by Xu et al. in dysarthric speech classification [240]. Instead of collecting perceptual labels, the authors utilized validated and computational-based measures (e.g. GOP, OHM, OAM, and cepstral peak prominence) as the prediction target, saving the laborious task of perceptual label collection. The authors applied Shapley values to explain the contribution of each clinically-interpretable measure to the assessment [241], [242]. The studies demonstrated that the use of predicted measures improved both the model’s accuracy and its interpretability.

When using human-provided perceptual labels as prediction targets, they must be carefully obtained to improve model robustness during both training and validation. For example, inexperienced listeners have been shown to assess voice quality in pathological speech differently from experienced listeners [243]. Collecting low-quality labels can lead to misleading validation results and poor performance when the algorithm is deployed in practice. To mitigate this, clinical label collection should be informed by guidance documents from professional organizations [244], [245], [121].

2). Prediction models based on language transcripts:

Transcribing clinical speech data into text data allows developers to derive NLP-based clinical measures. While NLP-based speech measures are traditionally derived from word or phrase statistics, such as frequency and types of words used, semantic relevance, syntactic complexity, etc. [62], [246], recent state-of-the-art text analysis has shifted from the statistical approach to data-driven approach. Text embeddings extracted from models such as word2vec, GloVe, and newer transformer-based models can capture semantic information from text [247], [248], [249]. The relationship between words or phrases can be reflected through their similarity (e.g. cosine similarity) computed in the embedding space. Using the similarity measure, valid and interpretable speech measures can be obtained for the analyses. The pipeline of deriving speech measures using NLP is illustrated in Fig. 11.

Fig. 11.

Using natural language processing (NLP), we can derive clinically-relevant speech measures from cosine similarities between text embeddings. These similarity scores capture rich semantic, syntactic, and discourse-level patterns in spoken language. They were shown to associate with cognitive status, neurological disorders, and mental health conditions, etc.

For example, semantic clusters and switches, which describe verbal fluency of a speaker, were measured using the cosine similarities between word2vec embeddings [250]. Semantic clusters reflect how the speaker elicits semantically related groupings of words over time, while the semantic switches measure the frequency of a person producing words that are semantically unrelated. In the assessment of early-stage psychosis, cosine similarities between adjacent pairs of word embeddings were used to derive the cluster-related semantic similarity and switch-related semantic similarity respectively. To establish the clinical validity, the measured similarities were shown to significantly correlate to the Communication Disturbances Index [251], a clinically-validated summary score used to identify disorganization in natural speech. In [252], coherence, which refers to the semantic similarity between the interconnected flow of ideas, was measured from cosine similarities between word embeddings extracted from the FastText model to assess formal thought disorder [253]. Experimental results showed that the centroid-derived coherence measures, where each word embedding was compared with the summed embeddings of previous words, achieved satisfactory correlation with the clinically validated Thought and Language Disorder (TALD) ratings performed by human annotators. In [254], to assess speech from patients with schizophrenia, coherence was measured by cosine similarity between embeddings of adjacent utterance pairs, using the word2vec model and the GloVe model. Statistical analyses showed that the coherence measures were significantly correlated to various social cognition ratings such as emotion processing, mentalizing, and attribution bias.

VII. Speech-based AI Models for Clinical Label Prediction

Once clinically relevant speech representations have been established, the next step is to use them as inputs to models that predict clinical labels. These labels may reflect diagnostic status, prognostic outcomes, disease severity, or performance on standardized clinical assessments. Clinical speech models are designed to support decision-making by providing objective, automated predictions that align with these labels.

Developing such models involves key considerations, including the volume of available training data, the interpretability of model outputs, computational efficiency during inference, and generalizability across populations and recording conditions [255]. There is also considerable flexibility in how these models are implemented. This includes variations in the number and type of speech representations used, as well as the complexity of the underlying algorithms, which range from traditional machine learning models with relatively few parameters to deep neural networks with billions of parameters.

In the sections that follow, we first review classical machine learning approaches, including linear regression, decision trees, and other parametric models. We then discuss the use of deep learning architectures in clinical speech prediction, followed by an overview of emerging applications that leverage large language models (LLMs) for clinical label prediction.

A. Traditional AI Models

Traditional AI models such as linear regression, decision trees, and other parametric models are inherently interpretable when used with validated speech measures [256]. Their decision processes are directly encoded in their parameters and structure, making them easier for humans to understand. Unlike complex models that require secondary models to approximate model behavior (e.g., LIME), these models can foster trustworthiness as they can be interrogated to reveal which measures were most influential in the decision, providing clinicians with actionable insights. Fig. 12 illustrates the pipeline of clinical model prediction based on speech measures.

Fig. 12.

Workflows for traditional AI models in clinical label prediction: low-complexity classifiers process low-dimensional inputs constructed by a small set of speech measures. Prior to clinical deployment, clinical validation is performed to systematically evaluate whether the model pipeline and the outputs address the intended clinical questions and align with the predefined clinical purpose.

Validation and explainability are generally more straightforward for simple models that rely on well-defined, construct-driven measures validated for specific conditions. For example, when predicting clinical labels for Alzheimer’s disease, focusing on a specific set of clinically understandable speech measures, such as the increased number of pauses or reduced vocabulary size, ensures the model relies on inputs grounded in prior research. Using lower-dimensional inputs also allows developers to build models that are simpler, easier to interpret, and less prone to overfitting, leading to more generalizable solutions. Furthermore, the reduced complexity of these models allows for more thorough validation, as each input feature can be individually evaluated for its contribution to the model’s predictions.

There are several examples of such models in the literature. For example, in [257], the authors develop a model for predicting forced vital capacity (FVC) from speech data collected from maximum phonation task, aiming to monitor FVC longitudinally. A 3-dimensional input comprising maximum phonation time, age and gender was constructed. A linear model was then trained to predict the FVC. Clinical validation followed to ensure the trained model addressed the intended clinical question and met the predefined context of use, i.e. longitudinal tracking of FVC. The predicted FVC showed a strong Pearson correlation of 0.8 with ground truth measures. Test-retest reliability reported an ICC greater than 0.9 in FVC obtained from repeated samples. The model’s ability to detect longitudinal decline in FVC was validated by comparing the trajectories of predicted and observed FVC, where both trajectories had similar intercepts and declining patterns (see Fig. 13). The clinical validation suggested that the maximum phonation task could be a useful surrogate for vital capacity in both research and clinical settings, while it provided evidence for the efficacy, reliability, and interpretability of the FVC prediction model.

Fig. 13.

Observed (left) and predicted (right) FVC longitudinal trajectories. Figure adopted from [24].

In [3], the semantic relevance (SemR) of transcribed speech was utilized to predict cognitive decline in speakers automatically and longitudinally. SemR quantifies the relevance of a spoken response relative to a picture description elicitation task by calculating how much of a speaker’s response refers to key concepts or elements that are expected based on the picture. The automatically-derived SemR achieved a strong Pearson correlation of 0.84 with manually-extracted SemR. The SemR also exhibited moderate to good test-retest reliability, with an ICC of 0.73 [258]. The study combined SemR with other language features to predict Mini-Mental State Examination (MMSE) scores, a key clinical measure of cognitive function. Incorporating SemR significantly boosted prediction performance, increasing the correlation from 0.38 to 0.65. To further assess the clinical relevance of the predicted scores, the study examined longitudinal trajectories of SemR for cognitively impaired speakers, where results revealed that SemR declined with age and worsening cognitive impairment, providing a clear qualitative understanding of how semantic relevance reflects cognitive deterioration (see Fig. 14). These results demonstrated a rigorous validation process for developing the automated SemR measure, which mitigate the labor-intensive task, expedite research, and enhance clinical assessment in the context of pre-clinical detection of cognitive decline.

Fig. 14.

Longitudinal plot visualizing how clinical output of semantic relevance varies as a function of age. Figure reprint from [3].

In [259], verbal memory in individuals with mental illness was assessed by two features extracted from transcribed speech: common word types and word mover’s distance (WMD). These features were used as inputs for a linear regression model to predict clinician-rated accuracy scores in story recall tasks. The predicted scores based on both human transcriptions and ASR transcriptions showed agreement, achieving a Pearson correlation of over 0.95. Additionally, the ASR-derived features alone achieved a strong correlation of 0.82 with human annotations. The trained regression model provided valuable insights into the influence of each feature. The coefficients for common word types and WMD were 0.15 and −0.54, respectively. This means that a higher frequency of common word types was associated with better accuracy in story recall tasks, while a higher WMD (indicating greater divergence between the recalled and original text) was linked to lower accuracy. These findings validated the model’s predictive power and offered interpretable insights into how each input affects the prediction outcome. They also demonstrated the feasibility of using speech technologies for frequent verbal memory assessments in psychiatric clinical monitoring.

While the above examples mainly used linear regression models as the clinical label prediction, other nonlinear models such as decision tree (DT) are also popular due to their inherent interpretability. For instance, expanding a trained DT reveals how each input feature contributes to the hierarchical decision-making process. In [260], speech intelligibility, computed by word recognition rate (WRR), was used to classify speakers into different severity groups. The trained DT visually illustrated how WRR thresholds were used to split speakers into severity categories, providing insights into the decision process. In [261], part-of-speech (POS) measures extracted with NLP techniques were used to assess speakers with primary progressive aphasia (PPA). A random forest classifier was trained to classify speakers into PPA subgroups. By analyzing feature importance from the trained RF, the study identified key POS measures for distinguishing between nonfluent PPA, logopenic PPA, and semantic PPA. Specifically, the noun-to-adverb ratio, count of determiners, and count of nouns emerged as the most important measures for classification.

The selected studies reviewed above highlight how simple AI models, paired with expert-designed low-dimensional inputs, can effectively address some clinical tasks while also enabling interpretation.

B. Deep Learning Models

In Section VI we reviewed how deep learning can extract clinically relevant measures from traditional spectral features. Similarly, recent research has applied deep learning to uncover abstract, latent, and complex relationships between speech inputs and clinical labels. These studies focus on identifying neural network architectures to improve prediction performance. For example, in [262], speech extracted from 3-second sustained vowel samples was used to train a classical feedforward network (FNN) for a binary classification task that detects voice disorders. The paper demonstrated that FNN outperformed traditional classifiers such as Gaussian mixture models (GMM) and support vector machines (SVM). More recently, research has progressed towards adopting deeper architectures from advancements in other domains, such as computer vision, to enhance prediction performance in clinical speech AI tasks. In [263], a CNN was used to classify intelligibility levels in dysarthric speech, outperforming FNNs. The study highlighted the importance of configuring CNN kernels to jointly process time and frequency information, which was more effective than handling these domains separately. In [264], a CNN combined with long short term memory (LSTM) network was used to classify patients with ALS, Parkinson’s disease, and healthy controls using spontaneous speech, DDK tasks, and sustained phonation. The LSTM’s ability to process long temporal sequences of frame-level features complemented the CNN’s strength in capturing local patterns, resulting in robust performance. Similarly in [265], a bidirectional LSTM (Bi-LSTM) was used to classify different types of voice disorders, further underscoring the utility of recurrent architectures in handling sequential speech data.

There has been a significant interest in the use of transformer-based speech foundation models, such as Wav2Vec2.0, HuBERT, and WavLM, across a wide range of speech-related tasks. These models are pre-trained in a selfsupervised manner on thousands of hours of speech data, enabling them to capture complex latent patterns in speech signals. Studies have demonstrated that when these foundation models are paired with a simple back-end model, e.g. FNN, fine-tuning the entire system with small data (e.g., as little as 10 minutes of speech for ASR [151]) can lead to substantial improvements in downstream task performance. This approach is particularly appealing here as clinical speech datasets are often limited in size. This model pairing has been successfully fine-tuned for a variety of clinical tasks, including pathological voice classification [266], voice disorder detection [267], stuttering classification [268], and dysarthria detection [161].

Many clinical AI models, particularly those based on deep learning to predict clinical labels, often exhibit a “blackbox” nature, making their decision-making processes difficult to interpret. This can pose significant challenges in clinical applications where interpretability is important for fostering trust with clinicians [269]. There are various model interpretation techniques proposed to interpret the decision mechanism behind these models. For example, many advanced speech models are integrated with the attention mechanism [270]; the visualization of attention is a common approach to understand which segments are prioritized by the trained model to make the prediction. For example, in [271], the attention was visualized to verify whether the speech segments of interest are utilized to predict perceptual evaluation of speech quality. Another popular interpretation approach is based on analyzing the gradient of the predicted class. The gradient measures how sensitive the model’s output (e.g. predicted clinical score or the probability of the predicted class) is to changes in the input. The key idea is that input features with the largest gradient magnitudes are the most influential for the model’s decision. For example in Parkinson’s disease prediction from speech [272], the gradient visualization demonstrated that the trained model utilized high-frequency regions for the detection, potentially reflecting attention to certain phones in elicited speech. Similarly, the gradient analysis was used in [228] to analyze the spectral properties of consonant-vowel that impact articulatory precision. Fig. 15 illustrates the saliency maps produced by the CNN-based acoustic model while analyzing consonant-vowel (CV) speech segments from both a control speaker and a speaker with dysarthria. Due to the clinical condition, the same CV sequence was articulated differently, and the saliency maps emphasized regions where articulation difficulties were evident. In Fig. 15(d)-(e), the substitution error involved voicing contrasts, as indicated by the model’s attention to the low-frequency spectrogram regions. Fig. 15(f) further confirmed that the CNN primarily focused on low-frequency components when processing these CV segments. In Fig. 15 (g)-(h), the substitution error was distinguished by the continuous turbulent airflow, Fig. 15(i) showed the model’s focus shifted to high-frequency regions associated with this turbulence, demonstrating that the trained acoustic model processed co-articulated segments as intended.

Fig. 15.

An example of utilizing saliency maps to explain how a CNN acoustic model processes consonant-vowel speech segments produced by the control speaker and the dysarthric speakers [228]. (a), (d) and (g) illustrate the speech signals. Their time-frequency representations are illustrated in (b), (e), and (h) respectively. The saliency maps produced by the CNN are depicted in (c), (f), and (i). Figure reprint from [228].

On the other hand, some studies utilize Shapley values [241] for interpretation, which aim to highlight the importance of individual input parameters in the model’s predictions. This is particularly useful when the deep learning models are used for clinical label prediction. For example in the task of speechbased dysarthria detection, Xu et al. [240] utilized the Shapley values to demonstrate that articulation measures were more important than voice quality measures in the model’s decision making. Similarly in the task of detecting Parkinson’s disease [273], the Shapley value analyses showed that the mean, standard deviation of the counted voiced segment were most important for the task. In particular, the Shapley values indicated that larger standard deviation of voiced segment length pushed the prediction towards healthy speech, whereas lower standard deviation pushed the prediction toward Parkinson’s disease. There are also other interpretation methods such as the Local Interpretable Model-agnostic Explanations (LIME) [274], where a simpler, interpretable model is constructed to approximate and visualize the decision boundaries of more complex deep learning models. It has been applied to the task of predicting arousal and valence from speech [275], synthetic speech detection [276], detecting dementia from speech transcripts [277], etc.

Recent efforts in deep learning have focused on designing models that inherently integrate interpretability. One promising approach is the neural additive model (NAM) [278], which combines the ability of DNN to capture complex data-label relationships with the interpretability of generalized additive models. By visualizing the learned shape functions between individual inputs and outputs, NAM provides clearer insights into how specific features influence predictions. For example, in [279], the NAM was used to classify Parkinson’s disease based on voice-related speech measures. As shown in Fig. 16, the model revealed that higher values of jitter, shimmer, and standard deviation in F0 were associated with an increased risk of disease, offering both predictive accuracy and interpretable results.

Fig. 16.

Visualizing learned shape functions in NAM to demonstrate how F0 measures are used to predict risk of Parkinson’s disease in male speakers. The red bands indicate normalized data density (with darker shades representing higher density), while the solid blue line shows the averaged curve derived from models across multiple cross-validation runs. Figure reprint from [279].

While the above model explanation techniques can offer valuable insights, such as confirming whether a trained model focuses on task-relevant speech segments, they are ultimately approximations rather than true reflections of how the model functions [256]. Moreover, the explanations they generate can be sensitive to the specific data used during interpretation, leading to inconsistencies or even conflicting conclusions [280]. This indicates the need for caution when applying interpretability techniques in clinical speech AI, given the high-stakes nature of the application. These limitations further emphasize the value of interpretable-by-design models, in which the inputs are individually validated and clinically interpretable, and the model structure is kept simple to facilitate transparent understanding of its decision logic.

C. Large Language Models (LLM)

Natural language plays an important role across the clinical domain, supporting everything from note-taking and documenting patient histories to generating diagnostic reports, aiding decision-making during consultations, and facilitating patient–provider communication. The growing accessibility of large language models (LLM), such as GPT-5 [281], Gemini [282], Llama [283], Mistral [284], etc., has therefore found application across various clinical use-cases. Their potential has been showcased in applications such as assisting surgical preparation [285] or analyzing computed tomography (CT) reports to extract oncologic phenotypes and retrieve critical clinical information [286], etc. Even though simple prompting to guide LLM behavior [287], these models were shown to outperform traditional AI approaches, particularly in zero-shot or few-shot scenarios with limited clinical data [288].

In clinical speech AI, there have been initial attempts to use LLMs for classification based on diagnostic labels. Given a transcript of a patient’s speech and an appropriate prompt, the LLM is instructed to produce a specific type of output, such as a diagnostic label, symptom summary, or risk assessment. For example, in assessing cognitive impairment based on transcribed speech, Wang et al. evaluated GPT-based LLMs for primary screening of MCI in a cohort of 174 participants from the DementiaBank database [289]. By prompt-tuning GPT-4 with carefully crafted text prompts, the study reported promising accuracy in distinguishing between individuals with normal cognitive function from those with MCI in the held-out test set. In [290], the LLMs were instructed to rate several macro-descriptors of Alzheimer’s disease, such as cohesion, lexical diversity, etc. These ratings were used to further construct an interpretable, low-dimensional vector to train a classifier to detect the presence of disease. For depression detection from speech [291], acoustic landmarks of speech were extracted and combined with speech transcripts to enrich the text data. The LLM was then fine-tuned by low-rank adaptation (LoRA) [292] on the enriched transcripts to enhance detection performance. In [293], LLM-based classifiers were trained to predict if an erroneous ASR transcript preserves the meaning in disordered speech. Experimental results showed that using LLMs delivers performance that is closest to human evaluation compared with other models. These examples have demonstrated the broad potential of applying LLMs in various clinical settings, ranging from lower stake tasks such as assessing the meaningfulness of transcripts and speech rating extraction, to higher stake tasks such as direct diagnostic label prediction.

D. Enhancing Model Robustness

Although increasing the size and diversity of clinical speech datasets is one of the most effective ways to improve model robustness and generalizability, such data collection efforts are often resource-intensive, time-consuming, and constrained by clinical protocols. As a result, a range of complementary strategies have been developed to enhance model performance in the presence of real-world variability. These include data augmentation techniques that simulate diverse acoustic conditions, regularization methods that improve embedding consistency and reduce overfitting, and domain adaptation approaches that facilitate generalization across datasets with differing characteristics.

Even without the need to collect additional data, several strategies have been developed to improve the resilience of clinical speech AI models to data variability. In [294], data augmentation techniques were used to train deep learning models for the automatic assessment of dysphonic speech (see Fig. 17). By simulating noisy factors unrelated to vocal quality, such as adding background noise to mimic different environments, applying impulse responses to simulate room reverberation and microphone effects, and randomly shifting the pitch to represent speech from unseen speakers, the models were better equipped to handle variability. These augmentation methods, widely adopted in speech recognition, speaker verification, and other clinical tasks [72], [295], [296], [297], [298], significantly improved model accuracy, particularly in cross-corpus evaluations.

Fig. 17.

Data augmentation can be applied to enhance clinical speech models’ noise robustness. Figure reprint from [294].

Another challenge in training deep learning models is the tendency to produce correlated embeddings, which leads to inconsistent model performance across datasets. Regularization techniques have been introduced to address this issue by reducing the model’s over-reliance on correlated features, thereby improving the consistency and repeatability of embeddings. In [299], a regularization term based on the intra-class correlation coefficient (ICC) was incorporated during DNN training. This approach minimized variability within each class, resulting in more repeatable embeddings and enhanced performance in tasks like speaker verification, zero-shot voice style conversion, and dysphonic speech detection. Other approaches directly tackle the co-linearity by modifying batch normalization [300], designing decorrelation-aware objective functions [301], or introducing dedicated regularization terms [302]. In the context of clinical speech tasks, such as dementia speech assessment using NLP, a regularization scheme was applied to fine-tune pre-trained BERT models [303]. By increasing variance and limiting covariance in each embedding dimension, this approach reduced the false alarm rate and improved the precision of severity score predictions compared to human ratings. Domain adaptation techniques further enhance model generalization across datasets. Methods like adversarial learning and fine-tuning on target domain data enable models to adapt to new environments by learning features invariant to domain shifts [304], [266], [305], [306]. This is particularly valuable in clinical settings, where data often comes from diverse real-world environments with varying recording setups and speaker populations.

E. Post-deployment Monitoring of Clinical Speech AI Models

The rigorous analytical and clinical validation during inhouse development establishes the evidence that the model is reliable and valid. As an additional layer, post-deployment monitoring in real-world settings is vital to reveal the model’s true clinical utility, such as its impact on patient outcomes, ensuring ongoing credibility and enabling refinements as needed. A recent report by the U.S. FDA highlights a tenfold increase in approvals for AI-embedded medical devices from 2018 to 2023 [307]. To ensure their long-term reliability and safety, continual post-deployment monitoring is essential. Studies reveal that pre-deployment evaluations often fail to predict long-term performance. An analysis of 32 datasets across four modalities found that 91% of deployed AI models experienced performance degradation over time [308]. In [309], AI models for early sepsis detection generated 43% more alerts during the COVID-19 pandemic due to unforeseen domain shifts in data. Similarly, in [310], an auto-segmentation model for cancer radiotherapy showed declining performance three years postdeployment.

Speech-based AI systems are also vulnerable to postdeployment degradation, especially when environmental conditions, patient populations, or recording equipment differ from those used during model development. For instance, [311] found that off-the-shelf ASR systems, such as Google Speech and OpenAI Whisper, performed poorly in noisy classroom environments, yielding high word error rates in naturalistic small-group conversation. Similar challenges are likely to arise in clinical speech applications, where changes in microphone types, recording settings, or patient demographics may reduce model accuracy over time if not continuously monitored.

The discussion above highlights general challenges associated with deploying AI tools in healthcare, many of which directly apply to clinical speech AI. Pre-deployment evaluations, typically based on curated test datasets, are insufficient to account for the full range of real-world variability. A finite test set cannot anticipate every possible domain shift or unforeseen condition encountered post-deployment. To address these challenges, strategies for ongoing performance monitoring are critical. Leveraging insights into factors that contribute to model degradation, such as domain shifts, data drift, and environmental changes, can help maintain model reliability over time. A major cause of performance decline post-deployment is data distribution shift, which can result from changes in the data collection process, software updates, input demographics, or other factors [312]. For instance, variations in algorithmic implementation between software versions may alter speech features, ultimately affecting model outputs. Distribution shifts may also arise from revisions or unexpected changes in speech data collection protocols [313]. Even changes in background noise conditions can significantly impact model reliability. Some of these shifts can be automatically tracked and mitigated. For example, software versioning can flag changes in system configuration, while background noise monitoring before data collection can alert users to environmental conditions that may affect model performance, recommending adjustments like relocating to a quieter space.

Other changes in input data distribution, such as demographic shifts in patient populations, are more challenging to address. For instance, the inclusion of new dialects or speech patterns not represented in the training data can significantly alter the distribution of speech features, making the model less reliable for certain populations. All of these differences contribute to domain shifts, increasing the disparity between the data used in training and the conditions encountered in deployment, ultimately affecting the reliability of model outputs. Recent studies in AI-driven diagnostics have demonstrated that existing models exhibit performance gaps across different demographic groups. For instance, in medical imaging research, the performance gap between demographic groups in diagnosing various cancer types ranges from 3% to 16%, as seen in studies of pathology models across patient populations with varying ethnic backgrounds [314]. Similar biases have been reported in clinical speech AI of mental health issues, where models underperform for certain demographic groups [315]. To address these biases that arise after the implementation of the model, a feedback policy can be beneficial for users to report problematic model behavior that allows developers to backtrack the relevant component of the model development pipeline [313] (refer to Fig. 3). In addition, statistical approaches which perform timely and automatic monitoring of model performance can also be used [316], [317].

For model developers, it is essential to routinely validate clinical models after hardware or software upgrades. New efforts are also emerging to support continuous model development post-deployment. Algorithms that can automatically detect distribution shifts and alert developers of potential performance declines [318], [319], or even automatically recalibrate models to adjust for the new data [320], [321], will be key to ensuring ongoing model reliability in clinical environments.

The reliability and validity framework for input speech measures, as discussed earlier (see Section VI-A), is also applicable in the model post-deployment. By continuously applying the same measurement-theory criteria to incoming clinical speech data, it becomes possible to monitor whether upstream factors (ranging from data distribution shifts to hardware changes or environmental alterations) are causing input measures to fall outside the boundaries in which they were originally shown to be reliable and valid (see Figure 5). Such ongoing measure-level monitoring facilitates early detection of model degradation but also enables targeted rootcause analysis.

VIII. Ethical, Privacy and Security Considerations in Clinical Speech AI

The safe deployment of clinical speech technologies brings forward new ethical, privacy, and security challenges. Issues like biased decision-making processes, security risks of AI models, and leakage of patient information can lead to harmful consequences for individuals.

An important ethical concern in clinical speech AI is the potential for biased decision-making stemming from homogeneous data collection. Insufficient demographic diversity in speech data can introduce bias into trained AI models [322]. It is known that speech characteristics are influenced by biological and socio-cultural factors. For example, F0 and formant frequencies vary between genders, [323], age groups [324], and lifestyles [325]. Articulation rate, pausing, and phrasing also differ across dialects [326], [327]. Ignoring these differences during data collection can result in models that perform well only on certain socio-demographic groups, leading to biased outcomes in applications like speech recognition and speaker recognition [328], [329]. In clinical contexts, similar biases during data collection, analytical validation, and/or clinical validation can compromise the accuracy of assessments. A recent study revealed that 71% of training data in published clinical AI models targeting American English patients came from just three states - California, Massachusetts, and New York - omitting data from the remaining states [330]. Such skewed representation can lead to inaccurate results for test subjects outside the specific populations included in the training data.

Open-sourcing clinical speech data and algorithms can raise new concerns due to the sensitive nature of the underlying information. Clinical datasets may contain identifiable health-related speech patterns, and the high clinical value of derived models makes them attractive targets for adversarial exploitation. Without appropriate safeguards, releasing these resources can expose systems to a range of cybersecurity threats that compromise both patient privacy and model integrity. For example, training speech data can be poisoned, significantly manipulating model behavior even when only a small portion of labels are altered [331]. Techniques such as active learning and knowledge distillation can be exploited to steal information from the original model, including hyperparameters, learned parameters, and model behaviors [332]. Once confidential information is compromised, attacks such as adversarial attacks, model inversion, and membership inference become possible [333]. Adversarial attacks involve injecting hand-crafted noise signals into the original input without being detectable by humans, misleading or manipulating the model’s decision-making process [334], [335]. In the clinical setting, such attacks could produce distorted speech representations and misleading clinical scores, leading to misallocation of resources, monetary waste, and significant threats to patient health, including delayed or invalid treatment and worsened symptoms. Model inversion attempts to reconstruct speech data from model parameters, potentially leaking patient privacy information during the process [336]. On the other hand, membership inference attacks determine if an AI model was trained on a specific data sample, exploiting the model to reveal the identity and related information of speakers in the training dataset [337]. These threats necessitate that algorithm developers incorporate robust detection and prevention strategies against such attacks in clinical speech AI.

These challenges have been the focus of research developments on voice anonymization methods in speech technology research [338], [339]. Anonymization refers to the goal of suppressing personally identifiable information in the speech signal, while preserving other attributes such as linguistic content and speaker intent. This ensures that the speech can still be used for analysis or system interaction without compromising privacy. Recent approaches to voice anonymization include voice conversion [340], where one speaker’s voice is transformed to sound like another speaker; and disentangled representation learning [341], which aims to separate the identifiable characteristics of the voice from its other features, allowing for selective suppression of speaker identity.

Speech carries clinically rich information, and our daily interactions with mobile devices leave extensive footprints of our health status. Recent clinical studies have utilized publicly available social media data for developing clinical applications. For example, users’ text data on Twitter has been used to detect depression [342], and speech data from YouTube has been employed to develop clinical AI models for evaluating oral cancer speech, dysarthria, and dementia [343], [344]. Speech recordings from celebrity publicly known disease were analyzed to showcase the potential of using speech changes as a function of disease progression [345]. While publicly available data offers cost-effective opportunities for algorithm developers and researchers, they also pose new risks such as the quality of speech data and the reliability of labels. Claims about users’ health statuses might be made without their consent. Analyzing high-profile public figures without consent can lead to the widespread dissemination of false narratives regarding their health conditions, potentially causing social and political crises beyond the control of researchers and clinicians.

The ethical, privacy, and security considerations in clinical speech AI are multifaceted and require comprehensive attention. To enable large-scale clinical research commercial applications, while protecting sensitive health data, robust collaboration between academia, industry, and regulators is essential. Recent initiatives demonstrate promising frameworks for ethical data collection and distribution. For example, the Speech Accessibility Project (SAP) [346] represents an advancement, collecting transcribed English speech from individuals with speech disabilities while explicitly allowing both research and commercial use, addressing the key limitation of research-only restrictions in the past efforts. SAP’s consent process ensures participant privacy through confidentiality measures, including protections against re-identification and the right to withdraw data. This approach has already improved commercial speech recognition systems, with Microsoft’s Azure AI Speech Service showing 18-60% accuracy gains across disability types [347]. Similarly, the Bridge2AI-Voice consortium [348] is compiling an ethically sourced clinical voice dataset, currently comprising 400+ speakers and 20,000+ recordings. The project uses privacy protections, removing all HIPAA Safe Harbor identifiers and excluding sensitive audio content while preserving non-identifiable static features for research.

IX. Discussion

As clinical speech AI moves from pilot studies to real-world applications, the field is presented with a new set of challenges. This paper has reviewed the end-to-end components of a technical pipeline for developing clinically valid, reliable, and interpretable speech-based AI systems. We highlight that designing such systems requires more than optimizing features or fine-tuning models; it requires alignment between clinical goals, speech task design, data acquisition protocols, model architecture, and validation strategies. In this discussion, we examine several cross-cutting challenges that shape this pipeline and outline opportunities to advance the field toward more robust and clinically actionable systems.

A central theme throughout the technical development process is the tension between flexibility in model and representation design and the need for rigor, transparency, and alignment with clinical practice. Decisions about how to frame a clinical problem, which speech representations to use, and what model complexity is warranted can all influence whether systems generalize across settings, remain interpretable to clinicians, and ultimately prove useful in practice. Despite encouraging performance reported in many studies, recurring patterns of overoptimism, ill-defined contexts of use, and a lack of prospective validation raise questions about real-world readiness of this technology.

At the same time, recent innovations in machine learning, particularly in LLM and spoken language models (SLM), present promising tools for scaling and enriching clinical speech pipelines. These models offer new capabilities for extracting high-level linguistic and acoustic features, but they also require careful adaptation to clinical constraints. Across the sections that follow, we discuss how the field can confront current limitations while embracing these new opportunities.

A. Usability of Clinical Speech AI

Empirical insights into the usability of clinical speech AI remain limited. It is a critical barrier to real-world translation. Researchers are encouraged to incorporate such usability considerations and user feedback during model development and deployment. For instance, Ahmed et al. [349] integrated a clinical speech model with an interactive mobile therapy game for childhood apraxia of speech (CAS). The survey showed that both children and speech-language pathologists provided high ratings on a 5-point Likert scale. A follow-up longitudinal study [350] further demonstrated notable improvements in speech outcomes and therapy engagement among pediatric patients. Roesler et al. [351] created a cloud-based conversational platform for early speech-based MCI detection, where user-experience surveys indicated that 90% of participants experienced strong engagement and satisfaction. More recently, Xu et al. [352] introduced a clinician-centric evaluation framework to address the gap between model interpretability and clinician trust. It simultaneously measures algorithmic performance, cognitive/professional workload, and perceived reliability using established healthcare usability scales. These efforts have highlighted an emerging research direction in clinical speech AI, that focuses on transforming validated models into practical, user-centered tools for routine clinical applications.

B. Overoptimism in Clinical Speech AI

Although high diagnostic performance is often reported in published studies, growing evidence has led researchers to question the reliability of supervised AI approaches for predicting clinical labels. For instance, studies in neuroimaging have shown that disease prediction models frequently report inflated accuracies that correlate negatively with sample size, suggesting overfitting and limited generalizability [353], [354]. Similar patterns have been observed in clinical speech AI. A meta-analysis of 59 speech-based classifiers for Alzheimer’s disease detection found a negative correlation between reported accuracy and dataset size [49] (Fig. 18), a trend that runs counter to expectations based on learning curve theory. Well-trained models should exhibit increasing accuracy with larger sample sizes, following a predictable power law relationship [50].

Fig. 18.

The negative association between reported accuracy and sample size for two types of classification problems across three meta-analyses. AD = Alzheimer’s Disease; CI = Other forms of cognitive impairment [49].

Several factors contribute to the inconsistencies observed in reported model performance, most notably data leakage and publication bias. Data leakage, where a model inadvertently accesses information from the test set during training, remains a critical concern in machine learning evaluations [52]. Notably, leakage can occur even without explicit training on test data; repeated use of the same test set during iterative development cycles may lead to implicit overfitting, as models gradually become tuned to the evaluation data. Furthermore, the flexibility afforded to developers in modeling choices can result in substantial variability in outcomes, even when the same dataset and research question are used. For example, a recent study involving 46 independent research teams analyzing the same dataset yielded notable variability in models and reported effect sizes [51]. Together, these findings highlight pervasive overoptimism in current approaches to clinical speech AI development.

C. Defining An Appropriate Context of Use

An important but often overlooked challenge in clinical speech AI development is clearly defining the context of use before data collection or model training begins. Many existing systems rely on supervised learning to classify speech into binary categories (e.g., “healthy” vs. “disease”) or to estimate severity from isolated speech samples. While this approach offers practical advantages particularly in label acquisition, it often reflects a narrow framing of the clinical problem and can limit the potential utility of the resulting models.

As outlined in [12], clinical speech AI has the potential to support a wide range of use cases beyond binary classification, including use of speech as a clinical endpoint, non-specific risk screening, monitoring for relapse or disease progression, and integration in digital therapeutics. Each of these applications entails different requirements for data, labels, and modeling approaches. Failing to define the intended use case early in the development process can lead to wasted resources, poor generalizability, and limited clinical relevance. Broadening the scope of clinical speech AI use cases introduces new design requirements for data collection protocols, model architectures, and evaluation metrics, ultimately supporting the development of systems that are more robust, generalizable, and clinically actionable.

D. Transitioning from Speech Features to Clinically-meaningful Speech Measures

Rather than relying on features that have performed well in other speech applications, clinical speech AI should begin with clinically relevant speech constructs and systematically operationalize them into measurable biomarkers of speech [355]. This approach strengthens construct validity, enhances interpretability, and ensures that models are aligned with the cognitive, linguistic, or physiological functions impacted by a given condition. Measures grounded in well-defined constructs also support clearer clinical oversight, facilitate error detection, and promote trust in model outputs.

Unlike traditional features derived primarily from engineering intuition or repurposed from other domains, clinically meaningful measures must be designed and validated in collaboration with domain experts [12]. Clinician involvement should extend beyond data labeling to include the identification of relevant constructs, the design of tasks that elicit appropriate speech to measure those constructs, and the validation of corresponding measures.

Importantly, this emphasis on construct-driven measures does not preclude the use of state-of-the-art models in clinical speech AI. Instead, it challenges researchers to use such models creatively and purposefully, especially when clinical data is limited. For example, deep learning models trained on healthy speech can be used to derive representations that operationalize constructs such as consonant-vowel transition fidelity or hypernasality [228], [9], which can then be applied and validated in clinical populations. As the scale and richness of labeled clinical data increase, the complexity of the models used to define, refine, and apply these measures can also grow.

E. Starting with Simpler and Interpretable Predictive Models

In parallel with efforts to develop clinically meaningful and interpretable input measures, another key consideration in clinical speech AI is the choice of modeling approach. A longstanding tension in the field involves balancing predictive performance with model interpretability. Conventional wisdom assumes that interpretable, low-dimensional models come at the cost of accuracy. However, recent work challenges this trade-off, suggesting that with structured and clinically relevant data, simpler models can perform comparably to more complex alternatives [256]. This view is supported by the Rashomon set argument, which posits that when many models achieve similarly high accuracy on a task, at least one of them is likely to be interpretable. In such cases, it may be both possible and preferable to select a simpler model that maintains performance while offering greater transparency, especially in high-stakes clinical contexts.

This emphasis on interpretability is particularly important in clinical speech AI, a nascent field characterized by limited labeled data and strong regulatory and clinical oversight requirements. In many applications such as diagnostic support or risk stratification, model decisions must be understandable to clinicians, auditable by regulators, and justifiable to patients. Simple models trained on well-defined, interpretable speech measures offer transparency, are easier to debug, and enable more effective collaboration with clinical stakeholders. They are also less prone to overfitting when data is scarce and can be updated or re-validated with fewer resources.

As larger and more diverse datasets become available and modeling goals evolve, researchers can gradually introduce advanced techniques (e.g. fine-tuning foundation models) when justified by task complexity and supported by available data and infrastructure. However, given the current lack of established clinical impact in this space, simpler models should first demonstrate clear utility in real-world settings. Their success can help build the case for investment in data collection, enabling more complex approaches and providing a validated foundation on which to layer additional model sophistication.

F. The Importance of Prospective Validation

For clinical speech AI to have real-world impact, retrospective validation alone is insufficient. While cross-validation on historical datasets can demonstrate initial promise, it does not capture the variability and complexity of real-world clinical deployment. Prospective validation - testing a fully developed model on newly collected real-world data - is essential to assess generalizability, identify hidden biases, and confirm clinical utility.

The V3 framework which includes verification, analytical validation, and clinical validation, offers a structured approach for evaluating models from signal acquisition to clinical decision-making [123]. While most current studies focus on retrospective analytical or clinical validation, prospective validation is the only step that tests whether a model remains robust under true deployment conditions, where microphones, environments, patient populations, and workflows vary over time and across sites. Prospective validation also ensures that model outputs remain meaningful and actionable when used by clinicians and patients, not just in carefully curated datasets. For speech-based systems, which are especially sensitive to recording conditions and speaker variability, prospective evaluation can uncover failure modes that are invisible during development.

While prospective validation is often treated as a post hoc evaluation step, it also opens up important and underexplored opportunities for engineering-driven research. For example, pre-deployment simulation of real-world variability, such as differences in microphones, environments, or speaker behavior, can help stress-test model robustness before rollout. Models can also be trained to explicitly account for metadata (e.g., device type, background noise levels, location), in order to improve their adaptability to deployment conditions. Additionally, developing proxy tasks or synthetic benchmarks that correlate with prospective performance would allow researchers to detect potential failure modes early, reducing reliance on resource-intensive clinical trials. There is also a need for realtime monitoring tools to detect model drift during deployment, as well as lightweight validation platforms that enable rapid, low-cost prospective testing in diverse populations.

G. Opportunities for Large Language Models and Spoken Language Models in Clinical Speech AI

One of the promising opportunities for LLMs in clinical speech AI is their ability to operationalize complex linguistic constructs that are otherwise difficult and time-consuming to annotate manually. For example, referential cohesion - the ability to maintain consistent reference to people or things throughout a narrative - is recognized as a useful linguistic marker of cognitive-linguistic decline in Alzheimer’s disease [102]. Yet, it is challenging to use in practice because its annotation requires detailed linguistic analysis, often by experts. LLMs, with their strong contextual language understanding and instruction-following capabilities, offer a new pathway for automating the extraction of such high-level discourse features from transcribed speech. This perspective offers a new dimension of exploration for developing construct-aligned speech measures.

More broadly, the emergence of LLMs and spoken language models (SLMs) - models trained to perform tasks directly on audio inputs via natural language instructions - presents a rich design space for clinical speech AI. These models can be prompted to perform specific analytic or diagnostic tasks using either transcribed text or raw speech signals, opening the door to multimodal pipelines that integrate acoustic and linguistic features. Adapter-based architectures [356], [357] further enhance flexibility, allowing researchers to bridge pretrained speech encoders with LLMs in modular systems that can assess constructs such as prosody, fluency, or lexical richness. While current limitations such as prompt sensitivity, input inconsistency, and limited multilingual performance present real challenges [288], [358], [359], they also represent opportunities for methodological advances in this space.

H. Regulatory Approval for Clinical Speech AI

A major challenge for AI technologies in healthcare is the persistent lack of regulatory approvals in the field’s rapid evolution [360]. Clinical speech AI faces the same problem. The growing availability of open-source tools, pre-trained models, source code, and datasets has dramatically lowered the barrier to development, enabling anyone to create the release these models without any regulatory approval. As a result, unapproved and inadequately validated models can proliferate widely, and their unsupervised deployment in clinical settings carries substantial ethical, safety, and clinical risks.

Mutual alignment between developers and regulatory bodies is essential to mitigate risks and accelerate safe translation. For example, Liss and Berisha [355] detailed the successful development and validation of an articulatory precision biomarker for ALS that earned FDA Breakthrough Device designation. This derived from the Healey ALS Platform Trial at Massachusetts General Hospital, which performed a prespecified analysis of a set of speech measures. It showed that participants taking pridopidine had improvements in speech measures compared to those taking placebo [361]. This result led to the decision to pursue a Phase 3 clinical trial of pridopidine in ALS [362]. On the other hand, multiple clinical speech AI startups are now routinely partnering with academic and clinical institutions to conduct prospective clinical trials prior to pursuing regulatory clearance for medical applications. These approach represents the practice and standard for responsible development. As emphasized by Mennella et al. [360], robust and adaptive governance frameworks are critical for fostering trust and enabling successful implementation of AI in healthcare. When developers rigorously follow regulatory-compliant roadmaps, regulatory agencies should reciprocate by expediting review of these high-quality submissions. This positive feedback loop accelerates the availability of validated, trustworthy tools in clinical practice.

X. Conclusion

This paper has presented an end-to-end overview of the components involved in developing clinical speech AI systems. Given the clinical stakes, this domain requires approaches that are tailored to specific populations and applications - balancing predictive performance with clinical interpretability and validity. We reviewed the mechanisms of speech production as a foundation for identifying relevant biomarkers, emphasizing the importance of selecting elicitation tasks that are appropriate to both the target condition and the patient population. Standardized data collection protocols were discussed as essential for ensuring consistency and data quality across sites and studies.

Selecting the right speech representation is among the most important design decisions in AI. We outlined the complementary roles of traditional spectral features and deep learning–based embeddings. Clinically derived speech measures offer a promising direction for bridging model performance and clinical utility, though their adoption depends on rigorous validation to establish both reliability and construct alignment.

We examined the progression of predictive modeling strategies - from conventional machine learning to deep neural networks and emerging large language models. This highlights the diagnostic potential these tools offer, as well as the new challenges they introduce in terms of validation, transparency, and deployment in real-world clinical workflows. Ethical considerations, including bias, security risks, and potential misuse, point to the need for stronger coordination between researchers, developers, and regulators to ensure safe and equitable use.

The paper also identified several key opportunities for advancing clinical translation. These include improving the transparency of performance reporting, better defining contexts of use, increasing the adoption of prospective validation, and addressing the trade-offs between model complexity and interpretability.

Taken together, this overview is intended to serve as a technical reference and a foundation for advancing the field. Realizing the full potential of clinical speech AI will depend on sustained collaboration across disciplines, careful attention to clinical realities, and the development of trustworthy tools that integrate seamlessly into healthcare delivery.

Fig. 10.

An example of inserting an interpretable DNN layer (highlighted in blue) that contains predicted perceptual measures of speech. W₁ is responsible for predicting speech measures given input X . W₂ is responsible for performing clinical diagnosis based on the predicted measures y₁. Figure reprint from [232].

TABLE II.

Illustrative examples of knowledge-driven speech measures for assessing cognitive and thought disorders. They are also applicable to broad conditions impacting Conceptulization, Formulation, and Articulation. Table revised from [62].

Category	Subcategory	Examples of Measures or Methods Used
Acoustic	Prosody	Temporal (pause rate, speech rate, pairwise voiced durations, etc.)
		Fundamental frequency variation (F0)
		Loudness variation
	Spectral	Formant trajectories (F1, F2, F3, etc.)
		VSA Spectral centroid
		MFCC statistics
	Vocal quality	Jitter, shimmer
		Spectral slope
		Harmonic-to-noise ratio (HNR)
		Cepstral peak prominence (CPP)
	ASR-related	Phone-level detection of filled pauses & temporal measures
Text	Lexical	Bag of words (BoW)
		Linguistic Inquiry & Word Count (LIWC)
		Lexical Diversity
		Lexical Density
		Part-of-speech (POS) tagging
	Syntactical	Constituency-based parse tree scores
		Dependency-based parse tree scores
		Speech graphs and attributes
	Semantic	Word & sentence embeddings: Latent semantic analysis (LSA) Neural word embeddings Neural sentence embeddings
		Topic modeling: Latent Dirichlet allocation (LDA) Vector-space topic modeling with neural networks
		Semantic role labeling
	Pragmatic	Sentiment analysis