Beyond Binary Decisions: Evaluating the Effects of AI Error Type on Trust and Performance in AI-Assisted Tasks

Abstract

Objective:

We investigated how various error patterns from an AI aid in the non-binary decision scenario influence human operators’ trust in the AI system and their task performance.

Background:

Existing research on trust in automation/autonomy predominantly uses the signal detection theory (SDT) to model autonomy performance. The SDT classifies the world into binary states and hence oversimplifies the interaction observed in real-world scenarios. Allowing multi-class classification of the world reveals intriguing error patterns previously unexplored in prior literature.

Method:

Thirty-five participants completed 60 trials of a simulated mental rotation task assisted by an AI with 70–80% reliability. Participants’ trust in and dependence on the AI system and their performance were measured. By combining participants’ initial performance and the AI aid’s performance, five distinct patterns emerged. Mixed-effects models were built to examine the effects of different patterns on trust adjustment, performance, and reaction time.

Results:

Varying error patterns from AI impacted performance, reaction times, and trust. Some AI errors provided false reassurance, misleading operators into believing their incorrect decisions were correct, worsening performance and trust. Paradoxically, some AI errors prompted safety checks and verifications, which, despite causing a moderate decrease in trust, ultimately enhanced overall performance.

Conclusion:

The findings demonstrate that the types of errors made by an AI system significantly affect human trust and performance, emphasizing the need to model the complicated human-AI interaction in real life.

Application:

These insights can guide the development of AI systems that classify the state of the world into multiple classes, enabling the operators to make more informed and accurate decisions based on feedback.

Précis:

We examine how AI errors influence human trust and performance. Some AI errors provided false reassurance, misleading operators into believing their incorrect decisions were correct. This leads to worsened performance and a significant decline in trust. Conversely, and paradoxically, some AI errors prompt safety verification, ultimately enhancing overall performance.

INTRODUCTION

Consider the following hypothetical scenario:

Sarah, a highly skilled pharmacist, is responsible for filling the medication bottles for each prescription order. Recently, the pharmacy Sarah works for has introduced an AI computer vision system¹ that can scan the filled bottle and identify the specific medication that has been filled. The AI system is introduced as another layer of verification before medication dispensing.

Today, Sarah receives a prescription order for one patient, Noah, who needs to take medication $\mathit{X}$.

Upon receiving the order, one of the following five cases could happen:

Case A: Sarah correctly fills the bottle with medication $\mathit{X}$. The AI system scans the bottle and correctly identifies that the filled medication is $\mathit{X}$ and signals a green light.

Case B: Sarah correctly fills the bottle with medication $\mathit{X}$. The AI system scans the bottle but incorrectly identifies that the filled medication is $\mathit{Y}$ (or any medication that is not $\text{X}$). As medication $\mathit{Y}$ is not the ordered medication, the AI system signals a red flag and indicates the need to double-check.

Case C: Sarah has a lapse when filling the bottle and incorrectly fills it with medication $\mathit{Z}$. Fortunately, the AI system correctly identifies the filled medication as $\mathit{Z}$. As medication $\mathit{Z}$ is not the ordered medication, the AI system signals a red flag.

Case D: Sarah has a lapse when filling the bottle and incorrectly fills it with medication $\mathit{Z}$. The AI system scans the filled medication and identifies it as medication $\mathit{Y}$ (or any medication that is not $\text{X}$ and not $\text{Z}$). As medication $\mathit{Y}$ is not the ordered medication, the AI system signals a red flag.

Case E: Sarah has a lapse when filling the bottle and incorrectly fills it with medication $\mathit{Z}$. The AI system scans the filled medication and incorrectly identifies it as medication $\mathit{X}$ and signals a green light.

This hypothetical scenario, which reflects recent developments in medical dispensing (Q. Chen et al., 2024; J. Y. Kim et al., 2025; Lester et al., 2021; Tsai et al., 2025; Zheng et al., 2023), presents unique characteristics that are largely overlooked in existing research on trust in and dependence on automation/autonomy and AI systems. Existing research typically involves a human subject performing tasks, with certain tasks being automated. For example, an automated combat identification (CID) aid can scan the environment, identify a friend or a foe, and make recommendations to soldiers (Du, Huang, & Yang, 2020; Guo, Yang, & Shi, 2023, 2024; Neyedli, Hollands, & Jamieson, 2011; Wang, Jamieson, & Hollands, 2009). To model the performance of the automation, most studies used the signal detection theory (SDT) (Hautus, Macmillan, & Creelman, 2021; Tanner & Swets, 1954). However, SDT categorizes the world into binary states—signal present or absent—which oversimplifies the complex dynamics and variances observed in real-world scenarios.

Let’s examine the five cases presented in the hypothetical scenario, focusing particularly on Case D and Case E shown in Figure 1b, which are especially thought-provoking. In Case D, although the AI erroneously identifies medication Y as Z, this error can be considered beneficial from a utilitarian perspective. The incorrect identification differs from the prescribed medication X, which rightly triggers a safety alert. Conversely, Case E also involves an error by the AI, but this mistake could potentially lead to catastrophic results, as it mistakenly confirms the incorrectly filled medication as correct, misleading the human operator and providing false reassurance. Interestingly, as shown in Figure 1a, if we assumed the prediction of the AI is only binary, Cases D and E would be combined into a single case.

Figure 1.

Difference in the performance patterns when choices are binary or non-binary. In the binary choice (a), the Wrong-Incorrect pattern always matches the reference. However, in the non-binary choice (b), the Wrong-Incorrect pattern could be categorized into two, based on whether incorrect predictions match the reference (prescribed pill).

This research aims to explore how people trust and depend on automation/autonomy when the predictions made by these systems extend beyond simple binary decisions. We first review the SDT, followed by existing studies on trust and dependence that utilize SDT within both single- and dual-task frameworks. After that, we introduce and detail the current study.

SDT, Trust, Automation Compliance and Reliance

SDT models the relationship between signals and noise, as well as the automation’s ability to detect signals among noise (Sorkin & Woods, 1985; Tanner & Swets, 1954). The state of the world is characterized by either “signal present” or “signal absent,” which may or may not be identified correctly by automation. The combination of the state of the world and the automation’s detection results in four possible states: hit, miss, false alarm (FA), and correct rejection (CR) (see Figure 2).

Figure 2.

Signal Detection Theory (SDT)

SDT is widely employed to examine trust in and dependence on automation/autonomy and AI systems. Typically, SDT is used to model the reliability of automation with which the human operators interact in both single-task (Bhat, Lyons, Shi, & Yang, 2022, 2024; Lacson, Wiegmann, & Madhavan, 2005; Madhavan, Wiegmann, & Lacson, 2006; Neyedli et al., 2011; Wang et al., 2009; Wiegmann, Rich, & Zhang, 2001) and multi-task paradigms (Chung & Yang, 2024; Du et al., 2020; McBride, Rogers, & Fisk, 2011; Wiczorek & Manzey, 2014; Yang, Unhelkar, Li, & Shah, 2017). The general findings indicate that as automation reliability increases, trust and dependence on automation increase accordingly.

When error types (FAs and misses) are examined more closely, findings suggest they lead to distinct behavioral patterns (Chancey, Bliss, Yamani, & Handley, 2017; Meyer, 2001, 2004). Dependence on automation can be categorized into compliance and reliance. Compliance is characterized by operators responding as though the error exists when the system indicates an error. FAs often result in commission errors when the operator follows the automation’s false indication. Reliance, by contrast, occurs when operators believe the automated system’s indication of safety. When operators fail to respond to events due to automation misses, omission errors arise (Meyer, 2001, 2004; Skitka, Mosier, & Burdick, 1999). Research suggests that FAs primarily influence compliance, though some studies indicate they may also affect reliance, whereas misses predominantly impact reliance (Chancey, Bliss, Liechty, & Proaps, 2015; Chancey et al., 2017; Dixon & Wickens, 2006; Dixon, Wickens, & McCarley, 2007; Meyer, 2001).

For example, Chancey et al. (2017) investigated automation reliability and error types (FA-prone vs. miss-prone) in a simulated flight task. Participants performed a primary task of maintaining level flight and a secondary resource management task supported by automated aids with high (90%) and low (60%) reliability. Trust was measured at an aggregate level halfway through each session. The results showed that reliability significantly affected compliance in the FA-prone condition, with participants more likely to comply with the highly reliable systems. In contrast, in the miss-prone condition, reliability primarily affected reliance, with greater reliance observed on highly reliable systems. Additionally, FAs had a stronger impact on trust than misses; trust mediated the relationship between FAs and compliance but not between misses and reliance (Chancey et al., 2017).

Other research indicates that misses may cause operators to trust automation less. For instance, in a study simulating an unmanned aerial vehicle mission with an automation-assisted secondary weapon-deployment task, Davenport and Bustamante (2010) investigated participants’ trust, reliance, and compliance following FA-prone and miss-prone aid. Trust was assessed through post-session questionnaires, revealing a greater trust decrement for systems with more misses compared to FAs. Compliance was greater for the miss-prone aid, while reliance was greater for the FA-prone aid (Davenport & Bustamante, 2010). Conversely, other studies indicate that operators may exhibit lower trust in systems with a high frequency of FAs, which may be attributed to the heightened prominence of FAs that reduces the credibility as operators need to actively verify the presence of errors, making the error more salient (Breznitz, 2013; Johnson, Sanchez, Fisk, & Rogers, 2004).

Some studies implicitly assessed compliance and reliance using performance and reaction time metrics. Dixon et al. (2007) examined error types in tracking and system monitoring tasks. Participants were grouped by automation aid type: no help, perfectly reliable aid, FA-prone aid (60% reliability), and miss-prone aid (60% reliability). Reliance was evaluated through performance deficits, while compliance was measured by comparing response times to those observed in the perfectly reliable aid condition. Results indicated that the FA-prone system reduced both compliance and reliance, while the miss-prone system only reduced reliance. In the FA-prone condition, failure detection rates and response times in the monitoring task were worse than in the perfectly reliable automation condition, reflecting reduced compliance. Both error types affected reliance as evidenced by poorer tracking task performance after silent automation trials. Notably, the FA-prone condition resulted in lower performance and longer detection times compared to the miss-prone condition (Dixon et al., 2007). In Sanchez, Rogers, Fisk, and Rovira (2014)’s multi-task simulation experiment, participants performed a collision avoidance driving task supported by 94.5% reliable automation and a secondary tracking task. Reliance was assessed based on whether participants chose to double-check after receiving support from the automated aid. Results revealed a trend of increasing reliance in the FA condition as participants became more familiarized with automation, whereas reliance decreased in the miss condition (Sanchez et al., 2014).

In high-stakes scenarios where the consequences of missing critical events could be catastrophic, automation thresholds are often calibrated to favor FAs to minimize the chance of overlooking abnormalities (Sanchez et al., 2014; Sorkin & Woods, 1985). However, excessive FAs may affect trust more than misses, leading to disuse (neglect or under-dependence), deviating from the original purpose (Chancey et al., 2017; Dixon & Wickens, 2006; Parasuraman & Riley, 1997). These studies treated the error type (FA-prone and miss-prone) as a between-subject variable. Future research could benefit from evaluating within-subject differences to provide a more comprehensive understanding of human behavioral patterns in response to automation errors.

Properties of Trust Dynamics

Over the past 30 years, many researchers have investigated people’s trust in automated/autonomous and AI systems. Historically, most studies have approached trust from a snapshot view, typically using end-of-experiment questionnaires to evaluate trust levels. Research within this view has identified many factors that can influence people’s trust in these systems (Hoff & Bashir, 2015; Kaplan, Kessler, Brill, & Hancock, 2023). More recently, acknowledging that a person’s trust can change dynamically while interacting with automated/autonomous technologies, there is a shift of research focus from snapshot trust to trust dynamics – how humans’ trust in autonomy forms and evolves due to moment-to-moment interaction with automated/autonomous technologies (de Visser et al., 2020; Wischnewski, Krämer, & Müller, 2023; Yang, Guo, & Schemanske, 2023; Yang, Schemanske, & Searle, 2023).

One body of research on trust dynamics explores how trust can be violated and restored after trust violation. Trust violations happen after negative experiences (false alarms and misses), which substantially decreases trust to a level below the original trust level (Baker, Phillips, Ullman, & Keebler, 2018; Esterwood & Robert, 2022; Esterwood & Robert Jr, 2023; P. H. Kim, Ferrin, Cooper, & Dirks, 2004; T. Kim & Song, 2021; Lewicki & Brinsfield, 2017; Sebo, Krishnamurthi, & Scassellati, 2019). Some trust violation research suggests that misses may lead to lower trust in automation. For example, Azevedo-Sa et al. (2020) investigated how failures in semi-autonomous driving systems influence drivers’ trust. Participants engaged in a non-driving task in a semi-autonomous vehicle, which occasionally provided FAs or missed obstacles. Trust dynamics were evaluated after each interaction, revealing that misses had a more harmful effect on trust than FAs (Azevedo-Sa et al., 2020). In contrast, other studies suggest no differences between error types. For instance, Guzman-Bonilla and Patton (2024) investigated participants’ responses to automation errors (FAs or misses) during a block pair matching task. Trust was measured after each block, and their findings indicated that error type did not affect the participants’ trust (Guzman-Bonilla & Patton, 2024). After trust violations occur, efforts can be made to restore trust, and these efforts often involve various actions/strategies such as apologies (Kohn, Momen, Wiese, Lee, & Shaw, 2019; Mahmood, Fung, Won, & Huang, 2022; Natarajan & Gombolay, 2020), denials (Kohn et al., 2019), explanations (Ashktorab, Jain, Liao, & Weisz, 2019; M. Faas, Kraus, Schoenhals, & Baumann, 2021; Natarajan & Gombolay, 2020), and promises (Albayram et al., 2020).

The second body of research is focused on developing real-time trust prediction models (Bhat et al., 2022; M. Chen, Nikolaidis, Soh, Hsu, & Srinivasa, 2018; Guo & Yang, 2021; Xu & Dudek, 2015). Examples of trust prediction models include the online probabilistic trust inference model (OPTIMo) by Xu and Dudek (2015), the Beta random variable model by Guo and Yang (2021), and the Bayesian model combining Gaussian processes and recurrent neural networks by Soh, Xie, Chen, and Hsu (2020). For a detailed review, please refer to Kok and Soh (2020).

The third body of research, which is examined in this study, uncovers general properties that govern how a person’s trust in automated/autonomous and AI systems changes over time. Recently, Yang, Guo, and Schemanske (2023) summarized three key properties of trust dynamics: continuity (Trust at the present moment i is significantly associated with trust at the previous moment i − 1.), negativity bias (Negative experiences due to autonomy failures have a greater influence on trust than positive experiences due to autonomy successes), and stabilization (A person’s trust will stabilize over repeated interactions with the same autonomy). The first two properties have been reported consistently in prior research (Lee & Moray, 1992; Manzey, Reichenbach, & Onnasch, 2012; Yang, Schemanske, & Searle, 2023; Yang et al., 2017). The last property, stabilization, was first empirically found in Yang, Guo, and Schemanske (2023). Using these properties, a predictive computational model was developed, which claims that trust at any point follows a Beta distribution (Guo & Yang, 2021). This model surpasses previous models in prediction accuracy while ensuring generalizability and explanatory power, indicating the importance of the three properties of trust dynamics.

The present study

For readability purposes, we denote the five cases mentioned in the hypothetical scenario using two letters. The first letter, $R$ or $W$, represents the participants making a right or wrong initial decision. The second letter, $C$ or $I$ or $I_{ref}$, represents the AI making a correct or incorrect prediction, given the human’s initial performance. We differentiate $I$ and $I_{ref}$, where “ref” emphasizes that the AI mistakenly confirms the incorrectly filled medication as the prescribed (i.e., reference) medication. Additionally, the outcome of the task can be denoted as $R$ or $W$. Altogether, we have ten different outcomes combining the five patterns (cases) and two possible outcomes for each pattern (see Table 1).

TABLE 1:

Description of the five cases and two outcomes. Bold text indicates potential incorrect outcomes resulting from automation errors.

Case	Pattern	Prescribed Medication	Filled Medication	AI Prediction	Final Outcome
A	RC	X	X	X	R: Dispense medication W: Error-correction refill
B	RI	X	X	Y/Z	R: Dispense medication W: Error-correction refill
C	WC	X	Y	Y	R: Error-correction refill W: Dispense medication
D	WI	X	Y	Z	R: Error-correction refill W: Dispense medication
E	$W{I}_{ref}$	X	Y	X	R: Error-correction refill W: Dispense medication

The aims of the study are multifaceted. The primary interest of the study is to investigate the influence of AI predictions following wrong initial decisions. Specifically, to compare between patterns $WC$ (i.e., Case C), $WI$ (i.e., Case D), and $W{I}_{ref}$ (i.e., Case E).

Both patterns $WI$ and $W{I}_{ref}$ involve the wrong medication being filled and subsequently misidentified by the AI system. The critical difference emerges in the outcome of the AI’s error: in pattern $WI$, the AI’s mistaken identification of medication Y as Z prompts a re-evaluation of the dispensed medication, potentially averting a mistake. Conversely, in pattern $W{I}_{ref}$, the AI incorrectly identifies medication $\text{Y}$ as $\text{X}$, misleading the human operator and providing false reassurance, thereby compounding the error. Therefore, we hypothesize:

H1: Pattern $\mathit{W}\mathit{C}$ will lead to the best performance, followed by pattern $\mathit{W}\mathit{I}$, then pattern $\mathit{W}{\mathit{I}}_{\mathit{r}\mathit{e}\mathit{f}}$.

In terms of trust change, we hypothesize both pattern $WI$ and pattern $W{I}_{ref}$ will lead to trust decrement compared to trust increment for pattern $WC$. In addition, there will be a more significant trust decrement for pattern $W{I}_{ref}$.

H2: Pattern $\mathit{W}\mathit{C}$ will have the highest trust increment, followed by pattern $\mathit{W}\mathit{I}$, then pattern $\mathit{W}{\mathit{I}}_{\mathit{r}\mathit{e}\mathit{f}}$. This trend should be observed regardless of the final outcome of the task (Note: a larger trust decrement is considered a smaller trust increment).

Regarding the time taken to make a final decision after receiving the AI’s prediction, we hypothesize an interaction between the three patterns and the final outcome. As mentioned earlier, pattern $W{I}_{ref}$ misleads the human operator and provides false reassurance. Therefore, we speculate that when participants are indeed tricked by pattern $W{I}_{ref}$, they would react very fast. On the contrary, it would take the longest time for a participant to recognize and rectify the error in pattern $WI$ because the incorrect prediction does not align with either the reference or the initial answer.

H3a: It would take a shorter time for people in pattern $\mathit{W}{\mathit{I}}_{\mathit{r}\mathit{e}\mathit{f}}$ to commit an error, compared to patterns $\mathit{W}\mathit{I}$ and $\mathit{W}\mathit{C}$.

H3b: It would take a longer time for people in pattern $\mathit{W}{\mathit{I}}_{\mathit{r}\mathit{e}\mathit{f}}$ to recognize an error, compared to patterns $\mathit{W}\mathit{C}$ and $\mathit{W}\mathit{I}$.

Along with the primary interest, we wish to provide further evidence for the continuity (H4), negativity bias (H5), and stabilization (H6) properties of trust dynamics.

H4: Trust at the present moment $i$ is significantly associated with trust at the previous moment $i-1$.

H5: The magnitude of trust decrement due to incorrect predictions ($\mathit{R}\mathit{I}$, $\mathit{W}\mathit{I}$, $\mathit{W}{\mathit{I}}_{\mathit{r}\mathit{e}\mathit{f}}$) will be greater than the magnitude of trust increment following correct predictions ($\mathit{R}\mathit{C}$, $\mathit{W}\mathit{C}$).

H6: The magnitude of trust adjustment will decrease as a function of time for both the correct and the incorrect AI predictions.

METHOD

This research complied with the American Psychological Association code of ethics and was exempt from the Institutional Review Board oversight at the University of Michigan (HUM00230326 ). Informed consent was obtained from each participant.

Participants

A total of 35 University of Michigan undergraduate and graduate students (average age = 22.43 years, SD = 3.27) participated in the experiment. Participants were required to have a normal or corrected-to-normal vision. Participants received a compensation of 10 dollars base rate with a bonus of up to 10 dollars based on their performance. Participants completed a three-dimensional spatial visualization and recognition task (J. Kim, Chen, Lester, & Yang, 2023; J. Y. Kim, Richie, & Yang, 2024; Vandenberg & Kuse, 1978), assisted by an AI. In each trial, participants were presented with a reference image selected from a pool of 30 images. They were then shown five options and asked to choose the one that matched the reference. Following their selection, the simulated AI predicted the shape of the participant’s chosen image. Finally, participants indicated whether their initial selection was correct. This workflow mirrors a medication dispensing task, where a pharmacist first receives a prescription (e.g., vitamin C) and retrieves the corresponding medication from an inventory containing hundreds of options. The retrieved medication is then verified for accuracy by pharmacy technicians or an AI system, comparing it to the prescription order. Based on the verification, the pharmacist decides whether to dispense the medication or to rectify errors by retrieving the appropriate item.

Experimental apparatus and stimuli

Stimuli development.

The study employed a simulated mental rotation task (MRT) and we followed the study of Shepard and Metzler (1971) to develop stimuli. We created 30 three-dimensional (3D) objects, half with 10 cubes and another half with 8 cubes. For each object, we created a reference two-dimensional image of the object, positioned at a specific rotation (see Figure 3). Subsequently, for each reference, we created 14 rotated images by rotating the 3D object every 45 degrees along the horizontal and vertical axes. This resulted in 30 reference images, each with 14 corresponding rotated images, denoted as correct alternatives (of the reference).

Figure 3.

Illustration of stimuli development. After creating the (a) reference images, one (b) mirrored image, fourteen 45-degree (c) horizontal axis rotated figures, and fourteen 45-degree (d) vertical axis rotated figures were created for each reference image.

The difficulty of the stimuli was determined through a separate study using Qualtrics, and the participants were recruited via Amazon Mechanical Turk (MTurk). Following Shepard and Metzler (1971)’s mental rotation study, participants viewed multiple image pairs. Each left image was a randomly chosen reference image, and the right image was either a correct alternative, a distractor, or the reference image itself (Figure 4). Distractors were randomly chosen from the remaining 29 reference images, which included 14 images with the same cube amount as the reference image (i.e., 8 or 10) and 15 images with a different cube amount. Participants had 10 seconds to identify whether the image pairs were the same or different. Data from participants who failed to match the validation pairs (reference-reference image pairs) correctly were discarded. The mean hit rate determined stimuli difficulty.

Figure 4.

For the beta test, participants were asked to determine whether the image pairs are the same items in different rotations (by clicking “Yes, they are the same item”) or different items (by clicking “No, they are different”) within 10 seconds.

Experimental Task.

The task was adapted from a three-dimensional spatial visualization test (Vandenberg & Kuse, 1978) and was developed using the Python Tkinter package. Figure 5 shows the flowchart of the experimental task.

Figure 5.

Flowchart of the experiment

Before the experiment, participants completed a demographics survey (i.e., age and gender) and a trust propensity survey estimating their propensity to trust automation (Merritt, Heimbaugh, LaChapell, & Lee, 2013).

During the experiment, participants performed the MRT, which consisted of 60 trials presented in random order. Each trial contained several steps. First, participants were shown a reference image and five choices, containing one correct alternative and four distractors (Figure 6a). The correct alternative was identical to the reference image in structure but was shown in a rotated position. Two distractors were randomly chosen from rotated mirrored images of the reference and the other two were randomly chosen from rotated images of other reference images. Participants were asked to choose the image that portrayed the same 3D object as the one portrayed in the reference image, as accurately as possible and within 15 seconds, then to click the “Next” button. The 15-second duration was determined based on pilot testing. If no selection was made within the time limit, the text on the initial choice page displayed “You did not make a selection within the time limit” (Figure 6b), and participants needed to click the “Next” button to start the next trial.

Figure 6.

Participants were asked to make their initial answer choice within 15 seconds.

After making the initial choice, participants rated their confidence in their initial choice using a visual analog scale, with the leftmost point labeled “Not confident at all” and the rightmost point “Absolutely confident” (Figure 7).

Figure 7.

Participants rated their confidence using a visual analog scale.

Next, participants were shown the AI’s prediction, displayed below their initial selection to illustrate the rotation of the AI-predicted shape. In Figure 8a, the participant initially selected the third image (indicated by an arrow), but the AI misidentified the initial selection and presented the reference image of its predicted shape (i.e., Case B described in the Introduction section). In contrast, in Figure 8b, the participant initially selected the fifth image (indicated by an arrow). The AI misidentified the initial selection and displayed the reference image of the correct alternative (i.e., Case E described in the introduction).

Figure 8.

Participants were presented with the AI system’s recognition and chose between sticking with or rejecting their initial choice within 10 seconds. If no selection was made, they were skipped to the next trial.

After viewing the AI’s prediction, participants chose between sticking with their initial answer by clicking “I was right” or rejecting their initial answer by clicking “I was wrong” within 10 seconds. Similarly, the duration was determined based on a pilot study. If no selection was made in 10 seconds, the text on the final choice page displayed a message similar to Figure 6b, and participants needed to press the “Next” button to move to the next trial.

Participants were then presented with their performance feedback and the validity of AI’s prediction (Figure 9).

Figure 9.

Participants were presented with the performance feedback page, showing their performance on the initial and final answer choices and the validity of the AI’s prediction.

At last, participants rated their trust in and perceived reliability of the AI system using a visual analog scale (Figure 10). Following Yang, Schemanske, and Searle (2023)’s approach, participants of this experiment were asked to rate their trust after each trial. The leftmost anchor of the trust scale was labeled “I don’t trust the decision aid at all” and the rightmost anchor of the trust scale was labeled “I absolutely trust the decision aid.” The perceived reliability rating was on a 0 to 100 scale.

Figure 10.

Participants rated their trust and perceived reliability using a visual analog scale.

Experimental design

The experiment employed a within-subjects design. The independent variable was the patterns of performance (Tables 1 and 2). It is important to note that the presence of each pattern or outcome was participant-dependent, as the participant’s initial and final responses could not be manipulated.

TABLE 2:

Mean and sd of confidence rating, trust adjustment, final performance, final reaction time, number of patterns, and number of participants grouped by patterns and outcomes of performance. (Note: Performance is based on the final answer choice. Therefore, for each pattern, there is only one output.)

Patterns of Performance		Confidence Rating		Trust Adjustment		Final Performance (%)	Final Reaction Time (s)		N Patterns		N Participants
		Mean	(SD)	Mean	(SD)	Mean (SD)	Mean	(SD)	(n)		(n)
RC	R	74.58 (23.28)	75.48 (22.73)	1.17 (4.33)	1.21 (4.36)	96.96 (17.17)	3.64 (2.00)	3.54 (1.88)	889	862	35	35
	W		45.81 (22.84)		−0.07 (3.02)			6.70 (3.00)		27		13
RI	R	72.53 (24.53)	78.45 (21.57)	−3.67 (6.06)	−3.65 (5.89)	78.61 (41.11)	5.78 (2.47)	5.82 (2.53)	201	158	35	33
	W		50.77 (22.58)		−3.77 (6.71)			5.63 (2.26)		43		23
WC	R	50.41 (22.11)	44.93 (21.75)	0.76 (4.03)	1.01 (3.39)	62.33 (48.50)	5.71 (2.71)	5.96 (2.62)	592	369	35	35
	W		59.46 (19.63)		0.34 (4.90)			5.30 (2.83)		223		33
WI	R	51.68 (23.32)	43.38 (21.53)	−1.91 (5.39)	−2.23 (5.32)	56.85 (49.65)	6.34 (2.88)	6.25 (2.71)	197	112	35	32
	W		62.62 (21.07)		−1.48 (5.47)			6.45 (3.10)		85		30
WI Ref	R	53.71 (26.16)	42.29 (23.23)	−4.64 (8.03)	−4.46 (7.82)	25.00 (43.53)	4.56 (2.86)	6.79 (3.15)	96	24	34	15
	W		57.51 (26.12)		−4.71 (8.15)			3.82 (2.35)		72		31

To induce the occurrence of different patterns, the AI system is set as follows: The AI system made correct recognition 70% of the time, irrespective of whether the participant’s initial choice was right or wrong. Pattern $RC$ (Case A) occurred when the participant’s initial choice was correct, coupled with a correct recognition from the AI system; Pattern $WC$ (Case C) occurred when the participant’s initial choice was wrong, coupled with a correct recognition from the AI system. The AI system made incorrect recognition 20% of the time by recognizing the participant’s selected choice as another shape other than the reference shape. This led to pattern $RI$ (Case B) if the participant’s initial choice was correct and pattern $WI$ (Case D) if the participant’s initial choice was wrong. The AI system recognized the participant’s initially selected choice as the reference shape 10% of the time. This led to pattern $RC$ (Case A) if the participant’s initial choice was right and pattern $W{I}_{ref}$ (Case E) if the participant’s initial choice was wrong. Therefore, depending on each participant’s initial choice, AI reliability ranged from 70–80%. This reliability is above the baseline threshold level of around 70% to be perceived as useful (Wickens & Dixon, 2007). In the present study, participants were informed that the AI aid’s recommendations were imperfect, but they were not provided with specific information about its reliability level.

Measures

We are interested in the following measures.

Confidence.

After participants completed step 1 of each trial, they rated their confidence in their initial selection on a Visual Analog Scale from 0 to 100. The leftmost anchor was labeled “I am not confident in my answer at all,” and the rightmost anchor was labeled “I absolutely trust the decision aid.”

Trust Adjustment.

After each trial $i$, participants reported their $trust(i)$ in the decision aid. We calculate a trust adjustment as:

$$Trustadjustment(i)=Trust(i)-Trust(i-1),\text{where}i=2,3,\dots ,60$$

Since the moment-to-moment trust is reported after each trial, only 59 trust adjustments are obtained from each participant.

Performance.

Another dependent variable was the final performance of the experimental task, which was calculated as the percentage of correct final answers (determining the validity of the initial answers) for each pattern for each participant (Figure 8a).

Reaction Time.

The reaction time was measured in seconds from when the AI prediction appeared until the participant pressed either the “I was right” or “I was wrong” button (Figure 8a).

Experimental procedure

After participants signed the consent form and completed the online demographics and trust propensity surveys (Merritt et al., 2013), they watched a video explaining the experimental task. Participants were informed that the AI aid’s recommendations were imperfect and may or may not be correct. Each participant completed 60 trials. Upon completion, participants completed a post-experiment survey.

Statistical Analysis

We constructed linear mixed-effects models, accounting for random effects of participants, to examine our hypotheses using the “lme4” package in R (Bates, Mächler, Bolker, Walker, et al., 2014). Mixed-effects models have advantages in dealing with repeated measures and missing values and are, therefore, particularly appropriate for our analysis. We used an iterative approach to construct the models. Following the standard procedure for building mixed-effects models (Field, Field, & Miles, 2012), we started simple and gradually added complexity to our models. Using the likelihood ratio test (LRT), comparisons of models were conducted, and the simpler models were used when no significant differences were found. The level of significance for this study was set to α =0.05.

The mixed-effects model equation is:

$$y_{ij}=\left({\beta }_0+u_{0j}\right)+{\beta }_1\cdot {\text{CaseB}}_{ij}+{\beta }_2\cdot {\text{CaseC}}_{ij}+{\beta }_3\cdot {\text{CaseD}}_{ij}+{\beta }_4\cdot {\text{CaseE}}_{ij}+{ϵ}_{ij}$$

Where:

$y_{ij}$: The outcome variable for participant $j$ under case pattern $i$.

${\beta }_0$: The fixed effect intercept.

$u_{0j}$: The random intercept for the $j\text{th}$ participant.

$Case{B}_{ij}$, $Case{C}_{ij}$, $Case{D}_{ij}$, $Case{E}_{ij}$: Dummy variables for the case patterns, comparing each case to the reference case (i.e., Case A).

${\beta }_1$, ${\beta }_2$, ${\beta }_3$, ${\beta }_4$: The fixed effect coefficients for the case patterns (B, C, D, and E) relative to the reference case A.

${ϵ}_{ij}$: The residual error for participant $j$ in case pattern $i$.

Post-hoc t-tests were conducted to analyze the differences between the patterns. The Kenward-Roger method of estimating degrees of freedom was used, which adjusts the variance-covariance matrix of the fixed effects using a Taylor series expansion, then calculates degrees of freedom (Kenward & Roger, 1997). This method is widely recognized in mixed-effects models for its ability to mitigate Type 1 errors, generate precise p-values, and maintain reliability across varying sample sizes (Kuznetsova, Brockhoff, & Christensen, 2017; Luke, 2017; McNeish & Stapleton, 2016).

RESULTS

After all trials were completed, the number of occurrences for each pattern coupled with the outcome was calculated. As the occurrence of each pattern can only be determined posteriorly, participants might not necessarily display each performance pattern. Table 2 shows the number of occurrences for each performance pattern and the descriptive statistics of all the measurements.

Confidence

Analyzing participants’ data on confidence can be regarded as a check of data quality. We expected that participants would have higher confidence when their initial choices were correct. We organized the patterns into two classifications: right human initial choice ($RC$, $RI$) and wrong human initial choice ($WC$, $WI$, $W{I}_{ref}$). The confidence rating revealed significant differences between the right (Mean = 72.74, SD = 1.98) and the wrong (Mean = 56.77, SD = 2.00) initial choices (t(1962.72) = −20.20, p < .001).

Comparing Patterns $WC$, $WI$, and $W{I}_{ref}$

Significance indicators are displayed on figures to highlight the statistical significance between patterns of performance.

Trust adjustment.

There was a significant difference between the three patterns on trust adjustment (F (2, 875.68) = 61.2, p < .001). Pattern $WC$ had a significantly larger trust increase than pattern $WI$ (t(875) = 6.57, p < .001) and pattern $W{I}_{ref}$ (t(878) = 9.96, p < .001). Pattern $W{I}_{ref}$ had a significantly greater trust decrement than pattern $WI$ (t(876) = −4.46, p < .001) (Figure 11).

Figure 11.

Trust adjustment by patterns of performance. The error bars represent +/− 2 standard errors.

Performance.

There was a significant difference between the three patterns in the final performance (F (2, 858.11) = 29.43, p < .001). Post-hoc results show that pattern $W{I}_{ref}$ had a significantly worse performance compared to $WC$ (t(856.62) = −7.67, p < .001) and to $WI$ (t(856.47) = −5.73, p < .001).

Reaction Time.

For the statistical analyses of reaction times, a log scale transformation was applied to response time data. Both patterns (F (2, 852.35) = 26.82, p < .001) and outcomes (F (1, 862.59) = 24.64, p < .001) significantly affected reaction time. In addition, a significant interaction effect was observed (F (2, 855.15) = 10.38, p < .001). Because of the significant interaction effect, we compare the three patterns within each level of outcome. When the final outcome was right (Figure 13a), there was no significant difference between the three patterns (F (2, 475.49) = .35, p = .71); When the final outcome was wrong (Figure 13b), patterns significantly affected reaction time (F (2, 363.47) = 22.09, p < .001). Post-hoc comparisons revealed $W{I}_{ref}$ resulted in significantly quicker reaction time than $WC$ (t(362.36) = 5.82, p < .001) and $WI$ (t(358.65) = 6.14, p < .001).

Figure 13.

Reaction Time by patterns of performance. The error bars represent +/− 2 standard errors.

Evaluating Continuity, Negativity Bias, and Stabilization

To examine the two properties of trust dynamics, the patterns were organized into two classifications: correct AI predictions (i.e., patterns $RC$ and $WI$) and incorrect AI predictions (i.e., patterns $RI$, $WI$, and $W{I}_{ref}$).

Continuity.

Comparing correct AI predictions to incorrect AI predictions shows that AI success led to a trust increment. Conversely, AI prediction failures led to a decline in trust. In addition, trust at the present moment t is highly correlated with trust at the previous moment $t-1$. To investigate the relationship between the present rating of trust and its historical rating over time, we computed the autocorrelation of trust ratings. The results, depicted in Figure 14, show that trust at time $t$ is highly correlated with trust at the previous moment $t-1$. In addition, the mean autocorrelation decreased as the time gap between ratings increased, indicating a decline in correlation with longer time intervals.

Figure 14.

Autocorrelation of trust as a function of time separation. The error bars represent +/− 2 standard errors.

Negativity bias.

To evaluate the property of negativity bias, we compared the magnitude of trust increment and that of trust decrement. Results show significant differences between the correct (Mean = 1.01, SD = 0.13) and the incorrect (Mean = −3.16, SD = 0.22) AI prediction (t(1968.00) = −16.58, p < .001).

Stabilization.

To evaluate the stabilization property, an analysis using a linear mixed model of trust adjustment as a function of time. Results show a significant decrease in the magnitude of trust adjustment over time for both the correct AI predictions (β = −0.035, t(1448) = −4.58, p < .001) and the incorrect AI predictions (β = −0.203, t(469.27) = −3.50, p < .001).

DISCUSSION

In this section, we discuss the findings in relation to our aims and hypotheses. The primary interest of the study is to compare between patterns $WC$, $WI$, and $W{I}_{ref}$. The results partially support H1, in which we hypothesized that pattern $WC$ will lead to the best performance, followed by pattern $WI$, then pattern $W{I}_{ref}$. Our results showed that pattern $W{I}_{ref}$ had a significantly worse performance compared to $WC$ and $WI$. Out of the 96 occurrences of pattern $W{I}_{ref}$, the outcomes of 72 were wrong. This highlights the high likelihood of making errors when automated decision aid erroneously confirms participants’ wrong initial answer as correct, making participants falsely believe their initial mistakes were correct. In H1, we also hypothesized that the performance of pattern $WC$ would be higher than pattern $WI$. To our surprise, we found a non-significant difference between patterns $WC$ and $WI$, which suggests the strong benefits of making such errors from a utilitarian point of view. This counter-intuitive finding–that certain AI errors are, in fact, beneficial–only surfaces when we go beyond the binary categorization of signal detection theory (SDT).

In prior work examining human-automation/autonomy interaction, risk has been identified as a significant factor influencing trust, dependence, and task performance (Hoff & Bashir, 2015). The risk of an event is the product of the probability of an event and the consequence of that event (Sheridan, 2008; Slovic & Lichtenstein, 1968). Mathematically, Sheridan (2008) expressed risk $R$ as the “product of the probability $P_E$ of an initial reference event $E$ and the sum of products of probability $P_i\mid E$ of each consequence occurring given that E occurs, as well as the cost of each consequence $C_i$.” Therefore, $R=P_E\times {}_i{}^{\text{"}\pounds }\left(P_i\mid E\times C_i\right)$, where ${\mathit{Consequence}}_E={}_i{}^{\text{"}\pounds }\left(P_i\mid E\times C_i\right)$. Applying the equation to the present study, $P_E$ is the probability of each case (i.e., $P_{case}$), and $P_i\mid E$ can be considered the probability of final outcome (i.e., $P_{success}\mid case$ and $P_{failure}\mid case$). In the experiment, participants were encouraged to perform the task as accurately as possible. We did not manipulate the cost of an error (i.e., one medication error could be harmless and another could lead to patient death) and of a success. Therefore, $C_{success}$ and $C_{failure}$ are constants. The findings pertinent to H1 can be summarized as $P_{success}|\mathit{WC}\approx P_{success}|\mathit{WI}>P_{success}|{\mathit{WI}}_{\mathit{ref}}$. Furthermore, if we are willing to assume $C_{success}=1$ and $C_{failure}=-1$, we could calculate the average consequences of the three cases: $Consequenc{e}_{WC}=0.24$, $Consequenc{e}_{WC}=0.14$, and $Consequenc{e}_{WC}=-0.5$.

H2 is fully supported that pattern $WC$ led to the highest trust increment, followed by pattern $WI$, then pattern $W{I}_{ref}$. Prior research has consistently shown that automation performance is a key determinant of the direction of trust change – automation success leads to trust increment, and failure leads to trust decrement (Lee & Moray, 1992; Manzey et al., 2012; Yang, Schemanske, & Searle, 2023; Yang et al., 2017). We simultaneously observe two seemingly competing theories of how people interpret automation performance. On the one hand, if automation performance is assessed based on the automation’s prediction performance per se, we would expect to see a trust increment for pattern $WC$, and a trust decrement for both patterns $WI$ and $W{I}_{ref}$. In addition, the amount of trust decrements should be similar because, for both cases, the automation made a wrong prediction. Alternatively, if automation performance is assessed based on the “utility” of a prediction, we would expect to see a trust decrement for pattern $W{I}_{ref}$ and a trust increment for both patterns $WC$ and $WI$, as both patterns prompt a re-evaluation of the initial selection. Further, the amount of trust increment between the two patterns should be similar because the final performance of the two patterns was comparable. Our results revealed neither theory is entirely right or wrong. Instead, the two forces – the AI’s performance per se and the “utility” of an AI prediction – together determine trust change. In prior research based on SDT, the automation’s prediction performance and its “utility” are aligned. However, in multi-class scenarios, the prediction performance and the “utility” of $WI$ are in opposite directions, causing the intriguing finding. Specifically, the direction of trust change, increment or decrement, is largely determined by the AI performance. Consistent with prior literature (Lee & Moray, 1992; Manzey et al., 2012; Yang, Schemanske, & Searle, 2023; Yang et al., 2017), a wrong prediction leads to a trust decrement. However, the magnitude of trust decrement is further influenced by the “utility”: a wrong prediction leading to good outcomes has a smaller trust decrement.

Regarding the time taken to make a final decision after receiving the AI’s prediction, we hypothesized an interaction between the three patterns and the final outcome. The results supported H3a – when the human operators were tricked by the incorrect reassurance from the AI, they reacted very fast, $W{I}_{ref}$ resulting in significantly quicker reaction time than $WC$ and $WI$. The results did not support H3b. We found a non-significant difference between the three patterns when participants successfully recognized the false reassurance. This lack of significance could be due to the small sample size, and future research is needed to examine this further. In addition, we noticed a pattern of speed-accuracy trade-off, where faster responses were associated with increased errors. In a practical work environment, whether time or accuracy has a higher priority hinges on the payoff structure. For example, in medical settings where a wrong medication may lead to catastrophic outcomes, it might be more beneficial to prioritize accuracy over time.

Along with our primary objectives, we also wished to validate the existence of the three properties of trust dynamics, namely, continuity (H4), negativity bias (H5), and stabilization (H6). The continuity and negativity bias has been reported in the literature consistently (Lee & Moray, 1992; Manzey et al., 2012; Yang, Wickens, & Hölttä-Otto, 2016). The stabilization property was recently reported in the studies of Yang, Guo, and Schemanske (2023) and Guo and Yang (2021). Our study provides further validation of this property – as a person uses an automated technology over and over again, this person’s trust will stabilize over time. It is important to note that the three properties are summarized based on aggregated data from empirical research (i.e., the trust change is averaged over a group of participants). Therefore, the three properties represent the population average behaviors.

CONCLUSION

Most studies on trust in and dependence on automation/autonomy and AI systems used SDT to model the performance of the systems. However, SDT categorizes the world into binary states—signal present or absent—which oversimplifies the complex dynamics observed in real-world scenarios. This binary categorization fails to address the recognition of complexity inherent in multi-class classification scenarios, which are increasingly prevalent in advanced AI systems across various sectors. Examples include medical diagnosis through image classification (classifying chest X-rays into normal, pneumonia, or lung cancer), object detection in autonomous vehicles (recognizing various objects encountered on the road), and product categorization in e-commerce (classifying products as electronics, clothing, home appliances, or books), among others (Gao et al., 2018; Kozareva, 2015; Rahman, Desai, & Bhattacharya, 2008). As far as we know, we are one of the few, if not the only, studies that examine trust in and dependence on automation/autonomy and AI systems beyond the binary classification of the state of the world.

Our study revealed the difference in performance, response time, and trust between three patterns: $WC$, $WI$, and $W{I}_{ref}$. These findings emphasize the need for careful consideration of error patterns in automated systems. In scenarios where AI falsely reassures the human operator, such as in pattern $W{I}_{ref}$, we observed high likelihood for operators to make expedited decisions that often led to errors. This highlights a significant danger in AI design—systems that mislead operators can severely undermine safety and efficiency. Conversely, errors that lead to safety checks and verifications, even though incorrect, can enhance the overall reliability of human-AI systems by preventing potential mishaps. Our results also illustrate the generalizability of the three properties of trust dynamics, continuity, negativity bias, and stabilization, illustrating how people’s trust can change over time due to moment-to-moment interaction with AI.

There are several limitations to this study. First, in the present study, we did not manipulate the cost of an error or the cost of success (i.e., $C_i$ in the risk equation), which often varies in safety-critical domains. For instance, Aldhwaihi, Schifano, Pezzolesi, and Umaru (2016) found that although most dispensing errors caused no harm, some errors had high consequences that could potentially cause harm or even death, depending on the dosage and patients’ vulnerability. Future research could incorporate varying costs into experiments to examine the impact of various AI error types. Also, future research should be conducted to validate the results in more realistic contexts, particularly in safety-critical domains. Second, inspired by the medical dispensing examples described at the beginning of the article, the testbed we developed focused on a mental rotation and recognition task. Future studies could replicate the study using testbeds focusing on other human information processing components such as memory and decision-making. Third, there were only 96 $W{I}_{ref}$ cases, which could have caused the lack of significance in H3b. Future studies could benefit from a larger sample size.

Figure 12.

Performance by patterns of performance. The error bars represent +/− 2 standard errors.

Figure 15.

Trust adjustment by patterns of performance. The error bars represent +/− 2 standard errors.

Keypoints.

• AI systems that offer a multi-class classification of the state of the world can lead to types of errors that SDT does not traditionally capture.

• Varying error patterns from AI significantly impacted performance, reaction times, and trust adjustments.

• AI errors can provide false reassurance, misleading operators into believing their incorrect decisions were correct. This leads to worsened performance and a significant decline in trust. Conversely, and paradoxically, some AI errors can prompt safety verification, ultimately enhancing overall performance.

• We illustrate the generalizability of the three properties of trust dynamics: continuity, negativity bias, and stabilization.

Biographies

Jin Yong Kim is a Ph.D. candidate in the Department of Industrial and Operations Engineering (Human Factors) at the University of Michigan Ann Arbor. He obtained a BSE in Industrial and Operations Engineering from the University of Michigan in 2021.

Corey Lester is an Assistant Professor in the College of Pharmacy at the University of Michigan Ann Arbor. He received his PharmD from the University of Rhode Island in 2012 and completed a PGY-1 Community Practice Residency at Virginia Commonwealth University. He received his PhD from the University of Wisconsin-Madison, School of Pharmacy in 2017.

X. Jessie Yang is an Associate Professor in the Department of Industrial and Operations Engineering and the Department of Robotics at the University of Michigan Ann Arbor. She obtained a PhD in Mechanical and Aerospace Engineering (Human Factors) from Nanyang Technological University, Singapore in 2014.