Understanding the experiences of self-injurious behavior in autism spectrum disorder: Implications for monitoring technology design

Abstract

Objective

Monitoring technology may assist in managing self-injurious behavior (SIB), a pervasive concern in autism spectrum disorder (ASD). Affiliated stakeholder perspectives should be considered to design effective and accepted SIB monitoring methods. We examined caregiver experiences to generate design guidance for SIB monitoring technology.

Materials and Methods

Twenty-three educators and 16 parents of individuals with ASD and SIB completed interviews or focus groups to discuss needs related to monitoring SIB and associated technology use.

Results

Qualitative content analysis of participant responses revealed 7 main themes associated with SIB and technology: triggers, emotional responses, SIB characteristics, management approaches, caregiver impact, child/student impact, and sensory/technology preferences.

Discussion

The derived themes indicated areas of emphasis for design at the intersection of monitoring and SIB. Systems design at this intersection should consider the range of manifestations of and management approaches for SIB. It should also attend to interactions among children with SIB, their caregivers, and the technology. Design should prioritize the transferability of physical technology and behavioral data as well as the safety, durability, and sensory implications of technology.

Conclusions

The collected stakeholder perspectives provide preliminary groundwork for an SIB monitoring system responsive to needs as articulated by caregivers. Technology design based on this groundwork should follow an iterative process that meaningfully engages caregivers and individuals with SIB in naturalistic settings.

INTRODUCTION

Autism spectrum disorder and self-injurious behavior

Autism spectrum disorder (ASD) is a prevalent pervasive developmental disability, affecting an estimated 1 in 59 youths.¹ In addition to atypical behaviors related to socialization and communication, a majority (68%) of children with ASD exhibit behaviors that can have physical consequences.^2,3 Such behaviors often fall under the umbrella term of self-injurious behavior (SIB).^3,4 SIB can include repetitive and rhythmic behaviors such as head banging and self-hitting,⁵ and is a leading cause of hospitalization for children with ASD.⁶ Parents and educators (hereafter referred to as caregivers) are often unable to control these episodes,⁷ especially as children reach adolescence.

Early childhood interventions, such as applied behavioral analysis, can support individuals with SIB and mitigate challenges associated with SIB. Many providers engage in applied behavioral analysis consisting of functional assessments to detect triggers (“antecedents”) of SIB before developing a management plan to guide interventions that would address such triggers.^8–10 Functional assessments are based on the well-established history of direct observational assessments.¹¹ In completing functional assessments, detailed events preceding and following SIB are recorded and described, along with other potentially relevant information such as environmental factors.¹⁰ Traditional functional assessments are often time-consuming to complete, due to the required attention to details in observation and extensive note taking, as well as the data complexity.¹¹ A further drawback associated with such assessments is that the behavior of interest (eg, SIB) may not be observed, owing to factors such as the time window of assessment or an absence of triggers in a specific environment.¹² Assessments may therefore need to be repeated, adding to the required time for completion. Smart behavioral tracking methods could help alleviate this workload from clinicians or caregivers by capturing relevant behaviors automatically vs manually.

Clinicians can complete functional assessments or train caregivers to perform them. Functional assessments performed by clinicians are informed by their substantial and specialized training,¹³ yet results can suffer from low ecological validity because they are unable to capture behaviors commonly occurring outside of the clinic, such as the home.¹³ Triggers identified from a functional assessment in the clinic may not accurately represent the triggers of SIB in other environments, and thus functional assessment results may not transfer across environments.¹⁴ When caregivers help complete analyses in more than 1 context, behavioral data are often more ecologically valid than when data are limited to 1 setting.¹⁴ Caregivers can face challenges with demands on attention (eg., other children or students, or other household or classroom responsibilities) while attempting to adhere to analysis protocols. Caregivers may not be able to record events while trying to safely control an SIB episode⁷; consequently, triggers may only be recalled afterward and be subject to recall bias, affecting accuracy.¹⁰ Monitoring technology could mitigate such challenges, and facilitate a more comprehensive and objective approach to functional assessments.

Monitoring technology for ASD

Sensors that record activity and other health-related data^15–17 could support monitoring beyond the clinic and advance functional assessment for ASD. Such sensors could provide continuous monitoring of SIB across contexts, extending into nonclinical settings such as the home and school. Continuously collected and accessible data could also inform decisions about time-sensitive injury risk to individuals or caregivers.^18–21 Technology may also enable people with ASD to self-monitor (eg, alerting them to recognize emotional states typically leading to SIB, and proactively suggesting replacement activities),²² offer warnings of SIB onset with management suggestions to caregivers, or help caregivers understand children’s behavioral and diurnal trends.²³Technology-based interventions are feasible and effective for ASD^24–26 but have not been studied for SIB.

Caregiver inclusion

Caregivers should be critical contributors to the development of monitoring technology, given their expertise in the day-to-day realities of SIB monitoring and management.²⁷ The importance of caregiver involvement has been demonstrated in systems design for people with cognitive disabilities²⁸ and children.²⁹ To the best of our knowledge, studies on the design of monitoring technologies for ASD have not included both educators and parents.^24–26 Failure to include caregivers during the design process may lead to inaccessible or undesired products and misused (or unused) innovations, in addition to unintended consequences.^30,31

Caregiver experiences related to monitoring and managing SIB have been explored from a psychological, but not technology design, perspective. Prior work exploring family experiences with SIB⁷ and anxiety³² among individuals with ASD found that such experiences related to financial strain and harmful effects on family well-being,⁷ and that child anxiety was often more of a concern than the diagnosis itself.³² While providing valuable insight related to the effects of SIB and anxiety on parents and home caregivers, these studies did not explicitly address technology-related needs. Furthermore, previous studies did not simultaneously consider parents and educators as caregivers, nor did they account for multiple contexts of use (eg, home, school). Technology needs and effectiveness may differ across settings,³³ and different perspectives are likely to lead to more robust designs that can meet the needs of different users.²⁵ To create effective SIB monitoring technology, interdisciplinary informatics research should assess needs, including social, environmental, and organizational factors affecting lived experience with SIB.^28,34–38 The purpose of this study is to assess multiple caregiver perspectives across settings for SIB monitoring technology to explicitly elicit guidance for a system design.

MATERIALS AND METHODS

Sample and setting

Participants were recruited from rural and urban areas within the Mid-Atlantic United States. Recruitment occurred at a private school offering ASD services and through researcher networks. Individuals who self-identified as either parents or guardians (hereafter referred to as parents) or educators of children and students with ASD and SIB participated. Consistent with prior work, the term children and students refers to individuals with ASD and SIB between f 2 and 30 years of age.³⁹ Although aggression is more prevalent in children under 9 years of age,² it often persists as children age.⁴⁰ Data collection emphasized associated caregivers, because many individuals with SIB were expected to be minimally verbal or “preverbal” based on pilot data and prior work.⁴⁰ All adult participants provided informed consent before data collection, and research procedures were approved by the University of Virginia Institutional Review Board for Social and Behavioral Sciences and the Virginia Tech Institutional Review Board. Participants received $20 gift cards and travel compensation.

Data collection

To enable triangulation through data collection methods, interviews and focus groups were held concurrently using the same questions for both methods of data collection. Individual interviews were conducted with caregivers who were recruited via snowball sampling, whereas focus groups were conducted with individuals who were recruited through community partners (eg, schools or parent support groups). Individual interviews lasted approximately 1 hour, and focus groups 1.5 hours.⁴¹ Remote interviews occurred over the phone, whereas in-person interviews and focus groups occurred in private conference rooms in public locations (eg, public libraries). Focus groups were conducted separately for parents and educators. An initial set of broad, open-ended questions built on prior work related to aggression and ASD⁷ and previous literature emphasizing the context of technology use^25,42 was developed to capture a wide range of responses.⁴³ Subject matter experts, including a parent, an educator, and a clinician, evaluated the initial interview questions before use to ensure that they captured relevant discussion points, used preferred terminology, and comprehensively covered important characteristics of SIB and monitoring technology. Questions spanned topics of SIB and its management, current and projected use of monitoring technology, and related benefits and challenges. The initial set of questions was used in the first 3 parent and educator interviews. Analysis of results from this first stage led to a second version of interview questions capturing additional discussion topics, specifically related to technology (eg, incorporating nondigital technology), child sensory preferences, and the wide variety of SIB definitions. Questions were again checked by experts and remained unchanged for the remainder of the study.

The interviews and focus groups concluded with an introduction of 2 potential forms of monitoring technology. The moderator explicitly stated that the devices were only examples and that the researchers had neither affiliation nor investment with such products. These technologies consisted of on-the-market remote data collection devices explored specifically for ASD. Each highlighted different potential technology: (1) accelerometers without attachment methods (isolated from a wrist band), representing wearable devices⁴⁴; and (2) a combined depth and video camera, representing nonwearable devices.^9,45

Demographic questionnaires were distributed after session completion to collect information on age, sex, race, and ethnicity, and all participants except for 4 educators completed them. Several questions differed between the parent and educator demographic questionnaires. For example, parents were asked about their relationship to the child and to rate the degree to which SIB presents a challenging problem. Educators were asked to describe their roles and experience in education. Several interview questions also differed between parents and educators to accommodate for appropriate settings (home and school) and terminology (family and students). For example, parents were probed about the impact of SIB in their personal lives and asked in detail about their children’s specific preferences or aversions and range of behaviors. Educators were asked to describe the children they taught in terms of perceived diagnosis severity and about the range of behaviors encountered throughout their years of teaching. Please see the Supplementary Appendix for demographic questionnaires and interview guides.

Data analysis

All interactive aspects of data collection were audio-recorded, professionally transcribed, and subsequently de-identified. The first 6 interviews were used both as part of the main analysis and to establish the codebook. Three members of the team (K.D.C.-G., C.D.K., D.M.G.F.) repeatedly read through the texts to examine the dataset as a whole.⁴³ Data analysis was informed by qualitative content analysis methods,⁴⁶ specifically Hsieh and Shannon’s⁴³ procedure for conventional content analysis. This approach was used to generate descriptive themes and categories, but modified to organize results for the design community.⁴⁷ Data analysis was conducted using QSR NVivo v10 (QSR International, Doncaster, Australia).⁴⁸K.D.C.-G., C.D.K., and D.M.G.F. independently coded transcripts before joining to develop a synthesized list of codes. Afterward, with R.S.V., these codes were placed on separate index cards for card sorting which led to the development of themes and categories. Card sorting was used to identify main themes that were conceptually distinct. Cross-cutting themes, relevant to all of the main themes, were identified during the discussion that resulted from card sorting. After developing the initial codebook, the researchers divided the remaining transcripts and completed analyses, regularly discussing questions, concerns, and potential discrepancies. The content of regularly occurring discussions was captured by the K.D.C.-G., and mapped collaboratively to appropriate main and cross-cutting themes. Through discussions and consensus building, the codebook was iteratively updated to promote consistent coding of remaining data. The codebook included documentation of labels, definitions, and relevant examples for inclusion and exclusion in each main theme and category.⁴⁹ Simultaneous coding allowed multiple meanings to be captured if several were represented within 1 textual sample.⁴⁹ The final codebook is presented in Supplementary Appendix.

RESULTS

Participants

Thirty-nine individuals participated (Table 1). Parent participants discussed children ranging from 6 to 26 years of age (mean 14.1 ± 6.7 years of age; 12 sons, 3 daughters), whereas educators discussed students whose ages were estimated as 3-22 years. Parents’ ratings of the degree to which SIB presents a challenging problem ranged from 1 (minor problem) to 3 (severe problem), with a mean of 1.8 and mode of 2. Educator responsibilities varied widely and included teaching, programmatic, and administrative roles. Educators often had applied behavioral analysis training.

Table 1.

Summary information on parents and educators

Participants	Gender profile	Age (y)	Years of experience	Number of focus groups
Parents	2 fathers,	31-62		2
	14 mothers	(45.1 ± 8.1)	NA	(2 and 3 people)
Educators	8 men,	22-46	2-24	2
	15 women	(31.1 ± 7.8)	(7.6 ± 6.2)	(7 and 10 people)

Values are range (mean ± SD).

NA, not applicable.

Main themes

Participant responses revealed 7 main themes. Six specifically described SIB: (1) triggers, (2) emotional responses, (3) SIB characteristics, (4) management approaches, (5) child/student impact, and (6) caregiver impact. Responses involving current technology motivated the theme of (7) sensory/technology preferences. The Supplementary Appendix provides themes, subthemes, and supporting quotes for each main theme, as summarized subsequently.

Triggers

When asked to discuss patterns of SIB, participants described known or uncertain triggers that were potential antecedents (events and conditions). Triggers were variable across children or students and within their daily presentation. Although some were concretely established, such as SIBs occurring directly after a specific event (1 [R1]) (Supplementary Appendix), others were more vaguely described (eg, fatigue [R2] or communication [R3]).

Emotional responses

Participants commented on emotional responses from the child that preceded SIB, describing reactions that led to or occurred during SIB. Emotions, often discussed in terms of direct causes of the behavior, involved feelings of upset, stress, frustration, or anger (R4). Positive emotions, like enjoyment or excitement, were discussed as both a cause of (R5) and a response to (R6) the behavior.

Self-injurious behavior characteristics

Participants described hurtful or harmful actions the child took toward him/herself that could or did lead to injury. Participants defined SIB and described intensity, duration, and frequency (R7), including a variety of SIB types with variable presentations. Parents noticed SIB onset at different ages, starting as young as 2.5 years of age, before an ASD diagnosis (R8). Characteristics of SIB often changed over time (R9). Participants reported that their children or students showed SIB in certain settings (eg, in the home but not at school) or used certain parts of that setting to self-injure (R10).

Management

Participants discussed current and past methods or approaches they or others use to change aspects of SIB. They described managing SIB in terms of sense-making (data collection) of the behavior, which often informed management methods. Both parents and educators noted issues with data collection while simultaneously managing behavior, indicating conflict between management safety and data accuracy (R11). Participants described the inconvenience and safety risk of using supplementary material (pencil, clipboard, chart, and a timer) (R12). Management strategies and their effectiveness varied widely, depending on the approach as well as the circumstances (R13, R14). Examples of approaches included ignoring the behavior, redirecting the behavior (R15), and using a token system to reward positive behavior.

Child/student impact

Participants described observed meta-level effects of SIB on their children or students, who frequently experienced social isolation through exclusion and limitations in social life. They often aimed to help students adapt to social norms, such as teaching them to redirect SIB into alternative behaviors when with friends (R16). Long-term health effects, including tissue damage, eyesight loss, and concussions, and resulting hospital stays, were also discussed. Participants described effects of SIB on education, with limited options for schools and services due to safety concerns. Once in school, SIB prevented students from accessing the curriculum, often due to missed class time (R17).

Caregiver impact

Participants commonly reported meta-level effects of SIB on parents or guardians, family, and educators of people with SIB. Caregivers experienced emotions such as stress and despair (R18, R19) as well as concerns over finances and education (R20). Participants explained tension in relationships due to SIB, such as with loved ones who might experience a stressed caregiver or perceive SIB and its management differently (R21). Participants also described difficulty in bringing children or students with SIB into the community, feeling less in control of managing SIB, and feeling judged when in public.

Sensory/technology preferences

Participants described reactions (sensory-seeking or sensory-avoidance) to objects or events that indirectly or directly relate to technology through stimuli or technology use. Examples of preferred stimuli included sequins or soft material, repetitive cartoons or noises, scents, physical pressure, and vibration. Participants mentioned examples of desensitization for sensory aversions (R22), described current technology use (eg, cellphones and timers [R23]) and potential use (eg, preferred methods of attachment of smart health technology), and discussed available and commonly used manual monitoring techniques (R24).

After seeing examples of wearable and nonwearable technology, participants expressed reservations about individuals with SIB interacting with such technology. For example, participants discussed possibly losing the accelerometer, and suggested adhering it to clothing or directly to the body to avoid problems of disrobing. Participants also suggested safe and easy removal to support safety of children and caregivers (R25). Caregivers indicated reservations about students noticing the depth camera in the environment and its field of view. They indicated that a wearable device with flexible and discrete placement on the child or student (R26), or on the caregiver (eg, camera), could address individual sensory concerns while remaining inconspicuous.

Crosscutting themes

Data were also cross-coded with 2 underlying themes, uncertainty and state of experience (hypothetical vs existing experiences). These themes were evident throughout all 7 main themes, and capture projected needs and concerns. Feelings of uncertainty often appeared to motivate comments about SIB definitions and origins (R27, R28). Individual interviewees commonly asked the interviewer if the discussed behavior qualified as SIB. Further, limited communication between children or students with communication difficulties and caregivers led to caregiver speculation about triggers, child or student emotions, and long-term impact. Participants described experiences from the past and present, as well as anticipated experiences. For example, caregivers discussed hypotheticals about data collection systems (R29) and hopes for data collection future technology (R30).

Parent or educator differences

Parent or educator differences were not formally coded but were noted during team discussions. Although parents and educators shared similar SIB experiences with the same pervasive themes in Supplementary Appendix, there were several differences between the 2 groups. Educators described data collection as integral to their job, especially in private schools for students with ASD (R31). Parents, however, noted demands that often prevented data collection (R32). Both groups discussed ways their contexts (school or home) served as a barrier to understanding SIB. Educators also discussed trends observed in groups of students, while parents spoke of their individual child (R33). Financial impact was only reported in parent experiences (R34).

DISCUSSION

Parent and educator responses revealed 7 main themes (Supplementary Appendix). The first 4 encompassed experiences before, during, and after SIB (triggers, emotional responses, SIB characteristics, and management approaches). Two others comprised long-term effects of SIB on the child and the caregiver (child/student impact and caregiver impact). Interactions with technology, particularly as applied to potential monitoring, were captured by the final theme (sensory/technology preferences). Orthogonal to these 7 themes were the crosscutting themes of uncertainty and state of experience. Parent or educator differences were observed across multiple themes.

SIB onset and management

Our findings related to the first 4 themes characterizing SIB, its triggers, and its management support previous research in several respects. Triggers identified here have also been reported in prior work, including escape from demands and change in routine (cognitive rigidity).⁵⁰ SIB types, such as head banging and self-biting, and the variable SIB duration also reflect earlier findings.^5,50 In contrast to prior work,^5,50 however, caregivers discussed explicit uncertainty about triggers and definitions of SIB. This uncertainty could stem from the changing triggers and continuously evolving behaviors that caregivers described (ie, that SIB may change in form for a given individual), which requires additional behavioral monitoring and changes to management plans. Findings related to ambiguously defined and evolving triggers and behaviors have novel implications for design. For example, SIB monitoring should include adaptive (smart) monitoring technology to learn changing behaviors or new types of SIB, and provide such information to caregivers. Such technology could thereby help reduce workload associated with managing uncertainty and changes over time. Moreover, the specific timing of SIB could offer information about triggers (eg, if SIB occurs at 8:05 am, and the school bus comes at 8:10 am, an event related to the bus may be the trigger). Additionally, quantifying intensity (eg, through collecting accelerometry data and calculating acceleration derivatives of SIB movement) could help determine risks of the behavior, or whether it could be considered self-injurious (eg, based on repetitiveness or velocity).

Caregivers also discussed management approaches. Similar to the work of Holden et al³⁸ on chronic illness management, our findings suggest that SIB management depends on existing resources, caregiver engagement, and support systems. In an earlier report,²⁷ researchers concluded that caregiver participation in medical data systems is critical to obtaining accurate data. Different from prior work, our findings reveal a critical need to consider the implications of simultaneously collecting data and mitigating risk. Specifically, caregivers noted concerns with potential conflicts between child or student safety and data accuracy, which technology could address. Automatic detection of SIB and accessible resources for management could alleviate caregiver responsibility by reducing workload while collecting accurate data.^38,47 Technology could also incorporate warnings of escalating behavior through the phone to support child safety. Last, it could provide caregivers with suggestions for management “next steps,” similar to notifications and suggestions provided by smartphones, including individualized education programs.

Our results indicate that collaboration between parents and educators supports consistent management methods, typically resulting in more positive management outcomes. This finding supports the need for SIB monitoring harmony, reflecting the goal of a synchronized and multiple user–centered database²⁷ in which specified caregivers and individuals can input and access SIB data and trends. Such a database could address the need to stay informed about child or student progress, alert caregivers of potential episodes, provide features for scheduling team meetings, and highlight behavioral data of particular concern across settings.

Long-term effects of SIB

Two other themes emerged relating to the long-term impacts of SIB on the child or student and caregiver. While prior national surveys highlighted that adults with SIB suffer from social isolation,⁴⁵ our results extend this finding to youth with ASD. Different from previous findings, caregivers discussed the impact of SIB on accessing the curriculum, implying potential hindering of educational opportunities. Similar to earlier reports examining SIB or aggression in ASD, we found that SIB impacts caregivers through high levels of stress, strains on relationships and finances, and insufficient resources.^7,51 Monitoring technology could address effects of SIB (eg, seclusion, lack of resources) through integration of technology fostering social support and community. A feature could remind caregivers to care for themselves during periods of high SIB frequency that may elicit high caregiver stress. Future work should examine user preferences for community facets of SIB monitoring, specifically to address the timing and type of caregiver prompts or notifications when the technology detects an increase in SIB. A feature of SIB monitoring could connect caregivers who are familiar with managing SIB to other caregivers who are managing increased or worsened SIB, which could support the sharing of effective management methods in times of need.

Technology preferences

The last main theme, sensory/technology preferences, captured current technology use and prospective preferences about monitoring technology. Participant comments were of particular interest if cross-coded with uncertainty or a hypothetical experience; these themes often emphasized areas of need such as attachment location. Our work, combined with earlier evidence, indicates that a wide variety of technology is used among individuals with ASD in home and school settings.⁵² Designers should consider technology already integrated into daily routines to support a smooth transition into SIB monitoring. For example, SIB monitoring could be integrated into a portal for communication between parents and educators. A report of the behaviors occurring at school could be sent automatically to parents before pickup, which would support consistency of care and school or home transitions.

Our study contributes to a broad understanding of design options for ASD and SIB by exploring related sensory stimuli for technology use. For example, participants mentioned “soft” (nonelectronic) technologies, such as “chewy bands” for behavior redirection from biting oneself, which were excluded in other work examining technology-related interventions for ASD.²⁵ We suggest that “soft” technologies could be incorporated into the hardware design to encourage user acceptance or redirection, such as encasing wearables with such technology or adding separate attachments with soft technology options (see Supplementary Appendix for design suggestions).

Additional sensory preferences (eg, pressure or vibration) could also be critical if the system is designed to notify the person with SIB of their escalating state. Future work should consider methods of feedback to notify children and caregivers of oncoming SIB, specifically examining the effects of the type, duration, and timing of feedback (before, during, and after SIB) on child and caregiver attention (eg, eye gaze and response time). However, based on present findings, we recommend that designers and caregivers include sensory feedback with caution; participants noted that once accustomed to an input and resulting feedback, children or students could fixate on that input-output loop and even use it for sensory stimulation. Further, the device itself could be used for sensory stimulation (eg, repeatedly rubbing or biting preferable material), or for self-injury (eg, using a body-worn sensor to hit themselves, or environmental technology to aggress). Designs should be examined beyond abbreviated study sessions among children with ASD and SIB to consider such potential interactions with technology. Resulting designs could offer flexible customization such as selections of sounds, lights, prints or colors, and attachment locations and methods to account for individual specific sensory preferences, aversions, and potential stimulation or fixation.

Limitations

Parents included in the study predominantly discussed sons with ASD and SIB. Though we did not target a particular gender ratio, the resulting sample included children who reflected the widely cited 4:1 male-to-female ratio for ASD diagnoses.⁵³ Participants were recruited from a relatively broad geographic region, but this study could be augmented with future work from other regions. Future research should also include more fathers. Though we collected more male perspectives than in prior work,⁷ several of the mothers interviewed suggested that their parenting approaches often differed from the those of the other parent involved. Additional work could support design that represents diverse perspectives of parental caregivers.

CONCLUSION

Seven main themes emerged regarding SIB and related technology needs, which were used to offer design considerations for future monitoring technology. This study included caregivers of children with a wide age range (6-26 years of age), which supports both early intervention for SIB and continued support for young adults.^7,54,55 Because participatory design can promote long-term adoption,³⁶ subsequent studies should also include children in participatory design. Early design feedback can be obtained from caregivers using technology with children, and from children interacting with technology in the presence of caregivers. Interactions with such technology should be evaluated with careful attention to unforeseen uses (eg, a hand sensor impacting the head of a child hitting him/herself). Technology should also be evaluated in groups of students to examine social influences among children with and without the devices, which would be of particular importance in school settings. We expect that technology acceptance and effectiveness may be promoted by building technologies responsive to the needs articulated by caregivers in this and other studies. Once adopted, SIB monitoring could support remote care for a pervasive concern in an increasingly prevalent disability.

FUNDING

This research was supported by a National Science Foundation Graduate Research Fellowship (to KDC-G) and a 4-VA Collaborative Research Grant (to RSV and MAN). However, neither agency had any involvement in data analysis, interpretation, or the decision to publish.

AUTHOR CONTRIBUTIONS

KDC-G and RSV conceptualized the study with support from MAN, SWW, and SK. KDC-G led data collection and analysis efforts with support from CDK and DMGF. All data collection and analysis was overseen by RSV. KDC-G, RSV, and MAN led writing efforts with contributions from everyone else. All authors reviewed and approved the final manuscript.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online

CONFLICT OF INTEREST STATEMENT

The authors declare no competing interests.