Understanding the acceptability of a computer decision support system in pediatric primary care

Abstract

Objective

Individual users’ attitudes and opinions help predict successful adoption of health information technology (HIT) into practice; however, little is known about pediatric users’ acceptance of HIT for medical decision-making at the point of care.

Materials and methods

We wished to examine the attitudes and opinions of pediatric users’ toward the Child Health Improvement through Computer Automation (CHICA) system, a computer decision support system linked to an electronic health record in four community pediatric clinics. Surveys were administered in 2011 and 2012 to all users to measure CHICA's acceptability and users’ satisfaction with it. Free text comments were analyzed for themes to understand areas of potential technical refinement.

Results

70 participants completed the survey in 2011 (100% response rate) and 64 of 66 (97% response rate) in 2012. Initially, satisfaction with CHICA was mixed. In general, users felt the system held promise; however various critiques reflected difficulties understanding integrated technical aspects of how CHICA worked, as well as concern with the format and wording on generated forms for families and users. In the subsequent year, users’ ratings reflected improved satisfaction and acceptance. Comments also reflected a deeper understanding of the system's logic, often accompanied by suggestions on potential refinements to make CHICA more useful at the point of care.

Conclusions

Pediatric users appreciate the system's automation and enhancements that allow relevant and meaningful clinical data to be accessible at point of care. Understanding users’ acceptability and satisfaction is critical for ongoing refinement of HIT to ensure successful adoption into practice.

Background and significance

The adoption of health information technology (HIT) has been slower in pediatrics compared to other fields of medicine.^1–3 However, HIT has been diffusing into a range of applications, including patient kiosks, computerized medication dosing and order entry, web portals, and various computer decision support applications.^4–9 Thus, while the use of HIT is still relatively limited overall,¹⁰ the potential exists to increase uptake and assimilation of HIT to facilitate the delivery of pediatric healthcare.

The limited adoption of HIT in pediatrics may relate to the unique aspects of pediatric workflow.^{11 12} The typical primary care environment is high volume; about 50% is health maintenance and screening, with the average visit lasting 20 min.^{11 12} Moreover, providers are expected to sort through an enormous volume of guideline recommendations.¹³ However, providers continue to have difficulty assimilating guidelines into practice due to visit time constraints, the breadth of anticipatory guidance topics, and balancing parents’ concerns regarding their children's health, development, and behavior.^{13 14}

One example of HIT in pediatrics is the Child Health Improvement through Computer Automation system (CHICA). CHICA is a decision support system linked to an electronic health record (EHR) designed specifically to provide decision support in a busy pediatric office setting.¹⁵ To maximize successful integration of a system like CHICA within pediatric practice, the attitudes and opinions of healthcare providers toward HIT are of particular importance.¹⁶ Currently, there are a limited number of studies that have specifically examined pediatric providers’ perceptions toward the effect of HIT applications in ‘real world’ practice.^{1 17 18}

The objective of this study was to examine the attitudes and opinions of pediatric providers and clinical staff on the acceptability of CHICA in four busy pediatric community clinics. This study was part of an ongoing quality improvement process to refine CHICA that has been in existence since 2004 within one healthcare organization.

Methods

Overview of the CHICA system

The CHICA system is an innovative computer decision support system (CDSS) and EHR which has been described elsewhere.^19–21 Briefly, CHICA combines pediatric clinical guidelines encoded in Arden Syntax rules with a scannable and tailored paper-based user interface, and an HL7-compliant interface to an existing EHR.^{22 23} However, CHICA can operate as a standalone EHR system.

After a patient completes registration for an appointment, CHICA produces a tailored pre-screener form (PSF) that contains 20 health risk questions for the parent or patient (if 12 years or older) to complete¹⁹ (see figure 1 for a sample PSF form). The 20 questions are selected from a library of questions based on previous information contained in the patient's EHR and the age of the patient at the time of the visit. Because the number of possible questions exceeds what can be asked at a single visit, CHICA uses a unique prioritization scheme that takes into account the likelihood and seriousness of the risk as well as the effectiveness of intervening on the risk and the evidence to support it.²³ The PSF is completed in the waiting room before the medical encounter.

Figure 1.

Sample Child Health Improvement through Computer Automation (CHICA) pre-screener form.

Once completed, the PSF is scanned back into CHICA, the collected data are immediately integrated into the EHR, and a second scannable physician worksheet (PWS) is generated. The PWS has space to record the history and physical exam and represents the medical record for that encounter. It also has six tailored prompts based on information collected from the PSF and information contained in the patient's EHR (see figure 2 for a sample PWS). CHICA generates ‘just in time’ (JIT) handouts to supplement physician counseling for certain prompts or to collect additional information that can be scanned into CHICA. When initially developed, all PSF forms were only available in English. However, this was changed early in the life of CHICA, which now prints questions in English on one side and Spanish on the other.

Figure 2.

Sample Child Health Improvement through Computer Automation (CHICA) physician worksheet.

Setting and participants

The first version of CHICA, or CHICA V.1.0, was implemented in one large pediatric community clinic in November 2004. In 2008 and 2009, CHICA V.2.0 was developed on an open-source electronic medical record framework (http://www.openmrs.org) to support the expansion of CHICA into multiple sites. CHICA V.2.0 was eventually implemented in four pediatric community clinics between July 1, 2009 and May 13, 2010. From November 2004 through February 2013, CHICA supported the care of over 34 000 pediatric patients across 188 000 medical visits. CHICA was designed to prioritize and automate surveillance and screening, and facilitate physician decision-making and documentation within these busy clinical and educational settings. Because of the user-friendly scannable paper interface and the need to print and scan documents into CHICA at various points along the clinical workflow, the users of the CHICA system include not only residents and faculty, but also nurses, medical assistants, and front office support staff.

Data collection and analysis

A survey was developed to assess general acceptability and satisfaction from all pediatric users of the CHICA system despite the availability of a previously validated survey on technology acceptance.²⁴ Survey items were designed to collect information unique to the design of the CHICA system (eg, ‘handouts CHICA produces are useful’), as well as its impact on the daily workflow (‘CHICA tends to slow down clinic’) and decision-making of CHICA users (‘CHICA sometimes reminds me of things I otherwise would have forgotten’). Respondents were asked to rate their level of agreement to a series of statements using a 5-point Likert scale ranging from 1=strongly agree to 5=strongly disagree. Items were sometimes positively worded (‘CHICA makes documentation easier’) or negatively worded (‘I would rather not use CHICA’) to prevent a ‘halo effect’. Additional demographic information about respondents was collected, including role in the clinic (physician, nurse, medical assistant, front desk, or other), time in clinic (full or part time), average number of children 18 years or younger seen per week (1–25, 26–50, 76–100, or >100) and—if a physician—whether he/she was a faculty or resident physician and his or her primary specialty (pediatrics, internal medicine/pediatrics, other). One open-ended question invited each respondent to provide any suggestions or comments about CHICA. The survey was administered to all CHICA users annually, starting in 2011, as part of an ongoing quality improvement process. No identifiable information was collected from participants so as to encourage candid responses and suggestions for improvement. All data were entered into an Excel database. During analysis, survey items response categories rated on a 5-point Likert scale were collapsed. Therefore, ‘strongly agree’ and ‘somewhat agree’ were collapsed to ‘agree’, and ‘somewhat disagree’ and ‘strongly disagree’ were collapsed to ‘disagree’; neutral responses did not change. Descriptive statistics and additional quantitative analysis comparing the change in reported means using the Wilcoxon rank-sum (Whitney–Mann) test to compare changes between wave 1 and 2 among independent samples was performed using Stata V.11. Free text comments from the surveys were transcribed, reviewed anonymously, and organized under themes that emerged using conventional content analysis.²⁵ Findings presented represent data from two waves of administration in 2011 and 2012. This study was reviewed and approved by the Indiana University Office of Research Administration.

Results

In 2011, all 70 eligible pediatric users in the four pediatric community clinics using CHICA returned the survey for a 100% response rate. Another round of surveys was done in 2012. Surveys were completed by all but two CHICA users (64 out of 66 eligible) for a 97% response rate. Just over half of the sample was made up of physician users (59% and 53% in wave 1 and wave 2, respectively). See table 1 for sample characteristics.

Table 1.

Sample characteristics

Sample characteristics	Wave 1 (2011) N=70 (%)	Wave 2 (2012) N=64 (%)
Role in clinic
Physician	41 (59)	34 (53)
Nurse	6 (9)	6 (9)
Medical Assistant	6 (9)	8 (13)
Front desk	9 (13)	11 (17)
Other	5 (7)	5 (8)
Time in clinic
Full time	34 (49)	37 (58)
Part time	28 (40)	25 (39)
Physician respondents
Faculty	23 (33)	23 (36)
Resident	17 (24)	11 (17)
Physician primary specialty
Pediatrics	32 (46)	28 (44)
Internal Medicine/Pediatrics	8 (11)	7 (11)
Other	8 (11)	6 (9)
Average number of children seen per week
1–25	25 (36)	20 (31)
26–50	12 (17)	9 (14)
51–75	6 (9)	10 (16)
76–100	3 (4)	4 (6)
>100	8 (11)	1 (2)

General satisfaction survey data suggest that CHICA users initially held mixed feelings about CHICA, but in just 12 months, users reported increased satisfaction with various aspects of the system. Users felt that CHICA sometimes reminded them of things that might otherwise have been forgotten (54% in 2011 vs 72% in 2012); made documentation easier (46% in 2011 vs 61% in 2012); uncovered issues with patients that otherwise would not have been found (44% in 2011 vs 63% in 2012); felt the reminders were consistent with recommended practices (57% in 2011 vs 72% in 2012); and that the CHICA handouts were useful (66% in 2011 vs 83% in 2012).

There were other aspects of CHICA that users were less satisfied with initially, including: perceptions that the system slowed down the clinic workflow (51% in 2011 vs 47% in 2012); perceptions that the system often made mistakes (43% in 2011 vs 31% in 2012); perceptions that the system had too many technical problems (46% in 2011 vs 21% in 2012); and disagreement over advice CHICA printed (29% in 2011 vs 6% in 2012). Between wave 1 and wave 2, mean satisfaction ratings for CHICA showed increased acceptability of the system (see table 2).

Table 2.

Comparison of mean Child Health Improvement through Computer Automation (CHICA) satisfaction ratings on 5-point Likert scale from 2011 to 2012

Survey item	2011 mean	2012 mean	p Value
CHICA sometimes reminds me of things I otherwise would have forgotten	2.5	2.1	0.06
CHICA makes documentation easier	2.7*	2.3*	0.03*
CHICA has uncovered issues with patients that I might not otherwise have found out about	2.7*	2.3*	0.01*
The handouts CHICA produces are useful	2.2*	1.8*	0.03*
The reminders CHICA produces are consistent with recommended practices	2.4	2.1	0.10
Technical support for CHICA is very good	2.7*	3.3*	0.01*
I rarely if ever use CHICA	4.1*	4.4*	0.05*
CHICA tends to slow down the clinic	2.6	2.8	0.20
CHICA often makes mistakes	2.6*	3.1*	0.03*
I would rather not use CHICA	3.1*	3.9*	0.01*
CHICA has too many technical problems	2.7*	3.3*	0.01*
I often disagree with the advice CHICA gives	3.1*	3.5*	0.01*
CHICA makes lots of errors	3.2	3.5	0.17

5-point Likert scale responses (1=strongly agree, 3=neutral, 5=strongly disagree).

*Differences between two independent samples were clinically significant at p≤0.05 by the Wilcoxon rank-sum (Mann–Whitney) test.

Free text comments reflected various themes that supported these ratings and are explained in more detail in the following paragraphs.

CHICA provides information for patient care

The survey reflected recognition of CHICA's primary goal of capturing and displaying patient information that would be helpful at the point of care in a number of positive comments in both waves. Some of these came from nurses, as well as physicians: ‘factual data is organized about the child on one page’, ‘[It is] able to anticipate the evaluation by physician due to asterisked areas’, ‘questions about child health are initiated by questionnaire’, and ‘it picks up things I may forget’. In addition, users felt the system ‘assists the doctors with capturing routine information that needs to be done on every patient per regulatory requirements’.

CHICA calculates values and customizes alerts

The most commonly cited positive aspect of CHICA was its ability to perform calculations related to growth and vital sign information required for each medical encounter. CHICA has the ability to calculate height and weight percentiles and body mass index: ‘[I] like that the percentile of the patient is in with weight, height and calculates body mass index’ and ‘[that] questions pertain to patient age and sex’.

Advantages to computer printed and scanned materials

Users recognized advantages to printed and scanned materials. Once CHICA forms are scanned, TIFF images of them are archived and can be brought up through the results viewing system of the electronic medical record: ‘I like that I can pull up the encounter on my computer. No need for [pulling the paper] chart’. Other comments indicated an appreciation that computer-printed information is clearer than if it were handwritten. Users also commented on liking the automation process of the system.

Along with the positive comments about CHICA's usefulness and its ability to facilitate daily workflow, there were other comments and themes that reflected ongoing criticism of the system and areas needing refinement.

CHICA was too early in development

In the first wave, there were several comments that CHICA had potential and that many of the concerns and complaints may be related to the newness of CHICA. There was acknowledgment that perhaps clinicians were not used to it yet. Comments included, ‘Give CHICA a little bit more time’ and ‘change—the process of getting everyone on board to where it feels like it is running smoothly’.

Critiques of format of CHICA forms

The most prominent theme of all comments related to specific critiques of the format of the CHICA PWS form. Some of these comments were readily addressed by the software technical team (such as addition of the patient's address and phone number on the form, or changing the units of the height and weight measurements). However, some were not easily remedied because changes would render the system non-functional—for example, asking that the check box responses to the PWS prompts be removed. Some comments reflected that users did not always understand how CHICA captured data. For example, there was a concern that families needed to fill in the circles on the PSF completely, but this is not the case. Commonly, users felt the space available on the form for free-text notes was too limited. Interestingly, even though there was space for more notes on the back of the PWS, user comments made it clear that turning the paper over was considered a nuisance.

CHICA slows check in, wastes time and money

Some critical opinions of the CHICA system were that it wasted time and money. Although the CHICA team conducted informal time-flow studies in one of the clinics showing that CHICA did not create significant delays, this perception persisted well into the second wave of surveys. Some users specifically pointed to the experience of burden during key aspects of clinical workflow: ‘it slows the process of checking in and checking out patients’, and ‘there are a lot of words [on the forms] and it takes time to read questions to make sure [I] don't miss important points on the form…it is hard to work it into the normal flow of the exam’. One comment made in the second wave acknowledged improved efficiency, but also the need for continual improvement: ‘Though wait times have been improved [of printing of the scannable paper forms], there could still be faster ways developed. I believe some of the support staff does not understand how waiting (even 1 minute) seriously affects the flow of the clinic’.

Problems with prompts

A few of the comments related to the prompts CHICA prints on the PWS. Some users wanted a checkbox choice such as ‘already done’ or a blank one to write in a response. Some felt that the prompts, though based on authoritative American Academy of Pediatrics recommendations, were not appropriate for the patient population. There were mixed requests for either more or less prompts for specific content areas (developmental milestones, school history).

Wording of questions and prompts

There were a few concerns about the specific wording for prompts and pre-screener questions. One user raised concerns that the prompts did not fit into the natural flow of the encounter, another raised concerns that the parents might not understand the questions. Some felt certain prompts were redundant (eg, dental).

Problems with scanning and paper interface

While users clearly saw benefits to CHICA's innovative use of scannable forms, there were recognized liabilities to a paper interface that requires printing and scanning. The scanning process takes time and effort by the staff, and sometimes fails: ‘Sometimes it does not read the scanning paper and it gives errors. I have scanned at least four times for the same patient’. Sometimes if physicians do not fill out the forms correctly, there is additional effort to correct them.²⁶

CHICA isn't in Spanish or other languages

At the time of CHICA's initial release in 2004, the parent questions were written only in English. There were initially concerns that Spanish-speaking parents would not be able to use CHICA: ‘I have a lot of bilingual patients that I don't think understand the questionnaire’. Once Spanish translations became available, one user was concerned that patients speaking other languages were not accommodated.

Too many/too few handouts

CHICA is designed to print ‘just in time’ handouts (JIT) that the physician might need based on the parent responses to the PSF. JITs are designed either for the physician to aid in decision-making or for the family with phone numbers for community resources. A JIT may also be a standardized screening instrument such as a Vanderbilt Attention Deficit Hyperactivity Disorder (ADHD) Rating scale.²⁷ These are printed before the encounter so the physician will have the JITs in hand should they be necessary. However, some users felt the JITs were not always needed: ‘CHICA unnecessarily prints forms/papers which is a waste—it should only be printed if asked to’. Nonetheless, physicians also asked for other handouts, depending on the topic they were most interested in (eg, obesity).

Issues with logic

As CHICA matured, physicians began to recognize that CHICA depends on a sequence of logical steps to link parent and physician responses on the forms to alerts and reminders at subsequent visits. Once this was understood, they commented on the logic choices such as using very sensitive but not specific questions about topics (eg, asthma) to generate alerts, re-asking questions over time, or how competing prompts for the same patient have been prioritized. One user commented in the second wave, ‘Does not always triage well which issue to put on the main page [PWS] if the [PSF] questionnaire uncovers multiple red flags’. Another user noticed that despite checking the box that the patient does not have asthma, CHICA ‘seems to spit out the same incorrect handouts—such as an asthma action plan’.

Involvement of clinical team in design

One of the more recent suggestions submitted by users is the need to engage the practices more actively in decisions about CHICA's design and as new modules were being built into the system. The concern that the relationship between the CHICA development team and the clinics is ‘top down’ reflected a perception that decisions are made about CHICA's design without adequate input from the clinics. While CHICA was largely designed for clinical use, CHICA's development group also utilizes the system to conduct research. Some of this research includes analysis of secondary data or implementing new rules and testing the effectiveness of CHICA modules to improve the quality of care for certain pediatric health conditions, such as ADHD, smoking cessation, maternal depression screening, and autism screening.^28–30 Some of the users requested a more formal method for learning about publications and that ‘these publications be shared with the group once a year [along with] review of the most practice-oriented improvements that can be garnered from the studies’.

Physicians want access to data

As physicians came to understand that the information captured by CHICA was stored in a database, they became interested in having access to the data for purposes of documenting their own quality of care for maintenance of certification (MOC) or earning salary bonuses.

Discussion

The practice of primary care pediatrics is challenging, given the array of topics to be discussed, visit time constraints, and the increasing prevalence of complex and time-consuming issues that pediatric providers are encountering in outpatient practice. Given the push to deliver high quality, evidence-based care that follows recommended practice guidelines, HIT has become increasingly integrated into the process of healthcare delivery. At our institution, we have implemented a novel CDSS integrated with an EHR to improve the detection and care of a variety of pediatric conditions within the constraints of busy outpatient pediatric practice. This study highlighted the growing acceptability of the CHICA system and the ongoing challenges or ‘growing pains’ that are encountered whenever a large-scale system change is implemented. Study findings are best understood using the diffusion of innovation framework.³¹ It is striking that repeated evaluations of the system over the course of a year showed both evolving user understanding of the system and its potential and more sophisticated critiques of its performance. Users’ comments helped to illustrate the process by which innovations in practice can be viewed as too complex at first; however with time and exposure, users came to value the relative advantages afforded by CHICA through its ability to prioritize meaningful patient care issues in practice.

To date, there have been limited studies examining the opinions and attitudes of users of pediatric CDSSs and other forms of HIT, even though these individual-level factors influence whether adoption of HIT takes place in practice.¹⁶ As demands on outpatient pediatric providers grow, the expanding role of HIT in facilitating healthcare delivery will only continue to increase. While our study focuses on a specific CDSS, like prior studies,^32–34 it highlights the critical need to take into account the perceptions of all users of any HIT application and to involve key stakeholders early in the process of its development to facilitate the diffusion of HIT innovation into practice.

The use of satisfaction surveys is but one method of eliciting users’ perceptions and measuring its adoption in ‘real world’ practice. Comments provided by users have led to continual refinement of the existing system and the development of new modules and other technical enhancements. For example, workflow concerns and the need to scan the PSF and PWS in a busy clinic are valid. The CHICA development team is in the process of transitioning to an electronic format of PSF and PWS forms such that it will eliminate the need to scan these forms. It is possible that once this transition is complete, it will improve the perception of burden on clinic workflow. A second example is that CHICA is now programmed to generate a color-coded growth chart based on the child's sex with growth data points pre-plotted for the clinicians’ quick reference. One last example is that as the MOC process relies on showing quality improvement in standards of care, it became clear that CHICA data could be used for this purpose. The CHICA development team has, in fact, applied for MOC credit for several of the activities that are automated by the CHICA system.

The initial deployment of CHICA V.1.0 and the eventual transition to CHICA V.2.0 was aided by ongoing enthusiasm for its use by hospital administration and clinical staff. CHICA also has a strong technical support team that includes two pediatric health informaticians, three software engineers, and two clinic technical liaisons who are available by pager and often interface daily with the clinical staff. In addition, the CHICA team has instituted quarterly CHICA Users Group (CHUG) meetings, attended by the technical team and clinic personnel, which provide a forum for ongoing communication of any issues specific to each clinic. At the request of CHICA users, the formation of a paid advisory panel of physicians representing each clinic using CHICA was done. This group meets monthly with one of the health informaticians. Moreover, the development team reviews weekly user reports to assess scanning rates of both the PSF and PWS, which is integral to updating CHICA's database and the EHR. There is a CHICA listserv that sends out periodic information on current and future system updates. This ongoing and dynamic process was critical to identify potential disruptions in workflow as early as possible whenever new rules were introduced or system changes as suggested by its users were implemented.

Certainly, technological innovations are not a panacea. Previous studies have examined barriers to successful adoption of HIT in practice. These include ensuring HIT does not hinder the usual flow of the clinic and users do not perceive additional burden, anticipating user needs, and delivering information in real time at the point of care, and monitoring and maintaining the system.³⁵ While the CHICA system is not ‘perfect’, the development team has strived to make CHICA a practical interface that streamlines clinical processes to fit into the workflow. The use of a paper interface may have helped facilitate early uptake of the system since the format is similar to the traditional paper chart. However, unlike a paper chart, checkboxes must be marked in order for CHICA to capture essential encounter data and integrate it into the larger EHR. The CHICA team is undertaking a separate study of the associated human factors that impede or facilitate healthcare decision-making at the point of care to glean a deeper understanding of how to improve on the functionality of CHICA.

The results in the present study may be limited in several ways. Our findings are related to a specific form of HIT, a CDSS within one clinical healthcare system. There are a variety of other HIT applications designed to facilitate healthcare delivery, such as telemedicine, patient portals, and electronic prescribing. Even though the CHICA platform is built on open source software and may be linked to any EHR, clinics outside our home institution have yet to adopt this technology. However, lessons learned from our study, like others, can apply to the process of implementing alternate forms of HIT. While our sample consisted of pediatric users from four community clinic sites within one institution, the total sample at each wave was not small and we obtained opinions from all eligible users in 2011 and all but two in the subsequent year.

Conclusion

The use of HIT to facilitate pediatric outpatient practice is likely to continue to increase in the coming years. We have highlighted the evolution of a CDSS within our healthcare organization and the growing acceptability and satisfaction toward the system among its users. As with any major systems change, challenges are expected, however, with time, appreciation of the system grew and users began offering suggestions that reflected a greater understanding of its logic and functionality. Stakeholders interested in adopting HIT to improve the process of healthcare delivery within their own practices could take lessons learned from our specific experiences with the CHICA system to optimize implementation of HIT into ‘real world’ practice in other healthcare organizations.