Adolescent Satisfaction with Computer-Assisted Behavioural Risk Screening in Primary Care

Abstract

Background

This study measures patient satisfaction with a computerised mental health and risk-behaviour screening tool and predictors of satisfaction.

Method

Youth, aged 11–20, were recruited to use a laptop-based screening system in nine primary care clinics. The study assessed correlations between satisfaction with the system and selected predictors.

Results

Most users were satisfied with their experience. Multivariate logistic regression found perceived ease of use, perceived usefulness, and trust to be significantly associated with high satisfaction. Satisfaction was not related to computer experience or risk behaviour status.

Conclusions

Adolescent patients, even those at risk, accept computer-assisted screening in primary care.

Key Practitioner Message

Screening adolescents for behavioural and mental health issues in the primary care setting is time consuming. Computer assisted screening has the potential to improve screening. This study finds that youth will accept computerised screening if the system is perceived to be useful and easy to use.

Introduction

Because adolescent morbidity and mortality are largely associated with high-risk behaviours, preventive counselling is critical for adolescents visiting their primary care clinicians. Current guidelines, supported by the American Academy of Pediatrics, recommend that well-teen visits include screening or discussion of many conditions and risk factors including health and safety issues, injury and violence prevention, mental health, nutrition, oral health, sexuality, and substance use (Green & Palfrey, 2002). Meeting this standard is a challenge, given the high patient volume and high productivity expectations faced by today’s paediatricians. The median length of a paediatric visit in the US is 14 minutes, which must include addressing patient and family concerns, history, and physical examination in addition to the extensive anticipatory guidance recommended (Woodwell & Cherry, 2004).

Some, but not many, primary care clinicians ask teens to complete paper and pencil self-report instruments to screen for behavioural risk factors (Gardner et al., 2003). However, scoring, transcribing, filing and storing paper and pencil screening instruments is resource intensive. Recognising that US adolescents’ use of computers to search for health information is growing rapidly (Lenhart, Madden, & Hiltin, 2005), computerised screening in the waiting room may be a comfortable and even fun way to engage youths in their health care, and may overcome some of their concerns about talking directly to clinicians about personal behaviours. In fact, many studies have found that adults and adolescents are more likely to acknowledge sensitive personal information including domestic violence, sexual abuse, and alcoholism to computers than in personal interviews (Bagley & Genuis, 1991; Lucas et al., 1977; Rhodes et al., 2002).

Recent research shows that most patients understand and accept computer-assisted self-interviewing technology (Porter et al., 2004; Rhodes et al., 2001; Sciamanna, Novak, & Marcus, 2005; van den Brink et al., 2005; Williams, Templin, & Mosley-Williams, 2004; Williams, Boyles, & Johnson, 1995). Psychosocial specific computer-assisted interviews, such as the computerised diagnostic interview survey for children, the Dominic Interactive, and TeenScreen have been used in settings including schools, emergency departments, specialty care sites, and research environments (Cawthorpe, 2001; Columbia University, 2005; Smolla et al., 2004). While a few studies have assessed the use of computer-assisted self-interviewing by youth in a paediatric care setting with promising results (Chisolm, Young, & McAlearney, 2005; Paperny, 1997; Paperny et al., 1990; Zuckerbot et al., 2007), more information is needed on how specific adolescent attitudes and expectancies affect satisfaction and acceptance.

This study examined patient acceptance of the Health eTouch System, a web-enabled, tablet-style laptop personal computer on which patients answer questions about injury risk, depression, and substance use. The goal of the Health eTouch system is to increase the identification of high-risk behaviour and to help youth access behavioural health services. This analysis studied the factors associated with Health eTouch user satisfaction to guide the development and implementation of future applications.

Method

Setting, system, and procedure

Setting

Nationwide Children’s Hospital implemented the Trial of Automated Risk Assessments of Adolescents (TARAA) project beginning June 2005. As part of a larger study, TARAA tested a web-based technology to facilitate screening for high-risk behaviours in youth aged 11 to 20 years in nine paediatric primary care settings serving a predominantly low-income population.

System

Each clinic was equipped with secure wireless network access and wireless laptop computers that presented web-based screening questions to youth while they were waiting to be seen by the clinician. The Health eTouch computer, depicted in Figure 1, has a 10 inch display panel, weighs less than three pounds, and has approximately 3.5 hours of battery life. The computer is charged in its docking station between patients.

Figure 1.

Teen uses a stylus to select a response to an item on the Health eTouch tablet-style personal computer

Procedure

Potentially eligible adolescents were invited to participate in the study by clinic registration staff or research staff while in the primary care waiting room. For youth under age 18, consent was obtained from the parent or guardian accompanying the patient and assent was obtained from the youth. Youth aged 18 to 20 provided their own consent. The consent process informed patients that all information entered on the computer would be made available to both their clinician and the research staff.

Once consent was obtained, the adolescents completed the Health eTouch screener in the waiting room or in other areas of the clinic if they were called back to the clinical area prior to completion. After the patient completed the screening items, the system generated a ‘lab report’ that clearly stated whether the patient screened positive for injury risk, depression risk, and/or substance use risk. If the screening was positive for any risk area, the report provided the patient’s responses to individual items in that section. For the trial component of the study, the nine clinics were randomised into one of two conditions: screening results printed in the clinic immediately or screening reports printed after a 3-working-day delay. The computer system satisfaction questions were asked before the patient saw the clinician or completed his/her office visit, thus randomisation to ‘immediate’ or ‘delayed’ feedback was not considered in the analysis. The Nationwide Children’s Hospital Institutional Review Board reviewed and approved the study.

Instrumentation

Items for the risk assessment were drawn from existing validated measures. Injury risk was assessed using age-group appropriate items from the Youth Risk Behavior Survey (YRBS) (Brener et al., 1995) assessing helmet use, seat belt use, physical fighting, carrying a weapon, and driving or riding with someone in a car under the influence of alcohol. Depression was assessed using the Centers for Epidemiologic Studies Depression Scale for Children (CES-DC), a 20-item depression screening tool (Faulstich et al., 1986; Fendrich, Weissman, & Warner, 1990). The assessment also included a single question on serious suicidal ideation from the Patient Health Questionnaire for Adolescents (Johnson et al., 2002). Substance use was measured using items drawn from the Comprehensive Addiction Severity Index for Adolescents (CASI-A) (Meyers et al., 1995). All youth completing Health eTouch were presented with the CASI-A items measuring use of tobacco, alcohol, marijuana, and inhalants. If the adolescent acknowledged use, he or she was asked about extent of use. All youth who reported use of alcohol, marijuana, or inhalants were also asked questions about use and, if positive, extent of use of other illicit substances (e.g. hallucinogens, cocaine, heroin). These youth also received questions about the consequences of their substance use (Meyers et al., 1995) and their readiness for change (Rollnick et al., 1992).

Patients were defined as having a positive drug screen if they acknowledged use of alcohol, marijuana, or illegal inhalants. For this analysis, patients were defined as depressed if they scored 25 or higher on the CES-DC. This cut-point, which is higher than the generally used 16, has been used in other primary care screening studies to limit stress to the system associated with low level depression common in adolescents (Walker, Oakley, & Townsend, 2000). Patients who acknowledged seriously considering taking their own lives in the past 30 days were considered positive for serious suicidal thoughts.

As the survey branched to different items based on the user’s age and reported behaviours, the actual number of items completed by users ranged from 45 to 101. Youths completed Health eTouch in an average of 15.1 minutes (SD = 9.6 minutes), but this includes time required for the informed consent and the questions about satisfaction that were not part of the clinical screen. Exact times for the completion of the screening questions are not available but, based on observation, most youths completed the screening items in 6 minutes or less.

Satisfaction measurement

Items measuring satisfaction with the use experience and attitudes toward the system were presented to the patients on the Health eTouch computer after they had completed all of the screening items. Satisfaction, the outcome variable of interest, was measured on a 4-point scale with values of ‘Very dissatisfied’, ‘Somewhat dissatisfied’, ‘Somewhat satisfied’, and ‘Very satisfied’.

User perceptions and characteristics

This analysis measured the relationship between patient characteristics and satisfaction. Two user attitude domains that have been consistently associated with acceptance of health care technology were studied: perceived ease of use and perceived usefulness (Chen, 2001; Chismar & Wiley-Patton, 2002; Venkatesh et al., 2003; Wilson & Lankton, 2004). Additionally, trust was studied as a correlate of satisfaction because of the socially undesirable and, sometimes, illegal nature of the behaviours screened in this study. Perceived ease of use was measured using two items: perceived ease of using the system and perceived ease of learning the system. Perceived usefulness was measured by two items that asked whether youths thought the system would help them communicate with their clinicians and get better care. Finally, trust was measured with items that asked whether youths believed the data would remain private, that the data would be used only for health care, and whether youths were comfortable with having their health data stored on a computer. All of these items were measured on 5-point Likert scales ranging from ‘Strongly agree’ to ‘Strongly disagree’. This study hypothesized that perceived ease of use, perceived usefulness, and trust will each be positively associated with satisfaction. The relationships between satisfaction and patient age, sex, race, payor category, degree of computer experience, and risk screening results were also assessed.

Analysis

Univariate frequencies for all descriptive variables are presented to describe the study population. Preliminary analysis of the satisfaction variables found low response frequencies for dissatisfaction; therefore, the lower three response options (‘Very dissatisfied’, ‘Somewhat dissatisfied’, and ‘Somewhat satisfied’) were collapsed into the category ‘Moderate/Poor Satisfaction’. All analyses compare ‘High Satisfaction’, those endorsing ‘Very Satisfied’, to ‘Moderate/Poor Satisfaction’. This focused analyses on identifying factors related to maximal satisfaction. Bivariate relationships between satisfaction and demographic variables were tested using chi-squared tests. Summary measures of perceived ease of use, perceived usefulness, and trust were calculated as the mean of the variables in each domain, each of which was measured on a 1 to 5 scale, with 1 indicating strongly disagree and 5 indicating strongly agree. The Wilcoxon non-parametric test for comparison of means was used to test the relationship between these variables and satisfaction because of non-normal distributions. Finally, logistic regression, with forward selection, was used to explore the multivariate relationship between high satisfaction and variables found to be significant in bivariate testing. For all tests, p-values < .05 were considered statistically significant.

Results

Study population description

The study population included 1021 adolescent, first-time users of the system between 1 June 2005 and 7 March 2006. Ninety-two responses were excluded because the respondents were not able to complete all of the screening questions before seeing the clinician, leaving 929 cases for analysis. Failure to complete the tool was most often attributed to short waiting room time or technical problems. Analysis comparing those 92 cases who started the computer questionnaire but did not finish and the 929 who completed the tool found no significant difference on gender, race, or payor status. User age was significantly associated with inclusion. Adolescents aged 18–20 were most likely to be excluded for failure to complete the tool (18%) followed by children in the youngest user category, ages 11–13 (11%).

Study population characteristics are presented in Table 1. Respondents were predominantly young adolescents, aged 11–15 (77%) and were slightly more likely to be girls (54% female). The racial distribution was 55% Black, 37% White, and 8% other race. Most respondents (75%) were covered by Medicaid, the publicly funded insurance program for eligible low income Americans. Only 3% of the respondents described themselves as having no computer experience. The screening system identified 35% of youth as positive for depressions, substance use, or suicidal thoughts.

Table 1.

Satisfaction with use of the Health eTouch system by user characteristics

Characteristic	N† (%) ‡	% Moderate/poor satisfaction	% High satisfaction	X2
Overall	929	40.4	59.6
Gender
Male	425 (45.7)	45.4	54.8	7.53*
Female	504 (54.3)	36.3	63.7
Age group
11–13	444 (47.8)	40.3	59.7	8.31*
14–15	268 (28.8)	45.1	54.9
16–17	181 (19.5)	37.0	63.0
18–20	36 (3.9)	22.2	77.8
Race category
Black	343 (55.2)	40.8	59.2	2.99
White	513 (36.9)	38.8	61.2
Other	73 (7.9)	49.3	50.7
Primary payor
Medicaid	700 (75.3)	40.9	59.1	4.77
Commercial	148 (15.9)	33.8	66.2
Other	81 (8.7)	48.1	51.8
Computer experience
No	29 (3.1)	44.8	55.2	0.15
A little	303 (32.7)	44.5	55.4
A lot	596 (64.2)	38.1	61.9
Substance use
Negative	845 (91.0)	40.0	60.0	0.52
Positive	84 (9.0)	44.0	56.0
Depression
Negative	697 (75.0)	40.3	59.7	<0.01
Positive	232 (25.0)	40.5	59.5
Suicidal thoughts
Negative	803 (86.4)	39.9	60.1	0.65
Positive	126 (13.6)	43.7	56.3

^*Statistically significant (p < .05).

^†Some categories do not sum to 929 due to missing responses. Percentages are based on non-missing responses.

^‡Some columns do not total to 100% due to rounding error.

User satisfaction

Overall, patients evaluated the Health eTouch System favourably, with 60% describing themselves as highly satisfied. An additional 35% were somewhat satisfied. Table 1 presents degree of satisfaction by demographic characteristics. Both user age and gender were associated with satisfaction. In girls, the rate of high satisfaction was 64% compared to 55% in boys. Highest satisfaction was found in the oldest age group. There was no significant relationship between computer experience or screening result and satisfaction.

Correlates of satisfaction

Perceived ease of use, perceived usefulness, and trust were each significantly associated with strength of user satisfaction in bivariate analysis. Among highly satisfied users, the mean perceived ease of use score was 4.59 compared to 4.09 in less satisfied users. Perceived usefulness had an even stronger association with satisfaction. Highly satisfied users had a mean perceived usefulness of 4.35 compared to 3.66 in those less satisfied. Youths with high satisfaction had a mean trust score of 4.43 compared to 3.82 in those less satisfied.

Table 2 presents a logistic regression model built using forward selection of all items found significant in the bivariate analysis. Age group did not enter the final model. Perceived ease of use (Adjusted Odds Ratio = 1.48, 95%; Confidence Interval 1.18–1.86), perceived usefulness (Adj. OR = 2.18, 95%; CI 1.63–2.62), and trust (Adj. OR = 3.31, 95%; CI 2.00–3.32) were associated with high satisfaction.

Table 2.

Logistic regression model of for user satisfaction

Effect	β	Std. error	Pr > Chisq	OR (95% CI)
Female*	0.31	0.16	0.05	1.36 (1.00–1.85)
Perceived ease of use*	0.39	0.12	<0.01	1.48 (1.18–1.86)
Perceived usefulness*	0.78	0.12	<0.01	2.18 (1.72–2.75)
Trust*	1.20	0.16	<0.01	3.31 (2.40–4.56)

^*Statistically significant (p < .05)

Conclusions

The promise of computer-assisted self-interviewing lies in the ability of the technology to improve both the quality and the quantity of health and behaviour history information available to physicians and patients during the clinical encounter (Agency for Healthcare Research and Quality, 2002). Using the Health eTouch system, 35% of primary care youth screened positive for substance use, depression, or suicidal thoughts. As with all screening, this group probably includes some false positives. Physicians, who are given the information from the system, remain the final decision-makers on whether the screening result is valid and whether follow-up care is needed.

Although this enhanced case recognition has the potential to improve the healthcare and health status of youth, it must be noted that it also has the potential to increase demand in an already strained US health system. In fact, as this study began, clinicians were added to the system’s behavioural health clinic to meet projected increases in referrals. Identification of new cases can only improve care if system-level changes are enacted, including increased availability of physicians and ancillary staff (i.e. nurse practitioners, social workers, case managers), expanded community and school-based treatment programs, and financial incentives that reward not just number of patients seen but the comprehensiveness of their treatment.

This study found that computer-assisted screening was well accepted in an urban, predominately low income, paediatric primary care setting. Satisfaction was generally high and was not associated with sociodemographic factors including race, payor status, or computer experience. Girls and older youths were more satisfied with the system, perhaps because they had slightly more positive views of the system’s ease of use or because of unmeasured differences in literacy. The age association did not remain significant after adjusting for perceptions of the technology. Given the socially undesirable nature of some of the risk behaviours screened for by Health eTouch, there was a concern that youths with such risk factors would have negative reactions to being asked about them. These data showed that concern to be unfounded. Users who acknowledged substance use, users with depression, and users with suicidal ideation were no less satisfied with the system than their lower risk counterparts.

The finding of high satisfaction in this study is important because computer-assisted screening may increase problem identification and treatment in two ways. First, the personal distance created by answering questions on a computer rather than face-to-face may increase the likelihood of reporting problem behaviours. The effect has been shown previously in computer based screening for domestic violence (Rhodes et al., 2002). Second, using computer-assisted screening in the waiting room allows for screening to be done during ‘dead time’, thus freeing time within the encounter for the clinician to focus on assessing the severity and response to reported behaviours. In short, computer-assisted self-interviewing allows clinicians to spend more time reacting to more complete information and thus allowing for better care.

Data also indicated that, even in an urban setting with disproportionately high concentrations of low-income and minority patients, where a ‘digital-divide’ (Hsu et al., 2004) might be expected, diverse users felt comfortable with the system. This may be because the system was designed to require no previous experience with a computer. The system required no keyboarding skills and relied on closed-ended questions, which have been shown to maximise data collection (Porter & Kohane, 2001).

Results of this study validate the importance of factors from traditional technology acceptance models as predictors of adolescent attitudes toward computer-assisted screening. Specifically, perceived ease of use and perceived usefulness were significantly associated with satisfaction. These findings are consistent with findings across a diverse range of technologies and populations (Chen, 2001; Featherman & Pavlou, 2003; Shin, 2003; Vijayasarathy, 2004). Because the application of this technology has the added complexity of asking questions about personal and even illegal activities, trust was also hypothesized to be a significant predictor of satisfaction and this hypothesis was supported by the study data. The finding that trust was significantly associated with satisfaction indicates that the use of computer-assisted screening does not relieve the health care provider of the responsibility to build trust with the patient. Instead, this study suggests that satisfaction with an information technology system, and possibly willingness to provide honest responses, is contingent on the belief that personal information will be treated with care and will be used only for the purpose for which it was provided.

Limitations

This study was conducted within a single health care system in a large metropolitan area and may not be generalisable to other populations. However, in studying an urban, low income population, focus was placed on a group at high risk of relevant problems who were likely to have computer literacy issues, concerns about the health care system, and issues of trust. The fact that so many youths completed the assessments and were satisfied with their experience suggests that computer-assisted screening holds promise in many settings. For this study, satisfaction was assessed on the computer table immediately after the risk items were completed. Assessment after the patient saw the physician and received treatment or referral for identified concerns may have yielded different results.

Another limitation was that not all patients seen in the study clinics used the system. Reasons for not being recruited into the study included refusal to participate, lack of a guardian to sign consent forms, insufficient wait time to complete the appraisal, acute symptoms, inability to read English at 4th grade level and, most commonly, limited clinic resources available to focus on study recruitment. Outright refusal to participate was low but it is possible that potential users with more negative attitudes toward information technology might have been less likely to consent to the study. These results do, however, provide strong insights into the perceptions of those willing to try the system.

Given that, as long a system is optional, some individuals will choose not to become users, the goal in design and implementation must be to maximise the experience of those who choose to use the system. The study found that 9% (92 of 1021) of those who started the tool did not complete the satisfaction questions, which were placed at the end of the questionnaire. While formal data were not kept, anecdotally the most common reason for stopping before completing those items was the beginning of the clinical visit. Youth in the oldest age group were most likely to stop before reaching the satisfaction items, followed by those in the youngest group. Higher non-completion rates in the youngest group may suggest readability issues that require further assessment.

A third limitation is that all of the data collected were self-reported by the adolescents who completed the screener. Biases in reporting of satisfaction, perceptions of the system, or risk behaviours could impact estimates of behaviours and the relationship between satisfaction and behaviours. Finally, because this was a cross-sectional study, causality could not be tested. It cannot be determined whether high perceived usefulness, ease, and trust caused high satisfaction. In fact, it is possible that high satisfaction with use led to positive expectancies because of a halo effect, meaning that people who had an overall favourable opinion of the system were likely to give positive responses for all questions about the system.

In summary, early recognition and response to high-risk behaviours in adolescents can help avert long-term negative outcomes. Therefore, health care providers must continue to develop mechanisms to improve the accuracy and efficiency of screening. This study shows that a computer-based screening system used in the primary care waiting room was well accepted in an urban clinic system. Even youths with acknowledged high-risk behaviours and with limited computer experience were generally satisfied with their use experience. Additional research is underway to determine the impact of the Health eTouch system on physician identification of youths at high risk and referrals for treatment and to measure the agreement between Health eTouch identified problems and physician identified problems. Such research will be important in documenting the value of further expansion of computer-assisted screening. Future research will also assess the impact of screening on health services use and associated cost effectiveness issues.