Lessons Learned: The Effect of Prior Technology Use On Web-Based Interventions

Abstract

This study examined the role of regular prior technology use in treatment response to an online family problem-solving (OFPS) intervention and an Internet resource intervention (IRI) for pediatric traumatic brain injury (TBI). Participants were 150 individuals in 40 families of children with TBI randomly assigned to OFPS intervention or an IRI. All families received free computers and Internet access to TBI resources. OFPS families received Web-based sessions and therapist-guided synchronous videoconferences focusing on problem solving, communication skills, and behavior management. All participants completed measures of depression, anxiety, and computer usage. OFPS participants rated treatment satisfaction, therapeutic alliance, and Web site and technology comfort. With the OFPS intervention, depression and anxiety improved significantly more among technology using parents (n = 14) than nontechnology users (n = 6). Technology users reported increasing comfort with technology over time, and this change was predictive of depression at followup. Satisfaction and ease-of-use ratings did not differ by technology usage. Lack of regular prior home computer usage and nonadherence were predictive of anxiety at followup. The IRI was not globally effective. However, controlling for prior depression, age, and technology at work, there was a significant effect of technology at home for depression. Families with technology experience at home (n = 11) reported significantly greater improvements in depression than families without prior technology experience at home (n = 8). Although Web-based OFPS was effective in improving caregiver functioning, individuals with limited computer experience may benefit less from an online intervention due to increased nonadherence.

INTRODUCTION

We have been involved in an ongoing research and development program to create an online family problem-solving (OFPS) intervention for families of children with TBI.^1,2,3,4 Just as psychologists and other health care providers must develop their own technical skills to harness these new technologies,⁵ they must also be cognizant of the technical skills of their patients and clients. Although most psychologists are sensitive to issues of education, ethnicity, gender, and class, they are less likely to consider technological sophistication as a moderating influence in research or therapy. Thus we set out to study the consequences of previous experience with computers in randomized clinical trials comparing two computer-based interventions for families of children with traumatic brain injury (TBI).

Despite documented caregiver burden and a demonstrated need for family-centered interventions that address pediatric TBI aftercare, empirically validated interventions are virtually nonexistent and community services are limited. Families often face barriers to treatment such as distance, limited availability of experienced providers, and the need to return for frequent rehabilitation treatment. Minorities and individuals with lower SES are disproportionately affected and are less likely to follow through with appointments.⁶ Online interventions may reduce these barriers and provide continuous access to treatment materials.

Given the multidimensional nature of family stress associated with developmental disabilities and chronic conditions and the range of factors influencing family adaptation, comprehensive approaches targeting multiple risks, moderators, and/or outcomes might have the greatest efficacy, particularly with families at heightened risk. Wade et al.⁷ tested communication and behavior management skills with families of children with moderate to severe TBI⁸ in a randomized clinical trial (RCT) comparing an Internet resources intervention (IRI) group to a group receiving an OFPS intervention.⁷ Families in both groups received a free computer, printer, high-speed Internet access, and our home page with links to brain injury Web sites and resources. Participants in the OFPS group also received an average of 10 Web-based sessions focused on brain injury education, behavior management, problem-solving and communication skills, and completed synchronous sessions with a therapist over the computer via video camera to review session content and implement the problem-solving process to address goals identified by the family.

Group differences in outcomes were assessed using ANCOVA, controlling for baseline levels on the outcome variable of interest. Consistent with our hypotheses, we found significantly greater improvement at followup in the OFPS group relative to the IRI group on measures of parental depression, anxiety, and global psychological distress.⁷ Moreover, examination of potential moderators of treatment efficacy revealed that families with varying levels of educational attainment, SES, and race were equally likely to benefit from this online treatment. Families were also positive in their responses about the helpfulness and ease of use of the Web site, suggesting that families from varying backgrounds could benefit from this type of online skill-building intervention.

Prior to our intervention, 35% of caregivers reported not using a home computer on a regular basis, of whom 77% did not own a home computer. Previous experience with computers has been linked to computer self-efficacy⁹ and lower levels of computer anxiety.¹⁰ Although recent studies suggest that computer self-efficacy beliefs may be more predictive of computer anxiety and anger than experience,¹¹ most of these studies have used undergraduate populations with few participants lacking home computer experience. Since we were bringing a computer into some families’ homes for the first time, we wanted to understand what effect, if any, lack of previous home computer experience had on parents’ comfort with and ability to benefit from the intervention. Thus, the goals of the current investigations were to examine the role of prior home computer use on the effectiveness of OFPS and to begin to identify the mechanisms through which prior computer use may impact treatment response. These findings are considered in the context of the potential costs and benefits of utilizing a Web-based intervention to provide treatment to underserved populations.

METHOD

Participants and design

Approval was obtained from the Institutional Review Board prior to study inception. Families with children ages 5 to 16 who sustained a moderate to severe TBI during the previous 24 months were recruited from the Trauma Registry of an urban children’s hospital and randomly assigned to receive either IRI or OFPS Internet resources regarding TBI. Using a computer program for random assignment, 87 individuals in 24 families were assigned to OFPS and 63 individuals in 20 families were assigned to the IRI group, with family being the unit of assignment and analysis.

Materials

All families received a new computer, printer, high-speed Internet access, and a home page providing access to brain injury Web sites and resources. Families in the OFPS group also received a Web camera. Families in both groups continued to receive any prior psychosocial treatments through out the intervention.

Procedure

The therapist met once with each OFPS family in their home to establish rapport, conduct a structured interview, and train OFPS participants in how to log on and navigate within the Web site. All subsequent sessions included a self-guided Web session and a subsequent one-to-one videoconference with the therapist (see Wade et al.³ for a complete description). The OFPS Web site had 14 separate sessions. Eight “core” sessions provided training in problem solving, communication, and antecedent behavior management skills for all enrolled families, while the remaining six sessions addressed content related to stressors or issues that affected some but not all families. Thus, the OFPS intervention can be thought of as standard problem solving plus training in other cognitive-behavioral skills relevant to coping with TBI.

IRI group

Families in the IRI group continued to receive any psychosocial care they were receiving prior to the intervention. IRI families received access to a home page of brain injury resources and links to Web sites with information and resources pertaining to TBI (identical to those given on the OFPS homepage), but they were not able to access OFPS session content. Web site usage was assessed in both groups via questionnaire at the end of the study.

Measures

Prior computer use

Parents completed 17-item self-report questionnaires pertaining to their prior computer use at home and at work. For each item (e.g., sent an e-mail message), parents indicated whether they had performed that action during the past week, past month, or more than a month/never. Responses were dichotomized such that parents who did not have a home computer or who indicated that they had not used the computer for any reason during the past week were identified as “nonusers.” Seventeen parents (8 OFPS and 9 IRI) were identified as nonusers on the basis of this criteria. Of these, 15 or 88% did not possess home computers prior to the intervention.

Parental depression and anxiety

We chose the Center for Epidemiologic Studies Depression Scale (CES-D) to measure depression because it is brief and has well-established psychometric properties.¹³ The CES-D contains 20 items that are rated on a 4-point scale ranging from rarely or none of the time (less than 1 day during the past week) to most or almost all of the time (5–7 days during the past week). Total scores range from 0 to 60, with higher scores corresponding to greater depression.¹²

The 10-item Anxiety Inventory (AI) has been widely used in psychological research in which repeated measures are taken, and limited time is available.¹³ Items pertain to the past week and include statements such as “I was jittery.” Response choices range from 0 (not at all) to 3 (very much), and total scores range from 0 to 30. Higher scores indicate greater anxiety.

Website Evaluation Questionnaire (WEQ)

The WEQ was adapted from the Website Evaluation Instrument.¹⁴ It contains a series of questions regarding the ease of use and helpfulness of the videoconferences and the components of the Web site rated on a 5-point Likert scale (1, not at all easy; 5, extremely easy; 1, not at all helpful; 5, extremely helpful). The sub-scales assessing ease of use of the Web site and videoconferences possessed high internal consistency (Cronbach’s α = 0.87 and 0.88 respectively), as did the subscales assessing helpfulness of the Web site and videoconferences (0.92 and 0.89 respectively).

Computer Equipment Comfort Rating (CECR)

The CECR was adapted from the Audiovisual Equipment Rating Scale,¹⁵ a measure of user perceptions of comfort and distractions associated with using audiovisual equipment. Items were modified to reflect the equipment and intervention design of the current project. The CECR consisted of 17 questions rated on a 5-point scale (1, not at all comfortable; 5, completely comfortable) regarding the user’s experiences utilizing the computer equipment during the online sessions. Cronbach’s alpha for the scale was 0.62.

Therapeutic Alliance Questionnaire

Parents completed a 12-item version of the Agnew Relationship Measure (ARM)¹⁶ to assess the quality of their relationship with the therapist. Items rate opinions about the therapist on a 7-point Likert scale (1, strongly disagree; 4, neutral; 7, strongly agree). Internal consistency for the 12 items was 0.84.

Missed sessions

To assess the relationship of prior computer usage to adherence, we created an index of the total number of missed sessions by summing the number of sessions missed because of cancellations and no shows. Technical problems were recorded and tallied to investigate group differences and impact on adherence.

Socioeconomic Status (SES)

The Hauser-Warren socioeconomic index was used to assess the parent’s occupational status. The Hauser-Warren index provides a measure of occupational SES derived from the education and income data from workers in the 1990 census and validated against the occupational prestige ratings from the 1989 General Social Survey.¹⁷ Higher scores correspond to greater SES. Parents were also asked about their educational attainment with choices ranging from less than 2 years of high school to a graduate degree.

RESULTS

Demographics

Forty-five families of children with moderate to severe TBI were randomly assigned to OFPS or IRI. OFPS attrition was 12% (n = 4; two were nontechnology users) versus 0% IRI; however, this difference was not statistically significant. The 17 girls and 28 boys with TBI were an average of 10.96 years of age (SD = 3.1) and were injured 13.73 months prior to baseline on average (SD = 7.10). Twenty-four percent of the children were African American, and 76% were Caucasian. SES and education level of primary caregivers varied substantially, with more than half having a high school education or less. Nearly two-thirds of families had home computers prior to enrollment; however, eight caregivers in the OFPS group and nine in the IRI group reported not using the computer regularly at home. The groups did not significantly differ with respect to prior computer usage, demographic or injury characteristics, or any of the parent outcome measures at baseline.

The relationship of technology use to treatment response

Twenty-one families participated in the OFPS intervention (one family was excluded from subsequent analyses because the child’s injury did not meet our severity criteria). ANCOVA, controlling for baseline scores on the dependent variable of interest, were used to examine the relationship of regular home computer use to treatment response. Consistent with our hypothesis, ANCOVA revealed significant group differences between technology users (n = 14) and nontechnology users (n = 6) in anxiety and depression at followup. After controlling for baseline levels of anxiety and depression, caregivers without prior technology use in the home reported significantly higher levels of anxiety at followup (M = 13.17, SD = 5.63) (F[1] = 9.72, p = 0.006) than did caregivers with prior technology use in the home (M = 7.57, SD = 3.75). Nontechnology users also reported significantly more depressive symptoms (M = 15.00, SD = 7.12) (F[1] = 7.04, p = 0.017) than technology users (M = 6.79, SD = 5.66). These findings suggest that primary caregivers who did not use a home computer regularly were less likely to benefit from the OFPS intervention

Unlike the OFPS intervention, the IRI did not significantly improve depression, F(1, 17) = 2.66, p = 0.12. Parental reports indicate that the IRI group had slightly, but not significantly, more experience with technology at work, and slightly, but not significantly, older children with TBI than the OFPS group. Because experience with technology at work and school could mitigate the consequences of experience with technology at home, we conducted an additional analysis controlling for prior depression, age, and technology at work. There was a significant effect of technology at home for depression t(22) = 2.24, p = 0.035, with families with prior technology experience at home (n = 11) reporting significantly greater improvements in depression than families without prior technology experience at home (n = 8). There was a nonsignificant trend in the same direction for anxiety t(22) = 1.01, p = 0.32. These data suggest that even with a less powerful intervention, the benefits of prior experience with technology are still evident.

The relationship of prior technology use to adherence

Both groups reported spending equivalent amounts of time on the Web site (nontechnology users, M = 2.33, SD = 1.03, versus technology users, M = 2.38, SD = 0.87; F[17] = 0.113, p = 0.82). Overall, 52% of sessions were completed at the first scheduled appointment (see Table 1). The number of sessions canceled due to technological problems did not significantly differ by group (nontechnology users, M = 1.50, SD = 1.52, versus technology users, M = 2.07, SD = 1.73; t[18] = 0.700, p = 0.65). Nontechnology users missed significantly more sessions over all (M = 16.33, SD = 11.29; t[18] = 2.43, p = 0.026) than did technology users (M = 7.64, SD = 5.06). Nontechnology users also “no showed” significantly more often (M = 6.67, SD = 6.31) than technology users (M = 1.78, SD = 2.63; t[18] = 2.49, p = 0.02). Using hierarchical regression analysis, we then examined the relationship of missed sessions to treatment response controlling for baseline symptom levels and prior technology use. The number of sessions rescheduled significantly predicted anxiety levels at followup (F change = 5.84 [1], p = 0.03), accounting for 15% of the variance. However, while nontechnology users also reported significantly more depressive symptoms (p = 0.017) than technology users, the number of sessions missed did not explain additional variance at follow-up (F change = 1.18 [1], p = 0.30). These findings suggest that nontechnology users were more likely to miss sessions, and in the case of anxiety, this nonadherence contributed to a poorer treatment response.

Table 1.

Adherence

	Prior technology use at home		p
Variable	Technology—No n = 6 Mean (SD)	Technology—Yes n = 14 Mean (SD)
Sessions
Scheduled	21.33 (4.37)	17.71 (4.25)	0.98
No-show	6.67 (6.31)	1.78 (2.64)	0.02
Canceled	3.0 (2.19)	4.07 (2.97)	0.24
Tech problems	1.5 (1.52)	2.07 (1.73)	0.65
Total missed	16.33 (11.29)	7.64 (5.06)	0.03

Treatment satisfaction

Ratings of the Web site, videoconference, and therapist did not differ by prior technology use in the home (see Table 2). Parents without regular prior computer experience did not rate the intervention as more difficult or less helpful than did more computer-literate parents. All but one participant (94.4%) found the Web site to be moderately to extremely easy to use. However, a higher proportion of parents (22%) rated the videoconferences as “not at all” or “a little” easy to use, suggesting potential drawbacks to the videoconferencing technology for some participants. Ratings of the helpfulness of the Web site were high with all but one participant rating all the content areas of Web site as moderately to extremely helpful. The videoconferences were rated as moderately to extremely helpful by all participants, but 22% rated it as “not at all” or “a little” helpful compared to face-to-face meetings. Followup analyses of the four parents who rated the videoconferences as not easy to use and not helpful relative to face-to-face sessions indicated that these parents did not report more distress at followup on the two parent outcome measures (AI and CES-D, p > 0.4), suggesting that ratings of helpfulness were not associated with the degree of improvement.

Table 2.

Web Site Ratings of Ease of Use and Helpfulness

Variable	Prior technology use at home
	NO (n = 6) Mean (SD)	Yes (n = 14) Mean (SD)	F	p
About the Web site
Ease of use	4.17 (0.98)	3.67 (1.50)	1.31	0.47
Overall helpful	4.17 (0.98)	3.92 (1.24)	0.11	0.67
About the videoconference
Ease of use	3.83 (1.47)	3.75 (1.14)	0.87	0.90
Overall helpful	4.50 (0.84)	3.92 (0.90)	0.87	0.20
About the therapist
Friendly toward	6.17 (1.33)	6.62 (0.65)	8.30	0.33
Supportive	6.17 (1.33)	6.62 (0.65)	8.30	0.33
Caring, understanding	7.83 (3.25)	9.42 (1.03)	11.68	0.13

Seven caregivers (33%) indicated they would prefer to meet with the therapist in person; however, the proportion of caregivers wishing to meet with the therapist in person did not differ significantly by prior technology use: four (29%) technology users versus three (50%) nontechnology users; χ²(3, N = 19) = 2.45, p = ns.

Therapeutic alliance

There were no significant differences in overall therapeutic alliance (M = 72.33, SD = 10.37 versus M = 76.77, SD = 9.90) t(17) = −0.90, p = 0.38). In fact, caregivers in both groups reported similarly high levels of therapeutic alliance with the therapist. However, comments from qualitative interviews suggest that technical problems potentially interrupted the “flow” of the intervention. Although the groups did not differ in terms of length of time to complete the intervention, several participants noted that the sessions were extended over a longer period of time due to technical delays and lengthened due to difficulties with logging on, poor video quality, or lack of sound.

Comfort rating of technology

Finally, we conducted a repeated-measures ANOVA, controlling for caregiver education, to examine group differences in comfort with technology over time. Comfort with technology was assessed at three time points over the course of the intervention: at the training (initial) session, the fourth videoconference, and the final videoconference. These analyses demonstrated a significant linear group-by-time interaction (F[2] = 5.04, p = 0.02). Post hoc analyses indicated that the groups did not differ in comfort levels at baseline (M = 3.81, SD = 0.38 vs. M = 3.66, SD = 0.40). However, the groups differed significantly at videoconference 4 due to increasing comfort among the technology users (M = 4.16, SD = 0.26, F[13] = −2.25, p = 0.04) accompanied by decreasing comfort in nontechnology users (M = 3.71, SD = 0.51). These differences were no longer significant at the final videoconference; however, statistical power was reduced due to missing data (n = 13). Over the course of the intervention, technology users reported becoming more comfortable with the technology (M = 0.488, SD = 4.77, F[3.8] = 3.07, p = 0.015).

We then examined the relationship between caregiver’s ratings of comfort with the technology and treatment response. Controlling for baseline symptoms, caregivers’ comfort with technology during the training session was predictive of anxiety at followup (F change = 6.57 [1], p = 0.03), accounting for 34% of the variance. Their change in comfort with technology from the training session to session 4 did not account for additional variance (F change = 0.28 [1], p = ns). Controlling for baseline symptoms, comfort at both the training session (F change = 9.08 [1], p = 0.009) and change in comfort with technology by the fourth session (F change = 7.57 [1], p = 0.02) was predictive of change in depression at followup. These findings suggest that prior technology use may influence treatment response through the caregiver’s comfort with the technology.

DISCUSSION

Consistent with our hypotheses, we found that participants who did not use a computer regularly in the home prior to the intervention were less likely to show improvements in anxiety and depression. To our knowledge, no previous investigations have reported differences in treatment response as a function of prior technology use. Moreover, consistent with previous findings,¹⁸ prior technology use was unrelated to therapeutic alliance, treatment satisfaction, and the desire to meet with the therapist in person. These findings have important implications for conducting online therapy or telehealth interventions. They suggest there is a subgroup of individuals who may derive limited benefit from technology-based interventions despite apparently high levels of treatment alliance and satisfaction.

This study differs from many telehealth interventions in that all participants were given home computers to use during the intervention and to keep afterwards, thereby bridging the “digital divide.”¹⁹ It also differs from many telehealth interventions in its requirement that users interface with two technologies (computers/Web sites and video cameras) to successfully complete the intervention. The use of multiple technologies may place greater demands on those who are not familiar with technology to begin with, thereby increasing their ambivalence toward the treatment. The current findings suggest that giving someone access to computers, Web cameras, and Web-based intervention materials may not be sufficient to enable them to benefit. Individuals without prior technology use may require more extensive training and technological support to address their unfamiliarity with using this type of technology. For example, check lists may be helpful in formalizing training to improve quality.

Several factors were examined as possible pathways through which prior technology use influenced treatment response. The number of sessions canceled due to technological problems did not differ significantly between groups, suggesting that both groups had similar technological difficulties over the course of the intervention. However, non-adherence, the total number of sessions missed due to cancellation or patient no-shows, was significantly higher among the nontechnology users. Additionally, nonadherence was a unique predictor of improvements in anxiety and depression with missed sessions contributing to higher symptom levels at followup. Taken together, these findings suggest that lack of prior technology use may contribute to nonadherence, particularly no-shows, which in turn results in less symptomatic improvement. Frequent missed sessions may disrupt the flow of the treatment and increase the sense of disconnection from the therapist and the therapeutic process. Although overall therapeutic alliance did not differ between technology users and nonusers, there was a trend for nonusers to perceive the therapist as less understanding, perhaps reflecting less rapport. Discomfort with technology may also have contributed to a caregiver’s likelihood of “no showing” for a scheduled session, although the correlation between our measures of computer comfort and adherence was not statistically significant.

Interestingly, comfort with the technology did not increase among the nontechnology users in the same way that it did for those with prior technology use. As a result, while those with prior technology use became significantly more comfortable over the course of the intervention, those who lacked prior experience became slightly, but not significantly, less comfortable over time. For both anxiety and depressive symptoms, initial comfort with technology was predictive of treatment response. Additionally, increases in comfort from session 1 to session 4 accounted for additional variance in depression at followup. These findings suggest it is important to monitor comfort with technology over time in Web-based interventions and to identify strategies for increasing comfort, particularly among those who may be less familiar with the technology. In addition to the initial training in using the technology, it may be helpful to have booster sessions or additional technological support throughout the intervention.

Other than prior technology use at home, nothing (including comfort with technology) distinguished this group of participants at baseline. This suggests that it is important for investigators and therapists to assess prior experience with technology before delivering a technology-dependent intervention and to provide additional training and support to increase computer skills or offer alternative interventions.

Technology use is confounded with other factors, such as education level, socioeconomic status, and race. Our groups did not differ in these demographics; however, technology use was related to caregiver education, SES, and race. These factors (poor, less education, minority status, or African American) are known barriers related to seeking treatment.⁶ While we removed one important barrier to treatment—the time and expense of traveling long distance—we must further improve accessibility by addressing those with limited computer experience. In future research, we can attempt to alleviate discomfort with technology by simplifying user interfaces such as logins and making the technology more engaging. Although feedback was solicited from family members on ways to improve our Web site, content, and process, individuals in our sample had difficulty articulating what would increase their comfort or improve our intervention.

It is worth noting that nearly all of the families in the nontechnology group had someone in the family (often a child or adolescent) who was familiar with computers, e-mail, and surfing the Web. This individual often took responsibility for interfacing with the computer during both the Web-based and videoconferencing sessions. As a result, the non-technology users were not, for the most part, required to directly interface with the equipment. This may account for the failure to find differences in ease of use between technology users and nonusers. Our findings suggest that having someone else in the house who is comfortable with the technology is not adequate to overcome the impact of the caregiver’s own discomfort on treatment response.

Several limitations of the current study should be noted. This is a preliminary study, and we did not have adequate statistical power to look at changes in comfort with the technology and adherence together in a mediation model.²⁰ However, the current findings provide preliminary evidence that both factors may mediate treatment response as measured by reductions in anxiety and depression. Although regular prior technology use proved to be a determinant of treatment response, it may be helpful to consider computer self-efficacy and anxiety to more thoroughly understand the role of prior technology use. Reliance on self-report measures and the lack of an extended followup pose further limitations. Despite these limitations, the current findings suggest it may be important to assess both prior technology use and comfort with technology when delivering Web-based interventions. Future telehealth research should look carefully at the issue of matching treatments and technological requirements to the needs and comfort of the end user. Psychologists may be accustomed to thinking about the “digital divide” as a social issue. Perhaps the most important lesson learned from this investigation is that prior computer use and technological sophistication are also issues affecting therapeutic outcomes.