Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Mar 1.
Published in final edited form as: Clin Pract Pediatr Psychol. 2019 Mar;7(1):9–19. doi: 10.1037/cpp0000261

Evaluating the Acceptability and Validity of Assessing Pain and Posttraumatic Stress Symptoms in an Adaptable eHealth System for School-Age Children

Nancy Kassam-Adams 1, Kristen L Kohser 2, Jeffery McLaughlin 3, Flaura Winston 4, Meghan L Marsac 5
PMCID: PMC6604622  NIHMSID: NIHMS1015867  PMID: 31275781

Abstract

Objective

To provide initial evaluation of the acceptability of a new eHealth system incorporating personalized self-report assessment of multiple health domains in school age children, and assess convergent validity of two brief measures presented via this system.

Methods

Ill or injured children (N=167) age 6 to 14 recruited in two pediatric health care systems used the prototype eScreen interface on a mobile device to select an avatar and complete brief assessments of pain and posttraumatic stress symptoms (PTSS). Children rated technology acceptability and completed validated measures for pain and PTSS.

Results

Children’s ratings indicated they found the eScreen interface easy to use (mean rating 4.4 on a 1–5 scale), potentially useful in helping them recover (mean=3.7), and would use / recommend it (mean=4.0). Among children age 6 to 8, mean ratings were: easy to use (3.7), usefulness (3.3), would use/recommend (3.4). Acceptability was largely consistent across child gender, family income, or usual access to mobile devices. eScreen measures showed strong convergent validity with established measures. The eScreen Pain Screener was highly correlated (r =.86 - .92) with, and evidenced strong agreement with, two validated pain measures. eScreen PTSS scores were strongly correlated with a validated PTSS measure (r=.67); a positive PTSS screen was associated with significantly higher PTSS severity.

Conclusions

Study results support the acceptability (ease of use, intention to use/recommend, perceived usefulness) of these tools for older school age children, and provide strong initial evidence for the validity of two brief measures presented in a novel digital modality.

Keywords: eHealth, pain assessment, posttraumatic stress assessment

Medical and psychosocial management of pediatric conditions often requires the health care team to be aware of symptoms or behaviors that occur outside the clinical setting; i.e., after hospital discharge or between clinical visits (Carlsen et al., 2017; Nkoy et al., 2013). These data may be needed to guide care, or to better understand the outcomes of care. For example, the health care team may need to adjust the medication plan based on a child’s evolving symptoms, or to evaluate patient-reported health outcomes after an initial episode of care to inform the post-acute or chronic treatment plan. However, achieving this type of repeated assessment presents many practical challenges, including cost, compliance, and ease of use. For many types of symptoms and experiences, the child’s own perspective (i.e. child self-report) is the gold standard for school age children and adolescents, and relying solely on parent or caregiver report can be problematic (Chambers, Reid, Craig, McGrath, & Finley, 1998; Kassam-Adams, Garcia-España, Miller, & Winston, 2006). Thus, a system for repeated assessment of child symptoms would ideally facilitate obtaining self-report data directly from pediatric patients.

Internet-based digital (referred to hereafter as “eHealth”) assessment tools for children may provide an effective solution to these challenges (Bromberg, Connelly, Anthony, Gil, & Schanberg, 2014; Carlsen et al., 2017; Fortier, Chiung, Martinez, Gago-Masague, & Sender, 2016). eHealth tools offer the promise of flexible delivery (time and place) and can incorporate age-appropriate features to promote engagement and use by youth. The use of eHealth tools for assessment is made increasingly feasible by the fact that the vast majority of US families have persistent internet access via mobile devices: in 2015 89% of US households with children had mobile devices and 85% had broadband access at home (Ryan, 2017). Even amongst school age children, mobile device use and online activity is increasing: A 2014 survey found that 26% of 8–10 year olds and 45% of 11–14 year olds had their own personal smartphone or tablet. In this survey, 73% of 8–10 year olds went online at least weekly and 29% spend > 1 hour per day online, and 95% of 11–14 year olds were online at least weekly and 52% spent >1 hour per day online (Lauricella, Cingel, Blackwell, Wartella, & Conway, 2014).

eHealth assessments may incorporate content from validated measures previously delivered in other modalities (i.e. paper-and-pencil or in person administration) but can also take advantage of the interactive nature of this modality to enhance engagement and ease administration burdens. When adoption of existing measures for eHealth delivery involves a straightforward presentation of verbal items on screen rather than paper-and-pencil, additional validation may be useful but not essential. But when the eHealth version of a measure adds features or elements that could alter the way in which a child interacts with the assessment measure, it is important to (re-)assess the validity of the measure as delivered via the eHealth platform.

Despite the promise of eHealth tools, to our knowledge there is currently no eHealth system that can incorporate validated assessment tools for a range of symptom domains for use with a wide range of pediatric populations. The long-term goal of this effort is to create an adaptable, personalized, engaging health assessment system, eScreen, for the collection of self-reported symptom data from school-age children facing illness or injury. The target users of these data are the inter-disciplinary health care team responsible for a child’s medical care, as well as health systems interested in optimizing care to improve child health outcomes at a population level. While eHealth tools have been designed for children with specific disease conditions (Bromberg et al., 2014; Carlsen et al., 2017), the goal for eScreen is to allow flexible configuration and inclusion of measures to meet the needs of different pediatric populations. The current project illustrates the process of selecting, adapting, and validating two measures for integration into eScreen, one, to assess pain; the other, posttraumatic stress symptoms. For pain intensity, we created a new measure that borrows key features from validated pain assessments but is optimized for delivery on a mobile device. For posttraumatic stress symptoms, we used items from a validated brief paper-and-pencil checklist. We incorporated personalization elements for each measure, with the aim of promoting child engagement. We included pain and posttraumatic stress measures in this initial assessment of eScreen components because both occur across pediatric populations (Friedrichsdorf et al., 2015; Price, Kassam-Adams, Alderfer, Christofferson, & Kazak, 2016), are often not assessed post-discharge although they appear to influence child recovery and functional outcomes (Gill, Drendel, & Weisman, 2013; Sabin, Zatzick, Jurkovich, & Rivara, 2006; Zisk, Frey, Medoff-Cooper, MacLaren, & Kain, 2008), and are best assessed via child self-report (Chambers et al., 1998; Kassam-Adams et al., 2006).

The objectives of the current study are (1) to provide an initial assessment of the acceptability and feasibility of several components of a new eHealth tool in development, and (2) to assess the convergent validity of two brief measures created or adapted for this tool and presented on a mobile device: a visual analog pediatric pain assessment tool (eScreen Pain Screener) and a screen for posttraumatic stress symptoms (Acute Stress Checklist for Children 6-item Short Form [ASC-6]).

Method

Overview of study procedures

Children age 6 to 14 years and one parent per child were recruited at inpatient and emergency department settings in two pediatric health care systems (one urban, one rural). We enrolled children with a recent injury or an illness-related event potentially involving pain; e.g., appendicitis or a pain crisis related to sickle cell disease. The study was conducted according to a protocol approved by the IRB at each institution. Parents provided consent and children provided assent for participation in the study.

Parents completed demographic questionnaires and reported on the child’s usual access to technology (smartphone or tablet) at home. Children used the prototype eScreen system interface on a mobile device (tablet) provided by a research assistant. Each child selected an avatar and then completed brief measures of pain and posttraumatic stress. Each study participant’s assessment data gathered on the mobile device were identified only by study ID with no personally identifying information. These data were stored in a password-protected file on the device and then transferred securely by study personnel to secure servers at each study site for analysis. After completing use of the tablet, children rated ease of use, perceived usefulness, and likelihood to use / recommend the eScreen system. In counter-balanced order across the sample, immediately before or after the tablet-presented measures, children also completed validated measures for pain and posttraumatic stress, administered by trained study personnel. We abstracted the child’s primary diagnosis and reason for medical care from the medical record.

Description of the eScreen System

Our team created a prototype version of several components of “eScreen”, an eHealth system designed for brief assessment of child symptoms or functioning. The goal for the eScreen framework is to allow (a) personalization for each child, and (b) modular incorporation of one or more brief measures (question items or other modes such as visual analog scales). In this study, the prototype eScreen system offered two components: personalization by allowing the child to choose an avatar, and presentation of brief measures of pain and posttraumatic stress symptoms. Faces based on the child’s chosen avatar were integrated into the pain assessment tool used by that child, as described below.

Measures

eScreen Pain Screener

This measure is a visual analog scale designed for delivery on a mobile device by adapting key elements of existing validated pain measures (McGrath et al., 1996) and adding personalization features. The measure appeared on screen as a visual analog ‘slider’ with a movable marker; the width and color intensity increased from bottom to top. The face of the child’s avatar (i.e. the avatar selected by the child to represent himself/herself in the eScreen system) anchored the lower and upper ends of the slider with a ‘no pain’ and a ‘most pain’ facial expression, respectively. We created these facial expressions for each avatar based on features in the lowest and highest pain faces of the Faces Pain Scale (Bieri, Reeve, Champion, Addicoat, & Ziegler, 1990; Hicks, von Baeyer, Spafford, van Korlaar, & Goodenough, 2001). In accordance with best practices in assessing pediatric pain intensity the faces do not include affective information such as smiles or tears (Chambers & Craig, 1998). Verbal anchors for the ends of the scale were based on the well-validated Colored Analogue Scale (McGrath et al., 1996), which is described below. On-screen instructions stated “The bottom of this scale is no pain, and the top is the most pain you can imagine. Slide the marker to show how much pain you feel right now.” The child used his or her finger on the touch screen to slide the virtual marker.

Acute Stress Checklist for Children 6-item Short Form (ASC-6)

Developed as a short form of the Acute Stress Checklist for Children (ASC-Kids), the ASC-6 has been previously validated as a paper-and-pencil questionnaire (Kassam-Adams & Marsac, 2016). Across multiple samples, it demonstrated concurrent validity with severity of symptoms as assessed via the full-length ASC-Kids checklist (correlations from .88 to .92) or a structured clinical interview (correlations of .61 - .62) (Kassam-Adams & Marsac, 2016). Also across multiple samples, a score of 6 or greater on the ASC-6 had strong sensitivity (.75 – 1.00) and specificity (.68 - .84) to detect concurrent acute stress disorder, and this cutoff was associated with more severe acute traumatic stress symptoms and more impairment from these symptoms (Kassam-Adams & Marsac, 2016). The ASC-6 was delivered on the mobile device (tablet) with the six items presented three at a time across two screens. The child’s avatar appeared on each screen. The child used the touch screen to select a response to each item on a 3-point Likert scale, scored as 0-1-2. We calculated the total summed severity score for the ASC-6 (possible range 0 to 12), and whether the child’s score met the previously established cutoff (≥ 6) for likely presence of significant posttraumatic stress symptoms. Internal consistency (Cronbach’s alpha) for the ASC-6 in the current study was .71.

Technology Acceptance Measure (TAM)

The technology acceptance model is a well-validated conceptual model for acceptance and adoption of new information technology tools (Davis, 1989; Venkatesh & Davis, 2000) that has been adapted for a variety of tools and types of users, including children as young as age 9 or grade four (Baulch, Chester, & Brennan, 2010; Bourgonjon, Valcke, Soetaert, & Schellens, 2010; Cheng, Lou, Kuo, & Shih, 2013; Hwang, Chu, Chen, & Cheng, 2014; Kuo, Chang, Yu, & Heh, 2013). The TAM scale used in the current study included 8 items assessing key constructs in the model: perceived ease of use (2 items, i.e. “the pain screener was clear and easy to understand”), behavioral intention (2 items regarding whether the child would use it or would recommend it to others), and perceived usefulness to achieve desired aims of the user (4 items, i.e. “Using this pain screener would make it easier for me to tell people how I am doing”). Children rated each item on a 5-point Likert scale from Strongly Disagree to Strongly Agree, scored as 1 to 5. The TAM is scored by calculating the mean item rating for each subscale: perceived ease of use, acceptability / intention, and perceived usefulness. Internal consistency (Cronbach’s alpha) for the TAM scale in the current study was .82.

Colored Analog Scale (CAS)

The CAS, developed with children age 5 through 16 (McGrath et al., 1996), has demonstrated reliability and validity as a pediatric pain measure. For example, the CAS had strong test-retest reliability and convergent and discriminant validity in a sample of 620 children age 4 to 17 assessed in emergency department settings (Tsze, von Baeyer, Bulloch, & Dayan, 2013). It consists of a plastic scale showing a wedge that increases in width and in color intensity from bottom (labeled as ‘no pain’) to the top (labelled as ‘most pain’), and a sliding marker that can be moved physically by the child. The child’s rating is read on the reverse side of the CAS tool, which shows ratings from 0 to 10 in 0.25 increments. Following standard CAS instructions, in the current study children were told that the “bottom end where there’s hardly any color is no pain, and the top end where it’s very red is most pain.” The scale was then handed to the child with instructions to “slide the marker to show how much pain you have right now.”

Numerical Rating Scale for Pain Intensity (NRSI)

The NRSI is a verbally-administered pain evaluation in which children are asked to rate their pain by choosing a number from 0 to 10 “that best tells us how much you are hurting, where 0 = no pain or hurt and 10 = the most or worst pain/hurt.” The NRSI has demonstrated strong convergent validity with other measures of pain intensity in children age 7 to 18 across multiple studies (Page et al., 2012; von Baeyer et al., 2009), as well as sensitivity to change over time (Page et al., 2012).

Child PTSD Symptom Scale for DSM-5 (CPSS-5) Self-Report Version

The CPSS-5 includes 20 items assessing posttraumatic stress symptoms aligned with the DSM-5 criteria for posttraumatic stress disorder (PTSD), rated by the child on a 5-point Likert scale, scored as 0 to 4 (Foa, Asnaani, Zang, Capaldi, & Yeh, 2018). The DSM-5 update of the CPSS builds on the well-validated DSM-IV version (Foa, Johnson, Feeny, & Treadwell, 2001; Meyer, Gold, Beas, Young, & Kassam-Adams, 2015). The DSM-5 self-report version has demonstrated strong internal consistency, test-retest reliability, convergent and discriminant validity in initial evaluations (Foa et al., 2018). We calculated the total summed severity score for the CPSS-5 (possible range 0 to 80), as well as severity scores for each symptom category (re-experiencing, avoidance, cognitive and mood alterations, and hyper-arousal). Internal consistency (Cronbach’s alpha) for the CPSS in the current study was .88.

Analyses

We first used descriptive statistics to summarize sample demographics, reason for medical care, and children’s usual technology access at home for children. In order to assess feasibility and acceptability, we (a) examined technology acceptance ratings on each TAM subscale, (b) conducted independent samples t-tests to assess the association of TAM ratings with child gender, reason for medical care (illness versus injury), family household income, and whether the child had access to technology at home, and (c) conducted an ANOVA to evaluate the association of TAM ratings with child age (in three categories). We evaluated the convergent validity of the eScreen-delivered measures with validated measures of child pain and posttraumatic stress symptoms in several ways: We examined correlations, described the difference in each child’s pain rating across the pain scales (all use the same metric), and conducted t-tests to examine differences in PTSS severity scores on the CPSS for children scoring above versus below the previously established cut-off score for significant PTSS symptoms (i.e., 6) on the ASC-6.

Results

Participant characteristics

Study participants were recruited at two sites; 98 (58.7%) were enrolled at a pediatric hospital located in a largely rural area, and 69 (41.3%) at a pediatric hospital in an urban setting that serves a primarily urban and suburban population. The 167 child participants were enrolled during an inpatient admission (n=148, 88.6%) or during an emergency department visit (n=19, 11.4%). Participant demographic characteristics, reason for medical treatment, and information on the child’s usual access to mobile technology are summarized in Table 1.

Table 1.

Characteristics of study participants (N=167)

N (%)
Child age
 6 – 8 years 36 (21.6%)
 9 – 11 years 52 (31.1%)
 12 – 14 years 79 (47.3%)
Child sex = male 89 (53.3%)
Child race
 Black / African-American 26 (15.6%)
 White 121 (72.5%)
 Other 20 (11.3%)
 Declined to report 1 (0.6 %)
Household income (reported by parent)
 ≥ $40,000 81 (48.5%)
 < $40,000 62 (37.1%)
 Declined to report 24 (14.4%)
Reason for current medical treatment
 Illness 97 (58.1%)
 Injury 70 (41.9%)
Child’s usual access to mobile technology at home
 Child personally has access to smartphone 132 (79.0%)
 Child personally has access to tablet 132 (79.0%)
Open in a new tab

Feasibility and acceptability

Children’s use of the prototype eScreen measures on the tablet, including selecting an avatar, took less than five minutes. Regarding acceptability as conceptualized in the technology acceptance model, children’s ratings for each TAM subscale (Table 2) indicated that they found the eScreen interface easy to use and that they would use or recommend it to others (i.e., TAM behavioral intention subscale). Children generally rated the eScreen system as potentially useful in helping them recover or communicate about their symptoms with others. We compared ratings on each TAM subscale based on child demographics. Independent groups t-tests indicated no statistically significant group differences in any mean TAM subscale ratings for boys versus girls, or for children from families with household income over US$40,000 versus lower incomes. For children with a recent injury versus illness, there was no difference in TAM subscales for ease of use or behavioral intention, but the injury group (M = 3.93, SD = 0.78) had a slightly higher mean rating than the illness group (M = 3.62, SD = 1.04) for perceived usefulness (t = 2.13, df = 165, p = .03, 95% CI for mean difference 0.02 to 0.61).

Table 2.

Children’s ratings of acceptability on Technology Acceptance Model (TAM) subscales, with results of ANOVA by age group

Total sample
(N=167)		 Age 6 to 8
(N=36)		 Age 9 to 11
(N=52)		 Age 12 to 14
(N=79)
M
(SD) 		M
(SD) 		M
(SD) 		M
(SD) 		F p
TAM subscale
Perceived ease of use 4.39
(0.89)		 3.69a
(1.18)		 4.50b
(0.79)		 4.65b
(0.62)		 17.42 <.0001
Behavioral intention 4.02
(1.00)		 3.36a
(1.20)		 4.12b
(0.89)		 4.26b
(0.89)		 10.83 <.0001
Perceived usefulness 3.75
(0.95)		 3.28a
(1.10)		 3.91b
(0.87)		 3.86b
(0.79)		 5.92 .003
Open in a new tab

Note: In each row, groups with different subscripts show a significant mean difference in post-hoc tests (Bonferroni)

Based on ANOVA results, we did observe a difference based on child age; post-hoc tests indicated that each TAM subscale’s mean rating was lower among the youngest group of children (age 6 to 8) compared to those age 9 to 11 or age 12 to 14 (Table 2). Also potentially relevant to feasibility and child age, four additional children (three 7 year olds and one 10 year old) were initially enrolled in the study but could not be included in the current study analyses. In these four cases tablet or questionnaire data was missing or of questionable validity due to the impact of physical symptoms (fatigue, pain) that became apparent after the child began study procedures, and/or due to the child’s difficulty with following study instructions.

To evaluate the possibility that acceptability could depend on a child’s usual access to mobile technology, we also used t-tests to compare TAM ratings for children with and without access to a smartphone or a tablet at home. We found no difference in mean TAM subscale ratings for these groups, except that those without access to a tablet (M = 4.13, SD = 0.65) had a higher mean rating than those with access to a tablet (M = 3.67, SD = 0.99) for perceived usefulness (t = 2.31, df = 157, p - .02, 95% CI for mean difference 0.15 to 0.77).

Concurrent validity of eScreen measures

Children’s pain ratings on each measure ranged from 0 to 10 (potential range for each measure is 0 to 10). Mean rating for each pain measure and correlations amongst pain measures are presented in Table 3. Regarding validity of pain assessment, scores from the eScreen Pain Screener were highly correlated with scores from the CAS and the NRSI. Because all pain measures in this study yield a rating on the same 0 to 10 metric, we calculated the difference between each child’s pain rating on the eScreen Pain Screener and his/her rating on the CAS or the NRSI. The mean difference between eScreen pain ratings and CAS ratings was 0.27 (SD=1.07); 121 (72.5%) of children’s ratings differed by less than 1 point, and 158 (94.6%) differed by less than 2 points across these two scales. Comparing eScreen ratings to NRSI ratings, the mean difference was 0.12 (SD=1.41); 112 (67.1%) differed by less than 1 point, and 148 (88.6%) differed by less than 2 points.

Table 3.

Children’s ratings of current pain on the eScreen Pain Screener, Colored Analogue Scale, and Numerical Rating Scale for Pain Intensity, with correlations amongst these pain measures

M (SD) Correlations (Pearson’s r)
CAS NRSI
eScreen Pain Screener 3.80 (2.61) .92 .86
Colored Analogue Scale (CAS) 3.54 (2.54) .91
Numerical Rating Scale for Pain Intensity (NRSI) 3.68 (2.66)
Open in a new tab

ASC-6 total scores ranged from 0 to 11 (potential range 0 to 12) and CPSS-5 total scores ranged from 0 to 71 (potential range 0 to 80). Mean ratings for PTSS severity on each measure are presented in Table 4. Regarding validity of posttraumatic stress symptom assessment, scores from the eScreen ASC-6 were strongly correlated with CPSS-5 scores (r=.67). Using the previously validated ASC-6 cutoff of 6 or greater, 53 (31.7%) of the child participants would have screened positive. To assess the utility of the eScreen-delivered ASC-6 in identifying children with clinically meaningful PTSS, we used t-tests to compare the mean CPSS severity scores for children below versus above this cutoff (Table 4). Those scoring above the cutoff on the ASC-6 had significantly greater PTSS severity overall and in each symptom category, as assessed by the CPSS.

Table 4.

Children’s ratings of posttraumatic stress symptoms, and results of t-tests comparing symptom severity based on eScreen Acute Stress Checklist - 6 item Short Form screening result

Total sample (N=167) ASC-6 ≥ 6
(N=53)		 ASC-6 < 6
(N=114)
M
(SD) 		M
(SD) 		M
(SD) 		t df p 95% CI for mean difference
eScreen ASC-6 3.99
(2.66)		 -- --
CPSS-5 total score 16.43
(12.67)		 25.87
(13.45)		 12.01
(9.53)		 7.55 161 <.001 10.23 – 17.48
CPSS-5 Re-experiencing score 3.85
(3.85)		 6.50
(4.57)		 2.61
(2.70)		 6.86 165 <.001 2.77 – 5.00
CPSS-5 Avoidance score 2.42
(2.31)		 3.98
(2.05)		 1.69
(2.06)		 6.69 164 <.001 1.62 – 2.97
CPSS-5 Cognition / Mood score 4.62
(4.82)		 7.85
(5.59)		 3.11
(3.54)		 6.56 161 <.001 3.31 – 6.16
CPSS-5 Hyper-arousal score 5.56
(4.20)		 7.64
(4.47)		 4.59
(3.71)		 4.60 161 <.001 1.75 – 4.38
Open in a new tab

Note: eScreen ASC-6 = Acute Stress Checklist - 6-item Short Form delivered in eScreen system; CPSS-5 = Child PTSD Symptom Scale for DSM-5

Discussion

This study provides initial support for the delivery of eHealth assessments of pain and posttraumatic stress symptoms via a new adaptable system for collecting school age child self-reported symptoms beyond the confines of the clinic or hospital. By incorporating child-friendly design and personalization, the eScreen system aims to enable assessment of phenomena such as pain, physical or psychological symptoms, behaviors, and functioning via child self-report. Study results support the acceptability (ease of use, intention to use or recommend, perceived usefulness) for children age 9 to 14 of key components of the eScreen platform, i.e., avatar selection plus personalized presentation of two measures on a mobile device. These results also provide strong initial evidence for the validity of two screening measures presented in a novel digital modality.

Other eHealth tools aimed at assessment of specific pain, symptoms, or behavior have been developed for school age children with specific conditions such as cancer, juvenile idiopathic arthritis, or inflammatory bowel disease (Bromberg et al., 2014; Carlsen et al., 2017; Fortier et al., 2016). The aim of the eScreen system under development is somewhat different: to provide a modular and adaptable system applicable across patient groups that incorporates elements of personalization to increase engagement and allows assessment modules to be selected based on the aims of each practice, setting, or clinical context. Thus the current study enrolled children with a range of types of illness or injury, and examined two different assessment tools that were incorporated within the prototype system.

This sample of school age children from two locations (one, urban; the other, rural) with varying demographic characteristics amongst the sample allowed us to examine potential age, gender, and socio-economic differences in the acceptability of the eScreen system. This is particularly relevant to the feasibility of implementing such tools in a broad pediatric population, as demographic factors might influence child or youth comfort with technology such as eHealth tools (Dolan, 2016). For example, having less opportunity for regular use of mobile devices could make it harder for some children to use a new mobile application such as an eHealth tool when it is presented to them. There is good news in these study findings, in that we did not observe differences based on gender, family income, or usual access to mobile technology at home in how children rated ease of use, or their intention to use or recommend tools like this. Perceived usefulness also did not differ amongst these groups, except that usefulness was rated more highly by children who did not have access to a tablet at home. The only observed difference in acceptability ratings between children treated for illness versus injury, a slightly higher perceived usefulness rating by those with injury, does not appear to be a clinically meaningful difference in ratings. As many prior eHealth tools have been developed for specific pediatric populations, future studies with tools, such as eScreen, that are not linked to a specific child health condition could add to our knowledge base. By enrolling children facing a variety of health conditions, including illnesses and injuries, studies can explore similarities and differences among pediatric populations in the acceptability and use of eHealth tools.

These results do point to areas in need of further development to achieve improved engagement of younger children. The ratings of children age 6 to 8 indicated that they had lower perceived ease of use, usefulness, and intentions to use these tools. (We also noted that three of the four consented participants who were unable to complete study measures due to physical symptoms and/or difficulty following study instructions were in this younger age group.) Thus, this type of system may need to be modified for children 8 and under. One advantage of the iterative development process for eHealth tools and systems is that it allows findings such as this to inform ongoing design and be incorporated into next stages in the development of system features. This is an area ripe for additional research and development. While studies that include children as young as 7 years old have indicated overall feasibility of using eHealth tools for repeated assessment (Heron, Everhart, McHale, & Smyth, 2017), few studies have evaluated eHealth tools specifically amongst this youngest school age group. Enabling parent support to help children to provide self-report ratings (for these youngest children) may be useful, with the caveats that parent-report of child PTSS can be biased by parents’ own experiences (Kassam-Adams et al., 2006), and that parents often utilize their child’s visible external behavior, rather than quiet distress, as an indicator for child pain level (Zisk et al., 2008).

The anticipated real-world use of the eScreen system is to assess for problematic symptoms, behavior, or functioning in children whom the health care team has reason to follow, but who are not present in the clinic or hospital. Thus, we focused our evaluation on convergent validity with the sort of established measures of pain or posttraumatic stress severity that might be administered if health care teams were able to conduct a brief in-person assessment. For example, we wanted to assess how the eScreen-delivered pain scale would compare to the pain rating that a child might give to a validated pain measure administered in person by a member of the health care team, and how a validated PTSS measure would compare to the eScreen-delivered PTSS measure. On both counts, these results are promising. In addition to strong correlations, the eScreen pain ratings showed very good agreement with ratings on well-established pain measures. Children who scored above the cutoff on the eScreen ASC-6 had substantially higher PTSS. The correlation of the eScreen ASC-6 symptom severity score with the CPSS was similar to that observed in a prior study for the ASC-6 with a clinical interview for PTSS (Kassam-Adams & Marsac, 2016).

This study had several limitations which should be addressed in future research. We did not conduct a full usability evaluation of the eScreen system; this will be necessary in future studies as the system is developed. While the technology acceptance model provides a useful and validated framework for assessing acceptability, the literature does not include prior use of the TAM scale with children under age 9. Thus its results for the youngest group of children in the current study should be interpreted with caution, and future studies might benefit from additional methods (i.e. interviews) to better understand children’s views on the acceptability of these tools. Other limitations include the cross-sectional design, with a single assessment point. Future research will test the feasibility of the intended use of the eScreen systems for repeated assessment with children both inside and outside the health care setting, and the convergent validity of those assessments with other indicators of changes in symptoms or functioning over time. Finally, because lower income families who own mobile devices may still face additional challenges to maintaining connectivity of those devices, socio-economic barriers require additional attention in future research.

Implications for clinical practice

These results affirm the promise of eHealth tools for feasible and validated assessment of school-age children. Further development of this system will aim to aid health systems and multi-disciplinary health care teams, including pediatric psychologists, by making it feasible to gather clinically-relevant data from regular eHealth assessments of self-reported symptoms, behavior, or functioning in a range of different pediatric patient groups.

Conclusions

Study results support the acceptability of key components of the eScreen platform for children age 9 to 14, and suggest the need for additional development and testing for feasibility and acceptability with younger school-age children. The study provides strong initial evidence for the validity of two screening measures for pain and posttraumatic stress presented in a novel digital modality, with personalization features.

Implications for Impact Statement.

Key parts of an adaptable eHealth screening system were rated positively by children age 9 to 14; younger children’s ratings suggest the need for additional development for their use. Results support the validity of brief pain and posttraumatic stress measures presented via this system and personalized with an avatar chosen by the child.

Acknowledgments

This work funded in part by a grant from the National Institute of Child Health and Human Development (R42HDHD087021).

Contributor Information

Nancy Kassam-Adams, Children’s Hospital of Philadelphia and University of Pennsylvania Perelman School of Medicine.

Kristen L. Kohser, Children’s Hospital of Philadelphia

Jeffery McLaughlin, Radiant Creative LLC.

Flaura Winston, Children’s Hospital of Philadelphia and University of Pennsylvania Perelman School of Medicine.

Meghan L. Marsac, Kentucky Children’s Hospital and University of Kentucky

References

  1. Baulch J, Chester A, & Brennan L (2010). Adolescent and parent content preferences and predictors of intention to use an online healthy weight website for adolescents. Electronic Journal of Applied Psychology, 6(1), 18–26. [Google Scholar]
  2. Bieri D, Reeve R, Champion G, Addicoat L, & Ziegler J (1990). The faces pain scale for the self-assessment of the severity of pain experienced by children: Development, initial validation and preliminary investigations for ratio scale properties. Pain, 41, 139–150. [DOI] [PubMed] [Google Scholar]
  3. Bourgonjon J, Valcke M, Soetaert R, & Schellens T (2010). Students’ perceptions about the use of video games in the classroom. Computers & Education, 54(4), 1145–1156. [Google Scholar]
  4. Bromberg MH, Connelly M, Anthony KK, Gil KM, & Schanberg LE (2014). Self‐reported pain and disease symptoms persist in juvenile idiopathic arthritis despite treatment advances: An electronic diary study. Arthritis & Rheumatology, 66(2), 462–469. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Carlsen K, Jakobsen C, Houen G, Kallemose T, Paerregaard A, Riis LB, … Wewer V (2017). Self-managed eHealth disease monitoring in children and adolescents with inflammatory bowel disease: a randomized controlled trial. Inflammatory bowel diseases, 23(3), 357–365. [DOI] [PubMed] [Google Scholar]
  6. Chambers C, & Craig K (1998). An intrusive impact of anchors in children’s faces pain scales. Pain, 78, 27–37. [DOI] [PubMed] [Google Scholar]
  7. Chambers C, Reid C, Craig K, McGrath P, & Finley G (1998). Agreement between child and parent reports of pain. Clinical Journal of Pain, 14(4), 336–342. [DOI] [PubMed] [Google Scholar]
  8. Cheng Y, Lou S, Kuo S, & Shih R (2013). Investigating elementary school students’ technology acceptance by applying digital game-based learning to environmental education. Australasian Journal of Educational Technology, 29(1), 96–110. [Google Scholar]
  9. Davis FD (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS quarterly, 319–340. [Google Scholar]
  10. Dolan JE (2016). Splicing the divide: A review of research on the evolving digital divide among K–12 students. Journal of Research on Technology in Education, 48(1), 16–37. [Google Scholar]
  11. Foa E, Asnaani A, Zang Y, Capaldi S, & Yeh R (2018). Psychometrics of the Child PTSD Symptom Scale for DSM-5 for trauma-exposed children and adolescents. Journal of Clinical Child & Adolescent Psychology, 47(1), 38–46. [DOI] [PubMed] [Google Scholar]
  12. Foa E, Johnson K, Feeny N, & Treadwell K (2001). The Child PTSD Symptom Scale: A preliminary examination of its psychometric properties. Journal of Clinical Child Psychology, 30(3), 376–384. doi: 10.1207/S15374424JCCP3003_9 [DOI] [PubMed] [Google Scholar]
  13. Fortier M, Chiung W, Martinez A, Gago-Masague S, & Sender L (2016). Pain Buddy: A novel use of m-health in the management of children’s cancer pain. Computers in Biology and Medicine, 26, 202–214. doi: 10.1016/j.compbiomed.2016.07.012 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Friedrichsdorf SJ, Postier A, Eull D, Weidner C, Foster L, Gilbert M, & Campbell F (2015). Pain outcomes in a US children’s hospital: A prospective cross-sectional survey. Hosp Pediatr, 5(1), 18–26. doi: 10.1542/hpeds.2014-0084 [DOI] [PubMed] [Google Scholar]
  15. Gill M, Drendel AL, & Weisman SJ (2013). Parent satisfaction with acute pediatric pain treatment at home. The Clinical journal of pain, 29(1), 64–69. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Heron KE, Everhart RS, McHale SM, & Smyth JM (2017). Using mobile-technology-based ecological momentary assessment (EMA) methods with youth: A systematic review and recommendations. Journal of Pediatric Psychology, 42(10), 1087–1107. doi: 10.1093/jpepsy/jsx078 [DOI] [PubMed] [Google Scholar]
  17. Hicks C, vonBaeyer C, Spafford P, vanKorlaar I, Goodenough B. (2001). The Faces Pain Scale-Revised:Toward a common metric in pediatric pain measurement. Pain, 93(2), 173–183. [DOI] [PubMed] [Google Scholar]
  18. Hwang G, Chu H, Chen B, & Cheng Z (2014). Development and evaluation of a Web 2.0-Based Ubiquitous Learning Platform for Schoolyard Plant Identification. International Journal of Distance Education Technologies, 12(2), 83–103. [Google Scholar]
  19. Kassam-Adams N, Garcia-España J, Miller V, & Winston F (2006). Parent-child agreement regarding children’s acute stress: The role of parent acute stress reactions. Journal of the American Academy of Child and Adolescent Psychiatry, 45 1485–1493. [DOI] [PubMed] [Google Scholar]
  20. Kassam-Adams N, & Marsac M (2016). Brief practical screeners in English and Spanish for acute posttraumatic stress symptoms in children. Journal of Traumatic Stress, 29(6), 483–490. doi: 10.1002/jts.22141 [DOI] [PubMed] [Google Scholar]
  21. Kuo R, Chang M, Yu C, & Heh J (2013). A pilot study of the situated game for autistic children learning activities of daily living. Research and Practice in Technology Enhanced Learning, 8(2), 291–315. [Google Scholar]
  22. Lauricella AR, Cingel DP, Blackwell C, Wartella E, & Conway A (2014). The mobile generation: Youth and adolescent ownership and use of new media. Communication Research Reports, 31(4), 357–364. [Google Scholar]
  23. McGrath P, Seifert C, Speechly K, Booth J, Stitt L, & Gibson M (1996). A new analogue scale for assessing children’s pain: An initial validation study. Pain, 64, 435–443. [DOI] [PubMed] [Google Scholar]
  24. Meyer R, Gold J, Beas V, Young C, & Kassam-Adams N (2015). Psychometric evaluation of the Child PTSD Symptom Scale in Spanish and English. Child psychiatry and human development, 46(3), 438–444. doi: 10.1007/s10578-014-0482-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Nkoy F, Stone B, Fassl B, Uchida D, Koopmeiners K, Halbern S, … Greene T (2013). Longitudinal validation of a tool for asthma self-monitoring. Pediatrics, 132(6), e1554–1561. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Page MG, Katz J, Stinson J, Isaac L, Martin-Pichora AL, & Campbell F (2012). Validation of the numerical rating scale for pain intensity and unpleasantness in pediatric acute postoperative pain: sensitivity to change over time. Journal of Pain, 13(4), 359–369. doi: 10.1016/j.jpain.2011.12.010 [DOI] [PubMed] [Google Scholar]
  27. Price J, Kassam-Adams N, Alderfer M, Christofferson J, & Kazak A (2016). Systematic review: A reevaluation and update of the Integrative (Trajectory) Model of Pediatric Medical Traumatic Stress. Journal of Pediatric Psychology, 41, 86–97. doi: 10.1093/jpepsy/jsv074 [DOI] [PubMed] [Google Scholar]
  28. Ryan CL, J. (2017). Computer and Internet Use in the United States: 2015 American Community Survey Reports, ACS-37. Washington, DC. [Google Scholar]
  29. Sabin J, Zatzick D, Jurkovich G, & Rivara F (2006). Primary care utilization and detection of emotional distress after adolescent traumatic injury: Identifying an unmet need. Pediatrics, 117(1), 130–138. [DOI] [PubMed] [Google Scholar]
  30. Tsze DS, von Baeyer CL, Bulloch B, & Dayan PS (2013). Validation of self-report pain scales in children. Pediatrics, 132(4), e971–979. doi: 10.1542/peds.2013-1509 [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Venkatesh V, & Davis F (2000). A theoretical extension of the technology acceptance model: Four longitudinal field studies. Management science, 46(2), 186–204. [Google Scholar]
  32. von Baeyer CL, Spagrud LJ, McCormick JC, Choo E, Neville K, & Connelly MA (2009). Three new datasets supporting use of the Numerical Rating Scale (NRS-11) for children’s self-reports of pain intensity. Pain, 143(3), 223–227. doi: 10.1016/j.pain.2009.03.002 [DOI] [PubMed] [Google Scholar]
  33. Zisk R, Frey M, Medoff-Cooper B, MacLaren J, & Kain Z (2008). The squeaky wheel gets the grease: Parental pain management of children treated for bone fractures. Pediatric Emergency Care, 24(2), 89=96. [DOI] [PubMed] [Google Scholar]

RESOURCES