Working out the kinks: Testing the feasibility of an electronic pain diary for adolescents with arthritis

Abstract

BACKGROUND:

Current approaches to evaluating pain in children with chronic arthritis suffer from methodological problems. A real-time data capture approach using electronic diaries has been proposed as a new standard for pain measurement. However, there is limited information available regarding the development and feasibility of this approach in children.

OBJECTIVES:

The aim of the present study was to pilot test the e-Ouch electronic pain diary in terms of compliance and acceptability in adolescents with arthritis to further refine the prototype.

METHODS:

A descriptive study design – with two iterative phases of testing, modifying the prototype and retesting – was used. A purposive sample of 13 adolescents with mild to severe pain and disability was drawn from a large rheumatology clinic in a university-affiliated pediatric tertiary care centre in Canada over a four-week period in December 2004. Participants were signalled with an alarm to use the diary three times per day for a two-week period. Adolescents completed an electronic diary acceptability questionnaire.

RESULTS:

Overall mean compliance rates for phases 1 and 2 were 72.9% and 70.5%, respectively. Compliance was affected by the timing of data collection and technical difficulties. Children rated the diary as highly acceptable and easy to use. Phase 1 testing revealed aspects of the software program that affected compliance, which were subsequently altered and tested in phase 2. No further technical difficulties arose in phase 2 testing.

CONCLUSIONS:

Feasibility testing is a crucial first step in the development of electronic pain measures before use in clinical and research practice.

Chronic pain is a significant problem for adolescents with juvenile idiopathic arthritis (JIA) (1); it negatively impacts quality of life (2,3). Assessment is the cornerstone for effective management of pain and is essential for evaluating the effectiveness of a wide array of pain-relieving therapies. However, current approaches (ie, questionnaires and paper pain diaries) to evaluating pain in children with chronic arthritis suffer from methodological problems (ie, recall bias, noncompliance) and lack measurement of the multiple dimensions of pain (4,5). Electronic pain diaries using a personal digital assistant (PDA) that employ methods for real-time data capture (RTDC) have been proposed as a new standard for pain measurement, because they circumvent many of these problems (6–10).

RTDC differs from traditional methods of assessment because it is based on sampling moments from peoples’ lives to more precisely capture the variability of the pain experience, and to reduce memory and other biases associated with recalling such experiences (8). RTDC approaches encompass the Ecological Momentary Assessment and Experience Sampling Methods, which are ambulatory monitoring techniques (8). Ecological Momentary Assessment is a broader concept than Experience Sampling Methods, and includes the self-report of subjective behaviours and physiological measures. There are three main types of sampling schedules – random, in which assessments are sampled at random; event-based, in which particular events or occurrences trigger assessments; and time-driven, in which assessments are conducted according to a preset schedule. The sampling design is based on the research question and the nature of the pain problem being examined (8).

Electronic diaries have several advantages over traditional paper-based approaches. They are convenient and pocketsized; they time-and date-sample all entries; they have flexible programming that allows for branching of question administration that is dependent on answers to previous questions (ie, if no pain is reported, then it automatically skips all other painrelated questions); they usually have touch-sensitive screens, making for convenient data entry; they have a relatively long battery life and/or are easily recharged; they have large memory capabilities for extended monitoring; and they can be programmed to upload data at selected times, either through a modem or wireless network (8). Furthermore, electronic diaries have the potential to maximize the compliance (11–13), validity (11–14) and reliability (8,14,15) of pain assessment data, and substantially reduce the time needed for data management (10). Electronic pain diaries hold particular promise for use with children because they are generally more familiar with and confident in the use of computerized technologies than adults (16). To date, however, research studies evaluating electronic pain diaries have primarily been conducted in adults with chronic pain (11–15).

Although electronic diaries are becoming more prevalent, there is very little known about their use in children (4). Electronic diaries need to be comprehensively studied with children, giving particular consideration to how developmental issues (eg, age, stage of development) influence the validity of results, and the usability (intuitiveness of the user interface) and feasibility (ie, compliance and acceptability) of this method in pediatric populations (4). Research on the use of electronic pain diaries in children is limited to a single study in which researchers examined the compliance, accuracy and acceptability of this approach using end-of-day ratings (17). Results indicated substantially higher rates of compliance in children using the electronic (83%) compared with the paper (47%) diary. Moreover, the electronic diary appeared easy to use, efficient and accurate. However, the reliability and validity of the electronic diary, including its ability to detect change, were not evaluated. Also, end-of-day ratings were used, which relied on the patient to recall pain experienced throughout the day. There was also no information on whether usability or feasibility testing of the electronic diary measure was undertaken before use in this research study.

The purpose of our program of research was to develop and evaluate the psychometric properties of a multidimensional electronic pain diary, called the e-Ouch, that employs an RTDC approach for adolescents with JIA. The success of RTDC electronic diary approaches depends on a high degree of participant compliance with the sampling protocol and a user-friendly interface. Noncompliance (12,13,18) and unintuitive user interfaces (19,20) can seriously threaten the validity of the assessment schema. Usability (intuitiveness of the user interface) and feasibility (acceptability and compliance) testing is recommended using iterative phases (eg, testing the electronic diary in natural settings to identify problems, making adjustments to the prototype and retesting) to refine the prototype before electronic diaries are put to research and/or clinical use. Therefore, we developed and evaluated the e-Ouch electronic diary using a sequential phased approach.

In phase 1, we used a qualitative approach with semistructured, audiotaped interviews (n=10) during two iterative cycles to determine the usability of the user interface of the electronic pain diary. Overall, participants found the diary very easy to use, learn and understand, and stated that it was satisfying to complete (19). The first iterative usability testing cycle identified aspects of the user interface that were suboptimal (eg, the visual analogue scale [VAS] slider), which impeded the performance of certain tasks, and adolescents generated ideas on how the diary interface could be improved. This multifaceted usability approach provided important insight on the use of technology by adolescents with JIA and, more specifically, on understanding how adolescents can more effectively use an electronic chronic pain diary.

Feasibility testing is the next step to determine the factors that affect compliance with and acceptability of electronic diary approaches in children with chronic pain. Several researchers have found that children experienced technical difficulties (ie, power failure and PDA malfunctions) that affected the quality of the data (17,21). Furthermore, a recent systematic review found that technical malfunctions were the chief disadvantage of electronic diaries (10). These findings underscore the importance of technical support and troubleshooting support in ensuring the successful implementation of RTDC electronic diary approaches. Therefore, the aim of the present study (phase 2) was to evaluate the feasibility of the newly developed e-Ouch electronic pain diary in adolescents with JIA in terms of compliance and acceptability before determining this measure’s psychometric properties (phase 3) (22). This model of usability and feasibility testing could serve as a template for the development and testing of other electronic patient-reported outcome measures.

METHODS

Patient selection

Participants were drawn from one rheumatology clinic in a university-affiliated pediatric tertiary care centre serving metropolitan Toronto, and central and northern Ontario. Thirteen of the 20 adolescents who participated in an earlier study testing the usability of the e-Ouch electronic pain diary agreed to participate in the pilot testing of the RTDC system (19). This initial sample was purposively recruited to achieve maximum variation in the age of adolescents with JIA; disease severity (mild, moderate and severe disease), as determined by a self-report global disease severity VAS; and pain intensity (mild, moderate and severe pain), as determined by self-reported VAS. The eligibility and exclusion criteria have previously been reported (19).

Study design and procedures

The present study was approved by the Institutional Ethics Review Board and required all adolescents to understand and provide written informed consent for participation. A descriptive study design was used to determine the feasibility of the e-Ouch RTDC approach in terms of compliance and acceptability, and to further refine the prototype. This design approach was based on the concept of a “hermeneutical circle”, as described by Snodgrass and Coyne (23), which is an iterative process of implementing a design, learning and understanding from discussion and feedback, and subsequent design refinement. This iterative rapid design and development approach concentrated on testing the RTDC approach with a focus on enhancing compliance and acceptability.

After obtaining consent, participants completed a general information questionnaire on personal and experiential factors (use and level of comfort with computers), as well as the Health Record Data Collection Form to gather additional demographic and disease-related data. Adolescents were familiar with the e-Ouch electronic diary from their participation in the usability testing study in which they had been trained three months earlier on how to use the diary with standardized pain vignettes (19). None of the adolescents required further demonstration of how to use the diary.

Adolescents were then signalled by an alarm to complete the e-Ouch three times per day (when they woke up, after school and just before bed) for a two-week period. After the first week, adolescents were asked to record their least, average and worst pain intensity, unpleasantness and interference scores for the previous week, as well as current (pain right now) ratings using the Recalled Pain Inventory. At the end of the second week, the adolescents completed the Recalled Pain Inventory, as well as generic and disease-specific quality of life and pain coping questionnaires. These measures were administered because they will be part of the battery of measures used to determine the construct validity of the e-Ouch in future psychometric studies. The adolescents were also asked to complete the e-Ouch Evaluation Questionnaire on the diary, which examined what they liked and did not like about the electronic pain diary. This questionnaire was administered electronically on day 15 at 09:00. Adolescents were instructed to complete it at a convenient time during the day. Following completion of the two-week diary period, the diaries and questionnaires were picked up from the adolescents’ homes or sent by courier back to the research office.

Participants were given a recharger and an e-Ouch manual that outlined when to complete the questionnaires, information about troubleshooting and contact information. The diary entries were automatically stored on the PDA and memory card, and were also sent wirelessly (cellular telephone network with broad geographical coverage) to the research centre. These ratings were stored in a Microsoft Access (Microsoft Corporation, Canada) database that was only accessible by the central research office computer. To ensure confidentiality of the information stored in the diary, all data were encrypted in case the PDA was lost or stolen. Telephone assistance was available (ie, 24 h availability by pager) in case of technical problems. Participants were contacted by telephone on day 1 of the study to address any technical difficulties or questions related to the study protocol. All technical and equipment problems experienced were logged. In addition, participants were contacted weekly (on day 7 and day 14) to ascertain technical difficulties and to remind them to complete the weekly questionnaires. Following completion of the two-week diary period and return of the electronic diary, participants received a $10 movie pass.

Following the first iterative cycle of 10 adolescents, changes were made to the e-Ouch software program based on problems identified from the first phase of pilot testing. The changes made to the software program were then evaluated in the second iterative cycle with seven of the same and three new adolescents participating. Because no further problems were identified in the second iterative cycle, the study stopped after the second cycle had been completed and analyzed.

Measures

Demographic and medical data:

Data on personal (age, grade level, use and comfort level with computers) and medical status (disease onset subtype, current medications) were collected using the general information questionnaire and Health Record Data Collection Forms, respectively. These forms were developed by the researchers.

e-Ouch electronic pain diary:

The multidimensional electronic pain diary was developed to obtain three daily pain ratings and used a signal-contingent approach (audible alarms at fixed times with 30 min compliance windows). The diary measured the sensory-discriminant (pain intensity), affective-motivational (pain unpleasantness) and cognitive-evaluative (pain’s interference with aspects of health-related quality of life (HRQL) adapted with permission from the Brief Pain Inventory – Short Form) (24) dimensions of pain. It also measured other symptoms commonly experienced by children with JIA (ie, stiffness and fatigue) using a VAS in an electronic format (ie, 5 cm; 0 to 100 metric in units of 0.5 mm) (25). The Tungsten W (Palm Inc, USA) was used for electronic data collection and has wireless capability. All other computer functions (calendars, address book, games and internet) of the unit were locked out. The software, which managed the screen displays and then tabulated and stored the user responses to the displayed scales, was developed using AppForge (AppForge Inc, USA). AppForge was used to custom design a software program for the electronic diary called GraalPad (26). The specific components and usability testing of the e-Ouch electronic pain diary have previously been described (19). It was not possible to return to a previous screen to review or modify a response.

e-Ouch Evaluation Questionnaire:

To elicit adolescents’ perceptions of their experience with e-Ouch and problems that were encountered with its use, the e-Ouch Evaluation Questionnaire was administered through the electronic diary on day 15 by signalling participants at 09:00 and asking them to complete it at a convenient time during the day. The content of the questions elicited what they found easy or hard to use; found easy or difficult to understand; liked or disliked; thought about how long it took to complete the diary; thought about how much it interfered with activities or friends; would like to change; thought about how long they would be willing to use the diary in the future; and thought about any problems they had encountered while using the electronic pain diary.

Data analyses

The quantitative data from the questionnaires on the personal and experiential factors were coded, scored and entered into the SPSS database (SPSS Inc, USA) (27). Data from the e-Ouch (eg, pain scores, log-on and -off times) were uploaded into an Microsoft Access database and then imported into the SPSS database. These data were analyzed to determine measures of central tendency and the distribution of values. Compliance was calculated by examining the observed and expected number of diary entries completed overall for each phase. Compliance was defined as 100% when data were entered by participants three times a day for 14 days. The Wilcoxon signed-rank test was used to determine whether there were differences in compliance between the two phases, as well as by time of day and week in each phase. Data from the e-Ouch Evaluation Questionnaire were used to examine adolescents’ likes and dislikes (acceptability) of the electronic pain diary. The level of significance was set at P<0.05 for all tests.

RESULTS

Sample selection and participant characteristics

Adolescents who met the study criteria were invited to participate. These adolescents were part of a purposive sample who participated in a previous study testing the user interface of the electronic diary (19). A total of 13 adolescents participated in the two phases of pilot testing – 10 in phase 1, and seven of those who participated in phase 1 plus three new participants in phase 2. Three participants from phase 1 did not want to do the diary again for another two weeks due to holidays.

The mean (± SD) age of the sample was 12.70±2.58 years (range nine to 18 years) and the majority of participants were female (84.6%). The sex of the participants was representative of the rheumatology subspecialty population (28). Over one-half (77%) of the participants were in public school (ie, grades 1 to 8). Ninety-two per cent of adolescents had a computer at home, and 100% reported being comfortable or very comfortable using a computer. These findings are in accordance with recent data revealing that 88% of 15-year-old Canadian students have at least one computer at home and are comfortable using the computer (29).

Illness characteristics

The mean duration of illness was 4.74±4.88 years (range 0.55 to 11.9 years). The most common disease-onset subtype was polyarthritis (60%), followed by oligoarticular arthritis (30%); one adolescent had psoriatic arthritis (10%). The mean disease severity rating was 37.25±23.87 mm on a 100 mm VAS. Medication use ranged from no medications to five medications per adolescent, with an average of two medications. The most common arthritis medications were methotrexate (50%), followed by etanercept (30%), prednisone (20%) and nonsteroidal anti-inflammatory drugs (20%).

Characteristics of pain ratings

Means and SDs for each of the pain indexes assessed during the two phases are outlined in Tables 1 and 2, respectively. On average, participants reported mild to moderate pain intensity and unpleasantness, and mild interference with aspects of HRQL over the course of the two-week study periods. In terms of interference with HRQL, pain had the most interference with walking and had the least impact on sleeping in both phases. Participants typically reported mild to moderate levels of stiffness in both phases, mild to moderate fatigue in phase 1 and mild fatigue in phase 2 (which coincided with school vacation). On average, participants perceived that they had a fairly high ability to control their pain. The participants, on average, reported two painful joints (range 0 to five) per diary entry, with the most commonly affected joints being the knees. Participants used one-to two-word descriptors (range 0 to 11) to describe their pain; the most frequently used words were ‘hurting’ and ‘uncomfortable’.

TABLE 1.

Descriptive statistics of e-Ouch indexes for phase 1

	Week 1							Week 2
	n	M	Mdn	SD	Min	Max	Skewness	n	M	Mdn	SD	Min	Max	Skewness
Intensity	10	34.50	26.19	26.86	2.72	72.50	0.36	9	22.94	20.56	16.28	1.00	43.44	−0.01
Unpleasantness	10	35.39	32.55	21.18	3.33	90.00	0.68	9	27.70	19.67	21.95	2.64	63.75	0.58
Interference total	10	22.04	20.88	4.24	16.75	29.93	1.04	9	13.84	14.07	4.13	8.34	19.73	−0.03
Activities	10	24.72	14.10	25.59	0.44	69.40	1.14	9	17.42	14.29	21.04	0.40	68.75	2.12
Mood	10	22.15	10.14	26.72	1.25	79.25	1.51	9	15.38	10.31	21.78	0.00	70.38	2.45
Walking	10	29.93	19.14	30.33	0.00	81.05	0.75	9	19.73	16.15	22.69	0.00	64.06	1.12
Sleep	10	19.70	14.59	24.80	0.00	85.00	2.35	9	8.34	4.00	11.93	0.00	32.43	1.56
Schoolwork	10	16.75	5.33	26.16	0.00	76.67	1.85	9	12.67	5.00	20.99	0.00	65.71	2.48
Relations	10	20.17	7.22	25.85	1.25	76.00	1.44	9	9.28	0.83	16.69	0.00	47.00	1.90
Enjoyment	10	20.88	9.49	24.02	2.50	77.50	1.75	9	14.07	7.31	19.20	0.00	59.69	1.20
Stiffness	10	32.05	26.08	24.15	8.53	78.00	0.77	9	28.24	15.85	26.23	3.06	77.50	0.99
Tiredness	10	43.51	33.49	35.66	0.83	95.00	0.33	9	31.02	13.13	33.90	0.00	92.81	1.17
Control	10	54.83	54.42	40.88	0.00	100.0	−0.06	9	54.02	66.67	36.29	0.00	100.0	−0.16

M Mean; Mdn Median; Max Maximum; Min Minimum

TABLE 2.

Descriptive statistics of e-Ouch indexes for phase 2

	Week 1							Week 2
	n	M	Mdn	SD	Min	Max	Skewness	n	M	Mdn	SD	Min	Max	Skewness
Intensity	10	30.25	27.20	24.39	1.40	73.9	0.46	10	25.69	27.05	22.47	0.00	63.60	0.35
Unpleasantness	10	33.35	22.45	27.38	4.00	82.5	0.76	10	26.31	20.25	25.95	0.00	78.30	0.87
Interference total	10	10.29	11.29	5.76	1.40	17.19	−0.51	10	18.78	21.53	6.99	6.32	27.01	−0.96
Activities	10	25.42	15.03	30.41	0.00	87.33	1.57	10	22.10	12.95	23.74	0.00	78.33	1.63
Mood	10	23.45	13.89	33.12	0.00	91.44	1.59	10	21.53	13.31	26.97	0.00	89.17	1.98
Walking	10	26.74	17.19	31.71	3.50	91.89	1.57	10	27.01	18.50	27.06	0.00	86.67	1.31
Sleep	10	15.49	1.40	28.92	0.00	92.50	2.52	10	6.32	0.23	10.93	0.00	30.80	1.67
Schoolwork	10	20.03	8.72	31.07	0.00	92.78	1.89	8	13.18	7.50	14.78	0.00	33.14	0.51
Relations	10	20.75	4.52	32.27	0.00	85.00	1.61	8	18.09	17.69	19.01	0.00	55.45	1.03
Enjoyment	10	23.21	11.29	33.49	0.00	92.44	1.64	8	23.23	12.16	29.59	0.00	81.67	1.37
Stiffness	10	32.58	26.99	24.55	3.39	80.67	0.78	10	27.27	33.50	21.84	0.44	60.67	0.03
Tiredness	10	27.97	16.51	31.20	0.00	94.00	1.35	10	25.27	18.34	27.24	0.00	94.28	2.03
Control	10	52.26	60.25	39.38	0.00	100.0	−0.18	9	52.02	54.40	37.51	0.83	100.0	−0.11

M Mean; Mdn Median; Max Maximum; Min Minimum

Compliance

Table 3 outlines compliance by phase. There was no significant difference in overall compliance between phase 1 (73%) and phase 2 (70%). Compliance decreased from week 1 (phase 1 = 79%; phase 2 = 75%) to week 2 (phase 1 = 68%; phase 2 = 66%); however, this decrease was only significant in phase 1 (T=−2.257, P<0.05). There was a significant difference in compliance by time of day in phase 1, with compliance being significantly higher on weekday mornings (84%) than afternoons (72%; T=−2.588, P<0.05) and evenings (71%; T=−2.511, P<0.05). There were no other significant differences in phase 1 during weekend ratings. There were no significant differences by time of day in phase 2. Patients reported that their compliance was affected by the time of data collection (eg, holidays, examinations) and technical difficulties (alarms, wireless connection).

TABLE 3.

Observed and expected rates of compliance by phase

Phase 1	Observed/Expected (%)
	Morning	Afternoon	Evening	Total
Weekend	2.2/4 (55)	3.1/4 (78)	2.6/4 (65)	7.9/12 (66)
Weekday	8.4/10 (84)	7.2/10 (72)	7.1/10 (71)	22.7/30 (76)
Total	10.6/14 (76)	10.3/14 (74)	9.7/14 (69)	30.6/42 (73)
Phase 2	Morning	Afternoon	Evening	Total
Weekend	2.6/4 (65)	2.8/4 (70)	3.1/4 (78)	8.5/12 (71)
Weekday	7.6/10 (76)	6.6/10 (66)	6.9/10 (69)	21.1/30 (70)
Total	10.2/14 (73)	9.4/14 (67)	10/14 (71)	29.6/42 (70)

Acceptability

The majority of adolescents found it easy to remember to complete the diary three times per day. Most adolescents liked the way that the diary looked and found it quick and easy to complete. They also found that the diary was easy to use to rate their pain. The majority of adolescents found that the diary was not overly bothersome to complete three times per day for the two-week study periods, and that it interfered minimally with activities and friends. Almost two-thirds (75%) of the participants were willing to use the diary again for at least two weeks, while the remainder were willing to use it for a longer period of time (six weeks).

Technical difficulties

During phase 1, seven of the 10 participants experienced various technical difficulties with the PDAs that influenced compliance. Following phase 1, changes were made to the e-Ouch software prototype to address these difficulties, including timing of the alarms on the weekend (weekend morning alarms were set 1.5 h later than weekday alarms), timing of evening alarms (moved up 30 min because two of the younger participants went to bed earlier than the preset alarm time), PDA digitizer sensitivity causing skipping to next question without the participant tapping the screen (n=2) and freezing while sending data through the wireless network when the wireless network was unavailable (n=5). When this occurred, participants had to be instructed on how to reset the PDA. In addition, there were several problems with the wireless network, including network failure for a period of 8 h to 10 h during the study; inadvertent cancellation of the wireless plan for one of the units, resulting in data not being sent to the research centre; and creation of an additional e-mail gateway without notification. Given these network difficulties, the data obtained from the wireless network were compared with data stored on the memory card, and some discrepancies were found between them (ie, more data on memory card and units). No problems were identified with the PDA batteries. Changes were made to the software program to correct these problems and they were re-evaluated in phase 2. Changes to the alarms were made on all of the units. The wireless network provider was also contacted regarding the network problems, which were subsequently resolved. None of the technical difficulties experienced in phase 1 arose in phase 2. In addition, no new technical difficulties were identified in phase 2 as a result of the changes made following phase 1.

DISCUSSION

To our knowledge, the present study is the first to evaluate the feasibility (compliance and acceptability) of an RTDC electronic pain diary in adolescents with chronic arthritis. Formative evaluation of the compliance, acceptability and technical difficulties with the e-Ouch electronic pain diary prototype led to the identification and correction of errors within the software package (eg, alarms, run times, etc). Furthermore, some consideration had to be made to address the wireless carrier service, such as when data plans were inadvertently cancelled and an additional e-mail gateway was created without notice. These difficulties were addressed and pilot tested again to ensure that they had been successfully resolved. In addition, this approach enabled us to determine that the e-Ouch electronic pain diary was easy to use and understand, and efficient and satisfying to complete.

Through usability and feasibility testing, we were able to evaluate the user interface and basic functionality of the e-Ouch prototype, modify the prototype and re-evaluate the software program using iterative cycles until a satisfactory product was achieved. It is essential that such an iterative design process, which involves the users during the initial development of the RTDC diary, be used early in the development of such electronic diary measures. This involvement is crucial because, when undertaken early in the design phase, alterations are less expensive (8,19,20). Given the complexity of software programming for RTDC approaches, there is the potential for programming errors that may be hard to detect without extensive pretesting and validation of the questionnaires. Therefore, we propose that this model of usability and feasibility testing with iterative cycles, using both qualitative and quantitative approaches, be employed by researchers developing RTDC electronic measures.

The success of RTDC electronic diary approaches depends on a high degree of participant compliance with the sampling protocol. Noncompliance can seriously threaten the validity of the assessment schema (8,12,13). Little research has been undertaken to determine the compliance with RTDC electronic diary approaches in children. We found that an RTDC electronic diary approach is feasible in terms of compliance with multiple daily entries (three per day) and a two-week study protocol. The overall compliance rates in both phases were fairly high, and are in broad agreement with other studies conducted in children (17,21,30–32) and adults (11–15). However, there are important differences among studies in terms of sampling densities, time frames and study populations.

Palermo et al (17) compared the compliance and acceptability of an end-of-day electronic diary with a paper and pen diary in a study of 60 children with recurrent or chronic headaches and arthritis pain over a seven-day period. Children completed a total of 312 electronic diary entries out of 420 possible diary entries (74%). Children in the e-diary group completed significantly more daily diary entries than children using the paper format. However, compliance was confounded by the differential awareness of being monitored (more intensive training and follow-up in the e-diary group) and feedback about actual compliance. The sampling density in the present study (three times per day) yielded a similar compliance rate to Palermo et al (17), who used end-of-day ratings. This finding does not support the hypothesis that more intensive sampling densities result in low or unacceptable levels of compliance due to irritation with the frequent intrusions into adolescents’ daily routines. Further research on the impact of training, support and other measures (incentives) is encouraged for understanding how the quality of the data collected can be further improved.

As the field of momentary diary research grows, researchers need to know whether the number of ratings per day affects patient compliance with timely completion of the diaries, and whether duration of assessments are well tolerated so as to yield acceptable protocol compliance. We did not find a significant decrease in compliance over the two-week study periods. This was surprising because seven of the participants participated in both phases (four weeks). Similarly, Stone et al (13) found that in adults with chronic pain, overall compliance with a two-week electronic diary protocol was 94%, and was not related to sampling density (three, six or 12 prompts per day) or study duration. Two diary studies in children – one examining the relationship among anxiety, mood and activity, and the other exploring daily challenges with attention deficit hyperactivity disorder – also found that frequent prompts during the day (every 30 min during nonschool hours) over a relatively short period of time (four to seven days) yielded similar compliance rates. It is of utmost importance to conduct further research on the mechanism determining protocol compliance and to determine whether any personal (age, sex, developmental level and temperament) or environmental (setting where completing diary entries, where diaries are placed in home settings) factors affect compliance rates in children.

Compliance was affected by the timing of data collection (ie, during examinations and school holidays) and the timing of preset alarms. Compliance varied throughout the course of the day and across days, being highest during weekday mornings and lowest for morning entries on the weekends. The lower compliance with morning alarms on weekends during phase 1 was mainly due to the timing of preset alarms (every 15 min between 07:30 and 08:00). When this was changed (09:30 to 10:00) and retested in phase 2, compliance improved by 10%. In addition, overall compliance may have been affected by collecting data during the holiday period (end of December). Several of the adolescents noted in the diary that they did not complete the diary on several occasions (busy with activities) and/or for several days because they were away and forgot to take the diary with them. One way around this is to preprogram the diaries around the participants’ schedule during the study period to minimize disruption and enhance compliance. For example, Whalen et al (31) ascertained from parents their child’s scheduled events during the one-week study period (eg, soccer games) and programmed the diaries so that they would not alarm during these events. It is essential to conduct feasibility testing to ensure that the timing of signal prompts does not interfere with activities of daily living (19,20).

Questionnaire responses supported the successful implementation of the e-Ouch electronic diary measure. Participants were not upset by the associated burden or by the degree to which completing the diary interfered with their usual activities. The majority of adolescents found it easy to remember to complete the diary three times per day, liked the way that it looked and found it quick and easy to complete. Spontaneous comments to the evaluation questionnaire were uniformly positive and enthusiastic. Most adolescents enjoyed the opportunity to use the diary to track their pain and were ready to participate in another similar study. This is important, because clinical trials often require several weeks of data collection. Overall, completing the diary entries was not viewed as an unpleasant or overly burdensome task by most adolescents. These findings are in agreement with other studies in children (17,21,30–32) and suggest that, with appropriate procedures (training, telephone follow-up and technical support) and a design that accommodates human factors (signals, user-friendly interface and quick completion), future RTDC studies using similar parameters of sampling densities, study duration and subject population will be well tolerated by most adolescents. Furthermore, electronic diaries have been found to be uniformly preferred over paper diaries in adults and children with chronic pain (12,13,17).

Given that adolescents tend to be more confident users of computerized technologies (eg, computers, computerized games, cell phones), it was not surprising that the adolescents in the present study found the diary easy to learn, use and understand, as well as satisfying to complete. Although electronic diaries have been used to study gastrointestinal symptoms (21), high-risk adolescent behaviours (30), attention deficit hyperactivity disorder (31), anxiety (32) and pain in children (17), there was no mention of whether these measures underwent formal usability or feasibility testing. While the participants in all of these studies found the diaries acceptable and satisfying to use, several of these studies reported that some participants encountered technical problems such as difficulty logging on, as well as modem and PDA malfunctioning (21,32). These difficulties are important to explore, because additional cognitive demands may be placed on adolescents when the screens are complex and unintuitive, or when there are unexpected technical difficulties, which could result in reductions in patient compliance and accuracy of pain ratings.

Compliance was also affected by technical difficulties (eg, alarms, wireless connection) experienced during the present study. Dale and Hagen (10) conducted a systematic review of nine randomized and quasi-randomized controlled trials comparing electronic with paper and pen diaries in terms of feasibility, compliance, data accuracy and acceptability. Five studies reported on feasibility, and all reported technical difficulties with PDA technology. For example, in the study by Palermo et al (17), 14% of the PDAs broke down because of technical problems, with all data being lost and/or unusable. The study also reported several power failures with the units and other PDA malfunctions. Furthermore, one unit was damaged, three stylus pens were broken or lost, and two of the AC adapters needed to be replaced. While electronic diaries have been found to outperform paper and pen diaries in terms of compliance, data accuracy and subject acceptability, technical malfunctions were the main disadvantage with the PDA-based RTDC method. Mechanisms need to be in place to ensure regular backup of data, adequately charged batteries and that technical problems can be resolved quickly when they occur (24 h support). For these reasons, it is crucial to undertake formal usability and feasibility testing of RTDC electronic diaries to uncover these technical difficulties before implementing them in research or clinical practice.

While extensive pretesting and validation of RTDC electronic measures is recommended, few standards or guidelines exist to assist those considering developing and/or using RTDC methods (8). One of the important lessons learned was that it is essential to have a fallback plan for collection of data because software, hardware and network problems are inevitable. We recommend using multiple methods to back up the data (save directly onto the unit and a memory card), because wireless network issues may result in data being lost. Second, it is crucial that the quality of the data from the units themselves be compared with that received from the wireless network and examined for any discrepancies (ie, missing data). Finally, subject compliance reflects not only technical success (electronic diary functions as required in the field), but also whether the participants understood the training, how to use the diary and when to provide data. Therefore, technical success is necessary, but it is not the only prerequisite to subject compliance because other factors can influence this. We recommend the development of a standardized method to train subjects on how to use the diary, as well as the provision of 24 h technical support. Future research and the development of standards are required in the area of usability and feasibility testing methods to guide researchers in the development of these innovative technologies so that the potential of electronic RTDC approaches can be fully recognized.

A few potential limitations of the present study need to be addressed. First, the study was from one tertiary pediatric care centre and the sample was relatively small, which can be considered a threat to the generalizability of the results. With a representative sample of end-users, typically involving as few as five participants per cycle, the majority of problems and issues can be identified. It has been estimated that a single initial cycle of design-evaluation-redesign can lead to as much as a 10-fold reduction in usability and feasibility problems (33). Second, we used the same patients that participated in the usability testing phase, which may have influenced their compliance with the study protocol. In general, it is recommended that the prototype changes be tested in a new representative sample of participants (23). However, we were not able to do this, because the next phase of testing involved two large studies to evaluate the psychometric properties of this new measure and our target population was limited.

CONCLUSIONS

Findings of the present study underscore the feasibility (compliance and acceptability) of the e-Ouch as a multidimensional prospective daily diary measure for use with adolescents who have JIA and chronic pain. However, the advantages of electronic diaries are only realized if researchers and clinicians take the necessary steps to ensure quality assurance in its implementation. Researchers developing RTDC electronic diaries to assess and track chronic pain and other symptoms need to consider usability and feasibility testing of these diaries before they are released for general use or evaluated in expensive randomized clinical trials. Software and design validation are important elements for a successful RTDC approach using PDAs. Usability and feasibility testing in naturalist settings can uncover costly design flaws of the program early on. Pilot testing revealed aspects of the e-Ouch software and wireless network interface that were suboptimal, affecting overall compliance and acceptability. We recently finished the first series of studies testing the construct validity and feasibility of the electronic diary in adolescents with arthritis (22). Other studies are currently under way to modify the e-Ouch diary and establish its psychometric properties in a variety of recurrent and chronic pain populations (eg, sickle cell disease, headache). This will determine the ability of the e-Ouch electronic diary to detect change in a wide range of pharmacological, physical and psychological treatment interventions in children with chronic illness.