Abstract
BACKGROUND:
Parents of children with cancer must learn and retain crucial information necessary to provide safe care for their child. Smartphone applications (apps) provide a significant opportunity to meet the informational needs of these parents. We aimed to develop, refine, and evaluate a smartphone app, informed by the Children’s Oncology Group (COG) expert consensus recommendations, to support the informational needs of parents of children with cancer.
PROCEDURE:
We employed a user-centered iterative mixed-methods approach in two phases (prototype development/refinement and pilot testing). We engaged parents and clinicians in evaluating the app via qualitative interviews and standardized tools that measured app quality (Mobile Application Rating Scale [MARS]), usability (System Usability Scale [SUS]), and acceptability (System Acceptability Scale [SAS]). We evaluated early usage patterns after public release.
RESULTS:
Thirty-two parents and 17 clinicians participated. Mean (± standard deviation [SD]) scores for app quality, usability and acceptability were: MARS: 4.5 ± 0.7 on a 5-point scale; SUS: 86.7 ± 23.8 on a 100-point scale; and SAS: superior (61%); similar (28%); inferior (11%) to written materials. Qualitative findings largely confirmed the quantitative data. Downloads of the app during the first year following public release have exceeded 5,000.
CONCLUSIONS:
The COG KidsCare app prototype was found to be of high quality and received high usability and acceptability ratings. Further testing is needed to determine app effectiveness in improving parental knowledge regarding care of children with cancer.
Keywords: mHealth, patient-family education, pediatric oncology
INTRODUCTION
For parents of children with cancer, learning the crucial information necessary for providing safe care for their child at home is often overwhelming,1 and accurate retention of that information can be challenging, given the shock and distress related to the child’s diagnosis.2 The Children’s Oncology Group (COG) developed expert consensus recommendations that emphasize the importance of standardizing and streamlining informational content provided to parents when a child is diagnosed with cancer, as well as the need to tailor methods of delivering this information to address differences in parental learning styles.1 Implementation of these COG recommendations,3 was limited to conventional methods of information delivery (i.e., face-to-face teaching and provision of written materials), which may not translate to sustained learning for some parents.4
In settings that require intense parent education, provision of education solely during hospitalization can be suboptimal, with reduced recall of crucial information.5–7 Parents have difficulty with retention of information that is taught only once,5,8 and often seek out health information via smartphone (rather than relying on hospital-provided written materials) to address knowledge gaps regarding their child’s care.9,10 Smartphone applications (hereafter referred to as “apps”) provide a significant opportunity to meet the informational needs of parents across a wide spectrum of literacy levels, economic circumstances, information needs, and learning preferences,11,12 and allow incorporation of graphics, video and other smartphone features, thus broadening applicability of the app across varied populations.13 COG had called for provision of its educational materials in multiple formats, ranging from low to high-technology options, to make information available in the modality most aligned with parents’ preferred learning styles and literacy/health literacy levels.1,14,15 Development of an app addresses the call for a high-technology option for delivering essential information to parents of children with cancer.
While several apps have been developed for children and adolescents with cancer to target symptom management and other aspects of quality of life16, few are designed for parents17–20, and none address expert consensus recommendations for provision of information to parents of children with cancer. We hypothesized that a smartphone app designed for parents of children with cancer and informed by COG expert consensus recommendations had the potential to readily provide parents with access to reliable information crucial to their child’s care. Thus, we aimed to i) develop a smartphone app prototype that would deliver high-quality, relevant information in a functional and engaging format for parents of children with cancer; ii) evaluate quality, usability, and acceptability of the smartphone app during its development and pilot testing, and refine the app based on feedback received; and iii) evaluate early app usage patterns following its public release.
METHODS
Study Design
The study was conducted in two phases: Prototype development/refinement (Phase 1) and pilot testing (Phase 2) (Figure 1). We used a mixed methods convergent parallel design with concurrent collection and analysis of qualitative and quantitative data during each phase (Supplemental Figure 1).21 We incorporated feedback from key stakeholders (parents, nurses, clinicians, app developers) using an iterative approach that included assessments of app content, design, functionality, and usability/ acceptability throughout the development and testing process. We used the COG expert consensus recommendations for provision of education to parents of newly-diagnosed pediatric oncology patients1 as the framework to guide app design and content (Supplemental Table 1).
Figure 1.
Smartphone app development process
Participants
App testing was conducted at a tertiary children’s hospital that cares for approximately 120 newly diagnosed pediatric oncology patients per year. Research participants included both parents and clinicians. We employed purposive sampling with the goal of obtaining heterogeneity reflective of the larger representative groups of parents and clinicians.
Parents/guardians (hereafter referred to as parents) were eligible to participate if they spoke English and provided home care to a child who was: i) diagnosed with cancer at least 1 year (Phase 1 participants) or 2 months (Phase 2 participants) prior to study participation; and ii) age 0–17 at the time of cancer diagnosis. Parents who participated in Phase 2 of the study were also required to own a smartphone. Targeted sample size was 8–12 parents in Phase 1 and 20–30 parents in Phase 2.
Eligible clinicians (Phase 1 only) included pediatric oncology nurses, physicians and psychosocial professionals at the tertiary children’s hospital. Targeted sample size was 12 to 20 clinicians.
The study was approved by the Institutional Review Board at the University of Alabama at Birmingham (IRB-160819007). All participants provided informed consent, were enrolled between 1/28/19 and 2/25/20, and did not receive compensation.
We also engaged a Stakeholder Advisory Board (SAB; defined as persons or groups with expert knowledge and/or interest in project outcomes)22 to serve in a consultative capacity throughout app development. The SAB was comprised of eleven members: two parents of children with cancer, a pediatric oncologist, a pediatric psychologist, and five pediatric oncology nurses, including two outside consultants representing the COG Nursing Discipline. Parents and outside consultants on the SAB received compensation according to stakeholder engagement principles.23
Procedures
Phase 1
During Phase 1, the research team engaged key stakeholders, including study participants, SAB members, and app developers (programmers and graphic designers from a mobile app company [myocv.com] experienced in developing mobile apps in both iOS and Android platforms for non-profit organizations) in an iterative process. Initially, the research team identified necessary content for inclusion in the app by examining COG expert consensus recommendations1 and related literature2,24–26. SAB meetings were convened to confirm selection of app content and determine preferred layout and aesthetic features. The SAB reviewed the initial design and provided feedback to the app developers, including a strong recommendation to add features beyond informational content that would keep parents engaged with the app (e.g., journaling, appointment tracking). The app developers then produced the functional prototype.
The app prototype was alpha tested (to identify initial design problems) with Phase 1 study participants. Parents and clinicians met with a research team member to evaluate the app prototype, and were presented with scripted scenarios directing them to perform various tasks using the app on a phone designated for research. To evaluate the app’s functionality and intuitiveness of use, participants were observed using the app as they attempted to complete these tasks. Qualitative interviews were conducted with each participant following their alpha testing session, to ascertain overall feedback regarding prototype content, functionality, and design. Aggregate findings from alpha testing were then presented to the SAB to obtain further feedback and recommendations for refinement, which were communicated to the app developers.
After further prototype refinement, the app was beta tested (to assess overall quality) with the same group of parents and clinicians who participated in alpha testing. During beta testing, participants met with a research team member and explored the app over the course of one hour, during which time they were directed to think of questions that they had when their own child was newly diagnosed (or, for healthcare providers, questions they have been asked by parents of newly diagnosed children), and to use the app to attempt to address those questions. Participants were also asked to explore overall app content, functionality, design, and intuitiveness. Following the beta testing session, participants completed study questionnaires and participated in qualitative interviews. Aggregate findings from beta testing were presented to the SAB for additional feedback and recommendations conveyed to the app developers for further refinement, including addition of content and features. At the end of Phase 1, a mature app prototype was ready for pilot testing.
Phase 2
During Phase 2, the prototype was pilot tested (to evaluate and further refine the app in the clinical setting). Parents participating in this phase met with a research assistant and completed a short app orientation/tutorial. Parents were then instructed to use the app at home over the subsequent two weeks in conjuction with the written materials that they currently used to care for their child. Following home use of the app, parents completed the study questionnaire and qualitative interview. Aggregate findings from pilot testing were presented to the SAB to obtain additional recommendations for app refinement, which were conveyed to the app developers.
Data Collection
Questionnaires
Participant demographic characteristics and current cellular telephone and Internet use (based on Pew Smartphone Use Survey)28 were collected via questionnaire at study enrollment. Parent participants also self-reported their child’s diagnosis, time from diagnosis, and current treatment status (on/off therapy), and completed a single-item health literacy screen.29
App assessment tools are summarized below and further described in Supplemental Methods.
Phase 1 study participants completed the Mobile Application Rating Scale (MARS)30 via questionnaire immediately following beta testing. This scale is designed to assess the quality of mobile health apps during the design/development phase; two versions were employed: MARS31 (for clinician rating) and uMARS30 (for end-user rating). Each tool consists of four objective and one subjective scale; the clinician version also includes a perceived impact scale. Ratings range from 1 (inadequate) to 5 (excellent); mean overall ratings average 3.31
Phase 2 study participants completed the System Usability Scale (SUS) and System Acceptability Scale (SAS; Supplemental Figure 2) via questionnaire immediately following pilot testing. The SUS is a validated 10-item app usability scale32; a score of 70 is considered “good”.33,34 The SAS is a 5-item investigator-developed scale measuring app acceptability and preferences (written vs. app) regarding format of educational materials.
Qualitative Interviews
Qualitative interviews (Interview Guides in Supplement) were conducted by the research team with participating clinicians and parents after alpha and beta testing to obtain perspectives regarding app quality, content, functionality, and aesthetics; and after pilot testing to obtain perspectives regarding app usability and acceptability. All participants were asked to describe problems encountered while using the app, and suggestions for improvement were elicited. Qualitative interviews were audiotaped and transcribed by study staff.
Data Analysis
Quantitative measures
Statistical analyses were conducted using SPSS Version 28 (Chicago, IL). Descriptive statistics (means, medians, standard deviations, and ranges for continuous data; frequencies and percentages for categorical data) were used to summarize the quantitative measures. Chi-square or Fisher’s exact tests were used to determine differences between the groups, with a two-sided P-value of <0.05 considered to be statistically significant.
Qualitative measures
Transcripts of all qualitative interviews were coded and qualitatively analzyed by two study team members (HH and WL) for relevant categories and themes on an Excel platform using content analysis methodology.35 Thematic saturation was achieved, as evidenced by no new major categories identified in the final two interviews conducted within each phase.
RESULTS
Participant Characteristics
Parents
A total of 42 eligible parents were approached and 32 (76%) enrolled (12 of 18 [67%] in Phase 1, and 20 of 24 [83%] in Phase 2). Reasons for refusal included “not a good time” (2 parents) and “not interested” (8 parents). The majority of parents were 30–49 years of age (75%); most were female (81%) and non-Hispanic white (88%); over half (53%) had completed a 2-year college degree or higher; and all had adequate health literacy based on the single-item screen. Their children were primarily diagnosed with leukemia (72%) an average (±standard deviation [SD]) of 16± 9 months prior to study participation, and most (84%) were receiving cancer treatment at the time of study participation (Table 1).
Table 1.
Participant characteristics
| Characteristic | Entire cohort N (%) | Phase 1 N (%) | Phase 2 N (%) |
|---|---|---|---|
| PARENTS (Phases 1 and 2) | 32 (100) | 12 (100) | 20 (100) |
| Child’s diagnosis | |||
| Leukemia | 23 (72) | 6 (50) | 17 (85) |
| Lymphoma | 3 (9) | 3 (25) | 3 (15) |
| Solid/central nervous system tumor | 6 (19) | 3 (25) | 0 (0) |
| Time since child’s diagnosis (months) | |||
| Mean ± Standard Deviation (SD) | 16 ± 9 | 25 ± 4 | 11 ± 7 |
| Median (range) | 14 (2–32) | 25 (20–32) | 10 (2–30) |
| Child currently receiving treatment? | |||
| Yes | 27 (84) | 7 (58) | 20 (100) |
| No | 5 (16) | 5 (42) | 0 (0) |
| Parent age group (years) | |||
| <30 | 5 (16) | 1 (8) | 4 (20) |
| 30 to <40 | 14 (44) | 2 (17) | 12 (60) |
| 40 to <50 | 10 (31) | 7 (58) | 3 (15) |
| 50 to <60 | 3 (9) | 2 (17) | 1 (5) |
| Parent sex | |||
| Female | 26 (81) | 10 (83) | 16 (80) |
| Male | 6 (19) | 2 (17) | 4 (20) |
| Parent relationship to patient | |||
| Mother | 25 (78) | 9 (75) | 16 (80) |
| Father | 5 (16) | 1 (8) | 4 (20) |
| Other | 2 (6) | 2 (17) | 0 (0) |
| Parent race/ethnicity | |||
| Non-Hispanic white | 28 (88) | 11 (92) | 17 (85) |
| Non-Hispanic black | 4 (13) | 1 (8) | 3 (15) |
| Hispanic/other | 0 (0) | 0 (0) | 0 (0) |
| Parent educational level | |||
| High school diploma or equivalent | 3 (9) | 2 (17) | 1 (5) |
| Some college or post-high school trade school | 12 (38) | 3 (25) | 9 (45) |
| 2-year college degree | 4 (13) | 3 (25) | 1 (5) |
| 4-year college degree | 5 (16) | 2 (17) | 3 (15) |
| Graduate/professional degree | 8 (25) | 2 (17) | 6 (30) |
| Parent health literacy | |||
| Adequate | 32 (100) | 12 (100) | 20 (100) |
| Marginal/limited | 0 (0) | 0 (0) | 0 (0) |
| Household income | |||
| < $30K | 5 (16) | 2 (17) | 3 (15) |
| $30K-$74K | 11 (34) | 5 (42) | 6 (30) |
| $75K or higher | 16 (50) | 5 (42) | 11 (55) |
| CLINICIANS (Phase 1 only) | 17 (100) | 17 (100) | -- |
| Role | -- | ||
| Physician | 4 (24) | 4 (24) | -- |
| Nurse practitioner | 4 (24) | 4 (24) | -- |
| Nurse | 6 (35) | 6 (35) | -- |
| Psychosocial team member | 3 (18) | 3 (18) | -- |
| Age group (years) | -- | ||
| <30 | 5 (29) | 5 (29) | -- |
| 30 to <40 | 6 (35) | 6 (35) | -- |
| 40 to <50 | 3 (18) | 3 (18) | -- |
| 50 to <60 | 2 (12) | 2 (12) | -- |
| 60 or older | 1 (6) | 1 (6) | -- |
| Sex | -- | ||
| Female | 13 (77) | 13 (77) | -- |
| Male | 4 (24) | 4 (24) | -- |
| Race/ethnicity | -- | ||
| Non-Hispanic white | 15 (88) | 15 (88) | -- |
| Non-Hispanic black | 0 (0) | 0 (0) | -- |
| Asian | 1 (6) | 1 (6) | -- |
| Hispanic | 1 (6) | 1 (6) | -- |
| Educational level | -- | ||
| High school diploma or equivalent | 0 (0) | 0 (0) | -- |
| Some college or post-high school trade school | 0 (0) | 0 (0) | -- |
| 2-year college degree | 2 (12) | 2 (12) | -- |
| 4-year college degree | 5 (29) | 5 (29) | -- |
| Graduate/professional degree | 10 (59) | 10 (59) | -- |
Clinicians
All 17 eligible clinicians who were approached enrolled. Clinicians were a mix of nurses (35%), physicians (24%), nurse practitioners (24%), and psychosocial team members (including social work, child life, chaplaincy) (18%). Over half (53%) were 30–49 years of age; most were female (77%), non-Hispanic white (88%), and had a 4-year college or professional/graduate degree (88%) (Table 1). There were no significant differences between parent vs. clinician report of cellular telephone and Internet use (Supplemental Table 2).
Findings: Phase 1 (App Development/Refinement)
MARS:
Mean (±SD) MARS scores obtained at the completion of beta testing across all six rating scales ranged from 4.2±0.9 (subjective quality) to 4.8±0.4 (information), with an overall quality rating of 4.5±0.7 out of 5. Perceived impact, rated only by clinicians, received a mean score of 4.6±0.5 (Table 2).
Table 2.
Mobile Application Rating Scale (MARS) scores (Phase 1 – clinicians and parents)
| MARS scores | ||||||
|---|---|---|---|---|---|---|
| Domain | Clinicians (MARS)‡ N=14 | Parents (uMARS)‡ N=8 | Combined Scores N=22 | |||
| Mean±SD | Median (range) | Mean±SD | Median (range) | Mean±SD | Median (range) | |
| Engagement | 4.3±0.7 | 4 (2–5) | 4.3±0.6 | 4 (3–5) | 4.3±0.7 | 4 (2–5) |
| Functionality | 4.5±0.5 | 5 (4–5) | 4.7±0.5 | 5 (4–5) | 4.6±0.5 | 5 (4–5) |
| Aesthetics | 4.5±0.6 | 5 (3–5) | 4.5±0.6 | 5 (3–5) | 4.5±0.6 | 5 (3–5) |
| Information | 4.7±0.5 | 5 (4–5) | 4.8±0.4 | 5 (4–5) | 4.8±0.4 | 5 (4–5) |
| Subjective quality | 4.2±1.0 | 4 (1–5) | 4.3±0.7 | 4 (3–5) | 4.2±0.9 | 4 (1–5) |
| Perceived impact | 4.6±0.5 | 5 (4–5) | N/A* | N/A* | 4.6±0.5 | 5 (4–5) |
| Overall rating | 4.5±0.7 | 5 (1–5) | 4.5±0.6 | 5 (3–5) | 4.5±0.7 | 5 (1–5) |
Perceived impact is rated only by clinicians
There was no statistically significant difference between mean MARS scores for clinicians compared to parents
Abbreviations: MARS = Mobile Application Rating Scale; uMARS = end-user version of the Mobile Application Rating Scale; SD=Standard deviation; N/A=Not applicable
Qualitative Findings:
Qualitative findings from the alpha testing phase indicated that while parents and clinicians viewed the app as a potentially helpful tool, they suggested several improvements that encompassed the themes of usability (with a focus on functionality of the app: e.g., “have the healthcare team emergency contact information front and center”), content expansion (with a focus on identifying additional subject matter to enhance parental engagement with the app: e.g., “incorporate links from cancer.gov for specific diseases”), and accessibility (with a focus on the initial parental experience with the app: e.g., “a walk-through or orientation for the parent…like ‘Welcome to this App’”) (Table 3).
Table 3.
Qualitative categories/themes and details from interviews across app development phases
| Category/Theme | Proportion of Interviews | Details from interviews |
|---|---|---|
| Phase 1: Alpha testing (parents and clinicians) | ||
| USABILITY: Lab values/ measurements not intuitively displayed | 64% | - Need the labs to display the same way doctors and nurses talk about them (parent) - Add options to allow conversion to metric units for height (centimeters), weight (kilograms), and temperature (Celsius) (clinician) |
| USABILITY: Emergency information not immediately apparent | 48% | - It would be nice to have the healthcare team emergency contact information front and center (parent) - It would be helpful to have a big red button that you use when you need to call (parent) - Move the navigation buttons to the bottom of screen, with an emphasis on the ‘emergency call’ button (parent) |
| USABILITY: Need for improved appointment tracker functionality | 44% | - You need to be able to scroll to set appointment date/time (parent) - The appointment should save to my regular calendar (parent) |
| USABILITY: Need for expanded storage area for contact information | 36% | - A lot of times someone (like the social worker) will just hand me a card – and do you think I can find that card again? No, I can’t! (parent) - It would be helpful to have a place to keep the pharmacist’s phone number (parent) |
| CONTENT EXPANSION: Inclusion of EMR data | 20% | - It would be great if there could be a medication list in the app (clinician, parent) - Could the labs from the electronic medical record be integrated into the app? (parent) |
| CONTENT EXPANSION: Inclusion of disease-specific information | 12% | - You could incorporate links from cancer.gov for specific diseases, like Hodgkin lymphoma (clinician) |
| ACCESSIBILITY: Language, tutorial | 12% | - Will any of this be available in Spanish? (clinician) - Is there a walk-through or orientation for the parent the first time they run it, like ‘Welcome to this App’? (clinician) |
| Phase 1: Beta testing (parents and clinicians) | ||
| USABILITY: Re-organization of content | 50% | - It would be helpful to organize the information according to time…information you need in the beginning, and then as time goes by (parent) - I think it is very logical, very intuitive (clinician) |
| USABILITY: Improve search bar functionality | 20% | - Adjust search so that all informational content within the app is searched, and search results are labeled with the name of the section and sub-section in which the searched-for information has been located (clinician) |
| CONTENT EXPANSION: COG New Diagnosis Guide and Family Handbook | 10% | - Include all of the information from the Children’s Oncology Group Family Handbook (parent) - Add information from the Children’s Oncology Group New Diagnosis Guide and Family Handbook (clinician) |
| Phase 2: Pilot testing (parents only) | ||
| USABILITY: Ease of access to/ availability of information via app | 100% | “It is the same information [as the written materials], it was just easier to find…going through the app was less time consuming. Easier to find the information.” |
| “I like the app because if you are somewhere…and you’re not with your big binder or your stuff at home…and something happens, you can just look it up on your phone.” | ||
| “I love the emergency room card…I found it very helpful, the fact that it is right there, convenient.” | ||
| “You can click and scroll instead of frantically flipping through pages and trying to find it – it is right there, convenient.” | ||
| “I think it is real easy to use. It actually gave me a reminder last night to come here [to clinic] today…it was very helpful.” | ||
| “Nowadays I have everything on my phone, so it is just so much nicer to have it there…I would prefer the app to the paper.” | ||
| ACCESSIBILITY: Similarity between app and written materials | 65% | “I found [the app and the binder] very similar. I wouldn’t say that there is one better than the other. I mean, I like physical paper but at the same time, driving down the road and thinking of something, the app is better when you don’t have the physical paper in front of you.” |
| ACCESSIBILITY: Preference for written materials over app | 6% | “When it comes to the information that was in the binders, I liked that I could read, highlight, take notes on what the information was…I feel like this information is so important it deserves a book.” |
Following beta testing, parents and clinicians generally indicated that they were pleased with refinements made after the alpha phase and viewed the app favorably, but suggested further refinements, that encompassed the themes of usability (with a focus on structuring by treatment phase: e.g., “organize the information according to time…information you need in the beginning, and then as time goes by”), and content expansion (with a focus on incorporating COG resources: e.g., “add information from the COG New Diagnosis Guide36 and Family Handbook37”) (Table 3). The recommendation to incorporate content from COG was reviewed and endorsed by the SAB, after making a determination that the material was rigorously vetted and of high quality. Permission was obtained from the COG Nursing Discipline to proceed with addition of these COG educational materials into the app prior to commencement of pilot testing.
Findings: Phase 2 (Pilot Testing)
Usability:
Mean±SD (median, range) SUS scores across all five domains ranged from 80.0±13.7 (80, 60–100) for integration of functions to 89.4±11.2 (90, 60–100) for intuitiveness, with an overall usability score of 86.7±23.8 (100, 20–100) (Figure 2a).
Figure 2.
(a) System Usability Scale (SUS) Scores; (b) System Accessibility Scale (SAS) Ratings, Items 1–4 [Parents, Phase 2]
Acceptability:
In comparison to written materials, parents rated the app as the same as or superior to written materials for the following domains: Availability of informational content (100%); Ease of understanding information (95%); Ease of access to information (94%); Overall satisfaction (89%) (Figure 2b). When asked their preference for educational materials about their child’s care, 83% stated that they preferred access to both written and electronic (app) formats, 11% preferred app-only, and 6% preferred written materials only.
Qualitative Findings:
Qualitative findings from Phase 2 largely confirmed the quantitative acceptability data. Most parents indicated they were pleased with usability of the app (e.g., “It is the same information [as the written materials]…just easier to find”); some expressed that accessibility of information was largely similar to written materials (e.g., “I wouldn’t say that there is one better than the other.”), and a few expressed a preference for information in written form rather than in the app (e.g., “I feel like this information is so important it deserves a book.”) (Table 3).
Further App Refinement
Following study completion, the COG Nursing Discipline reviewed the app and consulted with COG leadership regarding its adoption for dissemination to parents across the >220 COG institutions. In preparation for public release, the app was reviewed by the COG Patient Advocacy Committee, its name was changed to COG KidsCare, and funding was secured for ongoing technical maintenance and support. Platforms for both Spanish and French versions were developed to accommodate pre-existing translations of the COG Family Handbook and New Diagnosis Guide, permission was obtained to incorporate illustrations from the COG Family Handbook and videos developed by the National Cancer Institute for parents of children with cancer, and instructions and tutorials were developed. Additionally, cloud data storage capability was secured and necessary privacy legal disclaimers were obtained. All modifications were incorporated into the app prior to its public release in August 2021 (Figure 3, Supplemental Figure 3).
Figure 3.
Screenshots of COG KidsCare App features at time of public release
Early Usage Patterns
Since its release in August 2021, the COG KidsCare App has been downloaded more than 5,900 times; the majority on the iOS platform (74%). Over 2,900 app accounts have been created. These accounts can be accessed by multiple caregivers per patient to facilitate information-sharing. Additionally, an average of 5,003 app sessions are initiated weekly, an indicator of continued engagement by those downloading the app (Supplemental Table 3). The most commonly accessed app features include Newly Diagnosed (32%), During Treatment (18%), Journal (18%), Emergency Room Card (8%) and Appointments (8%) (Supplemental Figure 4).
DISCUSSION
We used an iterative process that engaged key stakeholders in development of a smartphone application, guided by the COG expert consensus recommendations framework1, to deliver high-quality information to parents of children with cancer. Although smartphone ownership was not required for participation in the first study phase, we found that all participating parents and clinicians were smartphone owners, reflective of the ubiquitous presence of smartphones among adults in the United States.28 During alpha and beta testing, several critical content and design modifications were identified by participants and incorporated into the prototype prior to pilot testing. Upon completion of beta testing, the app prototype received consistently high ratings across domains on the MARS tool, indicating high perceived app quality by both parents and clinicians. After incorporation of most parent/clinician recommendations from alpha and beta testing, results from pilot testing indicated that parents found the app highly usable and acceptable. Qualitative interviews largely confirmed the quantitative findings across both study phases. Early usage patterns following public release indicate robust uptake.
We found that engaging a Stakeholder Advisory Board that included both parents and clinicians resulted in valuable feedback during the design phase, and led to early design changes (e.g., addition of journaling and appointment tracking features) instrumental in increasing user engagement. This strategy has been similarly used in other apps to successfully gauge end-user needs early in the project.19,38,39
We also found that engagement of both parents and clinicians in the alpha and beta testing phases provided critical feedback for app design. Parents indicated a strong preference for displaying emergency contact information on the app’s “home” screen, moving navigation buttons to the bottom of the screen, and developing a one-touch dialing function that would allow parents to quickly place a call in an urgent situation. Clinicians identified the need to convert journal data from metric to conventional units (e.g., for recording height, weight, temperature), as hospitals often record these measures in units with which parents may be unfamiliar. Clinicians also identified the need for incorporation of disease-specific information; this was addressed by adding links to the patient versions of the PDQ® Cancer Information Summaries for specific childhood cancers, located on cancer.gov website.40 Furthermore, both parents and clinicians identified the need to incorporate content from the COG Family Handbook into the app. As reported in the development of other smartphone apps, feedback during the alpha and beta testing phases is crucial in identifying design issues and further refining the app prior to pilot testing.20,41
Findings from pilot testing largely confirmed that design changes made during the alpha and beta phases of development enhanced app quality and resulted in high usability and accessibility ratings. We also found that although most parents were enthusiastic about the app and found crucial information more accessible with its use – particularly when away from home – not all parents shared this view. A small minority of parents (6%) indicated that they preferred receiving only written information to guide their child’s care, while the large majority (83%) indicated that they preferred both written materials and the app, and a small (11%) proportion of parents indicated that they preferred solely app-based information. Thus, despite the considerable investment in time and resources to maintian both formats, COG has committed to making both written and app-based information available to accommodate parental preferences. Additionally, the COG expert consensus recommendations1 provide the framework to guide future research targeted at improving parental knowledge regarding the necessary care of their child with cancer across the continuum of care, with a focus on understanding how improved knowledge may impact cancer-related outcomes (e.g., therapy-related toxicity, adherence to home-administered disease-directed therapy) that are sensitive to the care that the child receives at home.
We acknowledge some study limitations, including its single institution design, and composition of the parent sample – whose children primarily had leukemia – potentially limiting generalizability; however, we included representatives from other COG institutions on the SAB. Additionally, the primary outcome of this study (mature app) is intended to be tested within the larger COG setting. We also acknowledge that the study population was primarily non-Hispanic white with at least some college education and adequate health literacy. We did not collect demographic characteristics of those refusing study participation, but it is possible that those from minority groups and of lower literacy actively chose not to participate in the study. It is also possible that those less interested in technology chose not to participate. Additionally, due to funding constraints during initial app development, enrollment was limited to English-speaking participants, which may have also resulted in under-representation of minority groups. Testing of the Spanish and French platforms will need to be incorporated into the early phases of subsequent testing.
In conclusion, we employed a user-centered design to develop and test the COG KidsCare smartphone app, informed by COG expert recommendations, and found the app to be of high quality with high usability and acceptability ratings. Further testing is needed to determine the effectiveness of the app in improving parental knowledge regarding the care of children with cancer.
Supplementary Material
Acknowledgements:
Supported by a research grant from the Kaul Pediatric Research Institute at Children’s of Alabama, by NCTN Operations Center Grant U10CA180886 from NCI/NIH to the Children’s Oncology Group (COG), and by the St. Baldrick’s Foundation. The content is solely the responsibility of the authors and does not necessarily represent the official views of the organizations providing funding for this study, including the National Institutes of Health. We thank the parents and clinicians of the Division of Pediatric Hematology Oncology at the University of Alabama at Birmingham who contributed to this study, the COG Nursing Discipline Steering Committee, the COG Leadership (Doug Hawkins, Deborah Crabtree), the COG Patient Advocacy Committee (Kristy Devine, Kim Buff, Kamala Allen, Cat Paciente, Peggy Kulm, Kathleen Ruccione), the COG Foundation (Daniel Woods), COG Foundation Legal Counsel (John Mussman, Dror Futter), Terese Winslow (for medical illustrations), Eric Uchalik (for vector illustrations), the National Cancer Institute (Carol Sienché, Cindy Lollar, Nita Seibel) for provision of the video content included in the app, Holly Simpson of the Institute for Cancer Outcomes and Survivorship at UAB for logistical/administrative support, and the app development team at OCV Apps (Logan Howell, Maclain Aslinger, Josh Carter, Chase Watkins, Eric Halverson), Auburn, Alabama, who supported design, development and implementation of the COG KidsCare App.
Abbreviation Key
- App
Application
- COG
Children’s Oncology Group
- SAB
Stakeholder Advisory Board
- MARS
Mobile Application Rating Scale
- SUS
System Usability Scale
- SAS
System Acceptability Scale
- SD
Standard deviation
Footnotes
Conflicts of Interest: The authors declare no conflicts of interest.
Prior presentations: Presented, in part, at the Association of Pediatric Hematology/ Oncology Nurses Annual Conference (virtual), September 2, 2020.
REFERENCES
- 1.Landier W, Ahern J, Barakat LP, et al. Patient/family education for newly diagnosed pediatric oncology patients. J Pediatr Oncol Nurs. Nov/Dec 2016;33(6):422–431. doi: 10.1177/1043454216655983 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Rodgers CC, Stegenga K, Withycombe JS, Sachse K, Kelly KP. Processing information after a child’s cancer diagnosis-how parents learn. J Pediatr Oncol Nurs. Nov/Dec 2016;33(6):447–459. doi: 10.1177/1043454216668825 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Landier W, York JM, Wadhwa A, et al. A Structured Discharge Education Intervention for Parents of Newly Diagnosed Pediatric Oncology Patients. J Pediatr Hematol Oncol Nurs. Jan 18 2023:27527530221140058. doi: 10.1177/27527530221140058 [DOI] [PubMed] [Google Scholar]
- 4.Davis DW, Logsdon MC, Vogt K, et al. Parent Education is Changing: A Review of Smartphone Apps. MCN Am J Matern Child Nurs. Sep/Oct 2017;42(5):248–256. doi: 10.1097/NMC.0000000000000353 [DOI] [PubMed] [Google Scholar]
- 5.Kamimura-Nishimura K, Chaudhary V, Olaosebikan F, et al. Does Nursery-Based Intensified Anticipatory Guidance Reduce Emergency Department Use for Nonurgent Conditions in the First Month of Life? A Randomized Controlled Trial. Int J Pediatr. 2016;2016:8356582. doi: 10.1155/2016/8356582 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Jonsson L, Hallstrom I, Lundqvist A. “The logic of care” - parents’ perceptions of the educational process when a child is newly diagnosed with type 1 diabetes. BMC Pediatr. Oct 20 2012;12:165. doi: 10.1186/1471-2431-12-165 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Amar-Dolan LG, Horn MH, O’Connell B, et al. “This Is How Hard It Is”. Family Experience of Hospital-to-Home Transition with a Tracheostomy. Ann Am Thorac Soc. Jul 2020;17(7):860–868. doi: 10.1513/AnnalsATS.201910-780OC [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Logsdon MC, Davis DW, Stikes R, Ratterman R, Ryan L, Myers J. Acceptability and initial efficacy of education for teen mothers. MCN Am J Matern Child Nurs. May-Jun 2015;40(3):186–92. doi: 10.1097/NMC.0000000000000126 [DOI] [PubMed] [Google Scholar]
- 9.Logsdon MC, Davis D, Eckert D, et al. Feasibility of Two Educational Methods for Teaching New Mothers: A Pilot Study. Interact J Med Res. Oct 08 2015;4(4):e20. doi: 10.2196/ijmr.4583 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Orr T, Campbell-Yeo M, Benoit B, Hewitt B, Stinson J, McGrath P. Smartphone and Internet Preferences of Parents: Information Needs and Desired Involvement in Infant Care and Pain Management in the NICU. Adv Neonatal Care. Apr 2017;17(2):131–138. doi: 10.1097/ANC.0000000000000349 [DOI] [PubMed] [Google Scholar]
- 11.Gordon EJ, Sohn MW, Chang CH, et al. Effect of a Mobile Web App on Kidney Transplant Candidates’ Knowledge About Increased Risk Donor Kidneys: A Randomized Controlled Trial. Transplantation. Jun 2017;101(6):1167–1176. doi: 10.1097/TP.0000000000001273 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Albanese-O’Neill A, Schatz DA, Bernhardt JM, Elder JH. Educational Needs and Technological Preferences of Fathers of Youth With Type 1 Diabetes. Diabetes Educ. Apr 2016;42(2):209–19. doi: 10.1177/0145721716628649 [DOI] [PubMed] [Google Scholar]
- 13.Athilingam P, Osorio RE, Kaplan H, Oliver D, O’Neachtain T, Rogal PJ. Embedding Patient Education in Mobile Platform for Patients With Heart Failure: Theory-Based Development and Beta Testing. Comput Inform Nurs. Feb 2016;34(2):92–8. doi: 10.1097/CIN.0000000000000216 [DOI] [PubMed] [Google Scholar]
- 14.U.S. Department of Education. Adult Literacy in the United States. National Center for Education Statistics. 2019. [Google Scholar]
- 15.Broderick J, Devine T, Langhans E, Lemerise AJ, Lier S, Harris L. Designing health literate mobile apps. presented at: National Academy of Sciences; 28 Jan 2014; Washington, D.C. [Google Scholar]
- 16.Mehdizadeh H, Asadi F, Mehrvar A, Nazemi E, Emami H. Smartphone apps to help children and adolescents with cancer and their families: a scoping review. Acta Oncol. Jul 2019;58(7):1003–1014. doi: 10.1080/0284186X.2019.1588474 [DOI] [PubMed] [Google Scholar]
- 17.Berntsen E, Babic A. Cherry: mobile application for children with cancer. Stud Health Technol Inform. 2013;192:1168. [PubMed] [Google Scholar]
- 18.Wang J, Yao N, Shen M, et al. Supporting caregivers of children with acute lymphoblastic leukemia via a smartphone app. CIN: Computers, Informatics, Nursing. 2016;34(11):520–527. doi: 10.1097/cin.0000000000000265 [DOI] [PubMed] [Google Scholar]
- 19.Slater PJ, Fielden PE, Bradford NK. The Oncology Family App: Providing Information and Support for Families Caring for Their Child With Cancer. J Pediatr Oncol Nurs. Mar/Apr 2018;35(2):94–102. doi: 10.1177/1043454217741874 [DOI] [PubMed] [Google Scholar]
- 20.Mueller EL, Cochrane AR, Campbell ME, Nikkhah S, Miller AD. An mHealth App to Support Caregivers in the Medical Management of Their Child With Cancer: Co-design and User Testing Study. JMIR Cancer. Mar 16 2022;8(1):e33152. doi: 10.2196/33152 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Cresswell JW, Plano Clark VL. Designing and conducting mixed methods research, 2nd ed. Sage Publications, Inc.; 2011. [Google Scholar]
- 22.Burton H, Adams M, Bunton R, Schroder-Back P. Developing stakeholder involvement for introducing public health genomics into public policy. Public Health Genomics. 2009;12(1):11–9. doi: 10.1159/000153426 [DOI] [PubMed] [Google Scholar]
- 23.Deverka PA, Lavallee DC, Desai PJ, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J Comp Eff Res. Mar 2012;1(2):181–194. doi: 10.2217/cer.12.7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Haugen MS, Landier W, Mandrell BN, et al. Educating Families of Children Newly Diagnosed With Cancer. J Pediatr Oncol Nurs. Nov/Dec 2016;33(6):405–413. doi: 10.1177/1043454216652856 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Rodgers CC, Laing CM, Herring RA, et al. Understanding effective delivery of patient and family education in pediatric oncology: A systematic review from the Children’s Oncology Group. Journal of Pediatric Oncology Nursing. Nov/Dec 2016;33(6):432–446. doi: 10.1177/1043454216659449 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Withycombe JS, Andam-Mejia R, Dwyer A, Slaven A, Windt K, Landier W. A Comprehensive Survey of Institutional Patient/Family Educational Practices for Newly Diagnosed Pediatric Oncology Patients. J Pediatr Oncol Nurs. Nov/Dec 2016;33(6):414–421. doi: 10.1177/1043454216652857 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.U. S. Department of Health and Human Services. What and why of usability. U.S. Department of Health and Human Services, https://www.usability.gov/. Accessed 9-7-17, https://www.usability.gov/get-involved/blog/2006/03/complete-picture-with-usability-testing.html [Google Scholar]
- 28.Pew Research Center. Mobile Fact Sheet. Accessed 11/15/2022, 2022. http://www.pewinternet.org/fact-sheet/mobile/
- 29.Wallace LS, Rogers ES, Roskos SE, Holiday DB, Weiss BD. Brief report: screening items to identify patients with limited health literacy skills. J Gen Intern Med. Aug 2006;21(8):874–7. doi: 10.1111/j.1525-1497.2006.00532.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth. Jun 10 2016;4(2):e72. doi: 10.2196/mhealth.5849 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. Mar 11 2015;3(1):e27. doi: 10.2196/mhealth.3422 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.U. S. Department of Health and Human Services. System Usability Scale (SUS). U. S. Department of Health and Human Services; https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html. Accessed 9-18-17, 1986. [Google Scholar]
- 33.Nielsen J The usability engineering life cycle. Computer. 1992;25:12–22. [Google Scholar]
- 34.Cox CE, Wysham NG, Kamal AH, et al. Usability Testing of an Electronic Patient-Reported Outcome System for Survivors of Critical Illness. Am J Crit Care. Jul 2016;25(4):340–9. doi: 10.4037/ajcc2016952 [DOI] [PubMed] [Google Scholar]
- 35.Krippendorff K Content Analysis: An Introduction to its Methodology. Sage Publications; 2003. [Google Scholar]
- 36.Sullivan J, Tomlinson K, eds. Children’s Oncology Group family handbook: New diagnosis guide. Children’s Oncology Group; 2018. [Google Scholar]
- 37.Murphy K, ed. The Children’s Oncology Group family handbook for children with cancer. Children’s Oncology Group; 2011. [Google Scholar]
- 38.Crosby LE, Ware RE, Goldstein A, et al. Development and evaluation of iManage: A self-management app co-designed by adolescents with sickle cell disease. Pediatr Blood Cancer. Jan 2017;64(1):139–145. doi: 10.1002/pbc.26177 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.Marsolo K, Shuman W, Nix J, Morrison CF, Mullins LL, Pai AL. Reducing Parental Uncertainty Around Childhood Cancer: Implementation Decisions and Design Trade-Offs in Developing an Electronic Health Record-Linked Mobile App. JMIR Res Protoc. Jun 26 2017;6(6):e122. doi: 10.2196/resprot.7523 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.National Cancer Institute. PDQ® Cancer Information Summaries: Pediatric Treatment. National Cancer Institute, https://www.cancer.gov/publications/pdq/information-summaries/pediatric-treatment. Accessed 18 Nov 2022, https://www.cancer.gov/publications/pdq/information-summaries/pediatric-treatment [Google Scholar]
- 41.Feldman AG, Moore S, Bull S, et al. A Smartphone App to Increase Immunizations in the Pediatric Solid Organ Transplant Population: Development and Initial Usability Study. JMIR Form Res. Jan 13 2022;6(1):e32273. doi: 10.2196/32273 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.