First experience with electronic feedback of the Psychosocial Assessment Tool in pediatric cancer care

Abstract

Purpose

The Psychosocial Assessment Tool (PAT) is a brief family screener, identifying families at universal or elevated risk for psychosocial problems. This study aimed to determine the feasibility and usability of the electronic PAT (ePAT) in pediatric cancer care.

Methods

Eighty-six parents of newly diagnosed children with cancer (0–18 years) agreed to participate and registered at the website www.hetklikt.nu (58%). Seventy-five families completed the ePAT at approximately 1 month post-diagnosis. Answers were transformed into an electronic PROfile (PAT ePROfile) and fed back to the psychosocial team. Team members completed a semi-structured evaluation questionnaire. Feasibility was measured as the percentage of website registrations, completed ePATs, and PAT ePROfiles reviewed or discussed by the team. Usability included perceived match of the PAT ePROfile with the team’s own risk estimation, perceived added value, and perceived actions undertaken as a result of the PAT ePROfile.

Results

Feasibility was 70% for website registration, 87% for completed ePATs, 85% for PAT ePROfile reviewing, and 67% for ePROfile discussion. Team members reported that the PAT ePROfile matched with their own risk estimation (M = 7.92, SD = 1.88) and did not provide additional information (M = 2.18, SD = 2.30). According to the team, actions were undertaken for 25% of the families as a result of the PAT ePROfile. More actions were undertaken for families with elevated risk scores compared to universal risk scores (p = .007).

Conclusions

Implementation of the ePAT seems generally feasible, but it is not always clear how this screener adds to current clinical practice. Strategies should be developed together with team members to improve quick exchange of ePAT results and allocate care according to the needs of the families.

Introduction

Pediatric cancer survival rates have improved markedly in the past four decades [1]. Consequently, while in the past family psychological adjustment focused on the almost certain death of the child, today the focus is on adjustment to diagnosis and cancer treatment and adapting to uncertain and sometimes unpredictable health outcomes [2]. Children have to deal with the often lengthy and demanding treatment regimens, and the burden of daily care has an impact on the whole family. Thus, being diagnosed with childhood cancer remains an obvious stressful and potentially traumatic event for the entire family [3, 4]. Even though it is widely acknowledged that families experience significant distress, especially at diagnosis and during treatment [5–7], a substantial amount of research has shown that in the long term, adaptive psychosocial adjustment is common for the majority of affected families [6–9]. Nevertheless, a considerable proportion of the families experience severe emotional distress associated with pediatric cancer diagnosis, with the potential to result in major short- or long-term psychosocial problems [6, 7, 10]. For example, previous research has demonstrated that perceived stress in newly diagnosed children with cancer predicts children’s symptoms of anxiety and depression at 6 and 9 months post-diagnosis [11]. Furthermore, it has been shown that initial parental distress and negative coping styles can predict later parental distress and psychological functioning [5, 12]. Early screening for psychosocial distress and pre-existing problems in family members is an important first step to identify families at risk for problems and to provide psychosocial care according to the specific needs of the families and therefore aiming to prevent or target late psychosocial effects.

The internationally established pediatric oncology psychosocial standards of care endorse systematic psychosocial screening of children and their families in pediatric cancer care [13, 14]. Psychosocial screening should be brief, with minimal burden for children and their families, with the aim to identify families at risk for ongoing distress, such that they can be efficiently directed towards evidence-based treatments [15]. An example of a brief and well-validated screener for psychosocial risk in families of children with cancer [16–20] and other disease populations [21–23] is the Psychosocial Assessment Tool (PAT) [17]. The PAT is based on risk and protective factors that are known to be associated with distress in families of children with cancer, such as child characteristics (e.g., age, temperament, behavior); the illness and treatment; family structure and resources (e.g., being a single or teenage parent); financial problems; psychopathology of family members (e.g., parents or siblings); family functioning and social support; and parental cognitions about and stress reactions to the course and outcomes of treatment [24]. The PAT is based on the pediatric psychosocial preventative health model (PPPHM) [25] and classifies families into universal, targeted, or clinical categories of risk for developing psychosocial problems.

Studies on the effect of using the PAT showed it to be an efficient screener in the USA [17, 19], Canada [20], Australia [16], and the Netherlands [18]. For instance, several studies showed that by screening with the PAT at diagnosis, families could be identified as universal, targeted, or high risk for psychosocial problems [16–20]. Furthermore, PAT risk scores were shown to predict the intensity of social work services [19, 26] for the US pediatric cancer population. Also, a Canadian pilot study found that screening with the PAT resulted in reduced levels of family risk and an improved child HRQoL [27]. These findings address the importance of systematic and evidence-based screening in pediatric cancer care.

The English version of the PAT has been translated into Dutch [18], and, as part of the implementation of an electronic system for the routine monitoring of patient-reported outcomes (PROs) in children with a chronic disease or cancer [28, 29], an electronic version of the PAT (ePAT) was developed. This way, families could easily complete the PAT at home or in the clinic, and their answers could be transformed into an electronic PROfile (PAT ePROfile; see Fig. 1). Research shows that patients generally prefer electronic measures [30], which typically yield similar results to those of identical paper-pencil questionnaires [31]. The ePAT as described above has been found to be a reliable and valid screener for psychosocial risk in Dutch families of children with cancer, and families indicated it to be an acceptable and non-burdensome questionnaire to complete approximately 1 month post-diagnosis [18].

Fig. 1.

Example of electronic PAT PROfile with specific risk areas per subdomain (only general information and two PAT subdomains displayed), summary of total family psychosocial risk, and contribution of subdomains to total PAT score. Note. This figure was reproduced with permission from the Center for Pediatric Traumatic Stress (CPTS) at Nemours Children’s Health System © 2016–2017. All rights reserved. The PPPHM image may not be reproduced in any form for any purpose without the expressed written permission of CPTS. To obtain permission to use or reproduce the most recent version of the PPPHM, please contact CPTS at psychosocialassessmenttool@nemours.org

Now that the psychometric properties of the PAT have been quite well established, the next step is to incorporate this assessment into the flow of screening and psychosocial care. The aims of the current study were to (1) determine the feasibility of the use of the ePAT in Dutch clinical practice approximately 1 month post-diagnosis in terms of website registration, number of completed ePATs, and reviewing and discussion of PAT ePROfiles by the psychosocial team, (2) evaluate the usability of the PAT ePROfile in terms of perceived match with psychosocial team risk estimation, added value, and actions undertaken as a result of having the information from the PAT ePROfile available as indicated by the team, and (3) determine possible differences in feasibility and usability for families with a universal versus an elevated (targeted or a clinical) risk score.

Methods

Procedure

This study was conducted in the context of a larger study (IMPROVE), examining the routine monitoring of electronic patient-reported outcomes and screening for family distress in pediatric oncology practice [18, 28]. The Medical Ethics Committee of the Academic Medical Center in Amsterdam approved the study. The IMPROVE study consisted of two study periods. Participants were recruited between June 2012 and December 2013 for the first study period (ePAT validation phase), in which the ePAT was only assessed for research purposes and the psychosocial team did not see ePAT results [18], and the cohort described in the current manuscript took part in the second study period (ePAT implementation phase; January 2014–January 2015), in which PAT ePROfiles were sent to and reviewed by the psychosocial team of the hospital department. All children with a new diagnosis of cancer from the Emma Children’s Hospital AMC in Amsterdam, the Radboud University Medical Center in Nijmegen, and the VU University Medical Center in Amsterdam were approached to participate. Inclusion criteria were (1) confirmed first diagnosis of cancer in a child aged 0–18 years, (2) receiving treatment with curative intent, and (3) sufficient knowledge of Dutch to complete a family psychosocial screening questionnaire. Parents of eligible patients were approached by the study coordinators (two research psychologists) within 1 to 3 weeks after diagnosis either during inpatient hospitalization, during outpatient clinic visit, or by phone and were given both verbal and written information about the study. After written informed consent was provided, families were asked to register online at the KLIK website (Dutch acronym for quality of life in clinical practice; www.hetklikt.nu) [29] and received access to a secure section of the website. One parent per family completed the ePAT preferably within 1 month post-diagnosis.

At the start of the second study period (ePAT implementation phase), all members of the psychosocial team (i.e., psychologist, social worker, child life specialist) undertook a 1-h training [34, 35] concerning the theoretical background and use of the PAT ePROfile. Each member had access to a secure section of the website, which only allowed them to see the results for patients of their own hospital department. The development, functions, and security of the website have been thoroughly described previously [29, 32, 33]. After families completed the ePAT, psychosocial team members received an e-mail with a link to the PAT ePROfile. Attached to the e-mail was a “staff ePAT evaluation questionnaire” with the request to jointly complete the questionnaire within 2 weeks after they received the link.

Participants

Figure 2 describes the inclusion of participants. A total of 218 children were diagnosed with cancer between January 2014 and January 2015. According to our inclusion criteria, 167 families were eligible for participation. Of the eligible families, 147 were approached and 123 verbally agreed to participate (enrollment rate 84%). Thirty-seven families subsequently did not register on the website, and 11 families withdrew from the study after they had registered on the website. Finally, 86 families verbally agreed and registered on the website (response rate 59%), of which 75 completed the ePAT. Responders and non-responders did not differ with regard to age (p = .479), gender (p = .265), or diagnosis type (p = .462). Table 1 presents the background characteristics of the participants.

Fig. 2.

Flowchart of participants

Table 1.

Patient and parent sociodemographic and disease characteristics (N = 75)

Variables	N/M	%/SD
Patient characteristics
Gender (male)	49	65.3
Age (years)	8 .97	5.12, range 0.19–17.68
Cancer diagnosis
Leukemias/lymphomas	37	49.3
Solid tumors	26	34.7
Brain/CNS tumors	11	14.7
Other	1	1.3
Parent characteristics
Gender (male)	22	29.3
Age (years)	41.53	6.69, range 24.54–53.88
Nationality (Dutch)	73	79.3
Marital status
Married/partnered	70	93.3
Separated/divorced	3	4.0
Single	2	2.7
Education
Low	6	8.0
Intermediate	18	24.0
High	50	66.7
Missing	1	1.3

The italic entries represent the format of the headings above the table (N= upright, M= italic; %= upright, SD= italic)

M mean, SD standard deviation

Measures

Psychosocial Assessment Tool (PAT 2.0)

The PAT [17] consisted of seven subscales: Family Structure and Resources, Family Social Support, Family Problems, Parent Stress Reactions, Family Beliefs, Child Problems, and Sibling Problems. Each subscale had 3–15 items, which were scored dichotomously as indicators of risk (score 1) or no risk (score 0). Subscale scores were generated by calculating the proportion of items on the scale endorsed as “risk.” A total score (ranging from 0 to 7) was calculated by summing the subscale scores and considered as universal (<1), targeted (≥1 and <2), or clinical (>2). Internal consistency of total PAT score was acceptable (α = .70).

Staff ePAT evaluation questionnaire

The staff ePAT evaluation questionnaire was assessed to measure the feasibility and usability of the PAT ePROfile in clinical practice. After viewing the PAT ePROfile, staff completed a five-item semi-structured evaluation form on their experiences with the use of the PAT ePROfile. The items included were as follows: (1) Has the PAT ePROfile of this family been reviewed by the psychosocial team in the past month (“yes”/“no,” if no: “please specify”)?; (2) Has the PAT ePROfile of this family been discussed between members of the psychosocial team in the past month (“yes”/“no,” if no: “please specify”)?, follow-up questions (only if question 1 or 2 had been answered with “yes”); (3) Did the information from the PAT ePROfile match with your own risk assessment of this family (0 “not at all”/10 “very much”)?; (4) Did the PAT ePROfile give you any new information about this family (0 “not at all”/10 “very much”)?; and (5) Did the information from the PAT ePROfile influence the actions you undertook/are going to undertake for this family (“yes”/“no,” if yes: “please specify”).

Statistical analysis

All statistical analyses were conducted using the Statistical Package for the Social Sciences (SPSS) version 21. Descriptives were calculated for the PAT total scale (% of universal, targeted, and clinical families). The close-ended questions of the staff ePAT evaluation questionnaire were analyzed descriptively using frequencies, percentages, means, and standard deviations. Open-ended questions were categorized into themes to provide supplementary information on close-ended questions. Chi-square tests (reviewing and discussion of PAT results, actions undertaken) and t tests (perceived match of PAT risk score with psychosocial team assessment, added value of PAT results) were used to analyze possible differences in feasibility and usability outcomes for families with a universal versus an elevated (targeted or clinical) risk score.

Results

PAT distribution of risk scores

Of the 75 families, 68% were classified in the universal group, 26.7% in the targeted group, and 5.3% in the clinical group.

Feasibility of PAT ePROfile feedback in clinical practice

As shown in Fig. 2, 70% (N = 86) of the families who provided informed consent registered on the website, of which 87% (N = 75) completed the ePAT approximately 1 month post-diagnosis (M = 34.65 days post-diagnosis, SD = 15.30, range = 10–116). The following reasons were indicated by families that did not want to participate: “too much effort,” “too many other things on our mind,” “no desire to complete extra questionnaires,” and “we do not want to commit to anything extra than just the standard treatment.” In 85% of the cases (N = 64), the PAT ePROfile results had been reviewed by the psychosocial team. Results were discussed by the psychosocial team for 67% of the families (N = 50). Reasons declared by the psychosocial team for not reviewing or discussing the PAT ePROfile results were as follows: logistics (i.e., too busy and/or no team meeting), psychosocial team members did not think it was necessary to discuss results because of a universal PAT score, no additional information derived from the PAT that had to be shared with the team, and that the family was not known to anyone from the psychosocial team.

Usability of PAT ePROfile feedback to the psychosocial team

Where the PAT ePROfile had been reviewed or discussed by the psychosocial team, the psychosocial team reported that the information from the PAT matched well with their own risk estimation of the families (M = 7.92, SD = 1.88), and it did not give much additional information on what was already known (M = 2.18, SD = 2.30). The team indicated that the information from the PAT ePROfile had influence on 25% of the actions (N = 16) that they undertook for the families. Actions undertaken as mentioned by the team were as follows: “extra incentive to monitor this family closely,” “discussing the PAT ePROfile with parents and asking more explicitly if they need support,” “ask parents about sibling problems in greater detail,” “keep in mind family’s financial problems,” “social work is going to meet with parents,” “asking the pediatric oncologist if extra support is needed,” and “also asking father about coping/depression.”

Differences between ePAT universal, targeted, and clinical risk scores

When looking at results separately for families with universal and elevated ePAT scores, there appeared to be no influence of risk score on the likelihood the PAT ePROfile was reviewed (universal: N = 42 out of 52 (82%), elevated: N = 22 out of 24 (92%), χ ²(1) = 1.131, p = .288) or discussed (universal: N = 36 out of 51 (71%), elevated: N = 14 out of 24 (58%), χ ²(1) = 1.103, p = .294) by the psychosocial team. Also, no significant differences between PAT risk classifications were found regarding the perceived match (universal: M = 7.88, SD = 1.81, elevated: M = 8.00, SD = 2.08, p = .815) with psychosocial team risk estimation of the family or perceived added value of PAT results (universal: M = 1.80, SD = 2.15, elevated: M = 2.95, SD = 2.46, p = .068). Results of the chi-square test indicated that there was a significant higher (χ ²(1) = 7.18, p = .007, φ = .34) percentage of actions undertaken by the team as a result of an elevated (N = 10 out of 22, 46%) ePAT score compared to a universal ePAT score (N = 6 out of 41, 15%).

Discussion

While systematic screening for family distress has become widely endorsed in recent years as a standard of care [13, 34], questions remain about how it can best inform clinical practice and whether it adds value in terms of directing services to families most in need. The present study looked at the use of a PAT ePROfile in psychosocial care approximately 1 month post a pediatric cancer diagnosis. The majority of families fell into the universal risk group and could thus be seen as resilient, while a minority had elevated risk scores, meaning they were at risk for developing (severe) psychosocial problems. This finding is in line with previous pediatric cancer studies [6, 7, 10, 16, 18, 20, 35] and with the pediatric psychosocial preventative health model as a conceptual framework supporting the PAT [25].

Results from the present study showed that with the different steps involved for screening in clinical practice and the stressful environment that accompanies a pediatric cancer diagnosis, it is a challenge to have families completing a screening instrument shortly after the diagnosis. Yet, once families manage to register on the website within 1 month post-diagnosis, completion of the ePAT and reviewing of PAT ePROfile results by the psychosocial team are generally feasible. Other studies also showed that it was feasible to assess the PAT within 1 or 2 months [36] or even within 48 h [37] post-diagnosis. The latter study [37] was performed in a controlled experimental setting, while the study conducted by McCarthy et al. [36] was dependent on the implementation of the PAT by social workers, which gives more of a realistic representation of clinical practice. Barriers mentioned for (early) assessment by social workers were a delayed confirmed cancer diagnosis, families not returning the form, and extra workload. The results from the current study also show that, even with having an electronic system available, barriers for successful implementation are present. In our study, timing and workload issues relating to the assessment and discussion of the ePAT were at stake, since the average completion time was approximately 1 month post-diagnosis and only 67% of the available PAT ePROfiles were discussed by the psychosocial team. Yet, issues concerning the use of the PAT ePROfile can probably not be explained by logistical barriers alone. It could have been helpful if the healthcare professionals received direct advice for certain psychosocial interventions relating to the PAT screening outcome [27]. Other factors could also be at stake, such as possible cognitions of the psychosocial team members about the use and implementation of the PAT. For example, di Battista et al. [38] showed that PAT summary scores were found most useful by nurses and least useful by social workers. Future studies could look into positive and negative cognitions of healthcare providers and what can be done to overcome possible resistance to change.

According to psychosocial team members, the information received from the PAT ePROfile in general matched with their own estimation of risk from the family. However, in previous studies that used more “objective” measures comparing PAT and team risk estimations, only moderate concordance was found between staff (oncologist, nurse, social worker, psychologist, child life specialist) and parent report of psychosocial risk [28, 35, 39]. Furthermore, the timing of the PAT might have influenced the experienced added value by the psychosocial team members in our study. It can be speculated that the advantage of using the PAT might increase when the PAT is assessed earlier (i.e., within 1 or 2 weeks post-diagnosis). Strikingly, even though the psychosocial team members in this study indicated that the information from the PAT ePROfile did not give them a lot of extra information at 1 month post-diagnosis, still for 25% of the families, the team indicated that the PAT ePROfile results influenced the actions that were undertaken for the families. Previous studies on the effect of the use of the PAT showed that the intensity of psychosocial care was provided according to the intensity of the risk profiles of the families [19, 39]. The present study supports previous findings, as the psychosocial team indicated that they more often undertook actions as a result of an elevated PAT ePROfile, compared to a universal PAT ePROfile.

There are several limitations of the current study. First, even though the amount of patients that provided informed consent for this study was quite high, the actual amount of eligible families that completed the ePAT within 1 month post-diagnosis (51%) was rather low compared to other studies [16, 17, 19, 20] where the PAT was assessed in pediatric oncology (70–89%). Second, the majority of the parents who completed the ePAT were highly educated and were married or partnered. Families with lower socioeconomic status may have had more issues with completing an electronic version of the PAT, which could have possibly led to a lower response rate, and it might have influenced the representativeness of the study population. Future research could therefore study if response rates go up (notably in lower-educated families) if families have the choice for either online or paper-pencil versions of the PAT. Third, to save time and burden for psychosocial team members, we asked them to jointly complete the “staff ePAT feasibility and usability questionnaire,” so this was a summary score of how the PAT was perceived by the psychosocial team. Ideally, to prevent measurement bias, the questionnaire should be completed separately by each team member. Finally, we were not able to reliably distinguish how well the team already had responded to family issues before the ePAT was completed and if additional actions would have been necessary because of unmet needs. That is, the participating hospitals were in a transition phase from moving from a paper to an electronic patient information record system at the time of the study, so we did not have reliable data on what specific psychosocial services and the intensity of services provided to families within the first month post-diagnosis. This research should be extended with a randomized controlled trial that includes more “objective” data on psychosocial health service delivery from medical records or hospital databases, such that can be identified whether PAT ePROfile results lead to a higher team knowledge and actions regarding the specific issues reported by the families.

The internationally established pediatric oncology psychosocial standards of care encourage systematic psychosocial screening in pediatric cancer care [13, 14]. Psychosocial screening has the goal of identifying families at risk for ongoing distress, such that they can be efficiently directed towards evidence-based psychosocial care [15]. However, the role of the medical and psychosocial team in regard to psychosocial screening remains unclear. This study showed that some additional value of the PAT ePROfile was experienced on top of the already performed psychosocial screening by the psychosocial team. Clear guidelines should be developed on how to best implement psychosocial standards—such as systematic screening for psychosocial problems—in pediatric oncology practice and how to specifically map psychosocial interventions to screening results in pediatric oncology practice.

To conclude, this study looked at the feasibility and usability of the ePAT at 1 month post a pediatric cancer diagnosis. It showed that, within the context of clinical practice, it is challenging to screen for psychosocial risk factors within 1 month post-diagnosis—especially for families with lower socioeconomic backgrounds. Once families have registered on the website, implementation appears to be generally feasible and the PAT ePROfile seems to match well with the teams’ estimation. However, the timing of 1 month post-diagnosis might be too late to inform the team about ePAT results. More research is needed on the perceived value of the PAT ePROfile by psychosocial team members and on how the PAT ePROfile can best inform the care provided to the families of children with cancer.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest. S.A. Schepers has full control of the primary data and agrees to allow the journal to review the data if requested.