Development and psychometric testing of an instrument to measure the patient’s experience of external radiotherapy: The Radiotherapy Experience Questionnaire (RTEQ)

Highlights

• •A new instrument to measure the patient’s comfort and experiences during RT.
• •The instrument gains the patients’ perspectives of the RT procedures.
• •RTEQ has a possible application for evaluation of newly introduced techniques.

Abstract

Background

The patient’s perception of external radiotherapy (RT) procedures and equipment is important to evaluate as a complement to endpoints such as treatment outcome and reproducibility. There is a lack of a proper, psychometrically robust instrument to evaluate the patient’s comfort and experience of the external RT procedure. Hence, this study aimed to develop and test an instrument to measure the patient’s experience during external RT.

Material and Methods

A preliminary 34-item questionnaire was generated from research literature, expert consultations and patient interviews, and it was distributed to patients (n = 825) at 8 RT units in Sweden. The answers were subjected to item analysis and reduction by using exploratory factor analysis. The reliability of the final questionnaire was evaluated using Cronbach’s alpha. Mean scale scores were compared across gender, length of RT and treatment area.

Results

Most items were highly skewed towards positive responses. Scree plot analyses of the 34-item correlation matrix identified six underlying themes explaining 68% of the total variance. After item reduction, the 6 themes explained 73% of the variance in a 23-item questionnaire. Cronbach’s alpha was satisfactory for all themes (between 0.79 and 0.9). Significant differences between treatment areas were found for two scales: situational unease and situational repose.

Conclusion

The RT Experience Questionnaire is a tentatively valid and reliable instrument to measure how patients experience the external RT session process and the environment in the treatment room.

Introduction

Radiotherapy (RT) is one of the main treatment modalities for patients with cancer, and about 50% of all cancer patients receive external RT [1]. The technical environment and equipment play a major role in the patient’s treatment experience in the RT setting [2]. Patients express difficulties tolerating external RT, worry over the medical equipment, lack of information and express feelings of isolation and anxiety during external RT [3], [4], [5], [6], [7], [8]. This may sometimes lead to disruption of the treatment [9], which may have negative consequences for the outcome.

A reproducible position and restricted and/or controlled motion of the patient is a prerequisite for high-precision radiotherapy (RT) [10], [11]. Improved precision of RT is achieved by minimizing uncertainties in the treatment chain, starting from the definition of the target volume, patient immobilization and minimization of, or active control of, motion, and highly conformal treatment techniques and modalities. Therefore, different devices that restrain patient movements are often used. The devices used can be specially designed pillows, biting blocks or thermoplastic face masks, both for standardized or individual use [11]. New immobilization systems, technical interventions and patient set-up are carefully examined with respect to treatment delivery, high precision, motion management and reproducibility [10], [12], [13], [14]. Few studies include patient-reported outcomes such as comfort or experiences of the set-up [15].

There is a shift in health care towards person-centered care (PCC), where the patient is encouraged to take an active part in the care process. In the past, in the biomedical model, the patient was seen as a passive recipient of care [16]. PCC contains both personalized care and an environment that encourages shared decision making and letting the person behind the illness give their viewpoints of symptoms and behavior [17]. As Mullaney et al. [2] conclude, healthcare providers should rethink the patient experience within healthcare regarding environmental impact on health, especially within the external RT setting where technical environment and equipment play a major role in the treatment experience. Therefore, it is important to explore how patients experience the external RT process and the RT environment in order to improve the RT experience.

Patients who receive external RT for head and neck (H&N) cancer, immobilized in thermoplastic face masks, have reported the restraint to be one of the worst experiences during the RT period [5]. Ideally, evaluations of technical interventions and workflows should include the patient’s perception of the procedures and equipment.

A range of instruments has been used to evaluate different aspects of the patient’s experience in cancer care [8], [17], [18], [19], [20], [21], [22]. Research in RT settings has mainly focused on the patient’s anxiety level, using instruments such as State-Trait Anxiety Inventory (STAI) [18] and The Hospital Anxiety and Depression Scale (HADS) [19]. Other aspects that have been measured are: delivered information (European Organization for Research and Treatment of Cancer Quality of Life Information module, EORTC QLQ-INFO25), [20], Information need (RT Information Needs Scale), [21], Concerns about RT (RT Concerns Scale), [21], preparedness for cancer treatment (Cancer Treatment Scale, CaTS) [22], and person centeredness (Person-centered climate questionnaire, PCQ) [17]. In Japan an RT Categorical Anxiety Scale was developed [8], which focuses on anxiety, adverse effects of RT, environment of RT and treatment effects of RT. Despite these, in our view, there is a lack of a proper psychometrically robust instrument that is specifically designed to measure the patient’s experience of the RT process and the environment in the treatment room. There is a need for a quick and easy treatment specific tool that allows comparison between units, different workflows and to evaluate newly introduced techniques, from a patient perspective.

The aim of this study was to develop an instrument to measure the patient’s experience of the RT session process and the environment in the treatment room: The Radiotherapy Experience Questionnaire (RTEQ), and to evaluate its psychometric properties.

Material and methods

This study had two main phases: the construction of a preliminary 34-item questionnaire, and then psychometric evaluation to successively optimize the questionnaire down to 23 pertinent items [23]. Ethical approval was obtained from the Regional Ethical Review Board in Umeå, Sweden. (Dnr 2014/40-31).

Phase 1: Questionnaire construction

First, instruments that are used to measure patients’ experiences in the RT setting, and which are published in peer-review journals, were reviewed. The following 7 questionnaires included topics relevant to measure patient experiences in RT care and were used for item generation: State-Trait Anxiety Inventory (STAI) [18]; The Hospital Anxiety and Depression Scale (HADS) [19]; EORTC Quality of Life Information module, EORTC QLQ-INFO25 [20]; Information need (RT Information Needs Scale) [21]; concerns about RT (RT Concerns Scale) [21]; Cancer treatment scale, CaTS [22]; the person-centred climate questionnaire; PCQ) [17] and RT Categorical Anxiety Scale [8].

Second, to identify the most important issues considering patient experiences during the treatment session, short individual face-to-face interviews with patients were conducted. This involved thirteen patients undergoing RT for the following cancers: breast (n = 5), gynaecology (n = 2), prostate (n = 1), H&N (n = 1), and unknown primary cancer (n = 4). Patients ≥ 18 years old, regardless of treatment area, were consecutively asked to participate. Patients were included from five different university hospitals: Umeå University Hospital (n = 2), Akademiska University Hospital in Uppsala (n = 4), Örebro University Hospital (n = 3), Skåne University Hospital (n = 2) and Karolinska University Hospital (n = 2). Oncology nurses at each RT unit informed the patients about the study and asked if they would like to participate. After giving consent the patients were briefly interviewed by the same RT nurse. The interview question was “What questions do you consider are important for the staff to ask patients receiving their RT?” The duration of the interview was approximately 5 min, and the nurses took notes simultaneously. The interview data were analysed and resulted in several areas of discussion, such as: What is going to happen during the treatment and what kinds of side effects are common? Do you experience anxiety and/or claustrophobia? Is it important to meet the same staff during the RT-treatment period? Do you prefer listening to music during the treatment? Does the waiting room feel comfortable and inviting? Do you prefer lights on during treatment?

Third, the data from the interviews, together with the content from the identified instruments, were discussed in a workshop including 8 RT experts from 5 RT units in Sweden. The group included six oncology nurses, one physicist and one physician from 5 different university hospitals. The group identified items within the areas of physical comfort, physiological comfort, i.e. anxiety, claustrophobia, dignity, patient empowerment, relation to staff, informational needs and treatment environment experiences.

The whole item-generation process resulted in a preliminary questionnaire, which included 96 items, and all items were formulated as a 6-point Likert–type scale for response options, which ranged from 1 = I strongly agree to 6 = I strongly disagree. There was a possibility to answer “I cannot say/I don’t know” and that option was handled as missing data through pair-wise deletion. To maximize content validity, an additional expert group workshop was conducted with the same 8 participating professionals as in the initial group. They were asked to examine the pool of 96 items with respect to content, format and scaling, and to suggest improvements and reduction of items. Based on the received comments and suggestions, the first draft was revised and a preliminary questionnaire of 34 items about the patient’s experience of the RT session was constructed.

To test the content of the items, a sample of 10 patients (2 patients from each University hospital) were asked to fill in the questionnaire and to give feedback on the clarity and readability. A short face-to face interview was also conducted immediately after returning the questionnaire. The 34-item questionnaire was regarded as satisfactory by all 10 patients and no further refinements were performed.

Phase 2: Psychometric evaluation

The preliminary 34-item questionnaire was distributed to patients at 8 RT units in Sweden in May 2014. Before responding to the items, patients were asked to provide information about: age, gender, treatment area and number of RT sessions they had received. At the end of the questionnaire the patients were given the possibility to answer an open-ended question: “Is there anything else you want us to know? All adult outpatients with any type of cancer scheduled to receive curative external-beam RT were eligible to participate. Exclusion criteria were: incapacitating psychosis or cognitive disabilities,<18 years of age, and insufficient Swedish language skills. The RT nurses and/or receptionists at the RT units distributed the questionnaire to all eligible patients. Completed questionnaires were anonymously collected in a sealed box, on site, before the patients left the RT unit or at the next visit. A total of 937 questionnaires were distributed and 825 were returned, representing a response rate of 88%.

Descriptive characteristics (means, standard deviation and frequencies) were calculated for each item in the scale. Theoretical construct validity of the RTEQ was evaluated based on the hypothesis that this would be supported by a Principal Component Analysis (PCA) resulting in a statistically stable and theoretically meaningful solution explaining > 50% of the variance in the data. Content validity was explored by the previously described sample of 10 patients. The distributions of individual answers were sometimes highly skewed, predominantly towards the high end of the scoring. However, as the sample size for all subgroups was large (n > 30), the comparisons of means were done under the assumption of normal distribution.

The criteria used to indicate appropriateness during factor analysis was a Bartlett test of sphericity of significance and a Kaiser–Meyer-Olkin measure of sampling ≥ 0.7. PCA was performed to reveal underlying dimensionality of the data, i.e. to group the variables into different themes. Visual interpretation of the Cattell scree test plot was used to identify the number of underlying categories [24]. After the identification of the number of underlying categories, items with ambiguous interpretation, i.e. variables with significant loadings in more than one category, were excluded. Items with a weak association to the identified categories, i.e. variables for which a minority of the variation was explained by the underlying categories, were also excluded. The thresholds for exclusion were communalities < 0.5 and loadings > 0.3 in multiple categories. The variable reduction was performed in steps. Finally, Cronbach’s alpha was used as an estimation of the reliability of the estimation of the underlying categories. Cronbach’s alpha between 0.7 and 0.95 was considered satisfactory.

A one-way analysis of variance (ANOVA) was used to determine any statistical significant differences between gender, treatment area or number of sessions received and the means of the sub-themes. The answers to each item had been adjusted so that a positive answer to an item was worth 6 and a negative worth 1. A mean of the answers for the items in each group was used, for example, for the theme “Situational unease”, the score was calculated

$$\text{Situational Discomfort}={\displaystyle \sum _{\text{item}=1}^5}\frac{{\text{Answer}}_{\text{item}}}{5}$$ (1)

as 6 is the positive answer to all the first 5 items. For the theme “Level of trust and understanding” the formula used was

$$\text{Pshchological coping}={\displaystyle \sum _{\text{item}=22}^{23}}\frac{(7-{\text{Answer}}_{\text{item}})}{2}$$ (2)

as 1 is the positive answer to the items 22 and 23. In this way a theme score of 6 means that the patients answered as positively as possible, and a theme score of 1 as negatively as possible.

All statistical analyses were performed using SPSS version 22.

Results

Sample characteristics

The sample (n = 825) consisted of 42% men and 54% women with an average age ± standard deviation of 64.6 ± 11.6 (range 19–92). The most common treatment-sites were chest (36%), abdomen (29%) and H&N (15%). Five percent of the patients had just completed their first RT session, 56% had completed 2–15 sessions, and 36% had completed 16 or more sessions (Table 1).

Table 1.

Sample characteristics (n = 825).

	n (%)	Mean (standard deviation) n = 825
Age (years)		64.6 (11.6)
Gender
-Male	348 (42.2)
-Female	444 (53.8)
-Data missing	33 (4)
Treatment session
-Treatment 1	38 (4.6)
-Treatment 2–15	456 (55.3)
-Treatments ≥ 16	286 (34.7)
-Data missing	45 (5.5)
Treatment Area
-Brain, Head and Neck	123 (14.9)
-Chest/Thorax	297 (36)
-Pelvis/Abdomen	240 (29.1)
-Other	105 (12.7)
-Data missing	60 (7.2)

Item analysis

No questionnaires were eliminated due to missing data. Items with no responses were handled as missing data and varied between 28 and 71 for the single items (mean 46). The frequency of the response “I cannot say/ I don’t know” varied between 4 and 62 (mean 12) for the single items. Due to the high response rate we chose not to eliminate any questionnaires or items and handled the missing data through pair-wise deletion to use all information possible with each analysis. The Kaiser–Meyer-Olkin measure of sampling adequacy was 0.889, and Bartlett’s test of sphericity was highly significant (p < 0.001), thus a principal component analysis (PCA) model was deemed appropriate.

The initial explorative PCA model (varimax rotation) including all 34 initial items and indicated (scree test) the presence of 6 orthogonal underlying dimensions explaining 68% of the total variation. Five items were removed in the first 4 steps and 6 items were removed in the final step, resulting in 23 remaining items and 6 orthogonal dimensions explaining 73% of the variation in the material. Fig. 1 shows scree-plots for all 34 variables and the 23 remaining after the variable reduction. In both cases there is a distinct plateau between component 5 and 6 followed by a drop in explained variance to component 7 and then slowly decreasing information content for further components. This means that the smaller number of variables substantially captures the information contained in the original set of variables.

Fig. 1.

Scree plot.

The six components were further analysed, and themes for each component were identified. The corresponding themes and items, including the loading of the items, are presented in Table 2. A reliability analysis was performed, and it resulted in high or very high Cronbach’s alpha for all themes (between 0.79 and 0.9) as shown in Table 3. The internal consistency coefficient (Cronbach’s alpha) was satisfactory (> 0.7) for all themes.

Table 2.

The grouping of the items into themes with loading for each item.

	Situational unease	Physical discomfort	Situational repose	Informational needs	Treatment environment acceptance	Level of trust and understanding
I feel trapped during treatment	0.88	0.05	−0.07	−0.8	−0.4	−0.25
I feel isolated during treatment	0.86	0.11	−0.11	−0.07	−0.06	−0.06
I feel stuck during treatment	0.86	0.02	−0.08	−0.04	−0.06	0.01
I feel scared during treatment	0.74	0.21	−0.15	−0.05	−0.05	−0.13
I feel nervous during treatment	0.72	0.20	−0.25	−0.04	0.01	−0.08
I feel pain when the nurse adjusts my position	0.11	0.87	0.03	−0.07	0.00	−0.00
I feel pain when I lay down on the treatment table	0.07	0.87	−0.04	−0.06	−0.05	−0.01
I feel pain during treatment	0.10	0.79	−0.09	−0.03	0.02	0.06
I feel pain when I stand up after treatment	0.03	0.78	−0.05	−0.05	−0.03	−0.04
It feel chafed during treatment	0.23	0.78	−0.02	−0.02	−0.08	0.04
I feel calm during treatment	−0.21	−0.03	0.89	0.08	0.17	0.07
I feel relaxed during treatment	−0.20	−0.08	0.89	0.09	0.13	0.03
I feel safe during treatment	−0.15	0.08	0.84	0.11	0.20	0.14
I feel comfortable during treatment	−0.07	−0.09	0.68	0.17	0.14	0.02
I got enough information about the side effects	−0.04	−0.05	0.07	0.86	0.07	−0.017
I got enough information about how to deal with the side effects	−0.12	−0.03	0.08	0.82	0.10	−0.02
I got enough information about the benefit of the treatment	−0.02	−0.06	0.14	0.79	0.07	0.10
I got enough information about how radiotherapy works	−0.07	−0.08	0.13	0.78	0.11	0.10
The colouring of the treatment room is easy to endure	−0.04	−0.04	0.13	0.09	0.84	0.03
The smell of the treatment room is easy to endure	−0.04	0.00	0.20	0.12	0.83	−0.02
The light in the treatment room is easy to endure	−0.09	−0.08	0.24	0.13	0.83	0.09
I trust the personnel	−0.10	0.06	0.06	−0.05	0.03	0.91
I understand the procedure	−0.12	−0.02	0.15	0.22	0.06	0.87

^⁎Loadings over 0.3 in bold.

Table 3.

Characteristics of the themes.

	Situational unease	Physical discomfort	Situational repose	Informational needs	Treatment environment acceptance	Level of trust and understanding
# of items	5	5	4	4	3	2
Explained variance %	26%	15%	11%	9%	6%	6%
Cronbach’s alpha	0.90	0.87	0.87	0.85	0.83	0.79

Theme scores, as given in Table 4, Table 5, include the observed scores for the different themes and summarize the findings. A multivariate analysis revealed that the timing of the survey (after 1, 2–15 or ≥ 16 RT sessions) only had a minor effect on the result, and there were no significant differences (one-way ANOVA). However, the general tendency was that questionnaires collected on the first RT day gave slightly lower scores in all themes, except for Informational needs, than questionnaires handed out later in the RT period. The multivariate analysis (F statistic) did not show any significant difference between the genders, while the treatment area had a significant impact.

Table 4.

Observed theme scores on different days of handling out the questionnaire.

	Fraction 1 (n = 38) Mean (±SEM)	Fraction 2–15 (n = 465) Mean (± SEM)	Fraction ≥ 16 (n = 286) Mean (±SEM)
Situational Unease	5.01 (0.25)	5.31 (0.05)	5.43 (0.06)
Physical Discomfort	5.47 (0.18)	5.65 (0.03)	5.67 (0.05)
Situational Repose	4.95 (0.26)	5.27 (0.05)	5.31 (0.06)
Satisfaction with Information	5.35 (0.20)	5.23 (0.05)	5.33 (0.06)
Treatment environment acceptance	5.49 (0.20)	5.74 (0.03)	5.64 (0.05)
Level of trust and understanding	4.43 (0.34)	4.95 (0.07)	4.88 (0.10)

Table 5.

Relationships between RTEQ subscales, treatment and disease variables.

	Head & Neck (N = 123)	Chest/Thorax (N = 297)	Pelvis/Abdomen (N = 240)	Other (N = 105)	Total (N = 765)
Situational Unease	4.5 (0.13)*	5.41 (0.06)	5.60 (0.07)	5.64 (0.07)	5.37 (0.04)
Physical Discomfort	5.66 (0.06)	5.59 (0.05)	5.76 (0.05)	5.57 (0.08)	5.65 (0.03)
Situational Repose	5.03 (0.12)**	5.11 (0.07)**	5.51 (0.06)**	5.33 (0.10)	5.26 (0.04)
Satisfaction with Information	5.13 (0.11)	5.24 (0.06)	5.42 (0.07)	5.23 (0.11)	5.28 (0.04)
Treatment environment acceptance	5.59 (0.09)	5.62 (0.05)	5.82 (0.04)	5.66 (0.09)	5.69 (0.03)
Level of trust and understanding	4.67 (0.16)	5.01 (0.09)	4.93 (0.11)	4.82 (0.16)	4.90 (0.06)

^*Head & Neck score was significantly smaller than the other for situational unease (ANOVA).

^**Pelvic/Abdomen score was significantly larger than Head & Neck and Chest/Thorax for Situational repose (ANOVA).

Discussion

This study aimed to develop a valid and reliable instrument for evaluation of how patients experience external RT. The reliability and the theoretical construct validity of the instrument were verified, and the analyses indicate that the tool is able to detect variations in patients’ experiences. The result is an instrument, referred to as The Radiotherapy Experience Questionnaire (RTEQ) with 23 items. This instrument assesses the patient’s experiences of the external RT procedure and includes psychological stress, physical discomfort and coping during the external RT procedure thorough development and testing has shown that these questions are relevant for patients undergoing RT. At present, similar questions are measured with different tools and scales (HADS, STAI, EORTC-QLQ-Info 25, CaTS and PCQ), but most of those scales are not specifically designed for patients receiving external RT. Those scales measure one aspect at a time, e.g. anxiety or informational needs, whereas the RTEQ includes several aspects that are specifically pertinent to external RT.

RTEQ can be used as a single tool to measure the patient experience during external RT treatment or in conjunction with other instruments that measure reproducibility, precision and motion management when introducing new techniques or immobilization devices. RTEQ allows for comparison between units, different workflows and to evaluate newly introduced techniques, from a patient perspective.

The theoretical construct validity was estimated as satisfactory since PCA separated the items into six themes that explained nearly 73% of the total variance in the sample. The content validity of the scale was regarded as satisfactory since the scale contains items, which reflect the dimensions that are described in the literature and by patients as being central aspects of their experience during RT. For example, anxiety and feelings of isolation [4] are captured in Psychological discomfort and Level of trust and understanding. Informational needs became apparent from patient interviews and is covered by the question “What is going to happen during the treatment and what kinds of side effects are common?”. Claustrophobia is a distinct problem for H&N cancer patients [3], [5], [9], and it is captured under Situational unease. This negative experience of the treatment, as indicated in the analysis of the sub-categories, is most likely due to more uncomfortable fixations for this patient group. In conclusion, estimates of content and construct validity for RTEQ indicate satisfactory psychometric properties of the scale.

There was a slight, but non-significant, tendency for lower scores, i.e. the patients were more negative within all themes except for Satisfaction with Information, when the survey was completed on the first day of treatment than with responses later in the treatment period. There was no statistically significant difference when the survey was completed early (RT 2–15) or later (RT ≥ 16) during the treatment period. This allows different options on when to administrate this survey. One option could be to use specific days when all patients at the department receive the questionnaire. Another possibility would be to always hand out the questionnaire on a specific day for a given patient group. H&N scores was significantly lower than the other treatment areas regarding Situational unease whereas Pelvis/Abdomen scores was significantly higher than H&N and Chest/Thorax for Situational repose. Situational unease captures claustrophobia that is a distinct problem for H&N patients [3], [5], [9]. Claustrophobia is a negative experience of the treatment, as indicated in the analysis of the subcategories. Claustrophobia is most likely due to more uncomfortable fixations for this patient group, and it is captured within items in Physical discomfort subcategory. No differences was found regarding gender.

Using a Pearson correlation matrix with highly skewed responses may involve a risk for an incorrect estimation of the factor structure. However, it is a well-established method, and we consider the result to be reasonable. In the exclusion of items, we used orthogonal components instead of non-orthogonal factors, and that choice was based on a theoretical perspective where we assume that the factors are uncorrelated, e.g. that Situational unease is independent of Situational repose. We find the use of orthogonal components to be mathematically sounder from a theoretical perspective.

Establishing construct validity is an ongoing process, and our findings represent the initial testing. Since this study mainly focussed on the theoretical dimensions of construct validity using principal component analysis, further testing and cross-validation of the findings with new samples are necessary to establish psychometric properties. Before the next step of test re-test reliability investigation, an analysis of the free-text comments from the questionnaire will be conducted.

Limitations

The small size of the patient interview group may be considered as not representative for the population, although there was representation from different patient set-up and immobilization groups: breast cancer, gynaecological cancer, prostate cancer and H&N cancer. The lack of formal cognitive interviews may also be a limitation, although the short semi-structured interviews fitted well in the context. The results were based on patients that were able to read and understand Swedish, and no data were collected relating to language or ethnicity. Therefore, whether and how language or ethnicity may affect the results remains unknown. A translation of RTEQ to different languages with further studies would be valuable. Another limitation is that no data on patient educational status were obtained. The patients were not asked if the questions were difficult for them to understand. As our finding represent initial testing no test-retest reliability was measured.

Conclusion

The RTEQ is a tentatively valid and reliable instrument to measure the patient’s comfort and experiences of the RT procedure. It has possible application for comparison between units, between different workflows for evaluation of newly introduced techniques, and to gain the patients’ perspectives of the RT procedures. Further studies regarding such applications are needed.

Source of funding

VINNOVA (The Swedish Agency for Innovation Systems).

Lion's Cancer Research Foundation, Umeå University.

Conflict of interest

None of the authors report a conflict.