Developing a qualitative and quantitative ambulatory assessment-based feedback system within cognitive behavioural interventions for people with persecutory beliefs

Abstract

Background

Although the application of self-monitoring (ambulatory assessment) and visual feedback in psychological interventions has yielded promising results, there are currently no reports on using self-monitoring and feedback during a complete therapy. The online m-Path platform provides a tailorable framework for integrating self-monitoring and visual feedback within different psychological interventions.

Methods

Therapy-specific questionnaires and visual feedback were developed within the online m-Path platform as part of the Feeling Safe-NL trial (registration number: ISRCTN25766661) for regular CBT for psychosis (CBTp) and the Feeling Safe Programme combined with peer counselling (the Feeling Safe-NL Programme). The design process involved people with lived experience, psychologists, peer counsellors, researchers, and software developers. The design principles included that the system should be 1) easy to use, 2) suitable for use during a six-month therapy, 3) focussed on positive and goal-aligned outcomes, 4) understandable by patients and professionals, and 5) informing, guiding, and promoting therapy. Design principles were evaluated using compliance data and a patient questionnaire.

Results

The system was used by 21 patients, of which nine completed the questionnaires for the full therapy period, 168 days on average. Usability data from patients revealed that the system was easy to use, well-explained, and suitable for use over six months of therapy. The patients also reported that the questions overall positively affected their emotions and that the feedback was insightful.

Conclusion

The results support the successful application of the design principles to promote the integration of the self-monitoring and visual feedback system within specific CBTp interventions.

Highlights

• •Ambulatory assessment-based feedback systems can be tailored to therapy interventions.
• •Therapy interventions include regular CBT for psychosis and the Feeling Safe Programme combined with peer counselling.
• •Patients perceive a combination of qualitative and quantitative feedback as valuable.
• •The feedback systems can be used during a full therapy of six months.
• •Innovations include clock graphs, stacked graphs, and word clouds with sentences.

1. Introduction

Ambulatory Assessment (see, e.g., Mestdagh and Dejonckheere, 2021; Trull and Ebner-Priemer, 2020), which includes various data collection methods such as the experience sampling method (ESM: Delespaul and Devries, 1987; Myin-Germeys et al., 2018) and the daily diary method (Bolger et al., 2003; Lischetzke, 2014), hold significant promise for clinical practice (Bos et al., 2019). With a history dating back over 50 years (Schneider and Stone, 2016; Verbrugge, 1980), these self-monitoring techniques offer a glimpse into patients' experiences in their everyday lives. Nowadays, self-monitoring is often conducted via a smartphone, prompting users to report their thoughts, behaviours, and context multiple times a day for several consecutive days. This process results in intensive longitudinal data (Ebner-Priemer and Trull, 2009; Shiffman et al., 2008). These techniques can help to minimise errors related to memory or recollection and provide unique insights into thoughts, behaviours and feelings and their dynamics with contextual factors, such as social interactions, activities, and significant events (e.g., a fight with a best friend or start of a new job).

Due to these advantages, modern self-monitoring techniques have been widely applied in research on mental health problems (Bell et al., 2020; Hanssen et al., 2020; Myin-Germeys et al., 2011; Oorschot et al., 2009). Moreover, there is a growing trend of using these techniques in clinical practice. Ambulatory assessment methods offer a more fine-grained, contextual data collection than standard self-monitoring methods. Although therapy settings usually involve some form of self-monitoring, it is often less intensively and frequently conducted with paper and pencil (Cohen et al., 2013). Other techniques, like routine outcome monitoring, are typically performed even less intensively and not within the patient's natural environment, for example during weekly therapy sessions (Lambert et al., 2018).

While the independent use of self-monitoring has been demonstrated to be useful as such, for instance, by increasing patients' awareness of their well-being (Bakker and Rickard, 2018; Bos et al., 2022), personalised feedback based on self-monitoring data has shown additional advantages. For example, feedback based on personal data can serve as a reward, increasing motivation and, thus, compliance in self-monitoring studies (Hsieh et al., 2008; van der Krieke et al., 2015). Personalised feedback may also offer new insights (Bos et al., 2019; Kramer et al., 2014) and make mental health problems more tangible (Folkersma et al., 2021). Additionally, incorporating ambulatory assessment-based feedback into therapy sessions may develop a stronger therapeutic alliance, improving shared decision-making (Bos et al., 2019, Bos et al., 2022; Cohen et al., 2013). Therefore, there has been a surge recently in the development of personalised feedback based on intensive longitudinal data gathered with ambulatory assessment techniques (Bartels et al., 2023; Dietvorst et al., 2024; Rimpler et al., 2024).

Current ambulatory assessment-based feedback often focuses on descriptive graphs (Bastiaansen et al., 2018; Kramer et al., 2014; Wichers et al., 2011a). These personalised graphs show, for example, how positive affect varies in different contexts (Bartels et al., 2023; Wichers et al., 2011b). More recently, in addition to quantitative information, qualitative information has become a part of the feedback to enhance the personalisation of ambulatory assessment-based feedback. For instance, patients can describe, via an open text box, the most pleasant or unpleasant event that day (Bringmann et al., 2021). These qualitative descriptions are then displayed as part of a quantitative graph or summarised in word clouds (Bos et al., 2022; Stadel et al., 2023; von Klipstein et al., 2023). However, this feedback is often not given face-to-face or only in one feedback session, or the feedback is not directly related to therapy interventions but involves general clinically relevant information (Bartels et al., 2023).

We now used a different approach, in which we integrated self-monitoring and visual feedback into two separate psychological interventions. The feedback was specifically developed for people with persecutory delusions (unfounded threat beliefs that others intend harm), a common experience in people with severe mental health problems, occurring in more than 70 % of people with psychosis (Bebbington and Freeman, 2017; Coid et al., 2013). The ambulatory assessment-based feedback system was developed as part of the Feeling Safe-NL trial (trial registration number: ISRCTN25766661; Tolmeijer et al., 2023) for both regular cognitive behaviour therapy for psychosis (CBTp) and the Feeling Safe Programme (Freeman et al., 2021a) combined with peer counselling (called the Feeling Safe-NL Programme). A key novel feature of our approach is that the feedback is presented in modular dashboards so that patients and therapists (and peer counsellors within the Feeling Safe-NL Programme) can select the relevant feedback. This approach can replace traditional CBT registrations, including thought-behaviour records or other homework tasks (i.e., STIC tasks; Cohen et al., 2013). This differs from traditional ambulatory assessment questionnaires, which usually do not directly match tools used in CBT. Similarly, a novel feature of our feedback modules is that they are specifically tailored to CBT. In addition, the integration of self-learning multiple-choice questions and open text boxes enables efficient and meaningful contextualization of quantitative ambulatory assessment data, a feature not previously seen in clinical settings. Furthermore, we introduce new visualisations – tailor-made for CBTp – that enhance the feedback, such as word clouds containing short sentences instead of just words. Depending on individual wishes and needs, the feedback can be discussed weekly with the therapist (and peer counsellor within the Feeling Safe-NL Programme) or less frequently.

This paper describes the developmental process of the novel qualitative and quantitative ambulatory assessment-based feedback system for regular CBTp and the Feeling Safe-NL Programme. We will outline the design process and principles and focus on particularly innovative components, such as self-learning items that enable patients to create and save personalised answer options (von Klipstein et al., 2023; Mestdagh et al., 2023; Stadel et al., 2024) and novel graphical representations (e.g., clock graphs). Additionally, the paper describes the use of the novel system when integrated into clinical practice and presents usability data.

2. Methods

2.1. Ethical approval and consent

The research was approved by the medical ethics committee of the VU Medical Centre (registration number NL77046.029.21). All participants provided written informed consent at the start of the Feeling Safe-NL trial.

2.2. Participants

Forty participants of the Feeling Safe-NL trial were asked to complete a brief questionnaire after their post-therapy assessment. Of the 34 patients who agreed to participate in the interview, 21 had used the system. These participants (n = 21) had received either regular CBTp (n = 11) or the Feeling Safe-NL Programme (n = 10).

2.3. Design principles & process

The design principles that we took as the starting point were that the system should be 1) easy to use, 2) suitable for use over the course of a full six-month therapy, 3) focussed on positive and goal-aligned outcomes, 4) understandable by both patients and professionals, and 5) informing, guiding and promoting therapy. The design process involved people with lived experience, psychologists, professional peer counsellors, researchers and software developers and involved several phases. First, questions for the assessments were discussed during an iLab meeting (https://ilab-psychiatry.nl/) involving researchers and clinicians with expertise in gathering intensive longitudinal data. Second, options for the feedback were initially drawn on paper and later using the collaborative design software Figma (www.figma.com). Next followed an iterative cycle of the third and fourth phase. In the third phase, the designs in Figma were used to develop an interactive prototype in the m-Path software (www.m-path.io). The feedback graphs were programmed in Javascript using the charting library Recharts (www.recharts.org). Fourth, the system was piloted with four therapists, four researchers (the authors), four patients and one counsellor. Feedback was collected to make adaptations, and the new version was again piloted and discussed during the Feeling Safe-NL trial meeting. Following testing and further refinements, the system was considered technically reliable and usable for monitoring for a period of at least several months. The design process is described in detail in Appendix A. For a full list of the questionnaire items, see Appendix A Table 1.

Table 1.

Demographic characteristics of patients.

Characteristic	Patients (N)
Gender
Male	11
Female	10
Non-binary	0
Age
20–35 years	9
36–50 years	8
51–67 years	4
Education level1
University	1
University of applied science	2
Secondary vocational education	8
Pre-university	1
General secondary education	1
Prevocational education	7
Primary school	1
Years in treatment (M, SD)	9.8 (8.4)
Main chart diagnoses
Schizophrenia Spectrum and Other Psychotic Disorders	13
Depressive Disorders	2
Anxiety Disorders	1
Trauma- and Stressor-Related Disorders	3
Substance-Related and Addictive Disorders	1
Personality Disorders	1
Comorbid chart diagnoses
Schizophrenia Spectrum and Other Psychotic Disorders	3
Neurodevelopmental disorders	3
Depressive Disorders	6
Anxiety Disorders	3
Trauma- and Stressor-Related Disorders	3
Somatic Symptom and Related Disorders	1
Substance-Related and Addictive Disorders	2
Personality Disorders	4
Feeding and Eating Disorders	1
Obsessive-Compulsive and Related Disorders	2
Three or more diagnoses	7
Medication use
None	4
Antidepressants	8
Antipsychotics	16
Anxiety Medication	8
Sleep Medication	0
Amphetamines	1
Alcohol Antagonist	1
Opiates	1
R-GPTS2	44.95 (13.71)

¹Dutch equivalents are (in order of table listing): WO, HBO, MBO, VWO, HAVO, VMBO.

²The revised Green et al., Paranoid Thoughts Scale (Freeman et al., 2021b).

2.4. The ambulatory assessment and feedback software

The online m-Path platform was used to collect data and provide visual feedback since this software provides an intuitive and flexible framework for sending ambulatory assessment questionnaires and generating visual feedback (Mestdagh et al., 2023). The software contains free functionalities for individual practitioners or small research projects (e.g., visualise responses) and paid features for customised projects such as this one. Within the m-Path platform, questionnaires were developed in line with therapy manuals.1 For the Feeling Safe-NL Programme, the questionnaires focused on the six empirical maintenance factors of paranoia and important recovery areas, including social functioning. For regular CBTp, the questionnaires focused on traditional CBTp targets, including thoughts, feelings and behaviours. For both the Feeling Safe-NL Programme and regular CBTp, patients received one daily ambulatory assessment in the morning and one in the evening during the entire therapy period (approximately six months). Patients chose at what time they wanted to receive the morning and evening questionnaires. Both questionnaires were accessible for three hours. All questions and answer options are presented in Table 1 in Appendix A. Besides Likert scales, open text boxes and newly developed self-learning multiple-choice questions were used (Mestdagh et al., 2023). In the case of self-learning multiple-choice questions, participants could add new answer options, which were then saved as response options for future assessments. Answers to self-learning items can include complete sentences (e.g., ‘I spent time worrying about other people’). Using the open text boxes, people can elaborate on a previously provided answer. For example, if people indicate they had a bad day, the open text box allows them to elaborate on why it was bad. Participants were required to answer all questions in the ambulatory assessments, except for the open text box questions, which could be skipped if these were experienced as too time-consuming.

2.5. Design of the feedback modules

For regular CBTp, one feedback module was created in line with the CBTp for persecutory delusions-manual (e.g., graphs displaying conviction, preoccupation and distress scores associated with the persecutory belief and thought-behaviour records). For the Feeling Safe-NL Programme, feedback was provided in eight separate modules, one for each of the six maintenance factors of persecutory delusions (voice-hearing, worry, sleep, self-esteem, trauma-imagery and safety behaviours) and one for activity and social interactions. Four main types of graphs were used: line graphs, stacked graphs, word clouds and clocks. The graphs are presented here, using three pseudo-anonymised vignettes from three participants participating in the Feeling Safe-NL trial (see Box 1). Two participants received the Feeling Safe-NL Programme, and one received regular CBTp.

Box 1. Vignettes.

Emir (Feeling Safe-NL) is a 40-year-old male who lives in supported housing. In his childhood, he was severely bullied. Due to these experiences, he feels vulnerable and less valuable than other people. He also hears threatening voices that make him feel anxious, and he worries that a spiritual entity can control him. Because he does not feel safe, he spends most of his time at home. He has little contact with friends or family. He starts with the voice-hearing module because he wants to learn to feel safer despite hearing voices. After completing this CBT module, he started the worry module because he still spends much time worrying at home. Lastly, he chooses to work on self-confidence and feeling safe enough since he would like to feel better about himself and spend more time doing the things he enjoys, such as visiting family and friends.

Philip (Feeling Safe-NL) is a 55-year-old married man. He has always been a morning person. After he lost his job, he started to worry more and sleep worse. He would wake up in the middle of the night without being able to regain his sleep. He started to worry that people thought he did something wrong and that they were, therefore, following him. Because he feels unsafe, he stays at home without much to do. He does not feel physically tired at night. Even though he is not tired, he goes to bed around 5 pm because he wants the day to end. He chooses to start with the sleep module because he would like to be able to wake up in the morning again instead of at night. He would also like a more regular sleep pattern, sleeping about seven hours each night. After improving his sleep, he thinks about reducing worry and feeling safe enough to go shopping, cycle and take a train to his sons.

Zoey (regular CBTp) is a 30-year-old woman who lives independently. She works in a clothing store and a lunchroom. She has a history of physical and emotional abuse by her parents and brother. When she is alone, she experiences thoughts that other people might want to physically harm her when she is alone because they do not like her. She, therefore, checks the locks frequently and continuously monitors the windows. She also avoids being alone at home and stays outside until late in the evening with other people after work. When she is out, she uses substances to feel calmer when she arrives home. She would like to feel at ease at home and spend more time alone. She would also like to feel better about herself and improve her sleep.

Alt-text: Box 1

2.5.1. General settings for all graphs

For both therapies, each graph in the feedback modules can be adjusted to visualise a specific period. Three settings are available: 1) the interaction setting, which presents only the completed questionnaire data; 2) the day setting, which allows for identifying missing data since this setting presents each day on which a questionnaire was sent; and 3) the week setting, which presents a summary of the data over the previous week. All graphs also have the option to display additional information for a particular data point by hovering over it. Quantitative and qualitative data can be displayed: 1) the exact date and time of provided answers, 2) the corresponding scores, 3) the corresponding labels revealing the meaning of each score, and 4) answers provided to open questions, if applicable.

2.5.2. The four main types of graphs

• 1.Line graphs. Line graphs were developed for the regular CBTp feedback module and the eight Feeling Safe-NL feedback modules. Examples of line graphs used are presented in Fig. 1 (Feeling Safe-NL) and Fig. 2 (regular CBTp). In these line graphs, the x-axis indicates when the questionnaire was completed (day and time), and the y-axis indicates the scores in percentages. Scores are presented in percentages to compare questions with different answer scales. One question related to mood was answered on a slider scale from 0 to 100, whereas all other questions were answered on a Likert scale with five options from 0 (not at all) to 4 (a lot). Hovering over a data point provides additional information, such as the qualitative details entered through open text boxes, for example, the thought-behaviour record shown in Fig. 2. The clinical relevance of line graphs is described in Box 2.

• 2.Stacked graphs. Stacked graphs were only developed for the Feeling Safe-NL feedback modules. The stacked graph shows a proportional relationship between variables, meaning the variables always add up to 100 %. Stacked graphs were used to visualise how some of the maintenance factors addressed in Feeling Safe-NL (i.e., worry, self-confidence, voice-hearing and trauma imagery) relate to the experience of feeling safe enough. This was done in line with the rationale of the Feeling Safe-NL Programme that reducing the maintenance factors can promote the re-learning of safety. Similar to the line graphs, the x-axis indicates when the questionnaire was completed (day and time) and the y-axis indicates the scores in percentages. An example of a stacked graph used in the Feeling Safe-NL feedback module for worry is presented in Fig. 3. When a patient scores a one for worry frequency (i.e., 5 min to 1 h spent worrying) and a two for safety (i.e., only feeling safe sometimes) on a particular day, the graph will show 33.3 % for worry frequency and 66.7 % for feeling safe enough.2 Hovering over a data point of interest in the quantitative visualisation of the stacked graph will provide the scoring labels and potential comments (if provided) about people's experience of feeling safe enough.

Fig. 1.

Line graph in the Feeling Safe-NL feedback module for the maintenance factor worry.

Fig. 2.

Line graph in the regular CBTp feedback module with a thought-behaviour record.

Box 2. Clinical relevance of the line graphs.

Fig. 1(Feeling Safe-NL line graph for worry). Emir spends almost all his time worrying (dark red line), which he experiences as very distressing (bright red line). He is also unable to stop worrying (green line). After his therapist introduced the “worry period”, in which 15 min per day were scheduled at a specific time (8 pm) and a relatively uncomfortable place (a wooden chair) to worry, the graph illustrates that Emir is more able to postpone his worrying. This helps him to reduce the time he spends worrying and the associated distress. The graph motivates Emir, since it illustrates that he can control his worry and maintain progress after completing the module.

Fig. 2 (regular CBTp line graph for preoccupation and distress associated with persecutory beliefs). Zoey spends relatively little time worrying about others wanting to harm her (green line) because she spends most of her time outside her house or with others to feel safer (not visible in this graph). When she does think about her main worry, she generally experiences quite a lot of distress (red line). Zoey and her therapist use the thought-behaviour record to understand why particular events lead to specific thoughts, emotions and behaviours. They also use the record to think about the short- and long-term consequences of her reactions. After evaluating alternative explanations for events, Zoey is better able to identify alternative, less threatening, interpretations of what is happening. This helps her to calm down and feel more comfortable at home by herself.

Alt-text: Box 2

Fig. 3.

Stacked graph in the Feeling Safe-NL feedback module for the maintenance factor worry.

In the Feeling Safe-NL sleep module, the stacked graph was also used to visualise how people spent their time in bed (i.e., time spent falling asleep, sleeping, being awake at night, and staying in bed after finally awakening) since this is addressed in therapy (see Fig. 4). Hovering over a datapoint of interest will provide the number of hours for each time category and comments (if provided) about people's experience of feeling safe enough. The clinical relevance of stacked graphs is described in Box 3.

• 3.Word clouds. Word clouds were developed for the regular CBTp feedback module and the eight Feeling Safe-NL feedback modules. Examples of word clouds used in the feedback modules are presented in Fig. 5 (Feeling Safe-NL) and Fig. 6 (regular CBTp). The word clouds were developed to summarise the qualitative information collected (using the self-learning item questions). For the Feeling Safe-NL feedback modules, word clouds were used to visualise triggers, coping strategies, activities and social interactions. For the regular CBTp feedback module, word clouds were used to visualise planned activities, activities, helpful thoughts and behaviours and alternative explanations. Contrary to standard word clouds, the newly developed word clouds allow for visualising complete sentences to provide more context than just a single word. Visualised words or sentences are displayed larger when the answer option is chosen more often. Hovering over the sentences or words allows viewing of how often the answer options were exactly chosen. The clinical relevance of word clouds is described in Box 4.

• 4.
  Clocks. Clocks were only developed for the Feeling Safe-NL sleep problems feedback module. The clock is a new kind of graph in which the bedtime and time when the person gets out of bed are displayed. Each dot represents a bedtime or getting out of bed time, with the time interval rounded to the closest quarter (e.g., 10:16 pm becomes 10:15 pm, 10:55 pm becomes 11 pm). When the dots align, someone is going to bed or getting out of bed at around the same time every night. However, when the dots are dispersed, there is an irregular sleeping pattern. Hovering over the dots displays the date the data point was collected, the exact time associated with the data point and reasons for sleeping well or poorly (if provided). The time period settings for clocks differ from all other graphs and include day segments (i.e., morning, afternoon, evening and night) instead of the usual interaction, day and week setting. This was done to allow viewing of the data in line with the specific sleep pattern of each patient. Using the button at the top right corner of the clock, the numbers around the outer edge of the clock, representing the hours, can be changed in line with the specified day segment. For example, the hours in the morning range from 1 am to 12 pm, whereas the afternoon segment hours range from 9 am to 8 pm. The clinical relevance of the clocks is described in Box 5.

  Fig. 7.

  

  Clocks in the Feeling Safe-NL feedback module for the maintenance factor sleep problems. The dots are randomly placed on the radius of the circle.

Fig. 4.

Stacked graph in the Feeling Safe-NL feedback module for the maintenance factor sleep problems.

Box 3. Clinical relevance of the stacked graphs.

Fig. 3 (Feeling Safe-NL stacked graph for worry). Emir initially spends almost all his time worrying and feeling unsafe with a lot of horror scenarios playing in his mind. Using different techniques to reduce worry, including activities, he starts to leave his house more often and build new memories of safety. The therapist uses the stacked graph to discuss the role of worry (red area) in Emir's experience of feeling safe enough to do what he would like to do (brown area). Worrying less appears to positively affect his experience of safety, which motivates Emir to keep practicing the techniques to reduce worry.

Fig. 4 (Feeling Safe-NL stacked graph for sleep problems). Philip struggles with falling asleep (brown area) and being awake at night (red area). The amount of time spent sleeping (green area) varies greatly at the beginning of therapy. He does get up immediately after his final awakenings, so there is no area covering the time related to getting out of bed. The graph helps Philip and his therapist to understand his current sleep pattern and track progress towards his goal of having a more regular sleep pattern and getting seven hours of sleep.

Alt-text: Box 3

Fig. 5.

Word clouds in the Feeling Safe-NL feedback module for the maintenance factor worry.

Fig. 6.

Word clouds in the regular CBTp feedback module.

Box 4. Clinical relevance of the word clouds.

Fig. 5 (Feeling Safe-NL word clouds for worry). After completing the voice-hearing module, Emir, his therapist, and his peer counsellor evaluate the information already collected about his worry. They notice that both the voices and a lack of activities trigger his worries, whereas doing something active helps him to reduce worry. This information helps guide the therapeutic and recovery-oriented interventions by his therapist and peer counsellor, respectively. Over time, Emir notices that the worry period and thinking about something else, such as the best-case scenario, are the most helpful strategies to reduce worry.

Fig. 6 (regular CBTp word clouds for alternative explanations and helpful thoughts). Zoey and her therapist use the word clouds to learn about possible other explanations for situations in which she is worried about others wanting to harm her. She learns that in some situations, her fears and insecurity cause her to mistrust the situation, but there is no actual threat in these situations. She also notices that it is helpful to think about fun goals since this makes it somewhat easier to tolerate her fear and stay in a situation. Zoey and her therapist, therefore, think about ways to remind herself of the goals she is working towards.

Alt-text: Box 4

Box 5. Clinical relevance of the clocks.

Fig. 7 (Feeling Safe-NL clocks for sleep problems). Philip goes to bed early (around 5 pm) because he wants the day to be over, resulting in him waking up in the middle of the night (around 2 am), after which he cannot fall asleep anymore. He, therefore, gets up in the middle of the night and spends most of his time worrying and trying to watch some TV. Philip and his therapist work on gradually shifting his bedtime to his ideal bedtime of 9 pm. Philip can shift his bedtime to 7 pm but notices that going to bed later is more challenging. The clocks help Philip and his therapist gain insight into his sleep pattern, keep track of progress and discuss challenges. Philip also discusses the clocks with his peer counsellor. Since boredom is a reason for Philip wanting to go to bed early, they brainstorm about fun activities that would help him delay his bedtime.

Alt-text: Box 5

2.6. Usability evaluation

To evaluate the ease of use, the suitability for use over the course of a six-month therapy, focus on positive outcomes, understandability for patients, and the value of informing, guiding, and promoting the interventions (design principles 1–5), the patients' experiences with the ambulatory assessment-based feedback system were assessed using a brief questionnaire developed for this study. A more extensive evaluation of the experiences of both patients and professionals is presented separately (Tolmeijer et al., in preparation). Questionnaire data related to the patient evaluation of the ambulatory assessments and visual feedback was visualised through boxplots in the R software (R Core Team, 2023). To further evaluate design principle 2 (suitability for use over the course of a full therapy), the number of days the patients filled in the ambulatory assessments was visualised as a histogram in the R software using ggplot2 (Wickham, 2016).

3. Results

3.1. Participants

The demographics of the 21 patients at baseline are presented in Table 1. The trial had broad inclusion criteria, allowing for a diverse sample in terms of illness severity (Tolmeijer et al., 2023). This diversity is reflected in Table 1, which shows that patients had been in treatment for an average of circa 10 years (SD = 8.4), with one-third having three or more diagnoses. Furthermore, scores on the Revised Green et al. Paranoid Thoughts Scale (Freeman et al., 2021b) ranged from 22 to 68.3

3.2. Usability evaluation

The results are based on 21 patients, of which 11 received CBTp, and 10 received the Feeling Safe-NL therapy. The questionnaire about patients' experiences with the m-Path platform and associated responses are presented in Fig. 8, Fig. 9.4 Regarding design principles 1 (easy to use) and 4 (understandable), patients overall reported they found the m-Path platform (assessments and feedback) well explained by their therapist and easy to use. Regarding design principle 2 (suitable for long-term use), patients were overall able to complete the questionnaire on average for 95 days (see Appendix Fig. A2, A3 and A4). Nine out of 21 patients completed the questionnaires for the full period of therapy, with an average of 168 days.5 Additionally, both groups indicated that the questionnaires were perceived as not too burdensome for use over an extended period of up to six months.6 In line with design principle 3 (focus on positive and goal-aligned outcomes), answering the questionnaires mostly positively affected patients' emotions. Concerning the final design principle 5 (informing, guiding, and promoting therapy), patients indicated that they found the feedback on average insightful. For instance, in the case of Philip (Box 5), the clock visualisation helped him to gain insight into his sleeping pattern, thus further guiding therapy. A more in-depth qualitative evaluation of the ambulatory assessment and visual feedback by patients, therapists and peer counsellors can be found in Tolmeijer and colleagues (in preparation).

Fig. 8.

Patients' responses on the questionnaire about the ambulatory assessment and visual feedback used in the Feeling-Safe NL Programme (n = 10), displayed as boxplots. The plots also indicate the mean (grey dot), and 95 % confidence intervals (the grey whiskers).

Fig. 9.

Patients' responses on the questionnaire about the ambulatory assessment and visual feedback used in regular CBTp (n = 11), displayed as boxplots. The plots also indicate the mean (grey dot), and 95 % confidence intervals (the grey whiskers).

4. Discussion

This study presents the development and usability of one of the first ambulatory-based feedback systems within the context of different CBT interventions for individuals with persecutory delusions. What is particularly novel is that new types of clock and stacked graphs were used in the feedback to promote understanding of people's experiences such as fragmented sleep patterns. Additionally, word clouds containing complete sentences instead of single words were used to visualise activities, helpful thoughts (including alternative explanations for distressing situations), triggers and coping. Furthermore, the system allowed for digital administration and visualisation of thought-behaviour records, allowing records to be efficiently completed, stored, and visualised on a timeline. The newly developed ambulatory assessment-based feedback system could be meaningfully integrated within the CBT interventions being evaluated in the context of the Feeling Safe-NL trial (Tolmeijer et al., 2023), regular CBTp and the Feeling Safe Programme combined with peer counselling (the Feeling Safe-NL Programme). The combination of qualitative and quantitative information appeared particularly valuable for clinical practice since it allows for the contextualisation of scores. For example, scores can be contextualised by descriptions of events and activities that day. In line with the five design principles, usability data also revealed that the system was easy to use, well explained, suitable for use over six months of therapy, positively influencing emotions and providing new insights.

In line with previous research (Bell et al., 2020), most patients found the system easy to use and a valuable tool. Additionally, similar to Eisner et al. (2019) and Welch et al. (2022), about two third of the patients could use the system for an extended period of up to six months (or more).7 A factor that might have reduced compliance is the assessment length (Eisele et al., 2020), particularly the length of the evening questionnaires, which contained over twenty questions. The possibility to adapt the feedback (similar to the interactive graphs in Burns et al., 2011) and select relevant questions is one avenue to reduce the assessment length and improve the relevance of the feedback (e.g., Bos et al., 2022; Palmier-Claus et al., 2013; von Klipstein et al., 2023). Tailoring assessments and feedback can promote compliance and integration of the feedback within each therapy trajectory. For modular CBT programmes, the assessment and feedback can, for example, be adapted to include only the most relevant modules. Currently, there is little insight into how feedback should be provided, whether its frequency (e.g., a single session versus weekly review) influences its effectiveness, or how the manner in which feedback is given impacts outcomes. Understanding the feedback process better could further optimize feedback delivery and improve patient outcomes. Furthermore, it would be interesting to study in a larger sample whether patients who engage more or for a longer duration show clinical improvement in persecutory beliefs. Given our limited sample size, we cannot draw definitive conclusions about this.

Combining qualitative and quantitative feedback (e.g., distress scores and associated thought-behaviour records) was found to promote the contextualisation of quantitative information, similar to earlier work (Bringmann et al., 2021; von Klipstein et al., 2023). The self-learning items presented here, which contained complete sentences (multiple words), have the potential to substantially improve patients' ability to express themselves easily and provide more contextual information. Namely, answers provided earlier are saved for the self-learning items so they can be re-selected with just one click. This does raise the question of whether and when these new self-learning items are preferred over open text boxes to contextualise quantitative information. When it concerns brief and repetitive answers (e.g., routine activities), self-learning answers can be particularly well-suited since they allow for reselecting previously provided answers. However, open text boxes may be particularly useful where elaboration and more details are typically given (e.g., elaboration on why someone had a bad day). Therefore, our developmental study suggests that new self-learning items cannot completely replace open text boxes, and information collected with self-learning items and open text boxes can be valuable for linking to quantitative timelines. A further limitation of using self-learning items is that there are currently no models available for evaluating their psychometric properties, such as reliability, in the context of ambulatory assessment.

The present study supports the successful application of the design principles to promote the integration of the newly developed ambulatory assessment-based feedback system within specific CBTp interventions. Since many CBT tools naturally involve collecting qualitative and quantitative information, they can be easily incorporated into the ambulatory assessment questionnaires. Moreover, the modularity of the feedback system makes it adaptable to other psychological contexts beyond CBT, particularly contexts in which modular interventions have already been developed, such as low self-esteem (Korrelboom et al., 2012). It is our hope that this developmental study provides a basis for future research on the use of tailored ambulatory assessments and feedback within psychological interventions for people with various mental health problems.

Declaration of competing interest

M. Mestdagh and S. Verdonck are the co-founders of the m-Path platform, and M. Piot is the m-Path Chief UX officer and also an m-Path shareholder. D. van den Berg and A.B.P. Staring are board members of the non-profit Cognition and Psychosis Foundation and authors of several CBT books, including the Dutch CBTp manual ‘Gedachten Uitpluizen’. They also provide CBTp training for therapists. D. Freeman founded the Feeling Safe Programme, and F. Waite has been involved in the developmental work. L. Isham provides CBTp training and supervision at the Oxford Cognitive Therapy Centre. E. Tolmeijer is the coordinator of the Feeling Safe-NL trial. During the preparation of this work, the authors used chatGpT to check the grammar. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the publication's content.

Appendix A. Supplementary data

Supplementary material