Daily angina documentation versus subsequent recall: development of a symptom smartphone app

Abstract

Aims

The traditional approach to documenting angina outcomes in clinical trials is to ask the patient to recall their symptoms at the end of a month. With the ubiquitous availability of smartphones and tablets, daily contemporaneous documentation might be possible.

Methods and results

The ORBITA-2 symptom smartphone app was developed with a user-centred iterative design and testing cycle involving a focus group of previous ORBITA participants. The feasibility and acceptability were assessed in an internal pilot of participants in the ongoing ORBITA-2 trial. Seven days of app entries by ORBITA-2 participants were compared with subsequent participant recall at the end of the 7-day period. The design focus group tested a prototype app. They reported that the final version captured their symptoms and was easy to use. In the completion assessment group, 141 of 142 (99%) completed the app in full and 47 of 141 (33%) without reminders. In the recall assessment group, 29 of 29 (100%) participants said they could recall the previous day’s symptoms, and 82% of them recalled correctly. For 2 days previously, 88% said they could recall and of those, 87% recalled correctly. The proportion saying they could recall their symptoms fell progressively thereafter: 89, 67, 61, 50%, and at 7 days, 55% (P < 0.001 for trend). The proportion of recalling correctly also fell progressively to 55% at 7 days (P = 0.04 for trend).

Conclusion

Episode counts of angina are difficult to recall after a few days. For trials such as ORBITA-2 focusing on angina, daily symptom collection via a smartphone app will increase the validity of the results.

Graphical Abstract

Graphical abstract.

Introduction

The emphasis on symptom relief in stable angina has increased since the value of coronary revascularization for reducing cardiovascular outcomes like myocardial infarction and death was questioned.^1,2. It is important that symptom relief is measured in an accurate, reliable, and patient-centred manner.

Modes of assessment include exercise time, angina frequency, e.g. Seattle Angina Questionnaire (SAQ),³ and quality of life, e.g. EuroQOL-5D (EQ-5D). Symptoms can be self-reported or physician-assessed, but both rely on patients recalling their symptoms over a period of time.

ORBITA was a randomized placebo-controlled trial of percutaneous coronary intervention (PCI) for patients with single-vessel coronary artery disease.⁴ Participants were randomized to PCI or placebo in a 1 to 1 ratio. The primary endpoint of ORBITA was exercise time. ORBITA-2⁵ was designed in line with feedback from ORBITA participants and the clinical and scientific cardiology community. ORBITA-2 is recruiting people with symptoms of stable angina, evidence of ischaemia, and significant coronary stenosis in at least one vessel. In particular, the endpoint is an ordinal clinical outcome scale for angina which contains daily angina frequency.⁶

This paper describes the development of the smartphone application used for documenting daily symptoms, including a feasibility and acceptability assessment, and a comparison with subsequent recall.

Methods

The app was designed in partnership with patients who participated in the ORBITA trial which was completed in 2017. The aim of the app is to collect data for the blinded ORBITA-2 trial, i.e. to measure the clinical response to PCI or placebo. The trial methods have been previously described.⁵ The primary outcome of ORBITA was exercise time and angina was primarily measured with the SAQ. The primary outcome of the ORBITA-2 trial is an ordinal clinical outcome scale which includes daily angina frequency reported on the smartphone app and the SAQ is a secondary outcome.

Development overview

The app was developed using a user-centred iterative design and testing cycle. This involved a focus group of people with lived experience of stable angina and participation in clinical trials. They were asked what elements would adequately cover their symptoms for reporting to researchers. Their feedback was recorded as notes during the meetings. They were each given a prototype of the app to try recording symptoms, and again their feedback was recorded. The app was iteratively improved and retested by members of the focus group.

The first prototype of the app was based on feedback from ORBITA participants and investigators regarding the limitations of other angina assessments such as exercise testing and the Seattle Angina Questionnaire. The next iteration incorporated the results of a competitive analysis. The competitive analysis involved an appraisal of existing angina apps [Angina Control (Google Playstore), Angina Recorder (Apple App store), and Heart Keeper (Google Playstore)] and apps for other chronic pain conditions (Sora, Migraine Buddy, and RheumaBuddy for endometriosis, migraines, and rheumatoid arthritis, respectively) to inform the design of the smartphone app.

Participants record the number of episodes experienced each day. The app also records the severity of the episodes. Participants use a slider to grade their worst episode that day from mild to severe on a scale (which they can interpret as the intensity and/or duration depending on their personal preference). However, in the ORBITA-2 trial, only angina frequency is included in the primary endpoint.

Feasibility was assessed in two ways: first by focus group members using the prototype app and second with pilot data from actual ORBITA-2 participants. The pilot data included completion rates of daily symptom app entries, the proportion of participants requiring reminders to complete the app, and how many reminders were required. The app data dashboard indicated when a participant had not recorded their symptoms for 3 days and this triggered a reminder to be sent.

Acceptability was also assessed by focus group members using the prototype app and with pilot data from actual ORBITA-2 participants. Focus group member opinions were collected through verbal feedback and a written survey. ORBITA-2 participant’s feedback was collected verbally at the end of their involvement in the study.

Participants

The design focus group participants were participants from the previously completed ORBITA trial. All participants from ORBITA who agreed to subsequent contact were invited to focus group meetings.

The recall assessment group was a substudy of ORBITA-2, composed of ORBITA-2 participants who had their final visit between 1 August 2020 and 30 April 2021 inclusive.

The completion assessment group were all participants in ORBITA-2 who had their final visit before 30 April 2021.

The ability to use the app is not an inclusion criterion for ORBITA-2. All participants receive one-to-one onboarding with a researcher to teach them to use the app and answer any queries. All participants who consented to participate have been able to complete the app. It is possible that the use of an app is a deterrent to some potential participants, but almost all of those approached already had devices compatible with apps.

Focus groups

The aims of the focus groups were to gather feedback for the design of the ORBITA-2 trial which included the primary endpoint, smartphone app, patient-facing literature, and the patient experience.

Technical aspects

The app design team created a progressive web app, designed to be available on any smartphone, tablet, or computer. Usability was tested across these platforms.

The user interface was tested and iteratively improved by focus group members who could try it on their phones and iPads available during focus group meetings.

Maintenance of data integrity was tested by the app design team, including ensuring adequate logging of dates, times, users, and changes.

User experience

In this phase, we optimized features such as font, readability, the onboarding process, error messages, e.g. for lack of Internet connection, and the frequently asked questions section, with input from the focus group members.

Regulatory considerations

A data protection impact assessment was carried out and a GDPR statement was prepared. Users are required to agree to a privacy policy during onboarding.

According to Medicines and Healthcare Products Regulatory Agency (MHRA) Guidance: Medical device stand-alone software including apps (including IVDMDs) v1.08⁷ the app is not a device. This is contingent on the app not being used to make decisions about clinical care. It is merely a tool to collect data for research purposes.

Statistical analysis

Each day participants record the number of angina episodes. These data are used for angina frequency and are not transformed or adjusted. Data are summarized as n (%) and median (interquartile range). Trends in proportions were assessed using logistic regression. Analyses were performed using the open-source statistical environment ‘R’ Version 4.1.0.⁸

Results

We developed a working prototype web-based app tested across various Android smartphones, iPhones, tablets, and computer platforms.

Participants

The design focus group was composed of 10 participants of the previous ORBITA trial, of whom one was female. The recall assessment group consisted of 29 ORBITA-2 participants who had their final study visit between 1 August 2020 and 30 April 2021. The completion assessment group consisted of the 142 ORBITA-2 participants who had their final study visit by 30 April 2021.

Design focus group and app design

The design focus group had meetings during October 2018. They reviewed prototype apps on their phones and/or iPad tablets which were available for their use during the meetings. They tried out the onboarding process. Table 1 shows their comments and the resulting changes made to the app and the participant onboarding process.

Table 1.

Recommendations from the design focus group

Feedback from focus group	Researcher comments	App modification
There is a practice on an imaginary patient named Bob. Can there be practice on oneself as well?	The reason that the practice module refers to an imaginary patient was (i) to cover a wide range of possible answers, (ii) create no possible confusion between the patient’s genuine symptom status and the practice process, and (iii) minimize the duration of the practice session	Onboarding process is mandated to be done one-to-one with a researcher so that any queries can be resolved immediately
	All participants were able to enter their data after a single practice session
Can you add a Help button?	During the prototype design, whenever there was a possible need for a Help button, the design was changed to become more obvious so that participants would not need to rely on the Help	Added a button with reference information and contact phone number
Why can we not describe our symptoms in more detail?	The app is to document symptoms numerically in all participants for each day. It was paramount in the design of the app that the burden on participants should be minimal	A free text box for description of symptoms was added to the onboarding process only. We decided not to invite additional free text description in the daily entries because this would be burdensome, inconsistently entered and not usable as an endpoint
	Descriptions of the symptoms are obtained via separate questionnaires
Should not the app include instructions for use?	Every extra sentence in an app can make it look complicated to some participants and thereby reduce completion rates	The participant workflow was modified, with additional concisely worded instructions shown but only at the precise stage that they are required
		The onboarding process was augmented with data entry on an imaginary patient for a series of days of pre-specified symptoms that cover the full range of possibilities
		A Help button was added to provide additional instructions and a contact phone number

Design focus group and angina induction by personalized reference activities

Discussion with the design focus group reminded us that people may modify their activity to avoid episodes of angina. We therefore wanted ORBITA-2 to include questions about symptom responses to reference activities that were stable during trial participation but were individually tailored to the participant at enrolment.

The design focus group completed a written survey of activities that triggered their angina episodes. Their responses are shown in Table 2. The design focus group advised us that changes in symptom response to these particular stimuli would be the best indicator of whether the treatment had worked, since it was relevant to their life and was the reason they originally sought medical help.

Table 2.

Focus group member responses to survey of angina triggers

Respondent	I always have angina with	I sometimes have angina with	Other activities causing angina
1	Running 5–10 km	Walking up hill	No response
2	Cycling up hill	Walking up hill	Very cold weather
3	Walking up hill	Walking quickly	No response
4	Walking far	No response	No response
5	Walking up hill	No response	No response
6	Cycling	Walking	No response
7	Walking fast up hill	Stress	No response
8	Walking up hill with cold weather	Walking up hill	No response
9	No response	Walking up hill	No response
10	Walking up stairs	Walking 2.5 km	No response

We used their responses to provide in the ORBITA-2 app a list of activities. This is shown in Figure 1. During onboarding, participants select two activities that currently cause angina. The list also contains an option for participants to enter an unlisted activity in free text form. This is shown in Figure 1. Subsequently, every week during the trial, participants are asked whether they had angina when conducting the two pre-specified personalized activities.

Figure 1.

Options and examples of weekly provocation questions on app.

Completion assessment group

The design focus group completed a written survey of activities that triggered their angina episodes. Their responses are shown in Table 2. The design focus group advised us that changes in symptom response to these particular stimuli would be the best indicator of whether the treatment had worked, since it was relevant to their life and was the reason they originally sought medical help.

We used their responses to provide in the ORBITA-2 app a list of activities. This is shown in Figure 1. During onboarding, participants select two activities that currently cause angina. The list also contains an option for participants to enter an unlisted activity in free text form. This is shown in Figure 1. Subsequently, every week during the trial, participants are asked whether they had angina when conducting the two pre-specified personalized activities.

Reminders required over the duration of participation were not specific to the time period within the trial (Figure 2B).

Figure 2.

(A) Distribution of the number of reminders needed in the completion assessment group. (B) Reminders by week of trial in the completion assessment group.

Recall assessment group

The recall assessment group were asked at the end of their participation to recall their number of episodes in each of the seven previous days individually. All said they could recall the previous day’s symptoms, but the proportion declined for each day before that, reaching 10 of 18 (55%) for 7 days previously (Figure 3A, P < 0.001 for trend).

Figure 3.

(A) Stated ability and actual ability to recall numbers of angina episodes. (B) Stated ability and actual ability to recall numbers of angina episodes when at least one episode was recorded.

Recollections did not always match what they had entered in the app. For the previous day, 14 of 17 (82%) were able to give a correct recollection. This proportion declined for each day before that, reaching 55% for 7 days previously (Figure 3A, P = 0.04 for trend).

Rates of correct recall were lower in those who had had at least one episode (Figure 3B).

The recall assessment group were also asked to recall the total number of episodes in the past 4 weeks. Six of 29 (21%) said they could not recall. The remaining 23 participants reported a total of 157 episodes in the past 4 weeks, median 2 (IQR 0–12). Ten of 23 (43%) recalled correctly, of whom 8 of 10 (80%) had had no episodes.

Discussion

In this study, we found that recalling numbers of episodes of angina is difficult or impossible after a few days unless that number is zero. For patients who experienced angina, episodes that occurred more than 1 day ago were only 14% likely to be correctly recalled, while they were 29% likely to be incorrectly recalled, with the remainder stating that they could not recall.

The ORBITA-2 symptom app was developed with patients who had completed the first ORBITA trial, because they recommended symptoms to be the primary focus of ORBITA-2. In the completion assessment group, 99% completed data entry in full.

Avoiding confounding by patients limiting activity to avoid angina

The design focus group advised that simply counting episodes would understate the impact of the condition because they reported adapting their activity patterns to avoid experiencing angina. ORBITA-2, therefore, asked each participant at enrolment to identify two reference activities which characteristically caused them angina and which they would continue to carry out at least weekly throughout their participation. Separate from the main assessment which was daily episode count, the app asked them weekly whether they experienced angina on each of the two personalized reference activities.

Personalizing the nature of angina

Members of the design focus group, during their discussions, noticed that their individual experiences of angina were different although generally stereotyped within an individual. They wanted this to be formally documented somehow in ORBITA-2. We were mindful that repeatedly asking large numbers of categorical questions could be irritating and still leave the participant with the feeling that their individual symptom has not been adequately described. Certainly repeating the process regularly during the study would impair the participant goodwill on which the trial depends.

ORBITA-2 therefore asks participants to describe their symptoms in their own words. This personal definition of angina is available on the app for them to refer back to. This allows a single daily question to cover the participant’s own angina without the excessive wordiness of listing all possible variants.

Ability to recall

The ability to recall (correctly or incorrectly) episodes of angina declines progressively within a few days to reach only 55% at 7 days. Even this limited recall is bolstered by those who actually had zero episodes. For those with one or more episodes to recall, the ability to recall (correctly or incorrectly) reached only 25% at 7 days.

We were surprised by the rapidity of decline in these proportions. The incorrect recall was only a problem for symptoms in the very recent past. For symptoms that were even further back, participants would simply say they could not recall at all.

ORBITA-2 collects daily data via the symptom app so that its angina episode counts can be as representative as possible. The episode counts reported at the end of the trial occur only because these are part of conventional questionnaires. The inaccurate recall is a renowned issue in clinical research for chronic pain conditions and a daily smartphone approach known as Ecological Momentary Assessment has been used for conditions such as breakthrough cancer pain⁹ and fibromyalgia¹⁰ for this reason.

Implications

The app is being used for daily angina reporting in other studies assessing symptom relief for stable angina.^11,12 The use of the app for monitoring in a clinical setting warrants further investigation in future, e.g. for patients within cardiac rehabilitation programmes or monitoring response to therapy.

Study limitations

The design focus group only contains patients who participated in ORBITA. Thus, it does not include people who may not agree to participate in a trial or undergo a randomized blinded procedure. It may therefore not be representative of the general population of patients with angina. However, we were doing this work with the intention of applying to the ORBITA-2 trial and therefore we believe that the focus group is well qualified to advise.

The Recall Substudy did not include all participants in ORBITA-2 but rather all recruited within a particular time window. We have no reason to believe that these participants were any different from the generality of ORBITA-2 participants.

For the reference measurement, we used the daily reports from participants via the symptom app. We did not make daily phone calls nor ask them to keep a separate paper diary. Therefore, we cannot exclude the possibility that they entered incorrect information on the app. On the other hand, telephone calls would be unnecessarily irritating for participants, jeopardizing retention in the trial. With paper diaries, there is no way to tell that entries are made contemporaneously rather than just before a subsequent visit. Therefore, we consider the daily entries on the app to be a suitably contemporaneous reference data set.

We did not try multiple different protocols for reminding participants, for example, starting when they are only 1 day late. We settled with our focus group on a pattern that they considered acceptable.

While it is possible that patients may report more symptoms including potentially trivial symptoms via the app, contemporaneous reporting is the gold standard for symptoms and limits post hoc interpretation of the symptoms.

Conclusion

As time passes, it becomes increasingly difficult to recall episodes of angina accurately. When patients have had angina, accurate recall falls to 25% even within 1 week. For clinical trials focusing on angina endpoints, daily documentation is therefore advisable. All participants found a smartphone app easy to use. One-third of patients entered all their data without needing reminders, and the other two-thirds required a mean of 0.4 reminders per week of participation. Overall, data collection was 99% complete.

Research in context

Evidence before this study

Clinical trials of angina often use questionnaires at the end of a follow-up period to assess angina outcomes. These rely on participant recall of symptoms.

Added value of this study

The ability to recall symptoms correctly declines dramatically with time. Daily symptom reporting on a smartphone application was acceptable to clinical trial participants. Two-thirds of participants required reminders when they were behind schedule on data entry.

Implications of all the available evidence

Patients who have participated in randomized trials assessing angina are an ideal expert group to advise on how angina can be assessed without being unreasonably intrusive. We recommend this partnership model for the development of future symptom reporting apps for clinical trials of symptom relief. Daily symptom reporting and a protocol of reminders when behind schedule are acceptable to real-life clinical trial participants.

Funding

ORBITA-2 is funded by Imperial College NIHR BRC. A.N.N. is supported by the NIHR Academy.

Conflict of interest: R.K.A.-L. receives speaker’s honoraria from Phillips Volcano and Menarini Pharmaceuticals. A.N.N. is supported by the NIHR Academy. M.F. and C.R. are supported by the MRC. T.R.K. has research grants from Abbott Vascular, Zoll Corporation, and speaker fee honoria from Abbott Vascular, Astra Zeneca, and BD. We acknowledge support from the National Institute for Health Research Imperial Biomedical Research Centre and the Imperial British Heart Foundation Centre of Research Excellence. F.E.H. receives consulting fees from Regeneron, Johnson & Johnson/Jenssen, Annexon, Provention Bio, and AstraZeneca for unrelated projects.

Data availability

Data will be made available on request.