Abstract
Introduction
Hypoglycaemia is a frequent adverse event and major barrier for achieving optimal blood glucose levels in people with type 1 or type 2 diabetes using insulin. The Hypo-RESOLVE (Hypoglycaemia—Redefining SOLutions for better liVEs) consortium aims to further our understanding of the day-to-day impact of hypoglycaemia. The Hypo-METRICS (Hypoglycaemia—MEasurement, ThResholds and ImpaCtS) application (app) is a novel app for smartphones. This app is developed as part of the Hypo-RESOLVE project, using ecological momentary assessment methods that will minimise recall bias and allow for robust investigation of the day-to-day impact of hypoglycaemia. In this paper, the development and planned psychometric analyses of the app are described.
Methods and analysis
The three phases of development of the Hypo-METRICS app are: (1) establish a working group—comprising diabetologists, psychologists and people with diabetes—to define the problem and identify relevant areas of daily functioning; (2) develop app items, with user-testing, and implement into the app platform; and (3) plan a large-scale, multicountry study including interviews with users and psychometric validation. The app includes 7 modules (29 unique items) assessing: self-report of hypoglycaemic episodes (during the day and night, respectively), sleep quality, well-being/cognitive function, social interactions, fear of hypoglycaemia/hyperglycaemia and work/productivity. The app is designed for use within three fixed time intervals per day (morning, afternoon and evening). The first version was released mid-2020 for use (in conjunction with continuous glucose monitoring and activity tracking) in the Hypo-METRICS study; an international observational longitudinal study. As part of this study, semistructured user-experience interviews and psychometric analyses will be conducted.
Ethics and dissemination
Use of the novel Hypo-METRICS app in a multicountry clinical study has received ethical approval in each of the five countries involved (Oxford B Research Ethics Committee, CMO Region Arnhem-Nijmegen, Ethikkommission der Medizinischen Universität Graz, Videnskabsetisk Komite for Region Hovedstaden and the Comite Die Protection Des Personnes SUD Mediterranne IV). The results from the study will be published in peer review journals and presented at national and international conferences.
Trial registration number
Keywords: general diabetes, mental health, diabetes & endocrinology, information technology, health economics
Strengths and limitations of this study.
The development of the Hypo-METRICS (Hypoglycaemia—MEasurement, ThResholds and ImpaCtS) (Hypo-METRICS) app is based on a strong multidisciplinary collaboration between psychologists, diabetologists and people with diabetes.
The Hypo-METRICS app was designed for a specific target population (adults aged >18 years with diabetes using insulin) and adaptations may be required for other groups.
The Hypo-METRICS app will be used in a European multicountry clinical study, which will enable its psychometric properties to be examined.
As the app is designed to require the user to complete multiple daily assessments, there is a risk of participant burden and dropout, which requires further investigation.
The use of the novel Hypo-METRICS app in conjunction with continuous glucose monitoring will enable a detailed investigation of the day-to-day impact of hypoglycaemia on various areas of daily life, with minimal recall bias, and will yield a more thorough understanding of variation over time.
Introduction
Hypoglycaemia (low blood glucose) is an important and often burdensome side effect of insulin therapy for people with type 1 or type 2 diabetes (T1DM/T2DM).1 The reported frequency of severe hypoglycaemic episodes (where assistance of others is needed for recovery) has been estimated at 0.2–3.2 episodes per person per year among adults with T1DM and at 0.1–0.7 episodes per person per year in adults with insulin-treated T2DM.2 Self-treated episodes are much more common, yet their prevalence is harder to quantify due to inconsistencies in definitions (symptom-based vs glucose level-based) and the fact that they can be overlooked.3 The weekly prevalence has been estimated at 1–2 episodes in T1DM and 0.3–0.7 in T2DM.2 Hypoglycaemia can be life-threatening,4 is increasingly being associated with a higher risk of future complications and cardiovascular events,4–11 and negatively impacts on psychological well-being12 and quality of life (QoL).4 In order to manage their risk of hypoglycaemia, people with diabetes often adapt their diabetes management (eg, reduce insulin doses, avoid physical activity, increase caloric intake), which can negatively impact on their HbA1c, or adapt their lifestyle (eg, avoid being alone or situations in which hypoglycaemia may occur or cause embarrassment), which can negatively impact on their QoL.13 Hypoglycaemia is commonly seen as major barrier for achieving optimal blood glucose levels.14
Many studies focusing on the personal impact of hypoglycaemia have not examined the temporal relationship between hypoglycaemia and its impact on sleep, mood, cognition, energy levels, social interactions and work productivity, in non-clinical, real-life settings. The impact of hypoglycaemic episodes has typically been assessed retrospectively, with people self-reporting the typical or average impact over several weeks or months,15 which may be prone to underestimation or overestimation due to reduced recall.16 17 Furthermore, retrospective assessments, by definition, cannot assess the immediate effect of each hypoglycaemic episode, including within-person fluctuations over time.
Ecological momentary assessment (EMA) offers the opportunity to overcome some of these limitations and complement insights from retrospective assessments. EMA is a method of collecting data (typically using portable devices such as smartphones) in real-world environments (‘ecological’), addressing a current or very recent state (‘momentary’), measured either randomly, at specific times, or in relation to specific events, with multiple assessments to follow variation over time and across situations.18 Continuous glucose monitoring (CGM) can be considered an objective EMA assessment, capturing episodes of hypoglycaemia 24/7, including those of which the person with diabetes is otherwise unaware.19 Self-reported EMAs can be used to assess an individual’s current thoughts, feelings and behaviours, as well as the contextual factors that may affect them. For these self-reported factors, EMA methods may minimise recall bias, maximise ecological validity and document variation over time,18 providing an opportunity for timely assessment of constructs such as sleep, mood, cognition, energy levels, social interactions and work productivity, particularly when assessed via smartphones (or similar portable devices),20 in parallel with glucose levels assessed using CGM.
EMA research is urgently needed to improve our understanding of the impact of hypoglycaemia on day-to-day life. To address this need, the Hypo-METRICS (Hypoglycaemia—MEasurement, ThResholds and ImpaCtS) application (app) was developed. This paper describes the process of development of the app and the planned psychometric analyses.
Methods and analyses
This study is part of the Innovative Medicines Initiative 2-funded Hypo-RESOLVE (Hypoglycaemia—Redefining SOLutions for better liVEs) project.21 The three phases of the development and planned psychometric analysis of the Hypo-METRICS app are summarised in table 1.
Table 1.
Overview of Hypo-METRICS app development phases and activities
| Phase | Activities |
| Phase 1: Defining the problem | Establish working group and liaise with patient advisory committee |
| Conduct targeted literature review | |
| Develop conceptual framework | |
| Phase 2: Hypo-METRICS app: design and development | Establish general principles for design of the Hypo-METRICS app |
| Develop items and response options | |
| Conduct user-testing and debriefing of Hypo-METRICS app content | |
| Select app platform and design app | |
| Phase 3: Hypo-METRICS app: planning psychometric validation | Design study and key study details |
| Develop psychometric analysis plan |
Hypo-METRICS, Hypoglycaemia—MEasurement, ThResholds and ImpaCtS.
Phase 1: defining the problem
Establish working group and liaise with patient advisory committee (PAC)
A working group with expertise in questionnaire development and validation, medical psychology and endocrinology was established. The role of this group was to define the conceptual framework for the Hypo-METRICS app content, and identify relevant domains for inclusion in the app.
Patient and public involvement
During the 2-year development period, the working group worked collaboratively with the Hypo-RESOLVE PAC and sought monthly input from the wider Hypo-RESOLVE consortium. The PAC members played a key role in setting the agenda, participating in discussions about the content to be included in the app, and providing in-depth feedback on multiple versions of the items as they were developed. In addition to the PAC members, an independent group of people with diabetes without prior knowledge of the project was also invited to test the app content (see below).
Conduct targeted literature review
A targeted literature review was conducted to identify literature focused on the impact of hypoglycaemia. The review served to identify aspects of life and constructs (eg, emotional well-being), that were: (1) relevant to the potential or known personal impact of hypoglycaemia and (2) subject to temporal fluctuation (day-to-day changes).
The construct of QoL was used as a starting point to identify relevant areas of daily life.22 QoL has been defined as a subjective, dynamic and multidimensional construct; consisting of physical, psychological and social aspects.22 The WHO specifies 6 broad domains of QoL, including 24 more specific facets.23 It is important to emphasise that the goal was not to develop an app that measures QoL as a whole (or the impact of hypoglycaemia on QoL), but to use this construct to identify areas of life (in the literature) relevant to the daily, personal impact of hypoglycaemia.
Based on the literature review, the following areas of daily life were regarded as relevant for inclusion in the app. First, hypoglycaemia can impair sleep quality and sleep duration due to the biological consequences of hypoglycaemia and the sleep interruption resulting from managing24–26 or fear of27 night-time episodes. Second, hypoglycaemia can affect physical functioning in several ways; the most frequently reported include feeling tired and less energetic.28 29 Third, hypoglycaemia can be associated with negative emotions including decreased happiness,28 and increased irritability,29 anxiety28 and depressive symptoms.8 Fourth, hypoglycaemia can negatively impact cognitive functioning with reduced alertness29 decreased memory30 and lower concentration.31 Fifth, hypoglycaemia has been associated with higher levels of fear of hypoglycaemia, potentially impairing QoL.32 33 Concerns regarding hyperglycaemia (as a key risk factor for vascular complications) are also relevant due to these potentially leading to more hypoglycaemic episodes through repeated insulin correction doses being given in an attempt to avoid high glucose levels.34 Sixth, qualitative studies in people with diabetes found that fear of hypoglycaemia contributed to avoiding participation in, or disruption to, usual daily activities, such as social activities, driving, sports or work, and that this had a negative impact on QoL.35 Rigid routines, such as intensive glucose monitoring and meal planning, may limit the ability to engage in social activities,35 and hypoglycaemic events were also described as being socially embarrassing.35 Finally, it has been found that hypoglycaemic episodes have substantial economic consequences, causing a loss of productivity amounting to between US$15.26 and US$93.47 (US$2009) per self-treated hypoglycaemic episode and 8.3–15.9 hours of lost work time per month.36 Productivity losses have been reported to be highest for those individuals who experienced nocturnal episodes.36
Develop conceptual framework
Combining the outcomes of the working group discussions and the results from earlier studies into the impact of hypoglycaemia, a conceptual framework was developed (figure 1). It represents the overall constructs relevant to the personal impact of hypoglycaemia (inner circle) and the specific areas of daily functioning to be assessed in the Hypo-METRICS app (outer circle).
Figure 1.
Conceptual framework of the key areas of daily functioning that might be impacted by hypoglycaemia.
Phase 2: Hypo-METRICS app: design and development
Establish general principles for design of the Hypo-METRICS app
Phase 2 involved the development of the specific questions for the app. A group of items listed within an area is referred to as a ‘module’. For practical reasons, the conceptual framework titles were not used as module names, although each of the specific areas of daily life from the conceptual framework is represented across the modules. To start, a set of general principles for the app was developed. The app design process involved developing item content, response options (eg, check-box responses or labels for the scales) and response scales (numerical scales). An iterative approach was used involving multiple meetings between the main working group, PAC members and the wider Hypo-RESOLVE consortium, followed by refinement of the app, and circulation to stakeholders for feedback. After initial consensus regarding the app items, 3 user-testing sessions, involving 15 people with diabetes who had not been involved in the development phase, were held at King’s College London in March 2019. The purpose was to refine and ensure the feasibility of the app items. Once the app content was finalised, it was implemented into a smartphone platform provided by uMotif Limited (London, UK).
In the early phases of the app development, the following general principles were defined. It was agreed that the Hypo-METRICS app should:
Be suitable for use in clinical studies targeting adults (>18 years) with T1DM or T2DM to examine the potential direct impact of hypoglycaemia on daily life.
Be suitable to capture information about self-reported hypoglycaemia.
Be suitable for administration via a smartphone, providing user-friendly access, and optimised for use on both iOS and Android devices.
Include only relevant domains, with a view to minimising burden on participants, non-completion of specific items or timepoints, or study attrition.
Be suitable for multiple assessments per day, to ensure data collection as close as possible to hypoglycaemic episodes as they occur (thereby minimising recall bias) and at other times as required by a study protocol (in the absence of preceding hypoglycaemia).
Be optimised for collecting and storing data in accordance with data protection regulations to ensure confidentiality of participant information.
Use recommended language related to diabetes and people with diabetes (ie, non-judgemental and non-stigmatising).37 38
Develop items and response options
When developing app items, the working group considered that it might be difficult for the person with diabetes to determine whether and to what extent (un)recognised hypoglycaemia impacted on a certain area of life. For example, mood can be concurrently impacted by hypoglycaemia and many other factors, and separating these can be challenging.39 Therefore, it was decided that the majority of app questions should be phrased in a general way rather than being attributed to hypoglycaemia specifically. The questions would instead be asked frequently (three times daily) in a general manner (eg, ‘How is your mood right now?’), thereby enabling responses to be linked later with either person reported hypoglycaemia or CGM-detected hypoglycaemia, to investigate correlations with hypoglycaemia in its different manifestations (symptomatic and asymptomatic). Asking general questions routinely, regardless of hypoglycaemia, allows for a comparison between days (or nights) with versus without hypoglycaemia.
Another consideration for item development was the number of daily assessments. Existing literature does not provide clear consensus on the optimal number of assessments (called ‘check-ins’ in the Hypo-METRICS app) or sampling frequency.40 41 Building the app with three daily ‘check-ins’ was a consensus decision based on a compromise between capturing as much variation over the day as possible, while allowing for use of the app in studies with longer durations (ie, multiple weeks), wherein it is key to minimise participant burden, as this could impact on completion rates and attrition.
The frequency with which each app module or items are presented to the respondent throughout a given day was determined by: (a) the amount of variation expected throughout the day in the construct being measured and (b) the feasibility of responding to certain items at certain times of day; for example, the work-related items were only presented in the evening check-in (after work hours), while mood was assessed at every check-in.
Two modules: ‘self-report of hypos while asleep’ in the morning check-in and ‘self-report of daytime hypos’ in the evening check-in, were developed with a branching option. This means that respondents are only asked to respond to items in these modules if they have already reported a particular experience, for example, hypoglycaemia while asleep. In this instance, respondents would be asked additional questions specific to each event (ie, time reference, detection and management). These modules also consist of questions not specific to single episodes but to hypoglycaemia overall across the day or night (eg, loss of sleep due to hypoglycaemia and worries about going back to sleep). An additional item was developed for these two modules to assess how psychologically bothersome hypoglycaemia was overall. To expand the investigation of hypoglycaemia’s impact on daily activities, an overall item was included in the evening check-in asking, ‘how long was it before you were feeling your ‘usual self’ again?’.
While some modules were assessed with single items (eg, social interactions), others were assessed with multiple items (eg, mood and cognitive function). The number of items selected to measure each construct was dependent on the complexity and dimensionality of the concept. The items were developed as questions (eg, ‘how anxious do you feel right now?’) instead of potentially leading statements (‘I’m feeling anxious’). The goal was to use short and precise sentences and avoid double-barrelled statements (eg, ‘I woke up feeling fresh and rested’). Negatively phrased items, which could be leading for participants, were avoided when possible (eg, ‘how is your mood right now?’ instead of ‘how depressed do you feel?’). The aim was to use non-academic, everyday language; for example, instead of asking about ‘sleep quality’, participants were asked how they slept and how they felt when they woke up. The time attribution for each item was qualified with use of ‘right now’, ‘last night’, ‘today’, ‘later today’ and ‘while asleep’. For items about event timing, only approximate time points were requested to reduce the participant recall burden. The item order was modified slightly between the check-ins to minimise the risk of developing response habits and participants just ‘clicking through’.42 Several of these decisions were informed by experts in questionnaire development within the consortium.
Response scales were another integral aspect of item development. When considering the number of points/options on a response scale, it has been suggested that the quality of measurement does not seem to improve beyond 7–11 points on a numerical scale.43 It was decided to use an 11-point numerical rating scale, (0–10) to maximise sensitivity to (even minor) changes and to minimise floor and ceiling effects. Other app-based EMA studies have similarly used 11-point scales.44–46 To ensure both daily minor variations and the more extreme and rare cases of variation were captured, both unipolar (eg, ‘not at all—extremely’) and bipolar (eg, ‘extremely badly—extremely well’) response options were used. Numbers in the middle of the scale were not labelled.
Conduct user-testing and debriefing of Hypo-METRICS app content
A group of people with diabetes without prior knowledge of the app, was invited to provide their feedback on the draft items and response options. Participants were recruited via local diabetes clinics (King’s College Hospital for people with T1DM and a UK general practitioner clinic for people with T2DM). The user-testing occurred in parallel to the item development process and was an integral part of finalising the app content.
A total of seven people with T1DM (four women, three men, aged 19–55 years) and eight with T2DM (four women, four men, aged 59–72 years) using multiple daily insulin injections (at least two per day) participated in the user-testing sessions. Participants met as two separate groups in two sessions to provide feedback on the app content; however, participants only tested the questions as a paper-and-pencil version and not in the uMotif platform. All participants with T1DM experienced hypoglycaemia multiple times per week, while the reported experience in those with T2DM ranged from less than once per month to multiple times per week. Overall, participants expressed intact awareness of hypoglycaemia, although four of the people with T2DM reported that since they did not experience hypoglycaemia frequently, their partners often (one participant) or sometimes (three participants) would recognise a hypoglycaemic episode before they did.
The overall feedback on the item content was positive, and participants expressed the importance of addressing the day-to-day impact of hypoglycaemia. Participants reported that completion of the app items three times per day was a feasible task. A selection of the feedback is provided in table 2.
Table 2.
Feedback from user-testing sessions and the changes implemented in the app
| Suggested change from PPI session: | Changes implemented in the Hypo-METRICS app: |
| For the items asking, ‘at what time did this/these happen?’ (referring to the hypoglycaemic events), there was an option to ‘add extra timepoints if more than once’. Participants suggested to add an extra item instead asking, ‘how many hypos did you have?’. Further, there was a wish for more clarity on how to classify multiple events versus long-standing ones. | We did as was suggested and removed the ‘add extra timepoints’ option, and included an item asking, ‘how many hypos did you have?’ both in the morning and evening check-in. Further, we added an ‘add another hypo’ function, so participants could respond to the hypoglycaemia-specific items for each event. We wanted participants to judge the difference between multiple and long-standing events themselves, to learn more about how the events are perceived from the participants’ perspective; thus, no changes were implemented on this point. |
| For the items ‘during the night, did you have a hypo OR take action to prevent a hypo?’ and ‘did you have a hypo today OR did you prevent a hypo today?’ there was uncertainty about what is meant by ‘preventing’. For example, some participants were in doubt if this included having a snack before bed ‘just in case’ rather than preventing an imminent hypoglycaemic event. | We decided to add ‘…prevent a hypo that was about to happen’ to emphasise that we are not trying to capture the ‘just in case’ snacks or insulin reductions, but instead events that were just about to happen, and most likely would have happened if the participant had not taken corrective action. |
| For the item ‘how anxious/relaxed do you feel right now?’ with the bidirectional 11-point response scale from ‘extremely relaxed (0)’ to ‘extremely anxious (10)’, participants felt that these did not necessarily belong on the same scale. | We decided to change this item to ‘how anxious do you feel right now?’ with a unidirectional 11-point response scale from ‘not at all (0)’ to ‘extremely (10)’. We similarly adjusted other items to make response scales similar. |
| There was disagreement about the use of the word ‘burden’ in the item ‘how much of a burden was hypoglycaemia last night?’, as it was perceived as overly strong language. | We adjusted the wording of the question to ‘How bothersome was hypoglycaemia last night?’ |
| The items ‘how long did your hypo(s) (on average) prevent you from doing your usual activity’ and ‘how long was it (on average) before you were feeling your ‘usual self’ again?’ caused some confusion, and participants said these would need extra clarification. Further it was suggested not to ask on average, but for each event. | The first item was removed from the app and replaced by several items recommended by health economic experts within the Hypo-RESOLVE consortium to better capture the effect of hypoglycaemia on work and productivity. The last item was changed to ‘Overall… How long was it before you were feeling your ‘usual self’ again?’. |
| Since the item ‘did your hypo(s) today negatively impact your social activities?’ was placed right after the work-related items, participants were in doubt if the item was asking in relation to work or any activities during the day. | The item was separated from the work-related items and adjusted to ‘how well did you get along with other people today?’. The new wording more accurately captures the intention of the question. |
| For the cognitive function items asking, ‘how is your concentration/memory/attention right now?’ participants said they found it difficult to answer these items in the morning check-in since they had not done anything in the morning to really concentrate on or remember. Similarly, it was unclear what memory we are referring to (short term, long term or for specific tasks). Further the difference between concentration and attention caused uncertainty. | We changed the items into ‘how alert do you feel right now?’, ‘how well are you able to concentrate right now?’ and ‘how easy was if for you to remember things today?’ and decided to only ask the latter item in the evening check-in, so that participants could reflect on their day in order to make an assessment of whether they experienced any memory difficulties. |
A number of functionalities were suggested to include in the app including:
|
Unfortunately, the app platform did not support progress bars for question or study progress. For some items, we included an option with free-text field entry but decided not to include free-text options for all items, to minimise participant burden and to avoid large amount of qualitative data that would require extensive analysis. A diary function in the app would allow participant to write additional notes during the study. For the large-text option, we provided a description for how to adjust this in the smartphone settings. The app platform did not support ‘snooze’ functions. |
Hypo-METRICS, Hypoglycaemia—MEasurement, ThResholds and ImpaCtS; Hypo-RESOLVE, Hypoglycaemia—Redefining SOLutions for better liVEs; PPI, Patient and public involvement.
After an iterative design process, including debriefing of items and response options with potential users, a total of 29 unique items were selected to best represent the conceptual framework (figure 1), and were presented in the app via seven modules (table 3).
Table 3.
Items per module, and completion timepoints (‘check-ins’)
| Module names and items | Conceptual framework domain | Completion timepoints (‘check-ins’) | ||
| Morning | Afternoon | Evening | ||
| Sleep quality module (two items) | ||||
| 1. How well did you sleep? | Sleep quality | x | ||
| 2. When you woke up how did you feel? | Sleep quality | x | ||
| General well-being module (seven items) | ||||
| 3. How is your mood right now? | Mood | x | x | x |
| 4. How anxious do you feel right now? | Anxiety | x | x | x |
| 5. How is your energy level right now? | Energy levels | x | x | x |
| 6. How irritable do you feel right now? | Mood | x | x | x |
| 7. How alert do you feel right now? | Cognitive function | x | x | x |
| 8. How easy was if for you to remember things today? | Cognitive function | x | ||
| 9. How well are you able to concentrate right now? | Cognitive function | x | x | x |
| Fear of hypoglycaemia/hyperglycaemia module (four items) | ||||
| 10. How worried are you about having a hypo later today? | Fear | x | x | |
| 11. How worried are you about having high blood glucose later today? | Fear | x | x | |
| 12. How worried are you about having a hypo while asleep? | Fear | x | ||
| 13. How worried are you about having high blood glucose while asleep? | Fear | x | ||
| Social interactions module (one item) | ||||
| 14. How well did you get along with other people today? | Social interactions | x | ||
| Work and productivity module (four items) | ||||
| 15. How many hours did you work today? | Work/productivity | x | ||
| 16. How many hours did you miss from work for ANY reason today?(this includes health issues, vacation, holiday, etc.) | Work/productivity | x | ||
| 17. How many hours did you miss from activities other than work today for ANY reason (eg, study, housework, shopping, family or leisure activities)? | Leisure activities | x | ||
| 18. How productive were you while working today? | Work/productivity | x | ||
| Self-report of hypos while asleep module* (eight items) | ||||
| 19. During the night, did you have a hypo OR take action to prevent a hypo that was about to happen?† | NA | x | ||
| 20. How many hypos did you have? | NA | x | ||
| 21. At what time did this happen? | NA | x | ||
| 22. How did you detect your hypo or a hypo that was about to happen? (Select all that apply) | NA | x | ||
| 23. What happened? (Select all that apply) | NA | x | ||
| 24. Overall: How bothersome was hypoglycaemia for you last night? | Burden | x | ||
| 25. Overall: How much sleep did you lose due to hypoglycaemia? | Sleep quality | x | ||
| 26. Overall: How worried were you about going back to sleep? | Sleep quality | x | ||
| Self-report of daytime hypos module* (seven items) | ||||
| 27. Today, did you have a hypo OR take action to prevent a hypo that was about to happen?† | NA | x | ||
| 20.1 How many hypos did you have? | NA | x | ||
| 21.1 At what time did this happen? | NA | x | ||
| 22.1 How did you detect your hypo or a hypo that was about to happen? | NA | x | ||
| 23.1 What happened? | NA | x | ||
| 28. Overall: How bothersome was hypoglycaemia for you today? | Burden | x | ||
| 29. Overall: How long was it before you were feeling your ‘usual self’ again? | Daily living/usual activities | x | ||
*Several of these items are not part of the conceptual framework, but were included to capture details about the hypoglycaemic episodes
†These items have branching: if a hypo is reported, the items below are presented to the participant for completion.
Select app platform and design app
After the items and response options were finalised, they were implemented into a software platform provided by ‘uMotif Limited’ with a data capture application that can be used on iOS and Android compatible smartphones47 (see figure 2). 'uMotif Limited' was chosen due to its high data security and confidentiality policies that comply with current EU General Data Protection Regulation (GDPR) laws and has been used in other patient-centred data capture studies.48 49 In order to maximise feasibility, participants could only complete check-ins at predefined time intervals: from 06:00 to 12:00 (morning), 12:00 to 18:00 (afternoon) and 18:00 to 24:00 (evening). Participants could self-initiate the check-ins but were not able to complete the individual check-ins outside these time intervals. The app was further configured to provide automated notifications (at predefined times of day: 07:00 hours, 15:00 hours and 21:00 hours) inviting participants to complete check-ins in the morning, afternoon and evening, respectively. The wide time intervals were chosen to increase the likelihood of completion.
Figure 2.
Sample of screenshots of the Hypoglycaemia—MEasurement, ThResholds and ImpaCtS app on the uMotif Limited platform.
Phase 3: Hypo-METRICS app: planning psychometric validation
Phase 3 is focused on the planned investigation of the psychometric properties of the Hypo-METRICS app for the measurement of the day-to-day personal impact of hypoglycaemia.
Design study and key study details
The Hypo-METRICS app has been implemented for the first time in the Hypo-METRICS clinical study, a large, prospective multicountry study starting October 2020 and led by the Hypo-RESOLVE consortium.21 Briefly, participants are asked to complete three daily check-ins (morning, afternoon, evening) on their smartphone for 10 weeks, while wearing a blinded CGM to measure glucose values throughout the day and night. This study will enable largescale testing and psychometric analysis of the Hypo-METRICS app.
The target population for this study is European adults with T1DM or insulin-treated T2DM, and the sample of participants chosen to represent this population will consist of 600 adults (aged 18–85 years) recruited from eight specialist diabetes centres across five countries (Austria, Denmark, France, the Netherlands, the UK). The Hypo-METRICS app was developed in English and afterwards translated into the four other languages. The translation plan was developed and based on the principles for translating patient-reported outcomes as described by Wild et al.50 After providing informed consent, participants will attend a baseline visit (physically or online), where training in use of the app will be provided. Further details on the Hypo-METRICS clinical study, including the full list of objectives, are available online.51
Develop psychometric analysis plan
With the development of a new instrument, it is important to examine its validity and reliability.52 Using data from the Hypo-METRICS clinical study, including user-experience interviews with a subset of participants, the latent structure, internal consistency, test–retest reliability, construct validity, feasibility and acceptability, and completion rates of the app will be explored. One of the key aims of the Hypo-METRICS clinical study is to explore associations between CGM data and Hypo-METRICS app responses. To avoid double reporting of results, these analyses will not be included in the current validation study.
Latent structure, internal consistency, test–retest reliability and construct validity
The examination of the validity and reliability of the Hypo-METRICS app will start with an investigation of the latent structure of the app items to examine whether items can be grouped in factors. A multilevel factor analysis will be conducted separately for each of the three check-ins to avoid violating assumptions of independence between the repeated measurements.53 Further, internal consistency of items listed under each latent factor will be investigated using McDonald’s ω.54 To explore test–retest reliability, factor scores will be aggregated and compared (via correlation analysis) across two different weeks. To examine between-person and within-person variability on an item level, intraclass correlations55 and root mean square of successive differences56 will be calculated. Lastly, construct validity will be examined by analysing the correlations between the items or factor scores from the Hypo-METRICS app and validated self-report questionnaires (listed in online supplemental table S1).54 These questionnaires assess either constructs where a moderate-to-strong relationship (convergent validity) or weak relationship (discriminant validity) with the app items is expected. Although the app items and the validated questionnaires focus on different time frames, moderate correlations are still expected as they address the same constructs.
bmjopen-2021-051651supp001.pdf (146.5KB, pdf)
Feasibility and acceptability (via user-experience interviews)
Although the content (items and response options) of the Hypo-METRICS app has been tested by people with diabetes, the finalised Hypo-METRICS app (ie, following integration into the ‘uMotif Limited’ platform) has not yet undergone full user-testing. Semistructured interviews will be undertaken with approximately 20 participants of the Hypo-METRICs study to explore the acceptability and feasibility of the app, and their experiences of using the app in their daily lives. Participants will be purposively sampled to ensure diversity on the following characteristics: type of diabetes, sex, age and completion rate.
Completion rates
An analysis of completion rates and patterns of missing data from the clinical study will be performed on the full sample (n=600). The proportion of check-ins and items not submitted together with the number of skipped items (ie, where participants have submitted the check-in but ‘skipped’ an item) will be examined. Using multilevel analyses, factors that predict completion (eg, day of study, time of check-in, age, sex, type of diabetes) will be determined. Distribution of responses, including how long after the notification the participants on average respond and the distribution of responses for each item, will similarly be examined. This analysis may help to refine future versions of the app and to determine the types of studies/contexts suitable for use of the app.
Ethics and dissemination
Ethical considerations are pertinent to this work. The participants are not required to provide personal information when registering to use the Hypo-METRICS app; instead they will use study-specific email addresses (eg, participantnumber@gmail.com) and can enter their study number instead of their name. The participant requires access to a smartphone (iOS or Android system) and either WIFI or mobile data for entering responses.
For analytic purposes, all data will be handled as pseudonymised data. ‘uMotif Limited’ will only process encrypted data. Data are stored securely in accordance with GDPR at all times. The Hypo-METRICS clinical study has received ethical approval at the lead site and in all five European countries.
There is a risk that the completion of items (and additional questionnaires used for validation purposes) required for the study may over-burden participants or cause discomfort. In these situations, the participants can opt to skip questions and/or seek assistance from the healthcare professional at their local recruitment centre.
The results from the psychometric analyses and the semistructured interviews will be submitted to peer-reviewed and open access journals, and further presented at both national and international conferences.
Ethical approval was not required for the Hypo-METRICS app development. The Hypo-METRICS clinical study has received ethical approval at the lead site from the South Central Oxford B Research Ethics Committee (20/SC/0112) and in the other European countries (in the Netherlands by CMO Region Arnhem-Nijmegen, in Austria by Ethikkommission der Medizinischen Universität Graz, in Denmark by Videnskabsetisk Komite for Region Hovedstaden and in France by the Comite Die Protection Des Personnes SUD Mediterranne IV).
Discussion
Hypoglycaemia is an important complication of insulin treatment among people with diabetes. In this paper, the systematic development of the Hypo-METRICS app, tailored to determine the impact of hypoglycaemia on daily functioning, is described. The iterative design process, involving multidisciplinary teamwork between psychologists and diabetologists in close collaboration with people with diabetes, was key to the app development. The feedback from user-testing with people with diabetes (who had not been involved in the item development) was overall positive. They found it manageable to complete the questions across the planned three daily check-ins. In this paper, we also present the planned psychometric validation work that will be carried out with data from a multicountry clinical study, where the Hypo-METRICS app will be used for the first time by a large number of participants over a 10-week study period. This study will further allow for in-depth interviews with a subset of participants who have used the app.
It is anticipated that the Hypo-METRICS app will minimise recall bias, maximise ecological validity, document variation over time and allow for a more in-depth understanding of the day-to-day impact of hypoglycaemia. The app includes seven modules (29 unique items) assessing: self-report of hypoglycaemic episodes (during the day and night, respectively), sleep quality, well-being/cognitive function, social interactions, fear of hypoglycaemia/hyperglycaemia and work/productivity. Once the Hypo-METRICS app has undergone psychometric analysis, the authors anticipate that it will provide a novel tool for researchers to more accurately examine the impact of hypoglycaemia. The Hypo-METRICS app may be used as a key outcome in clinical trials evaluating new glucose lowering medications or new diabetes technology, but it can perhaps also be used in clinical settings to further optimise diabetes care and outcomes for individuals with diabetes. It must be emphasised that the Hypo-METRICS app has been developed for adults with diabetes (using insulin) in the UK, Denmark, the Netherlands, Austria and France, and that adaptations will be required for its use in other groups (eg, youth with diabetes, pregnant women with diabetes) and other countries.
Supplementary Material
Acknowledgments
The authors thank the people with diabetes and members of the Hypoglycaemia—Redefining SOLutions for better liVEs consortium who participated in the development and user-testing of the Hypoglycaemia—MEasurement, ThResholds and ImpaCtS (Hypo-METRICS) app items. We also thank all the people who contributed to translation of the Hypo-METRICS app and organization of local cognitive debriefing at each study site, including: Dutch site: Evertine Abbink (MD, PhD), Namam Ali (MD, PhD Student) and Ash Abbink (student); affiliated with Department of Internal Medicine of the Radboud University Medical Centre in Nijmegen, the Netherlands. Austrian site: Monika Cigler (MD, PhD), Daniel Hochfellner (MD), Amra Simiċ (MA in English) and Tina Pöttler; affiliated with Medical University of Graz, Austria. French site: Jérôme Place (MSc), Al Masri Manal (CRA), Omar Diouri (PhD) and Anne-Marie Marteil-Oudrer (MD); affiliated with Montpellier University Hospital, France. Danish site: Mette Valdersdorf Jensen (PhD student, M.P.H.), affiliated with University of Southern Denmark; and Stine Tving Kjøller (Research Nurse) affiliated with Nordsjællands Hospital, Hillerød Denmark.
Footnotes
Twitter: @uffehs, @BroadleyMelanie, @nzarembakcl, @janespeight, @drpratikc, @FransPouwer
Collaborators: Hypo-RESOLVE Consortium.
Contributors: All authors were involved in the conceptualisation of the app. US, MB, NZ, PD, CH, JS, PC and FP developed the app with expertise input and advice from AB, DJP, BdG, GN, RJM, UP-B and SA. NZ and PC conducted the user testing of app items and response options. US, MB, JS, CH and FP produced the first manuscript draft. US, MB and FP planned the psychometric analyses and semistructured interviews with advice from JS, CH, ZM, NZ and PC. All authors reviewed the manuscript at multiple stages and provided feedback. All authors approved the final draft of the manuscript.
Funding: This work was supported by the Innovative Medicines Initiative 2 Joint Undertaking (JU) under grant agreement No 777460. The JU receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA and T1D Exchange, JDRF, International Diabetes Federation and The Leona M. and Harry B. Helmsley Charitable Trust. The industry partners supporting the JU include Abbott Diabetes Care, Eli Lilly, Medtronic, Novo Nordisk and Sanofi-Aventis. JS and CH are supported by core funding to the Australian Centre for Behavioural Research in Diabetes provided by the collaboration between Diabetes Victoria and Deakin University.
Competing interests: UPB has received grants and personal fees from Novo Nordisk and personal fees from AstraZeneca, Bristol-Meyers Squibb, Sanofi and Zealand Pharma.
Provenance and peer review: Not commissioned; externally peer reviewed.
Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
Ethics statements
Patient consent for publication
Not applicable.
References
- 1. Leonard CE, Han X, Brensinger CM, et al. Comparative risk of serious hypoglycemia with oral antidiabetic monotherapy: a retrospective cohort study. Pharmacoepidemiol Drug Saf 2018;27:9–18. 10.1002/pds.4337 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Frier BM. Hypoglycaemia in diabetes mellitus: epidemiology and clinical implications. Nat Rev Endocrinol 2014;10:711–22. 10.1038/nrendo.2014.170 [DOI] [PubMed] [Google Scholar]
- 3. Little SA, Leelarathna L, Barendse SM, et al. Severe hypoglycaemia in type 1 diabetes mellitus: underlying drivers and potential strategies for successful prevention. Diabetes Metab Res Rev 2014;30:175–90. 10.1002/dmrr.2492 [DOI] [PubMed] [Google Scholar]
- 4. Fidler C, Elmelund Christensen T, Gillard S. Hypoglycemia: an overview of fear of hypoglycemia, quality-of-life, and impact on costs. J Med Econ 2011;14:646–55. 10.3111/13696998.2011.610852 [DOI] [PubMed] [Google Scholar]
- 5. Ahammed A, Pathan F, Afsana F, et al. The burden of severe hypoglycemia on quality of life among diabetes mellitus patients in a tertiary level hospital of Bangladesh. Indian J Endocrinol Metab 2018;22:499–504. 10.4103/ijem.IJEM_338_17 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Davis RE, Morrissey M, Peters JR, et al. Impact of hypoglycaemia on quality of life and productivity in type 1 and type 2 diabetes. Curr Med Res Opin 2005;21:1477–83. 10.1185/030079905X61929 [DOI] [PubMed] [Google Scholar]
- 7. Rombopoulos G, Hatzikou M, Latsou D, et al. The prevalence of hypoglycemia and its impact on the quality of life (QOL) of type 2 diabetes mellitus patients (the HYPO study). Hormones 2013;12:550–8. 10.14310/horm.2002.1443 [DOI] [PubMed] [Google Scholar]
- 8. Green AJ, Fox KM, Grandy S, et al. Self-Reported hypoglycemia and impact on quality of life and depression among adults with type 2 diabetes mellitus. Diabetes Res Clin Pract 2012;96:313–8. 10.1016/j.diabres.2012.01.002 [DOI] [PubMed] [Google Scholar]
- 9. Lopez JM, Annunziata K, Bailey RA, et al. Impact of hypoglycemia on patients with type 2 diabetes mellitus and their quality of life, work productivity, and medication adherence. Patient Prefer Adherence 2014;8:683–92. 10.2147/PPA.S58813 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Polonsky WH, Fisher L, Hessler D. The impact of non-severe hypoglycemia on quality of life in patients with type 2 diabetes. J Diabetes Complications 2018;32:373–8. 10.1016/j.jdiacomp.2018.01.014 [DOI] [PubMed] [Google Scholar]
- 11. International Hypoglycaemia Study Group . Hypoglycaemia, cardiovascular disease, and mortality in diabetes: epidemiology, pathogenesis, and management. Lancet Diabetes Endocrinol 2019;7:385–96. 10.1016/S2213-8587(18)30315-2 [DOI] [PubMed] [Google Scholar]
- 12. Hendrieckx C, Halliday JA, Bowden JP, et al. Severe hypoglycaemia and its association with psychological well-being in Australian adults with type 1 diabetes attending specialist tertiary clinics. Diabetes Res Clin Pract 2014;103:430–6. 10.1016/j.diabres.2013.12.005 [DOI] [PubMed] [Google Scholar]
- 13. Khunti K, Alsifri S, Aronson R, et al. Impact of hypoglycaemia on patient-reported outcomes from a global, 24-country study of 27,585 people with type 1 and insulin-treated type 2 diabetes. Diabetes Res Clin Pract 2017;130:121–9. 10.1016/j.diabres.2017.05.004 [DOI] [PubMed] [Google Scholar]
- 14. Shafiee G, Mohajeri-Tehrani M, Pajouhi M, et al. The importance of hypoglycemia in diabetic patients. J Diabetes Metab Disord 2012;11:17. 10.1186/2251-6581-11-17 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Pedersen-Bjergaard U, Alsifri S, Aronson R, et al. Comparison of the HAT study, the largest global hypoglycaemia study to date, with similar large real-world studies. Diabetes Obes Metab 2019;21:844–53. 10.1111/dom.13588 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Tan NC, Goh S-Y, Khoo EY-H, et al. Self-Reported hypoglycaemia in insulin-treated patients with diabetes mellitus: results from the Singapore cohort of the International operations hypoglycaemia assessment tool study. Singapore Med J 2020;61:129–36. 10.11622/smedj.2019081 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Blome C, Augustin M. Measuring change in quality of life: bias in prospective and retrospective evaluation. Value Health 2015;18:110–5. 10.1016/j.jval.2014.10.007 [DOI] [PubMed] [Google Scholar]
- 18. Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol 2008;4:1–32. 10.1146/annurev.clinpsy.3.022806.091415 [DOI] [PubMed] [Google Scholar]
- 19. Henriksen MM, Andersen HU, Thorsteinsson B, et al. Asymptomatic hypoglycaemia in type 1 diabetes: incidence and risk factors. Diabet Med 2019;36:62–9. 10.1111/dme.13848 [DOI] [PubMed] [Google Scholar]
- 20. Berkman ET, Giuliani NR, Pruitt AK. Comparison of text messaging and paper-and-pencil for ecological momentary assessment of food craving and intake. Appetite 2014;81:131–7. 10.1016/j.appet.2014.06.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. de Galan BE, McCrimmon RJ, Ibberson M, et al. Reducing the burden of hypoglycaemia in people with diabetes through increased understanding: design of the hypoglycaemia redefining solutions for better liVEs (Hypo-RESOLVE) project. Diabet Med 2020;37:1066–73. 10.1111/dme.14240 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Speight J, Reaney MD, Barnard KD. Not all roads lead to Rome-a review of quality of life measurement in adults with diabetes. Diabet Med 2009;26:315–27. 10.1111/j.1464-5491.2009.02682.x [DOI] [PubMed] [Google Scholar]
- 23. Billington R, et al. The New Zealand World Health organization quality of life (WHOQOL) group. Vol. 123, 2010: 65–70. [PubMed] [Google Scholar]
- 24. Biggers A, Sharp LK, Nimitphong H, et al. Relationship between depression, sleep quality, and hypoglycemia among persons with type 2 diabetes. J Clin Transl Endocrinol 2019;15: :62–4. 10.1016/j.jcte.2018.12.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Brod M, Christensen T, Bushnell DM. Impact of nocturnal hypoglycemic events on diabetes management, sleep quality, and next-day function: results from a four-country survey. J Med Econ 2012;15:77–86. 10.3111/13696998.2011.624144 [DOI] [PubMed] [Google Scholar]
- 26. Hayashino Y, Tsujii S, Ishii H, et al. High frequency of non-nocturnal hypoglycemia was associated with poor sleep quality measure by Pittsburg sleep quality index in patients with diabetes receiving insulin therapy: diabetes distress and care registry at Tenri (DDCRT 4). Exp Clin Endocrinol Diabetes 2013;121:628–34. 10.1055/s-0033-1355424 [DOI] [PubMed] [Google Scholar]
- 27. Martyn-Nemeth P, Phillips SA, Mihailescu D, et al. Poor sleep quality is associated with nocturnal glycaemic variability and fear of hypoglycaemia in adults with type 1 diabetes. J Adv Nurs 2018;74:2373–80. 10.1111/jan.13765 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Gold AE, Deary IJ, Frier BM. Hypoglycaemia and non-cognitive aspects of psychological function in insulin-dependent (type 1) diabetes mellitus (IDDM). Diabet Med 1997;14:111–8. [DOI] [PubMed] [Google Scholar]
- 29. Geelhoed-Duijvestijn PH, Pedersen-Bjergaard U, Weitgasser R, et al. Effects of patient-reported non-severe hypoglycemia on healthcare resource use, work-time loss, and wellbeing in insulin-treated patients with diabetes in seven European countries. J Med Econ 2013;16:1453–61. 10.3111/13696998.2013.852098 [DOI] [PubMed] [Google Scholar]
- 30. Sommerfield AJ, Deary IJ, McAulay V, et al. Short-Term, delayed, and working memory are impaired during hypoglycemia in individuals with type 1 diabetes. Diabetes Care 2003;26:390–6. 10.2337/diacare.26.2.390 [DOI] [PubMed] [Google Scholar]
- 31. Brod M, Rana A, Barnett AH. Impact of self-treated hypoglycaemia in type 2 diabetes: a multinational survey in patients and physicians. Curr Med Res Opin 2012;28:1947–58. 10.1185/03007995.2012.743457 [DOI] [PubMed] [Google Scholar]
- 32. Barendse S, Singh H, Frier BM, et al. The impact of hypoglycaemia on quality of life and related patient-reported outcomes in type 2 diabetes: a narrative review. Diabet Med 2012;29:293–302. 10.1111/j.1464-5491.2011.03416.x [DOI] [PubMed] [Google Scholar]
- 33. Shi L, Shao H, Zhao Y, et al. Is hypoglycemia fear independently associated with health-related quality of life? Health Qual Life Outcomes 2014;12:167. 10.1186/s12955-014-0167-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Singh H, Gonder-Frederick L, Schmidt K, et al. Assessing hyperglycemia avoidance in people with type 1 diabetes. Diabetes Manag 2014;4:263–71. 10.2217/dmt.14.3 [DOI] [Google Scholar]
- 35. Vanstone M, Rewegan A, Brundisini F, et al. Patient perspectives on quality of life with uncontrolled type 1 diabetes mellitus: a systematic review and qualitative Meta-synthesis. Ont Health Technol Assess Ser 2015;15:1–29. [PMC free article] [PubMed] [Google Scholar]
- 36. Brod M, Christensen T, Thomsen TL, et al. The impact of non-severe hypoglycemic events on work productivity and diabetes management. Value Health 2011;14:665–71. 10.1016/j.jval.2011.02.001 [DOI] [PubMed] [Google Scholar]
- 37. Dickinson JK, Guzman SJ, Maryniuk MD, et al. The use of language in diabetes care and education. Diabetes Educ 2017;43:551–64. 10.1177/0145721717735535 [DOI] [PubMed] [Google Scholar]
- 38. Speight J, Conn J, Dunning T, et al. Diabetes Australia position statement. A new language for diabetes: improving communications with and about people with diabetes. Diabetes Res Clin Pract 2012;97:425–31. 10.1016/j.diabres.2012.03.015 [DOI] [PubMed] [Google Scholar]
- 39. Eldar E, Rutledge RB, Dolan RJ, et al. Mood as representation of momentum. Trends Cogn Sci 2016;20:15–24. 10.1016/j.tics.2015.07.010 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40. Rosenkranz T, Takano K, Watkins ER, et al. Assessing repetitive negative thinking in daily life: development of an ecological momentary assessment paradigm. PLoS One 2020;15:e0231783. 10.1371/journal.pone.0231783 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41. Eisele G, Vachon H, Lafit G, et al. The effects of sampling frequency and questionnaire length on perceived burden, compliance, and Careless responding in experience sampling data in a student population. Assessment 2020:1073191120957102. 10.1177/1073191120957102 [DOI] [PubMed] [Google Scholar]
- 42. Silvia PJ, Kwapil TR, Walsh MA, et al. Planned missing-data designs in experience-sampling research: Monte Carlo simulations of efficient designs for assessing within-person constructs. Behav Res Methods 2014;46:41–54. 10.3758/s13428-013-0353-y [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43. DeCastellarnau A. A classification of response scale characteristics that affect data quality: a literature review. Qual Quant 2018;52:1523–59. 10.1007/s11135-017-0533-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44. Emerson JA, Dunsiger S, Williams DM. Reciprocal within-day associations between incidental affect and exercise: an EMA study. Psychol Health 2018;33:130–43. 10.1080/08870446.2017.1341515 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45. Kratz AL, Murphy SL, Braley TJ. Ecological Momentary assessment of pain, fatigue, depressive, and cognitive symptoms reveals significant daily variability in multiple sclerosis. Arch Phys Med Rehabil 2017;98:2142–50. 10.1016/j.apmr.2017.07.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. Engelen L, Held F. Understanding the office: using ecological Momentary assessment to measure activities, posture, social interactions, mood, and work performance at the workplace. Buildings 2019;9:54. 10.3390/buildings9020054 [DOI] [Google Scholar]
- 47. uMotif . uMotif. Available: https://www.umotif.com/ [Accessed 20 Jul 2020].
- 48. Lakshminarayana R, Wang D, Burn D, et al. Using a smartphone-based self-management platform to support medication adherence and clinical consultation in Parkinson's disease. NPJ Parkinsons Dis 2017;3:2. 10.1038/s41531-016-0003-z [DOI] [PMC free article] [PubMed] [Google Scholar]
- 49. Dixon WG, Beukenhorst AL, Yimer BB, et al. How the weather affects the pain of citizen scientists using a smartphone APP. NPJ Digit Med 2019;2:105. 10.1038/s41746-019-0180-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 50. Wild D, Grove A, Martin M, et al. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (pro) measures: report of the ISPOR Task force for translation and cultural adaptation. Value Health 2005;8:94–104. 10.1111/j.1524-4733.2005.04054.x [DOI] [PubMed] [Google Scholar]
- 51. Hypo-METRICS . Hypoglycaemia - Measurement, ThResholds and ImpaCtS (Hypo-METRICS), 2020. Available: https://www.clinicaltrials.gov/ct2/show/record/NCT04304963
- 52. Palmier-Claus JE, Myin-Germeys I, Barkus E, et al. Experience sampling research in individuals with mental illness: reflections and guidance. Acta Psychiatr Scand 2011;123:12–20. 10.1111/j.1600-0447.2010.01596.x [DOI] [PubMed] [Google Scholar]
- 53. Reise SP, Ventura J, Nuechterlein KH, et al. An illustration of multilevel factor analysis. J Pers Assess 2005;84:126–36. 10.1207/s15327752jpa8402_02 [DOI] [PubMed] [Google Scholar]
- 54. Forkmann T, Spangenberg L, Rath D, et al. Assessing suicidality in real time: a psychometric evaluation of self-report items for the assessment of suicidal ideation and its proximal risk factors using ecological momentary assessments. J Abnorm Psychol 2018;127:758–69. 10.1037/abn0000381 [DOI] [PubMed] [Google Scholar]
- 55. Theobald E. Students are rarely independent: when, why, and how to use random effects in Discipline-Based education research. CBE Life Sci Educ 2018;17:rm2. 10.1187/cbe.17-12-0280 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56. Woyshville MJ, Lackamp JM, Eisengart JA, et al. On the meaning and measurement of affective instability: clues from chaos theory. Biol Psychiatry 1999;45:261–9. 10.1016/S0006-3223(98)00152-8 [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
bmjopen-2021-051651supp001.pdf (146.5KB, pdf)