Table 2.

Feedback from user-testing sessions and the changes implemented in the app

Suggested change from PPI session:	Changes implemented in the Hypo-METRICS app:
For the items asking, ‘at what time did this/these happen?’ (referring to the hypoglycaemic events), there was an option to ‘add extra timepoints if more than once’. Participants suggested to add an extra item instead asking, ‘how many hypos did you have?’. Further, there was a wish for more clarity on how to classify multiple events versus long-standing ones.	We did as was suggested and removed the ‘add extra timepoints’ option, and included an item asking, ‘how many hypos did you have?’ both in the morning and evening check-in. Further, we added an ‘add another hypo’ function, so participants could respond to the hypoglycaemia-specific items for each event. We wanted participants to judge the difference between multiple and long-standing events themselves, to learn more about how the events are perceived from the participants’ perspective; thus, no changes were implemented on this point.
For the items ‘during the night, did you have a hypo OR take action to prevent a hypo?’ and ‘did you have a hypo today OR did you prevent a hypo today?’ there was uncertainty about what is meant by ‘preventing’. For example, some participants were in doubt if this included having a snack before bed ‘just in case’ rather than preventing an imminent hypoglycaemic event.	We decided to add ‘…prevent a hypo that was about to happen’ to emphasise that we are not trying to capture the ‘just in case’ snacks or insulin reductions, but instead events that were just about to happen, and most likely would have happened if the participant had not taken corrective action.
For the item ‘how anxious/relaxed do you feel right now?’ with the bidirectional 11-point response scale from ‘extremely relaxed (0)’ to ‘extremely anxious (10)’, participants felt that these did not necessarily belong on the same scale.	We decided to change this item to ‘how anxious do you feel right now?’ with a unidirectional 11-point response scale from ‘not at all (0)’ to ‘extremely (10)’. We similarly adjusted other items to make response scales similar.
There was disagreement about the use of the word ‘burden’ in the item ‘how much of a burden was hypoglycaemia last night?’, as it was perceived as overly strong language.	We adjusted the wording of the question to ‘How bothersome was hypoglycaemia last night?’
The items ‘how long did your hypo(s) (on average) prevent you from doing your usual activity’ and ‘how long was it (on average) before you were feeling your ‘usual self’ again?’ caused some confusion, and participants said these would need extra clarification. Further it was suggested not to ask on average, but for each event.	The first item was removed from the app and replaced by several items recommended by health economic experts within the Hypo-RESOLVE consortium to better capture the effect of hypoglycaemia on work and productivity. The last item was changed to ‘Overall… How long was it before you were feeling your ‘usual self’ again?’.
Since the item ‘did your hypo(s) today negatively impact your social activities?’ was placed right after the work-related items, participants were in doubt if the item was asking in relation to work or any activities during the day.	The item was separated from the work-related items and adjusted to ‘how well did you get along with other people today?’. The new wording more accurately captures the intention of the question.
For the cognitive function items asking, ‘how is your concentration/memory/attention right now?’ participants said they found it difficult to answer these items in the morning check-in since they had not done anything in the morning to really concentrate on or remember. Similarly, it was unclear what memory we are referring to (short term, long term or for specific tasks). Further the difference between concentration and attention caused uncertainty.	We changed the items into ‘how alert do you feel right now?’, ‘how well are you able to concentrate right now?’ and ‘how easy was if for you to remember things today?’ and decided to only ask the latter item in the evening check-in, so that participants could reflect on their day in order to make an assessment of whether they experienced any memory difficulties.
A number of functionalities were suggested to include in the app including: A ‘question progress bar’ to see how many questions remain in each check-in A ‘study progress bar’ to see how many days of the study they have left A text field entry field so participants could provide more context A ‘large text’ feature A ‘snooze’ function, so a reminder notification is sent out later.	Unfortunately, the app platform did not support progress bars for question or study progress. For some items, we included an option with free-text field entry but decided not to include free-text options for all items, to minimise participant burden and to avoid large amount of qualitative data that would require extensive analysis. A diary function in the app would allow participant to write additional notes during the study. For the large-text option, we provided a description for how to adjust this in the smartphone settings. The app platform did not support ‘snooze’ functions.

Hypo-METRICS, Hypoglycaemia—MEasurement, ThResholds and ImpaCtS; Hypo-RESOLVE, Hypoglycaemia—Redefining SOLutions for better liVEs; PPI, Patient and public involvement.