Evaluation of prototype risk prediction tools for clinicians and people living with type 2 diabetes in North West London using the think aloud method

Abstract

The prevalence of type 2 diabetes in North West London (NWL) is relatively high compared to other parts of the United Kingdom with outcomes suboptimal. This presents a need for more effective strategies to identify people living with type 2 diabetes who need additional support. An emerging subset of web-based interventions for diabetes self-management and population management has used artificial intelligence and machine learning models to stratify the risk of complications from diabetes and identify patients in need of immediate support. In this study, two prototype risk prediction tools on the MyWay Diabetes and MyWay Clinical platforms were evaluated with six clinicians and six people living with type 2 diabetes in NWL using the think aloud method. The results of the sessions with people living with type 2 diabetes showed that the concept of the tool was intuitive, however, more instruction on how to correctly use the risk prediction tool would be valuable. The feedback from the sessions with clinicians was that the data presented in the tool aligned with the key diabetes targets in NWL, and that this would be useful for identifying and inviting patients to the practice who are overdue for tests and at risk of complications. The findings of the evaluation have been used to support the development of the prototype risk predictions tools. This study demonstrates the value of conducting usability testing on web-based interventions designed to support the targeted management of type 2 diabetes in local communities.

Introduction

Diabetes is a metabolic disorder that can have long-term health implications for affected individuals. It is estimated that by 2045 there will be 693 million people worldwide living with diabetes between the age of 18 and 99 years.¹ In England, one of the regions with the highest prevalence of diabetes is North West London (NWL), particularly amongst ethnic minority groups of South Asian, African and Caribbean descent.² NWL represents a high population density region and one of the most socioeconomically diverse areas in the United Kingdom.² According to data held by Public Health England, in 2017, it was estimated that 179,798 people in NWL were living with diagnosed and undiagnosed diabetes, representing 9.3% of the population.³

Uncontrolled type 2 diabetes has long been associated with complications such as cardiomyopathy, amputation, renal failure and retinopathy. The impact of such complications on patient well-being is evident, with significant cost implications for the National Health Service (NHS). For example, it has been estimated that 1% of the NHS budget is spent on treating complications from diabetes such as ulcers and amputations which equates to approximately £900 million per annum.⁴ Earlier awareness and intervention for at-risk individuals could reduce the human and financial costs associated with uncontrolled diabetes.

Supported self-management is considered an effective approach to the management of long-term conditions such as diabetes. However, the psychological impact of receiving a diagnosis of type 2 diabetes has been associated with different attitudes towards the perceived seriousness of the diagnosis. In addition, the relationship and communication between people living with diabetes and healthcare professionals has been shown to impact patient adherence to clinical advice and treatments.^5–7

Diabetes self-management education has long been an essential part of supporting people living with type 2 diabetes, particularly to improve glycaemic control in the short term for newly diagnosed patients. However, the uptake of face-to-face educational programmes is as low as <10% of newly diagnosed people with type 2 diabetes in England.⁸ The emergence of digital health has presented an opportunity to overcome the challenges associated with low uptake of structured support, increase the scope of self-management beyond initial diagnosis as well as reduce costs associated with face-to-support.⁹ Web-based interventions for diabetes management have been in development and use since the 1990s, the designs of which have exponentially improved to increase the functional capabilities and interactivity.¹⁰

An emerging subset of web-based interventions for diabetes self-management and population management has used artificial intelligence and machine learning models to stratify the risk of complications from diabetes and identify patients in need of immediate support. Such models analyse large amounts of data extracted from electronic patient records (e.g. body mass index (BMI), biomarkers and smoking status) to estimate the probability of diabetes-related complications in the short term (e.g. hypoglycaemia) and long term (renal failure, myocardial infarction and retinopathy).¹¹ One of the challenges with regard to the uptake and long-term engagement of digital health technologies for diabetes self-management and population management is improving the usability of tools, ensuring that the intended user can navigate and utilise features within the tools independently.⁹

The aim of this study is to gather feedback on the usability and acceptability of prototype risk prediction tools on the MyWay Diabetes and MyWay Clinical platforms amongst people with diabetes and amongst general practitioners (GPs), respectively, in NWL.

Methods

MyWay Digital Health web-based interventions

MyWay Digital Health is a company that originated from the University of Dundee in Scotland. MyWay Digital Health has two web-based interventions in ongoing development; MyWay Clinical which is a clinician-facing platform for the provision of integrated care and performance reporting; and MyWay Diabetes which is a self-management platform for people living with type 2 diabetes that allows users to view their medical data and delivers personalise advice and self-management education. MyWay Diabetes has been designed to support people at any point in their clinical journey from diagnosis and regardless of their current complication status.

Alongside historical delivery of online education resources (MyWay Diabetes) and clinical data access and interpretation (MyWay Diabetes and MyWay Clinical), both platforms have been augmented using machine learning algorithms and large clinical datasets to serve different use cases; the algorithm in MyWay Diabetes predicts the user's risk of developing complications from type 2 diabetes; whereas the algorithm in MyWay Clinical presents the user with their patients’ short-term and long-term risk of developing complications from type 2 diabetes at the individual and cohort levels. The models used to develop the risk prediction elements of the platform were originally validated using NHS Scotland datasets. For the purpose of this study, the models were revalidated using the NWL Discover dataset hosted by Imperial College Health Partners. The dataset includes coded primary care, secondary, acute, mental health, community health and social care records for over 2.5 million patients who live in and are registered with a GP in NWL. The dataset is one of Europe's largest linked longitudinal costed datasets fed by data from over 400 provider organisations including over 350 GP practices, 2 mental health and 2 community trusts and all acute providers attended by NWL patients (in the form of Secondary Users Service (SUS) data).¹²

The MyWay Diabetes platform has been commissioned in NWL (under the Know Diabetes branding: www.knowdiabetes.org.uk) and MyWay Clinical platform was additionally tested as part of a Health Data Research UK (HDRUK)-funded project. However, the specific risk prediction functionality discussed here has not yet been rolled out at scale to intended users in NWL.

An overview of the prototype MyWay Diabetes and MyWay Clinical risk prediction features is shown in Tables 1 and 2, respectively.

Table 1.

MyWay Diabetes risk prediction tool.

Page element	Screenshot and description
Current and new risk calculation and visualisation	At the top of the page, the user's current risk of complications (left) and their new risk of complication (right) from diabetes are displayed. In the test environment of the prototype, the numerical value for the current risk was randomly generated. In practice, the current risk is calculated from the user's patient record data within the MyWay Diabetes platform. Each dot represents 1% of the population living with type 2 diabetes. The number of coloured dots in the new risk visualisation increases or decreases after the sliders below for Weight and Activity are adjusted by the user. The view of the page on 100% zoom means that users must scroll down to view the tool features which enabled them to adjust the weight and activity sliders (see below).
Weight and Activity should be capitalised sliders	The circles can be clicked and dragged to change the value of the user’s weight (kg) and activity (hours per week). Adjusting the sliders would change the new risk calculation and visualisation at the top of the page.
HbA1c and BMI display box.	The user's most recent HbA1c and BMI values are displayed in this static box at the bottom of the page.
Smoking and Statins toggle buttons	The user can adjust the toggle buttons to indicate their smoking status and/or if they are taking statins. Adjusting these buttons would change the new risk calculation and visualisation.
Weight box	An information box appears on the page when the user adjusts the Weight slider. The box contains information about how weight loss impacts HbA1c.

The MyWay IQ platform and designs are commercial property of MyWay Digital Health; Patent Pending. United Kingdom Patent Application No. 2217554.1. BMI: body mass index; HbA1c: glycated haemoglobin.

Table 2.

MyWay Clinical – risk prediction tool and future risks pages.

Page element	Screenshot and description
Risk Predictions	The Risk Predictions sidebar allows the user to adjust the slider button to view the patient's predicted risk of amputation, blindness, myocardial infarction and renal failure at 1+, 2+, 5+ or 10+ years. The Risk Prediction values are calculated from the patient record data populated in MyWay Clinical. An additional category, Mortality, is only visible when the user adjusts the slider to 10+ years.
Future Risks	A mouseover or mouse hover event across the segments of the data visualisations displays the risk category and percentage of the practice patients stratified to the respective category.
Future Risks – range	The range sidebar enables the user to change the visualisation to display the future risk of practice patients in 1+, 2+, 5+ or 10+ years.
Future Risks – key	The key on the Future Risks page is related to the percentage of the practice population who are low (green), medium (amber) or high risks (red) of complications from diabetes. In other parts of the MyWay Clinical tool, the same categorisation applies to practice compliance with key diabetes targets and key performance indicators.
Future Risks – population stratification feature (populated with synthetic patient data).	Clicking on a segment within the data visualisation takes users to a page in which they can view a list of practice patients in a specific risk category in X amount of years for a type of diabetes complication.

The MyWay IQ platform and designs are commercial property of MyWay Digital Health; Patent Pending. United Kingdom Patent Application No. 2217554.1.

Recruitment

Participation was sought from people living with type 2 diabetes who are registered with a practice in NWL, as well as GPs based in NWL. A sample size of at least five people per user group was sought as this number is considered sufficient for uncovering most of the usability problems in a round of usability testing.¹³ People with type 2 diabetes were identified and recruited via the NWL Health Research Register, a consent-to-contact research register for all adults in NWL. The Register has over 6500 registrants to date, including over 500 people living with type 2 diabetes. GP participants were identified through the research team's professional networks. Age and sex data were collected for participants who live with type 2 diabetes. No demographic data were collected for GP participants. Convenience sampling was used to identify people living with type 2 diabetes and GPs who were readily available to participate in the study on a ‘first-come, first-served basis’.

The research team contacted potential participants via email and/or phone call to inform them of the study. Those who expressed interest in the study were emailed a copy of the participant information sheet to read and had the opportunity to ask questions. Those who agreed to take part in the study signed an informed consent form to participate in the study and were invited to book an hour session with the research team.

Participants who live with type 2 diabetes were paid £30 per hour (£25 per hour for their involvement plus an additional £5 to cover WiFi/electricity) as per guidelines originally developed by the UK National Advisory group (known as INVOLVE) and reviewed by its successor the National Institute of Health and Care Research (NIHR) Centre for Engagement and Dissemination.¹⁴ GP participants were paid £100 per hour.

Study design

The think aloud sessions were conducted between April and May 2021. Due to the coronavirus disease 2019 (COVID-19) pandemic restrictions for face-to-face contact, this study was designed to be conducted remotely using Microsoft Teams to host the evaluation sessions. Prior to the sessions, a Know Diabetes account was set up for participants living with type 2 diabetes to allow them access to the main dashboard and risk prediction tool. For GP participants, a MyWay Clinical account was set up. During the sessions, participants were instructed to share their screen and were provided links to the platforms that were to be evaluated.

The ‘think aloud’ method was used during the sessions, in which participants were instructed to navigate the web pages and tell the research team what they are looking at, thinking, doing and feeling. The think aloud method is a commonly used method of usability testing as it is a direct way of understanding the cognitive processes that users go through while performing a given task.¹⁵ A study guide was developed by the research team and included prompts and actions for the participants to undertake during the session. The purpose of the guide was to standardise the activities that participants completed across the sessions and elicit insight into any cognitive or technical insights about the platforms.

Analysis

The think aloud sessions were video and audio recorded on Microsoft Teams. Audio recordings were transcribed manually by the research team using intelligent verbatim only to correct grammatical errors and sentence structure. Key themes from the sessions were derived using an inductive approach.¹⁶ The themes were then categorised and grouped according to the features of the MyWay Clinical and MyWay Diabetes to which participants referred (e.g. Weight and Activity sliders, Risk Predictions). The categorised themes were then re-ordered based on the hierarchy of the web applications (home page, main dashboard, web pages of specific metrics, etc.). Based on the findings across the key themes, recommendations were made to improve the usability of the applications. Informative quotes were compiled with identifiable information removed to protect participants’ identities.

Results

Participant characteristics

People living with type 2 diabetes

Six people living with type 2 diabetes were consented to participate in the study and have been included in the analysis. Most participants were female (4 females and 2 males) and the mean age was 53 years (range 33–70, median = 52, SD = 14.8). A total of 12 people living with type 2 diabetes were invited to participate in the study. Three people were excluded from the study: one person had not had their data transferred successfully from the GP record to Know Diabetes, and two people did not have an email address on their GP record. One person withdrew for medical reasons and two people were withdrawn from the study due to technical problems accessing a computer device and internet connection.

General practitioners

Six GPs based in NWL were consented to participate in the study and have been included in the analysis. A total of 14 GPs were invited to participate in the study. Three declined interest or did not respond to the invitation, with an additional five GPs choosing not to participate due to clinical commitments.

Themes from MyWay Diabetes analysis

The main themes derived from the think aloud sessions with people living with type 2 diabetes were the presentation of information and medical data, navigation, colour scheme, use of graphics and icons, autonomy and terminology. Overall, participants were supportive of the concept of the risk prediction tool; however, technical limitations of the test product and usability issues impacted the user experience during the sessions.

Presentation of information and medical data

One of the main observations within this theme was that the tool provided minimal instructions and there was a lack of information about the intended use of the tool.

While most participants understood that the risk prediction tool is intended to provide them with a trajectory of their health in relation to their current diagnosis of type 2 diabetes, some participants interpreted that the tool is aimed at prediabetic patients who are presented with their risk of becoming diabetic.

The current risk is where I am standing now, and the new risk is based on what I am working on for myself. – Person living with type 2 diabetes

It's giving me my risk towards being diabetic. – Person living with type 2 diabetes

Although most participants took the data that they were presented with at face value, one participant stated it could be useful to include more patient identifiers at the top of the webpage, for example, date of birth and NHS number. This was perceived to be a valuable addition for people with a common name (e.g. John Smith), and to reinforce the message to all users that the risk scores presented to them are based on their own medical data.

In case something bad happens, like being shown as high risk, if I have my name and age somewhere I know for sure it is me. – Person living with type 2 diabetes

Most participants commented that the relevant parameters related to the risk of complications from type 2 diabetes are displayed in the tool. Although one participant commented that there is nowhere on the tool to indicate a past history of smoking.

I think the thing all doctors take into account is being a previous smoker. – Person living with type 2 diabetes

Upon adjusting the Weight slider, a dark green dialogue box appeared on the left-hand side of the page (see Table 1, Weight box). The dialogue box was not immediately registered by most participants. However, participants who were prompted to comment on the information in the dialogue box thought it was very useful.

Was it there before? Did it change colour? – Person living with type 2 diabetes

It's useful because it tells you that you can reduce your blood glucose by reducing your weight. – Person living with type 2 diabetes

Navigation

The observation was that most participants needed to be prompted to scroll down to view the section of the tool where they can adjust the sliders. On 100% zoom settings for the page, participants could only see the visualisation of their current and new risks (see Table 1, Current and new risk calculation and visualisation). This resulted in participants scrolling up and down to adjust the sliders at the bottom of the page and view the new calculation of their new risk. Although participants thought the ability to adjust the parameters to see their new risk was useful, they suggested adjusting the layout of the page to make it clearer to them and other users as to how to use the tool.

It's the wrong way round, you need to fill out the bit below before you know what your new risk is. – Person living with type 2 diabetes

Maybe you could have the adjust section at the top and then the new risk at the bottom. I would be looking down to see where my answer goes. – Person living with type 2 diabetes

If I make a change here where are the result of what I am doing? It should show me exactly what I need. How is this helping in this box? – Person living with type 2 diabetes

Colour scheme

The observation from the sessions was that most participants either assumed or questioned if a traffic light system applied to dots in the current and new risk visualisation (see Table 1, Current and new risk calculation and visualisation). They expected that the colour of the dots would change between green, orange or red depending on their score, however, they did not observe a colour change during the sessions.

I assume it's a traffic light system and that might be low but you should endeavour to make it lower. Colour changes with risk. – Person living with type 2 diabetes

If the new risk was a bit more of a colour change it would be good to show the difference. Low is green which gives you some positivity that you’re doing better. More is red, which means just to be on alert that your risk could be higher. – Person living with type 2 diabetes

Use of graphics and icons

The feature of the risk prediction tool that participants liked the most was the visual representation of risk. Most of the participants understood that each dot represented 1% risk and that they should aim to keep the number of dots low. Some participants were unsure as to what the dots represented and would have appreciated further written explanation.

I like that diagrammatic thing. I think that's better than quite good. – Person living with type 2 diabetes

The dots are fine. Maybe graphs would be better? When you showed the graph on the previous thing I understood it, but this doesn't give that much of a picture. – Person living with type 2 diabetes

Autonomy

Although the aim of the tool is to support patient self-management, most participants had to be prompted to read the instructions for adjusting the sliders (see Table 1, Weight and Activity sliders).

At the top it should have an explanation of what to do. – Person living with type 2 diabetes

Some participants stated that they would want support from a clinician before attempting to set goals for their weight and other health-related parameters to ensure their goals are realistic.

It's not a good thing to tell people what you should lose. People will lose what they can. Everyone can decide they want to lose xkg but it doesn't happen like that. – Person living with type 2 diabetes

Most participants assumed that either (i) they needed to fill in the glycated haemoglobin (HbA1c) and BMI, (ii) they could adjust the HbA1c and BMI to see their new risk or (iii) these metrics would change when Weight and Activity are adjusted (see Table 1, HbA1c and BMI).

You wouldn't know your HbA1c and BMI so maybe that would be more difficult to fill in. That should be a click thing. If they put the changeable Weight and Activity things, then I would expect to change the things in the box too. – Person living with type 2 diabetes

You can't do it can you. I can't say what my new HbA1c is and with the BMI you should be able to put your height and weight in. – Person living with type 2 diabetes

Terminology

A consistent theme across the think aloud sessions was the meaning of ‘risk’ within the context of the risk prediction tools. Participants differed in their interpretation of the term ‘risk’. Some interpreted that risk related to complications from type 2 diabetes, others that risk related to their overall risk of becoming unwell, few more that risk related to how likely the intended user is to get diabetes.

Risk of conditions I’m going to develop. It could be foot ulcers if I don't control the diabetes. My eyesight can go bad. – Person living with type 2 diabetes

If I had just been told I was diabetic prediabetic and recommended this site, I would presume that that is what I was looking at prediabetic. If I went onto the sight and that came up as 50% and red that would mean you’ve already got it. Risk might not be the right term to use. – Person living with type 2 diabetes

Most participants assumed that the term ‘statins’ related to the management of their cholesterol. However, participants felt that the terms would be unfamiliar to some patients and thus lay terms should be used throughout the risk prediction tool.

That's for cholesterol, yes? I’m quite good at taking my medications so I know what statins is for but sometimes you have different names for it. – Person living with type 2 diabetes

Statin is the pills for cholesterol? I’m taking statin. Tell me if statin means taking cholesterol pills? – Person living with type 2 diabetes

Themes from MyWay Clinical analysis

The main themes derived from the think aloud sessions with GPs based in NWL were the presentation of practice and patient information, colour scheme, use of graphics and icons, clinical workflow and patient communication.

Patient and practice information

Most clinicians were able to interpret the information presented to them in the way that was intended for users. The risks described under Risk Predictions (see Table 2) were considered by some clinicians as general descriptions of the consequences of long-term deterioration. Such clinicians queried the relation of the risk percentages to the patients’ current test results.

They all sound very dramatic, it's making me think renal failure is a significant drop in eGFR – General Practitioner.

Most clinicians noticed that an additional Mortality parameter only appeared as a risk at 10+ years (see Table 2, Risk Predictions). Participants considered this an appropriate timeframe to present mortality risk. However, some clinicians suggested that Mortality should be shown at other timeframes with an ‘N/A’ instead of percentage risk to signal to users that there are five risks that can be viewed.

Another one, mortality, has come in so there might be others that aren't showing – General Practitioner.

So when I click through the risk, mortality is added when I go to 10 plus years. Would be nice to display all of the parameters, even if they’re not relevant, they can just be non-applicable at that time – General Practitioner.

One consistent theme throughout the think aloud sessions was the meaning of risk, in the context of the risk prediction and clinical analytics tools, and how it is calculated. Some participants stated that the term ‘risk’ is applied broadly across both tools and that more specificity would be needed when explaining risks to patients. Furthermore, participants sought more clarity on what value ranges applied to low-risk, medium-risk and high-risk categories (see Table 2, Future risks – range).

I’m wondering what it means by medium risk. A patient might want that put into more tangible terms. – General Practitioner

My question is how did we know [what risk they are] and what do you need to fall into that category. – General Practitioner

In the Future Risks, clinicians liked the tabular view of patient lists (see Table 2, Future risks population stratification feature) and were relatively confident using the table feature to export the list of patients and filter patients by Risk Score. Some clinicians attempted to click on other headings in the table. These participants stated that they would like to be able to sort and filter the data by other columns in the table.

I went here to see if it could order them by their last result. That's usually quite useful but looks like I can't reorder this. – General Practitioner

On the same page, one clinician highlighted that there is no indication on the webpage as to whether the patient list they are looking at is for 1+, 2+, 5+ or 10+ years.

I think that was in one year but it doesn't say up here. I might just go back and check… yes, one year. Might be helpful if it said up here ‘one year’. – General Practitioner

In addition, clinicians generally seemed to think that the Risk Score was useful to include in the table. However, they seemed unable to decipher the meaning of the Risk Score and its implications in their clinical practice.

It's given me a score, which is new. I don't know what that score means. – General Practitioner

Colour scheme

The traffic light system for the colour coding of categories was apparent to all participants, that is, green means low, amber means medium and red means high. As part of the evaluation, and viewing other features of MyWay Clinical, participants noted that the same colour scheme in MyWay Clinical was used for Future Risks and local compliance with key diabetes targets and key performance indicators.¹⁷ Therefore, clinicians sought more clarity in terms of how and what low, medium and high are referred to in the context of the data they were presented with (see Table 2, Future risks – key).

I don't know whether green is out of targets, somewhat within targets. – General Practitioner

Use of graphics and icons

Most participants liked the use of doughnut charts to show the proportion of patients who are at low, medium and high risk. However, some participants did not initially realise they could view the percentage or number of patients in each risk category by hovering the cursor over the segments (see Table 2, Future Risks).

Different segments over here would represent how many of those patients would be at higher risk of complications. – General Practitioner

Might’ve been a little bit more helpful if I’d maybe seen a percentage [hovered the cursor over the segments] Okay, I see it now. – General Practitioner

Clinical workflow

Some clinicians stated that the analytics presented in the Future Risks tool made them think of the QRISK algorithm¹⁸ that applies to patients aged between 35 and 74 years and predicts their cardiovascular and stroke risk over the next 10 years.

Because sometimes it's how do you equate, you know, we all know the diabetes is got all of these risk factors. But sometimes we use a QRISK too. But sometimes it's hard to quantify and to add more meaning to a patient. So this is helpful because, with a patient, I can show them the data I have reviewed and their risk of complications – General Practitioner.

The categories and percentages make me think of QRISK. But QRISK gives us a good idea as to what we do with those percentages – General Practitioner.

Some guidance would be needed, as to how to reduce these percentages. For example, QRISK is not only based on their cholesterol. It's their age, where they live, what conditions influence it and I don't know what influences amputation other than my common sense and that may differ from clinician to clinician – General Practitioner.

One clinician wanted to seek clarification on how the word ‘risk’ was defined in the context of the tools and how it is calculated from the patient data.

What do individual risks mean to them – I’m not really sure if this means absolute or relative risk (are we comparing them to someone of the same age with diabetes). I would need more information to counsel patients – General Practitioner.

Clinicians perceived the Future Risks to be particularly useful for managing their practice population, to bring patients overdue for tests and at-risk of complications for appointments.

Red would be the area of focus for the teams such as the primary care teams or community-based diabetes teams. – General Practitioner

If I had my diabetic clinic, which we are setting up, I would want to know who to fill in those clinics. And normally, I would be trying to remember who is poorly controlled and bring them back. Or some patients might not follow up. – General Practitioner

I obviously probably need to focus on the reds to make sure that they turn green. But then how do I go about doing that? – General Practitioner

Patient communication

The user interface of the Risk Predictions sidebar was intuitive to participants. The participants thought the tool would be very useful to help patients decide what medical or lifestyle options they would want to pursue to bring their risks of complications down.

We can move it and explain it to the patient. I can say to them in X-years time if these things stayed similar then their biggest risk is developing renal failure – General Practitioner.

It could help in terms of when counselling patients that when they start a medication. With this in front of them they might agree to take a medication – General Practitioner.

The Risk Predictions tool was considered a useful feature which could facilitate consultations with patients. However, participants stated that they wanted a way to visually demonstrate how improving their health would impact patients’ risk of complications. The main suggestions from participants were that this could be in the form of (1) allowing the user to adjust the patient's metrics to see how their risk predictions change or (2) adding an overlay to the patient metrics which shows the target values that should be achieved to reduce their future risk of complications.

The patient is going to ask how they can reduce that. I wonder if there is anything on this screen where if I can change their metrics so I can show them how their parameters might change – General Practitioner.

Would be good to click into Risk Prediction and slide down parameters to see the change or see another graph next to it to see the comparison – General Practitioner.

You can have a dynamic discussion with a patient, which, for example, the risk of complications increases as a number of years – General Practitioner.

Clinicians reflected on how the information under Risk Predictions could be presented to patients in a way that could facilitate a positive tone in consultations and reinforce desired behaviours for self-management, irrespective of whether the data showed they were at a low, moderate or high risk of complications.

I’m not entirely sure that is helpful because I guess they want to make our consultation a positive one. It sounds a bit grim – General Practitioner.

I may not say 0% probability worried as a patient, but you know, just drop everything and say, actually, that's absolutely fine – General Practitioner.

I think you can scare people with numbers – General Practitioner.

Discussion

Summary of findings

The aim of this study was to evaluate the usability and acceptability of the prototype risk prediction and clinical analytics tools on the MyWay Diabetes and MyWay Clinical platforms using the think aloud method. The key findings from the MyWay Diabetes evaluation were that users who live with type 2 diabetes would benefit from clarification around the intended use of the risk prediction tool and how to navigate the page. The graphic and interactive visualisation of ‘risk’ was the most popular feature of the risk prediction tool. Although participants did note that it would be useful to use lay terms instead of medical terms (i.e. cholesterol tablets instead of statins). In practice, some participants would prefer to use the tool with support from their clinician to set realistic goals. The key findings from the MyWay Clinical evaluation were that clinicians would benefit from more clarification as to what ‘risk’ pertains to and would want to use the Risk Predictions tool to facilitate a consultation with a patient that positively encourages them to manage their condition. The Future Risks tool was considered useful for populating diabetes clinics with the most at-risk patients in the practice.

Comparison to previous literature

This study identified improvements to usability features previously described by Shields et al.,¹⁹ however, participants of this study also described the same social contexts which could impact the acceptability of the tool (i.e. concerns regarding digital literacy, the use of medical terminology, setting ‘unrealistic’ goals).

The findings of this study demonstrate the potential for digital health solutions to support people living with type 2 diabetes to mitigate complications from the disease and enable healthcare professionals to provide accessible care at a lower cost.²⁰ However, digital literacy remains a challenge for the adoption of digital health solutions in diabetes care, particularly as digital literacy is reportedly low in the demographic groups that have a higher burden of type 2 diabetes, for example, older people and people from low socioeconomic backgrounds.²¹ Participants of this study liked the visual representation of risk, interactive features and being presented with their individual risk calculations based on their data. Such design strategies in the development of digital health solutions have been recommended to reduce the literacy burden and improve health literacy.²¹

Furthermore, the value of user-centred design and usability testing methods such as think aloud in the early stages of product development is further demonstrated by the study findings. Primarily, to improve the acceptability of patient-facing digital health solutions, the design and interface must enable autonomy and use lay terminology. The same challenges regarding digital literacy apply to health literacy in diabetes care and the impact on demographic groups with a higher prevalence of type 2 diabetes,²² as evidenced by one study which showed that 86.9% of diabetes website materials were too difficult for the average adult to read.²³ Findings from this study have already resulted in significant changes in the design and development of a version 2 prototype of the platforms.

The discussion around risk in the context of diabetes can elicit concerns regarding the impact of patient behaviour and adherence to lifestyle recommendations and diabetes self-management. Previous studies have identified the communication strategies employed by clinicians at the point of diagnosis to be a strong indicator of clinical outcomes, that is, reiterating the seriousness of the patient's diagnosis while reassuring them that their diabetes can be managed successfully, as well as providing resources and information and advising patients to read the material at their own pace.⁵ This study therefore supports the use of risk prediction tools within a clinical workflow, for example, a GP consultation, to ensure improved clinical outcomes.

Strengths and limitations

Participants living with type 2 diabetes were recruited from the NWL Health Research Register, most of whom had been living with the condition for several years. Therefore, while their comments were valuable, they may have had more contextual knowledge of the information presented to them in the tool compared to people who have been newly diagnosed. Although none of the participants reported that they had used a digital tool for diabetes management, the findings may have differed if the think aloud sessions were conducted with people who had been newly diagnosed.

There were technical limitations across both the MyWay Diabetes and MyWay Clinical platforms that the study team was aware throughout the evaluation. Due to time limitations to deliver the project findings, the MyWay Diabetes risk prediction tool generated a random number for current risk as opposed to a number based on the patient’s medical data. This random result was kept within the normal expected range for the actual model. To mitigate this, all participants were informed of the specific technical limitations prior to the evaluation and instructed to focus on providing feedback on the concepts behind the features. However, this could have impacted the perceived usability of the platforms.

In addition, this study was conducted during the COVID-19 pandemic which resulted in national and local lockdowns and prohibitions on face-to-face meetings. Therefore, the evaluations were conducted in a remote user testing environment. Some participants were less familiar with the remote environment and experienced challenges sharing their screen with the researchers. Providing guidance prior to the session, regarding how to access the meeting link and share their screen, streamlined the process and maximised the time allowed for the think aloud activities.

Future implications

The high prevalence and socioeconomical burden of type 2 diabetes in NWL calls for scalable solutions to support self-management and population management to monitor and mitigate the risk of complications associated with diabetes. This study demonstrates the value of seeking feedback on the usability and acceptability of a digital health solution developed in a different healthcare context, in this case, NHS Scotland, prior to implementation in a new healthcare setting (NWL). The findings from this study will be used to support ongoing product development and implementation strategy to ensure the MyWay Diabetes and MyWay Clinical platforms can be embedded and enhance diabetes care and risk stratification in NWL.