DiaDigital Apps: Evaluation of Smartphone Apps Using a Quality Rating Methodology for Use by Patients and Diabetologists in Germany

Abstract

There are several computer applications (apps) that can be installed on smartphones to assist patients with diabetes in treating their disease. Not only does the range of use of such apps vary greatly, their quality does as well. As part of the DiaDigital initiative of the German Working Group for Diabetes Technology (AGDT), apps are evaluated using a set of criteria and a seal of distinction is then awarded (www.diadigital.de). The information collected in this review process is made public on this website to ensure both the necessary transparency as well as to be able to rapidly adapt voting on further app development. Until now, no such approach on evaluating the quality of diabetes apps has existed in Germany. The hope is that this will also help improve the quality of apps.

Many patients with diabetes want support in coping with their disease during their everyday lives. The extensive availability of smartphones or tablets enables patients to use apps that provide such support. The number of diabetes-related apps is probably in the tens of thousands worldwide. In just one app store (Apple) more than 1000 such apps are available. There is a clear necessity for a way to help control the avalanche of apps.¹ The situation in Germany is better in terms of the number of apps, as only about 100 German-language apps are used frequently. However, for the interested patient this does not yet mean that it is easy to make a selection. There are ways to analyze how many diabetes apps have been downloaded, but none to determine if or how intensively they are used, and therefore how successful they really are.

Apps and their quality vary extensively; a broad spectrum exists in terms of the (programming) effort involved and the maintenance and further development of the app. These range from applications developed by individual patients or programmers who use them to spread their own approaches to apps by corporations which see them as the best way of communicating with their customers. Considering the total financial volume of apps nowadays—and not only medical apps—of more than 1.5 billion euros, app programming has become an industry. New developments in the field of patient treatment (such as automated insulin delivery) via apps will most likely be widely used in the future. The highly complex apps required for such an application, which should always be connected to the cloud, differ considerably from simple apps. In addition, the medical benefit for patients with diabetes varies depending on the app itself although assessing the benefit of the app depends greatly on the personal perspective.

To date, there have been few systematic and structured studies on whether the advantages, such as avoiding acute metabolic derailments and a general improvement in metabolic adjustment, are actually achieved through the use of apps. For example, there have been few publications in diabetological literature to date dealing with apps, but there are current initial critical evaluations, for example, of bolus calculator apps.²

The high frequency of use of apps shows that they present patients a concrete advantage in everyday life—otherwise patients would not use them. Whether and to what extent the use of apps has a demonstrable medical benefit remains to be clarified. We see a definite need for diabetologists and the diabetes teams to develop competencies when it comes to apps. Most of the inquires received by the German Working Group for Diabetes Technology (AGDT) of the German Diabetes Association come from patients, and the frequency of these inquiries indicates that patients feel that their diabetologists are unable to provide qualified answers to this topic.

The DiaDigital initiative of the AGDT aims to support all patients and groups interested in digitalization and especially in apps. This working group consists of a number of people of different backgrounds (physicians, diabetologists, diabetes nurses, scientists, app developers, and more than 30 diabetes patients who use such apps. We have developed a set of criteria for evaluating apps and providing a seal of approval for those that fulfil these requirements. It must be noted that this set of criteria has been developed by our working group representing a wide range of expertise and backgrounds and is not based on any published previous work.

Current Status of Apps

Apps are a way for patients

• to collect, analyze, and present relevant data

• to obtain information

• to receive warnings

• to be reminded of blood glucose measurements or insulin inputs by alarms

• to receive support in calculating the right insulin dose with the help of an integrated bolus calculator, etc

• to control devices

Good apps support users in an active and self-directed approach to their disease. They help patients adapt their lifestyle (eg, monitoring of nutrition and exercise) and therapy as well as facilitate communication with other affected persons. The rapid development of technical possibilities makes coaching via apps possible nowadays, that is, the patient can speak directly to a diabetes consultant.

Availability of Apps

Any programmer can add an app to an app store without any systematic control of functionality, security, and so on by an independent body. This is the reason for the number and diversity of apps available. However, if apps are used to manage diseases, these can sometimes pose a problem as there is no guarantee that the apps used are of a high quality or that their use does not pose a risk to the patients. If, for example, the algorithm used to calculate the insulin dose contains errors, this can result in life-threatening acute metabolic disorders.

Apps as Medical Products

To avoid these and other risks, all computer programs (including apps) that make statements about the therapy of patients in any form are considered a medical device and must have a CE marking in Europe. However, the process to get this marking takes both time (1-2 years) and money (upward of 500 000 euros), which is why only a few apps have a CE marking.

A general “disconnect” is visible between the regulatory world and the rapid development of technical solutions in the field of smartphones and diabetological technology. If products (here, apps) come to the market after the protracted approval phase, they may already be outdated. This raises the question of how the justified need for safety—paired with the proven benefits of the product—can be combined with the rapid availability of innovative products for patients. It is unclear whether and in what form the legal requirements will change (also at the European level) to take the developments mentioned into account.

Data and Data Security

Overall, data are the core of all apps, and appropriate data handling should benefit the patient. Within the concept of data, all relevant aspects must be considered, in particular data security and compliance with data protection regulations such as the EU General Data Protection Regulation (GDPR). On the one hand, this applies to access to the user’s personal and account information. On the other hand, it applies to associated data and a possible third-party use of this without the user’s consent. It is not absolutely certain that app data remains on the smartphone—it may end up in completely different databases.³ This is an important aspect when evaluating apps. Since data are considered to be gold today, the handling and interests of app providers must be clearly stipulated.

Apps and the Diabetes Market

In view of the potentially massive global market of hundreds of millions of patients with diabetes, there are a large number of commercial parties interested in the development and distribution of apps. Varying business models exist, including providers—among others—who have apps developed and make them available to patients free of charge to open up a communication channel to patients for interested pharmaceutical companies. It is no coincidence that companies such as Apple and Google are so interested in apps and their use as these global players see an important future market here.

Evaluation of Apps

Apps are widely rated on the Internet but the set of criteria used is often unclear or is viewed critically; this also applies to ratings that have been published in patient journals. Up to now, there has been no official body in Germany that carries out a systematic and independent evaluation of apps. The contribution of technological expertise is vital as well as providing the corresponding infrastructure or platform for the publication of results developed by the project partners (see below).

The question is, who can rate apps and how is it done? Can a diabetes app be rated by a person without diabetes? The very approach of wanting to evaluate apps in some way can, in principle, be seen critically. There are statements that such approaches, which have already existed to evaluate home pages dealing with medical topics, and many apps are probably implementations of home pages at this user level, are a difficult or even hopeless undertaking.

We basically see two different approaches:

1. The AGDT (ie, the members of the DiaDigital initiative) systematically evaluates apps and publishes them, for example, on its own home page. However, it is neither feasible nor possible to continuously evaluate all apps (and their updates).

2. Another option is that users (ie, patients with diabetes) evaluate apps transparently using a catalogue of criteria and the results are listed on the home page. Patients, health care professionals, and scientists must agree on an accompanied self-disclosure procedure for evaluating apps. Apps that fulfill such requirements are awarded a DiaDigital seal.

We consider the second approach to be the most sensible and have therefore drawn up a diabetic-specific catalogue of criteria (Table 1). The seal is awarded when the set of criteria (Codex criteria) is fulfilled (Table 2).

Table 1.

Criteria That Should Be Specified for the Respective App to Be Evaluated.

General aim of the app/product description
Product category	Prevention/supply/research	*
Business model	Opportunities: Advertising, subscription, download fees, sponsoring by medical technology or pharmaceutical companies, free of charge	*
Target group	Checkboxes: Patient/citizen, (caregiver) family member, (medical/nonphysician) service provider; a further restriction of the target group is to be discussed, in relation to age group and disease/health condition	*
Cost	Price in euros (possibly divided into basic and full version, this is the norm for some apps)	*
Reimbursement by health insurance	?
Updates	How often, when last?	*
Support available during normal office hours (for technical and content problems or comments)	Yes/no, at what times? By email or by phone?
Advertising	Advertising contains advertising or commercial marketing of data	*
Innovation	Is the app innovative? Special new approaches? What distinguishes the product?
App store description	Copy of the description of this version	*
Download (links) Apple & Google	Download path	*
Medical device classification	Yes/no, MP classification: I, IIa, IIb, III	*
ISO certificate for medical software XXXXX	Yes/no
Auxiliary means number in the GKV List of auxiliary means
Available in which languages?
Version information		*
Terms of use of the app
What accessibility measures have been implemented?	Programming according to Google, Apple, Microsoft standards?	*
Information about the manufacturer, qualification		*
Medical aspects.
Medical goal of the app	Defined or not?	*
Help in case of illness
Prevention	All four prevention areas: - addiction - movement - nutrition - stress management, relaxation
Help for emergency situations?	Eg, ketoacidosis, etc
Content goal-oriented? Does the app meet the medical goal?		*
Training as a medical goal?
Which parameters are documented?
Supports DMP (German disease management program)?
Are there studies on the app?
Interaction with the app user/interaction options for the app user.
Help functions
User feedback
Reminder function
Video & audio content
Motivation/games
Graphical display
Others	Eg, new interaction approaches
Data management.
Type of data
Data exchange and data transmission	With whom? Between persons and/or institutions?	*
	Local, D, EU, www
	Cable/near field/www
	Where to? Communication with other devices, app as front-end
Cloud storage	Encrypted/unencrypted	*
	Local/user decision, eg, Dropbox/Cloud
Local/online storage	App usable offline?	*
Backup of data		*
Are values calculated?		*
Information beyond the app	Eg, www search
User-related data	E-mail, name, pseudonym, personal data, medical data, otherwise.	*
Switching between systems possible, eg, iOS to Android		*
System requirements also for functions such as data transmission		*
		*
		*
	Declaration for handling of	*
		*
	?
		*
	Established in the company?
	Eg, CE

Self-disclosure by the manufacturer for the app. *Codex criteria with product details.

Table 2.

Verification of Self-Disclosure: The DiaDigital Group and the Technology Partner Check the Manufacturer’s Information for Accuracy: The Codex Criteria Considered Essential Must Be Met.

Target group from our point of view:
Group	Affected persons/health service providers
Age group
Type of diabetes	Including prediabetes
Form of therapy
Communication:
Technical: how the data exchange takes place technically
Personnel: exchange with other persons
Data entry:
Ergonomic	Usability
Automatic data transfer	Eg, from devices, from the smartphone
Bolus calculator yes/no	Functional? How does it work?
Measurement offline/online possible?	Offline use
Accessibility without barriers	Completely or partially available?
Help functions:
Tutorial available?	Yes/no, useful?
Support	Available, has replied, not tested
Functional test by DiaDigital:
Technical problems found?	Yes/no, serious?
Functional evaluation in medical professional care	Process-supporting? or obstructive?
Training connection:
Training on the topic available? How to evaluate?	Yes/no, sensibly designed?
Therapy support:
Assessment of therapy support	Is self-management really supported? Have you observed an increase in motivation?
Does the app meet the medical goal?	Yes/no, how?
Certificates available:
For safety	yes/no
On quality	yes/no
Usability
- User orientation/usability: how user-friendly and motivating is the app?
- Intuitive operation and customization of the app or user-defined configuration
- Barrier-free use?	Yes/no
- Expectation conformance (ie, the app behaves consistently for language and input)	Yes/no
- Motivating elements (eg, games)
- Diversity (video and audio, texts, …)
- Graphic representation/design (for iOS apps there are also fixed guidelines from Apple)
- App can also be used without a permanent internet connection?	Yes/no
- Performance requirements (response and load times)	Hardware, please specify
- Language (implementation in German, if necessary also in English)
Judgment/conclusion/review:
Without notes, define items
Wishes for the product:
Technical
Graphic
Intuitive usability
Codex criteria fulfilled yes/no
Seal to be awarded: yes/no

Evaluation by the DiaDigital Group (affected persons, practitioners, and technology partners).

The DiaDigital Seal

It is important that the evaluation of the apps and awarding of the seal follows clear and well-defined rules. In addition, the conflict of interest of all individuals and groups involved in this activity is transparent and the steps (as presented below) are documented.

Manufacturers that have programmed an app for patients with diabetes approach the DiaDigital working group for apps and ask for it to be evaluated. The manufacturers confirm compliance with the criteria by providing information themselves and the DiaDigital working group assesses the accuracy of the information. A technology partner (ztg GmbH; a governmental agency of the state of North Rhine Westphalia; www.ztg-nrw.de) assesses the aspects of IT security in app programming (appendix), such as the (non)encryption of data streams. Users are asked to provide feedback on the functionality and actual benefits. While the individual reviews do not provide complete validation, the considerable number of independent evaluators (at least five diabetes patients und five health care providers [HCPs]) is regarded as a sufficient safeguard to avoid biased views and prevent conflict of interest issues. During regular telephone conferences steered by MK and DD, the evaluation outcome of a given app is discussed among all parties involved in the evaluation process. The seal is awarded only if the criteria are met completely (100%). If the team refuses to award the seal, the manufacturer receives feedback and the opportunity to improve the product. The procedure for awarding the DiaDigital seal is seen as an accompanied, voluntary commitment model. For reasons of practicability and transparency, this process is carried out on an Internet platform where the results of the cooperative evaluation of the apps are also presented. To date, six apps have received a seal after passing the evaluation process successfully and six apps have not.

The manufacturer can declare the seal in the app store and the diabetes association provides the information to the public and their members. The aim is to help potential users to find useful apps and quickly separate the wheat from the chaff.

In the end, such a seal should help the market regulate itself because patients will download and use an app regularly only if it fulfils certain (constantly updated) criteria and is deemed good. This should also lead to an overall improvement in the quality of apps.

We see the following advantages in using such a seal (positive selection):

1. Not very vulnerable from a legal point of view because only positive statements are made in awarding seals

2. User perspective is clearly represented

3. Evaluator competence is developed

4. Community strengthens itself

5. Concrete communicable recommendations are made

6. Intensive reconciliation of interests

7. Probably delivers the most reliable results

8. Assessment performed jointly with the manufacturers

Problems With This Approach

Such a seal has its weaknesses. To name a few, a considerable organizational effort is necessary and a functioning structure must be maintained over a longer period of time. This is also associated with costs, covered up to now by the AGDT.

Apps can be seen as highly dynamic—a manufacturer can quickly react to detected defects by means of an update. It is important to consider such updates as part of evaluations. If these refer to older versions, they can quickly become obsolete if an update to the app is performed. For this reason, manufacturers are required to inform us about updates. If an evaluation unearths what seems to be a problem with an app, the manufacturer of this app can regard it as incorrect and take legal action for potentially damaging its business.

Summary

The systematic use of a catalogue of criteria is intended to help to determine the quality of diabetes-related apps. Patients with diabetes and HCPs perform joint evaluations. The information collected about the apps evaluated are presented and stored on the www.diadigital.de home page. Apps that meet the criteria are awarded with a seal.

Appendix

Statement of the Technical Project Partner

The purpose of analysis and app monitoring is to test the security of data streams and data transport, that is, to test whether the data runs over a secure https connection. This applies in particular to sensitive and personal data, such as passwords or health information. Charles Proxy is used to analyze the communication of the apps.

If communication is via an http protocol, this indicates that the communication is unencrypted, since the http protocol is used and not the https protocol.

If many data streams are displayed using an http protocol, this is an indication that this app should be looked at more closely. The decisive question is which data or communication processes are transported via an http connection. If these are only simple image files of the apps, and so on, and not personal data, this is less critical and does not pose a major problem.

Furthermore, the platform dependency of the apps is analyzed, that is, on the one hand it is examined whether the apps basically work on the two largest app platforms of Apple/IOS and Google/Android and are available for download. The results of this test may also differ from the information provided by developers and manufacturers. The analysis for the security of the data streams takes place accordingly on both platforms.

Furthermore, as a third step, it is checked whether it is possible to analyze to which location or into which country the data or in particular personal data are sent. This is important because the IT security aspects of the apps should at minimum comply with European law, better still, of course, with German law. On the Charles Proxy, the result is determined by calling up a page like https://geoiptool.com/de/, for example.

In addition, antivirus software (such as Kaspersky Internet Security) is used to scan apps for potential threats such as spyware (spyware, sniffer software), malware (malware) or viruses.

Finally, the general terms and conditions (GTC) of the respective app manufacturer are read through—with a focus on data protection and internal data handling—with the aim of being able to compare the previously analyzed results with the information in the GTC and to be able to filter out any existing contradictions or open questions to the manufacturer. This step rounds off the overall evaluation.

In this context and for legal reasons, it must be added that this procedure cannot be used to identify or track what specifically happens to the data collected or whether the manufacturer passes the data on to third parties. A resale of the data, for example, cannot be evaluated. However, it is an overall rather uncomplicated and comprehensible procedure which also makes it possible to test a larger number of apps.

Platform independence/platform

Data transport encrypted yes/no (https/http)

Use of analysis services (Google Analytics)

Locations/sites user data/analysis services

Threats/viruses

Analysis of the general terms and conditions/data protection data

Upshot