Alignment between outcomes and minimal clinically important differences in the Dutch type 2 diabetes mellitus guideline and healthcare professionals’ preferences

Abstract

To evaluate the clinical benefit of new medicines for type 2 diabetes mellitus (T2DM), the Dutch guideline committee T2DM in primary care established the importance of outcomes and minimal clinically important differences (MCIDs). The present study used an online questionnaire to investigate healthcare professionals’ opinions about the importance of outcomes and preferences for MCIDs. A total of 211 physicians, pharmacists, practice nurses, diabetes nurses, nurse practitioners and physician assistants evaluated the importance of mortality, macro‐ and microvascular morbidity, HbA1c, body weight, quality of life, (overall) hospital admissions and severe and other hypoglycemia on a 9‐point scale. All outcomes were considered critical (mean scores 7–9), except for body weight and other hypoglycemia (mean scores 4–6). Only HbA1c and hospital admissions were valued differently by the guideline committee (not critical). Other relevant outcomes according to the respondents were adverse events, ease of use and costs. Median MCIDs were 4 mmol/mol for HbA1c (guideline: 5 mmol/mol) and 3 kg for body weight (guideline: 5 kg weight gain and 2,5 kg weight loss). Healthcare professionals preferred relative risk reductions of 20% for mortality (guideline: 10%) and macrovascular morbidity (guideline: 25%) and 50% for other hypoglycaemia (guideline: 25%). The MCID of 25% for microvascular morbidity, hospital admissions and severe hypoglycaemia corresponded to the guideline‐MCID. Healthcare professionals’ preferences were thus comparable to the views of the guideline committee. However, healthcare professionals had a stricter view on the importance of HbA1c and hospital admissions and the MCIDs for mortality and other hypoglycemia.

Healthcare professionals’ preferences for outcomes and minimal clinically important differences in the evaluation of new type 2 diabetes mellitus (T2DM) medicines were comparable to the views of the Dutch guideline committee T2DM in primary care.

What is already known about this subject

• Selection of outcomes and minimal clinically important differences (MCIDs) for the evaluation of new medicines is important for the development of clinical guidelines.

• It is unknown how healthcare professionals evaluate the importance of outcomes and MCIDs used for the type 2 diabetes mellitus (T2DM) guideline in primary care.

What this study adds

• According to healthcare professionals, severe hypoglycemia and mortality are the most important outcomes for the evaluation of new T2DM drugs.

• Adverse events, ease of use and costs are additionally mentioned as important parameters. Median MCIDs according to healthcare professionals are in line with the MCIDs used in the Dutch primary care T2DM guideline.

1. INTRODUCTION

The last two decades, new pharmacological treatments have become available for the treatment of type 2 diabetes mellitus (T2DM), including dipeptidylpeptidase‐4 (DPP4)‐inhibitors, glucagonlike peptide‐1 (GLP1)‐ agonists and sodium‐glucose‐cotransporter 2 (SGLT2)‐inhibitors. Most of these drugs have found their ways into national and international clinical guidelines. ¹ , ²

To evaluate new pharmacological treatments, guideline committees have to specify the criteria the medicines have to meet. Therefore, the importance of outcomes and cut‐off values for a clinical benefit on these outcomes have to be established. The importance of outcomes can be scored according to the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach. GRADE recommends the use of a 9‐point scale. A score of 1–3 indicates limited importance, 4–6 important, but not critical, and 7–9 critical importance. ³ Subsequently, cut‐off values for a clinical benefit can be defined. ⁴ Those cut‐off points, also known as minimal clinically important differences (MCIDs) or minimal important differences (MIDs), are used to evaluate the clinical relevance of a difference between two treatments. ⁵

For the evaluation of pharmacological treatments for T2DM, a considerable number of outcomes can be relevant, varying from direct outcomes for clinical efficacy (e.g. mortality) to surrogate outcomes (e.g. HbA1c), safety outcomes (e.g. hypoglycemia) and patient reported outcomes measures (PROMs) like quality of life. ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ Validated MCIDs are not available for T2DM medicines. The decisions about importance of outcomes and MCIDs in treatment guidelines are therefore based on expert opinions and guideline committee consensus. ⁴ , ¹¹ , ¹²

The abovementioned approach for the definition of importance of outcomes and corresponding MCIDs was also followed in het process of updating the Dutch clinical guideline for the treatment of T2DM in primary care in 2018. ⁴ The outcomes mortality, macrovascular and microvascular morbidity, HbA1c, body weight, quality of life, (overall) hospital admissions, severe hypoglycemia, other hypoglycemia (not specified, mild or modest) and other adverse events were evaluated by the guideline committee for their relative importance. Subsequently, the guideline committee established the MCIDs for those outcomes. ⁴ The MCIDs were based on previously defined MCIDs in other national and international guidelines, ² , ¹¹ , ¹² non‐specific thresholds for relative risks and standardized mean differences (SMD) provided by GRADE ¹³ and expert opinion in the guideline committee. ⁴ An overview of outcomes, their relative importance and MCIDs used during the Dutch T2DM guideline development can be found in Table 1.

TABLE 1.

Overview of importance and MCIDs of outcomes used in the update of the T2DM guideline ²

Outcomes	Importance a	Cut‐off point for clinical relevance	MCID based on
All‐cause mortality	critical	RRR 10% (RR <0,9 or RR >1,1)	Expert opinion guideline committee
Macrovascular morbidity	critical	RRR 25% (RR <0,75 of RR >1,25)	GRADE 13
Microvascular morbidity	critical	RRR 25% (RR <0,75 or RR >1,25)	GRADE 13
Quality of life	critical	every statistically significant difference or SMD = 0,5	GRADE 13
Severe hypoglycemia	critical	RRR 25% (RR <0,75 of RR >1,25)	GRADE 13
Other adverse events	critical or important b	every statistically significant difference or SMD = 0,5	GRADE 13
Hospital admissions	important	RRR 25% (RR <0,75 of RR >1,25)	GRADE 13
Change in HbA1c	important	0,5% or 5 mmol/mol	NICE guideline Type 2 diabetes in adults: management 11
Change in body weight	important	5% in case of both treatments cause weight gain 2,5% in case of one treatment causes weight loss and the other causes weight gain (or had a neutral effect on weight)	Dutch guideline T2DM in secondary care 12
Other hypoglycemia (not specified or mild or modest)	important	RRR 25% (RR <0,75 of RR >1,25)	GRADE 13

Abbreviations: RRR, relative risk reduction; RR, relative risk; SMD, standardized mean difference.

^aMinor differences existed in relative importance between different healthcare questions. The importance shown is based on the importance for most healthcare questions.

^bDepending on the severity of the adverse event.

The treatment recommendations in the final guideline heavily depend on the classification of importance of outcomes and MCIDs. Since the final guideline is leading for the treatment choices healthcare professionals in primary care make, it is of particular interest to know the degree of alignment between the guideline committee and the end users of the guideline. There is limited or no evidence concerning the views of healthcare professionals about the importance of outcomes and MCIDs used in guideline development or in the evaluation of blood glucose lowering drugs. Therefore, the aim of this study was to investigate healthcare professionals’ opinions about the importance of outcomes and preferences for MCIDs used in the evaluation of new medicines for the T2DM guideline.

2. MATERIALS AND METHODS

2.1. Design

An online questionnaire was developed to investigate healthcare professionals’ opinions about outcomes and MCIDs used in the evaluation of new T2DM medicines. According to the Dutch legislation, neither obtaining informed consent nor approval by a medical ethics committee is obligatory for conducting research among healthcare professionals that does not include patient data. Therefore, no ethical approval was needed.

2.2. Participants

Participants for the online questionnaire were approached using the mailing list for newsletters of the Dutch Institute for Rational Use of Medicine (IRUM). The mailing list contained 12.115 email addresses of stakeholders in pharmaceutical care, such as healthcare professionals and policy makers. Since there was no information about the profession of the subscribers, the questionnaire was sent to all subscribers. The respondents were asked for their profession in the questionnaire. Therefore, the selection of relevant professions could be made afterwards.

2.3. Data collection

The invitation to fill out the questionnaire was sent by email with a link to the online questionnaire on 17 February 2020. All subscribers received one reminder after 10 days (27 February 2020). The online questionnaire was closed on 13 March 2020. Participants did not receive a financial compensation, although every 10th participant was offered a free online accredited course about the treatment of T2DM, which is part of the IRUM continuous medical education program.

2.4. Questionnaire and measurements

The questionnaire is available in Appendix 1. The content of the questionnaire was based on the outcomes and cut‐off points for clinical relevance used during the development of the T2DM guideline. ⁴ The questionnaire was developed by the researchers and fine‐tuned during several sessions. Thereafter, the questionnaire was pre‐tested by six healthcare professionals (a general practitioner, a public pharmacist, a hospital pharmacist, a practice nurse and two diabetes nurses). Based on their suggestions, an open‐ended question that asked for other relevant outcomes was added. As expected, the test panel experienced the most difficulties with the questions about MCIDs, especially about relative risks. To simplify these questions, some minor linguistic changes were made. Also, an option ‘I do not know/no opinion’ was added to all questions about MCIDs.

The final questionnaire was programmed in Enalyzer. The questionnaire consisted of 24 questions. Respondents were first asked whether they were actively involved in the management of T2DM patients in their daily clinical practice. Only healthcare professionals working with T2DM patients were asked to complete the questionnaire. They were asked to score the importance of the outcomes used for the evaluation of new T2DM medicines on a 9‐point scale, assuming they were a member of a guideline committee. Respondents could also (optionally) mention other relevant outcomes. The questionnaire then explained the situation where a new treatment was compared to a control treatment. Respondents were asked which difference they would define as MCID. Because of the expected difficulty of estimating relative risks, the questionnaire stated a fictional situation where an absolute number of patients in the control group of 1.000 patients experienced the outcome. Respondents were asked which (absolute) number of outcomes in the treatment group would demonstrate a clinical relevant difference. A fictional example was given for clarification purposes. All questions were open‐ended, but only reasonable values (based on expert opinion) were permitted. The last part of the questionnaire was used for validation purposes. The questionnaire mentioned the used MCIDs for clinical relevance for HbA1c and mortality in the Dutch guideline, and asked the respondents whether they agreed with these values. These responses were triangulated with the corresponding open ended answers.

2.5. Data analysis

Respondents were categorized by profession. Other professions than physicians, pharmacists, practice nurses, diabetes nurses, nurse practitioners and physician assistants (physician associates) were not included in this analysis, because they were not considered as end–users of the guideline who have either prescription authority (physicians, diabetes nurses, nurse practitioners and physician assistants) or a direct influence on prescription behaviour (practice nurses and pharmacists). No distinction was made between healthcare professionals in primary and secondary care.

Mean scores for importance of the different outcomes (on a 9‐point scale) were calculated. Differences in the scores for importance between outcomes were compared by paired samples t‐test, and differences between professions with One‐way ANOVA. Results were considered statistically significant when p < 0.05. The other outcomes mentioned in the open‐ended questions were categorized by two researchers (based on consensus). One independent researcher verified the categorization.

For the analysis of cut‐off points, one highly unlikely value for body weight decrease (a difference of 90 kg) was excluded from further analysis. Respondents who found every difference relevant were assumed to support the lowest difference possible (1). The distribution of the cut‐off points was plotted for all variables and medians were calculated. All results were analysed with IBM SPSS Statistics 24.

3. RESULTS

3.1. Characteristics of healthcare professionals

A total of 394 respondents started the questionnaire, of whom 329 were healthcare professionals working with T2DM patients. Other professions than physicians, pharmacists, diabetes nurses, practice nurses, nurse practitioners and physician assistants were excluded (n = 83, predominantly healthcare assistants and nurses other than practice nurses). Another 35 respondents dropped‐out before the questions about relevance of outcomes. Therefore, the final population consisted of 211 healthcare professionals, including 44 physicians (predominantly general practitioners), 55 pharmacists (predominantly community pharmacists), 69 practice nurses, 27 diabetes nurses, 14 nurse practitioners and two physician assistants. Data of nurse practitioners and physician assistants were combined in the analysis, due to the low number of respondents and the similarity in profession.

The distribution of sex, age, years of working experience and number of patient contacts per week is shown in Table 2. The location of the practices was well distributed among the Netherlands. The majority of the physicians and pharmacists was well‐experienced (≥20 years of working experience).

TABLE 2.

Characteristics of respondents

	Physicians (n = 44)	Pharmacists (n = 55)	Practice nurses (n = 69)	Diabetes nurses (n = 27)	Nurse practitioners/physician assistants (n = 16)
Female sex	20 (46%)	38 (69%)	68 (99%)	25 (93%)	11 (69%)
Age (y)
20–39	8 (18%)	22 (40%)	7 (10%)	0 (0%)	2 (13%)
40–59	27 (61%)	26 (47%)	41 (59%)	21 (78%)	10 (63%)
≥60	9 (21%)	7 (13%)	21 (30%)	6 (22%)	4 (25%)
Working experience (y)
<5	6 (14%)	9 (16%)	5 (7%)	0 (0%)	7 (44%)
5–9	6 (14%)	7 (13%)	10 (15%)	1 (4%)	4 (25%)
10–14	4 (9%)	6 (11%)	26 (38%)	5 (19%)	2 (13%)
15–19	4 (9%)	4 (7%)	21 (30%)	14 (52%)	1 (6%)
≥20	24 (55%)	29 (53%)	7 (10%)	7 (26%)	2 (13%)
Number of patients contacts per week
<5	21 (48%)	9 (16%)	2 (3%)	3 (11%)	11 (69%)
5–10	17 (39%)	9 (16%)	17 (25%)	4 (15%)	4 (25%)
11–20	5 (11%)	11 (20%)	32 (46%)	11 (41%)	1 (6%)
≥20	1 (2%)	26 (47%)	18 (26%)	9 (33%)	0 (0%)

3.2. Relevance of outcomes

Healthcare professionals valued severe hypoglycemia as the most important outcome measure (mean score 8.30), followed by mortality (8.14) and quality of life (8.13) (Table 3). All outcomes were considered less important than severe hypoglycemia, except for mortality (p = 0.074). However, small differences did not affect the importance according to the GRADE scaling: other hypoglycemia and body weight were seen as important outcomes (score between 4 and 6), all other outcomes were of critical importance (score between 7 and 9).

TABLE 3.

Mean (SD) importance of outcomes measures, scored on a 9‐point scale.

Outcome measure	Mean score importance (SD)	Importance according to GRADE scaling
Severe hypoglycemia	8.30 (0.818)	Critical
Mortality	8.14 (1.032)	Critical
Quality of life	8.13 (0.991)*	Critical
Macrovascular morbidity	8.00 (0.933)**	Critical
Microvascular morbidity	7.95 (0.911)**	Critical
Hospital admissions	7.65 (1.104)**	Critical
HbA1c	7.04 (1.388)**	Critical
Other hypoglycemia a	6.64 (1.625)**	Important
Body weight	6.46 (1.360)**	Important

^aMild, modest or not‐specified.

^* p = 0.01,

^** p < 0.001 (all compared to severe hypoglycemia).

There were some differences in the assessment of importance of outcomes between professions (Figure 1). Diabetes nurses gave the highest scores for many outcomes, meaning that they valued outcomes more important than other professions. Physicians and pharmacists most often gave the lowest scores. Statistically significant differences (p < 0.05) between professions were found for all outcomes, except for mortality (p = 0.716) and quality of life (p = 0.138).

FIGURE 1.

Mean importance of outcomes according to different professions. ◇ Physicians; X Pharmacists; □ Practice nurses; ○ Diabetes nurses; △ Nurse practitioners/physician assistants

Of the 211 respondents, 114 healthcare professionals (54%) mentioned additional parameters they considered relevant in the assessment of blood glucose lowering drugs. Table 4 shows the outcome measures mentioned by at least two respondents. Adverse events (44.7%), ease of use (41.2%) and costs (10.5%) were most often mentioned.

TABLE 4.

Other relevant outcomes mentioned by healthcare professionals

Outcome measure	Number of respondents (%)
Adverse events a	51 (44.7)
Ease of use	47 (41.2)
Costs	12 (10.5)
Renal effects b	8 (7.0)
Effects on insulin use	4 (3.5)
Drug interactions	3 (2.6)
Glucose parameters other than HbA1c	2 (1.8)

^aAdverse events include some specific adverse events, like gastro‐intestinal adverse events (n = 2), psychological adverse events (n = 2), lactate acidosis (n = 1) and fall risk (n = 1).

^bRenal effects include renal adverse events as well as use by patients with renal impairment.

3.3. MCIDs

MCIDs were investigated for HbA1c, body weight (increase as well as decrease), mortality, macrovascular and microvascular morbidity, hospital admissions, severe and other hypoglycemia. A considerable number of respondents found every difference clinically relevant or had no opinion (Table 5).

TABLE 5.

Response on MCID‐questions

	N	Every difference relevant (%)	No opinion (%)
HbA1c	192	19%	21%
Body weight increase	191	29%	15%
Body weight decrease	185	25%	11%
Mortality	156	27%	19%
Macrovascular morbidity	156	22%	22%
Microvascular morbidity	156	21%	24%
Hospital admissions	156	24%	20%
Severe hypoglycemia	156	28%	16%
Other hypoglycemia	156	13%	27%

Respondents who had no opinion were excluded from further analysis. The results of the remaining healthcare professionals can be found in Figure 2. Median MCIDs according to healthcare professionals were 4 mmol/mol for HbA1c, 3 kg for weight increase as well as decrease, 20% for both mortality and macrovascular morbidity, 25% for microvascular morbidity, hospital admissions and severe hypoglycaemia and 50% for other hypoglycaemia.

FIGURE 2.

(A‐I) Boxplots of MCIDs for outcomes. Dotted lines indicate the MCIDs used in guideline development

For validity reasons, respondents were asked whether they agreed with the MCID used in the clinical guideline for HbA1c (5 mmol/mol) and mortality (RRR = 10%). Figure 3 shows the correspondence of this answer (x‐axis) with the earlier preferred MCID, mentioned as open answer (y‐axis). Although the answers roughly correspond, the wide range of answers (especially for mortality) shows that there was a considerable number of respondents whose answers were not in line. For example, a substantial amount of the respondents preferred an MICD <10% for mortality according to the close‐ended question, but previously mentioned an MCID >10% in the open‐ended question. Most likely, this indicates interpreting difficulties with the estimation of MCIDs, especially for RRRs.

FIGURE 3.

(A and B) Correspondence of close‐ended (x‐axis) and open‐ended (y‐axis) questions about preferred MCIDs for HbA1c and mortality

4. DISCUSSION

In the evaluation of new T2DM medicines, healthcare professionals considered most outcomes used in the Dutch T2DM guideline in primary care as critically important. Exceptions were other hypoglycemia and change in body weight. Severe hypoglycemia was valued as the most important outcome, followed by mortality and quality of life. As additional parameters, adverse events, ease of use and costs were also seen as relevant. The preferred median MCIDs for HbA1c, body weight, macrovascular and microvascular morbidity, hospital admissions and severe hypoglycemia were comparable with the MCIDs used in the development of the Dutch T2DM guideline. For mortality and other hypoglycemia, healthcare professionals preferred higher median MCIDs. ⁴ However, this result should be interpreted with caution, because of the difficulties the respondents experienced with the estimation of MCIDs.

The views of healthcare professionals on importance of outcomes roughly correspond with the evaluation by the guideline committee. Compared to the guideline committee, only HbA1c and hospital admissions were valued differently (critical instead of important). The relevance of adverse events, ease of use and costs did also align. These outcomes were also considered during the process of the clinical guideline development, although at a later stage. ⁴

Remarkably, a safety outcome (severe hypoglycemia) was seen as most important, even more important than mortality and other efficacy parameters. Especially practice nurses, diabetes nurses and nurse practitioners/physician assistants valued the importance of severe hypoglycemia. Pharmacists gave the lowest scores for the importance of hypoglycemia (both severe and other). This difference might reflect the intensity of patient contacts among these professions. Healthcare professionals with many patient contacts will most likely have a more profound experience with hypoglycemia and thus are confronted with the impact of severe hypoglycemia on patients. ¹⁴ , ¹⁵ , ¹⁶ However, other explanations for the differences between professions cannot be excluded, since the distribution of sex, age and years of working experience were also markedly different between the professions.

The results of importance of outcomes are in line with a study by Mol et al., ¹⁷ that showed that physicians valued cardiovascular benefits of T2DM drugs as the most important aspect in making regulatory decisions. HbA1c, hypoglycemia and weight gain did also significantly affect physicians’ choices. ¹⁷ A study by Gauthier et al., ¹⁶ however, showed that prescribers considered the overall efficacy in achieving glycemic control as the most important factor in choosing a blood glucose lowering drug if a patient failed on metformin. Also, cost and insurance coverage, risk of hypoglycemia, weight gain, short‐ and long‐term adverse events and quality of life were valued as important considerations. Clinical efficacy outcomes, like mortality and macro‐ and microvascular morbidity were barely mentioned. ¹⁶ The differences between the results of Gauthier et al. compared to our investigation and the study by Mol et al. ¹⁷ might be explained by the setting. Gauthier et al. investigated considerations in prescribing blood glucose lowering drugs to individual patients, while the investigation by Mol et al. and our study focused on decision‐making at the population level in regulatory science and guideline development, respectively. In daily practice, decisions might be more influenced by short term outcomes on patient level, while clinical guidelines and regulatory agencies particularly focus on long term outcomes and population level. ¹⁸ , ¹⁹ Additionally, cultural differences and a shift towards valuing direct outcomes for clinical efficacy over surrogate outcomes during the last years could also have contributed. ⁶ , ¹⁰ Our study did not involve patients views on clinical relevance of T2DM drugs. However, their views have been investigated intensively elsewhere. Patients value glucose control, body weight, ease of use, hypoglycemia and other side effects important. ¹⁴ , ¹⁵ , ¹⁶ , ²⁰ , ²¹ , ²² The views of patients—as well as the views of healthcare professionals—are mostly in line with those of regulators. ¹⁷

Our study also showed that healthcare professionals experience difficulties with estimating MCIDs, as was already concluded during the development of the questionnaire and the responses of the test panel. Despite the changes made for reasons of understandability, approximately 20 percent of the respondents had no opinion or did not answer the questions about MCIDs. Moreover the wide range of answers given, especially for RRRs, also indicate difficulties with the interpretation of these relative outcome measures. ²³ , ²⁴ However, the validation questions show that—despite the difficulties—there was reasonable alignment and the answers therefore give an indication about the estimation of MCIDs by healthcare professionals. The median MCIDs for HbA1c and body weight decrease were very close to the MCIDs used in guideline committees. The distinction made by the guideline committee between MCIDs for body weight decrease and increase was not seen in our results: the median MCID according to healthcare professionals was the same for both situations. The median MCIDs for other hypoglycemia was obviously higher (50%) than for mortality and macrovascular morbidity (20%) and microvascular morbidity, hospital admissions and severe hypoglycemia (25%). This also aligns with the establishment of relative importance of those outcomes, since other hypoglycemia was, among these outcome measures, also seen as the least important outcome. Due to these interpreting difficulties, no further analyses were performed on the MCIDs according to type of healthcare professional.

To the best of our knowledge, this is the first study that investigated the views of healthcare professionals about MCIDs used in the evaluation of T2DM medicines. However, we previously reported the correspondence of preferred outcomes and MCIDs for COPD medicines between healthcare professionals and regulatory agencies. Healthcare professionals preferred higher cut‐off values for clinical relevance for COPD‐related PROMs than the MCIDs used by registration authorities. ²⁵ In addition, the need for focus on clinical relevance in addition to statistical significance is often highlighted, in the conducting as well as reporting and interpretation of clinical trials. ²⁶ , ²⁷ The difficulties in the interpretation of risks and clinical relevance found in this study also highlights the need for education of healthcare professionals about the interpretation of clinical benefit of (new) medicines. ²³ , ²⁴ , ²⁸ Moreover the clinical relevance of new medicines can be over‐ or underestimated by healthcare professionals if the used outcomes and MCIDs in the evaluation of those medicines are not clearly communicated, especially since the views of healthcare professionals do not necessarily correspond with those of regulators and guideline committees.

This investigation was meant as a first study to explore the opinion of healthcare professionals on outcomes and MCIDs used in the evaluation of new medicines in the Dutch T2DM guideline in primary care. Since this study is based on the opinions of healthcare professionals working with T2DM patients, it provides a clear view of how clinical relevance of new medicines is considered in their daily practice. A main strength of this investigation is the exploratory and open character which was stimulated by the questionnaire with open‐ended answers.

There are, nonetheless, some limitations of this study. First, the response rate seemed poor. This can be explained by the use of the mailing list for newsletters of the IRUM, which contains both email addresses of healthcare professionals and other stakeholders in pharmaceutical care. Since the profession of the subscribers was not known, it was not possible to target the invitation for the questionnaire. Although there was still a considerable number of 211 respondents, this approach might have limited the validity and generalizability of this study, also because only healthcare professionals that subscribed to the IRUMs newsletter and therefore will be interested in pharmaceutical care and IRUMs activities were included in this study. Second, no distinction could be made between healthcare professionals from primary and secondary care. Although most physicians and pharmacists were working in primary care, the work setting of the diabetes nurses and physician assistants/nurse practitioners was not known. Last, the questions about MCIDs, especially for RRRs were fairly difficult, as can be seen in the proportion of respondents that did not answer these questions and the wide range of answers. The examples given in the questionnaire for clarification purposes could thereby have influenced the respondents. However, from the results of the validation questions it can be concluded that the majority of respondents interpreted the questions correctly, and the results for MCIDs can therefore be interpreted, albeit with caution.

This study must be seen as a first exploratory investigation towards the alignment of outcomes and MCIDs between the guideline committee T2DM and end users of the guideline. This study suggests that the views of healthcare professionals on the evaluation of importance of outcomes and MCIDs for the evaluation of new T2DM medicines are in line with the views in the guideline committee. However, HbA1c and hospital admissions were more important according to healthcare professionals and the MCIDs for mortality and other hypoglycemia were higher than the MCIDs used in the guideline. For those parameters, healthcare professionals were therefore more strict in defining clinical relevance than the guideline committee. Future research should confirm these results by the use of a larger representative group of healthcare professionals. In the meantime, clinical guideline committees should clearly communicate about how clinical relevance is established, so end users of the guideline can easily track the way new medicines were evaluated.

CONFLICT OF INTEREST

Dankers M, Nelissen‐Vrancken MHJMG, Hart HE, Lambooij AC and Mantel‐Teeuwisse AK declare no conflict of interests. van Dijk L received unrestricted grants from TEVA Pharmaceuticals, AstraZeneca, Pfizer and AbbVie for research projects not related to this study.

AUTHOR CONTRIBUTIONS

Dankers M, Nelissen‐Vrancken HJMG, Hart HE, van Dijk L and Mantel‐Teeuwisse AK developed the questionnaire. Dankers M and Hart HE analyzed the data. Lambooij AC verified the data analysis. Dankers M wrote the first draft. All authors revised the manuscript.

APPENDIX 1. Questionnaire

INTRODUCTION

The Instituut Verantwoord Medicijngebruik (Institute for Rational use of Medicine) studies the added value of new medicines in primary care. This questionnaire focuses on the criteria that guideline committees use to evaluate the clinical relevance (added value) of new medicines for type 2 diabetes mellitus (T2DM).

Answering the questionnaire takes approximately 10 minutes. Every 10^th participant receives a free accredited online course on the treatment of T2DM. If you are interested in this course, please enter your e‐mail address at the end of the questionnaire. We will only use your e‐mail address to send the login code for the online course. All data will be processed anonymously.

Starting questions

We would first like to ask you some general questions about yourself and your working experience.

1. Are you involved in the daily treatment of patients with T2DM?

• Yes

• No (end of questionnaire)

2. What is your gender?

• Woman

• Man

• Gender‐neutral

3. What is your age?

• Younger than 20 years

• 20 to 39 years

• 40 to 59 years

• 60 years or older

4. What is your current profession?

• Physician (forward to 5)

• Pharmacist (forward to 6)

• Practice nurse (forward to 7)

• Diabetes nurse (forward to 7)

• Nurse practitioner (forward to 7)

• Physician Assistant (forward to 7)

• Other, namely {open field} (forward to 7)

5. What is your specialization?

• General practitioner with special interest in T2DM

• General practitioner (including general practitioner trainee and dispensing general practitioner)

• Internist

• Other, namely {open field}

(forward to 7).

6. What is your specialization?

• Community pharmacist (including community pharmacist specialist trainee)

• Hospital pharmacist (including hospital pharmacist trainee)

• Pharmacist in outpatient pharmacy in hospital

• Other, namely {open field}

7. How many years of working experience in your current profession do you have?

• Less than 5 years

• 5 to 9 years

• 10 to 14 years

• 15 to 19 years

• 20 years or more

8. What are the first 2 digits of the zip code for your working area? (We only use this answer to look at regional distribution).

{Open question: only answers between 10 and 99 allowed).

9. On average, how many T2DM patient contacts (for this condition) do you have per week?

• Less than 5

• 5 to 10

• 11 to 20

• More than 20

Outcome measures

The last years, new T2DM medicines have become available (DPP4 inhibitors, GLP1 agonists and SGLT2 inhibitors). In order to develop a clinical guideline, the guideline committee first determines medicine relevant effects. We call this the outcome measures. For example, a new medicine for T2DM can be evaluated on the outcome measure ‘mortality’, but also on ‘HbA1c’ or ‘hypoglycaemia’. The following questions concern your opinion on the importance of these outcome measures. In other words: should improvement of this outcome measure be included in the evaluation of a medicine?

You are a member of the guideline committee. According to you, how important are the effects of a blood glucose‐lowering medicine on the following outcome measures? Please give your answer on a scale from 1 to 9, 1 meaning limited importance, 9 meaning critical importance.

• Mortality

• Macrovascular morbidity

• Microvascular Morbidity

• HbA1c

• Body weight

• Quality of life

• Hospital admissions

• Severe hypoglycaemia

• Mild, moderate, or unspecified hypoglycaemia

{Answers on a scale of 1 ‐ limited importance to 9 ‐ critical}.

10. Are there other outcome measures you consider relevant when evaluating new medicine for T2DM? Which outcome measure(s)?

{Open question, not obligatory}.

Clinically relevant improvements

A guideline committee must also determine which difference in effect size between the new medicine and a control medicine is large enough to have added value for the patient. We call this difference or improvement clinically relevant. In the following questions you can indicate when you consider a difference to be clinically relevant.

11. HbA1c.

We compare a new medicine for T2DM with a control agent. The new medicine causes a larger HbA1c decrease in patients. What difference in HbA1c decrease between the control agent and the new agent do you consider clinically relevant?

For example, in the control group, the HbA1c decreases by 8 mmol/mol. In the group with the new medicine, the HbA1c decreases by 10 mmol/mol. Your answer is then 2 mmol/mol.

• ‐Give your answer in mmol/mol (in whole numbers).
• ‐Do you think every difference is clinically relevant? Then your answer should be "1".
• ‐If you don't know or don't have an opinion, your answer should be ‘0'.

{Open question, only answers between 0 and 25 allowed}.

12. Body weight gain.

Both the new medicine and the control agents increase body weight. What difference in body weight gain between the control agent and the new agent do you consider clinically relevant?

For example, in the control group the body weight increases by 5 kg. In the group with the new agent, the body weight increases by 2 kg. Your answer is then 3 kg.

• ‐Assume an average body weight of 100 kg.
• ‐Assume that both the control agent and the new agent increase the body weight.
• ‐Give your answer in kg (whole numbers).
• ‐Do you think every difference is clinically relevant? Then your answer should be "1".
• ‐If you don't know or don't have an opinion, your answer should be ‘0'.

{Open question, only answers between 0 and 99 allowed}.

13. Body weight decrease.

The new medicine reduces the body weight compared to the control agent. What difference in body weight reduction between the control and new agent do you consider clinically relevant?

For example, in the control group the body weight increases by 2 kg. In the group with the new medicine, the body weight decreases by 1 kg. Your answer is then 3 kg.

• ‐Assume an average body weight of 100 kg.
• ‐Assume the control agent increases or does not affect the body weight and the new agent decreases the body weight.
• ‐Give your answer in kg (whole numbers).
• ‐Do you think every difference is clinically relevant? Then your answer should be "1".
• ‐If you don't know or don't have an opinion, your answer should be ‘0'.

{Open question, only answers between 0 and 99 allowed}.

Other outcomes

We compare a new T2DM medicine with a control agent.

1,000 patients use the control agent, of which 100 patients experience the outcome of interest.

1,000 other patients use the new medicine.

For how many outcomes in the group with the new medicine do you consider the difference to be clinically relevant?

For example, 100 out of 1,000 patients in the control group die. You think the difference in mortality is clinically relevant if only 20 out of 1,000 patients with the new medicine die. Your answer is then 20.

• ‐Give your answer in number of outcomes (whole numbers).
• ‐Do you think every difference is clinically relevant? Then your answer should be "99".
• ‐If you don't know or don't have an opinion, your answer should be ‘0'.

14. Mortality {Open question, only answers between 0 and 99 allowed}.

15. Macrovascular morbidity {Open question, only answers between 0 and 99 allowed}.

16. Microvascular morbidity {Open question, only answers between 0 and 99 allowed}.

17. Hospital admissions {Open question, only answers between 0 and 99 allowed}.

18. Severe hypoglycaemia {Open question, only answers between 0 and 99 allowed}.

19. Mild, moderate or unspecified hypoglycaemia {Open question, only answers between 0 and 99 allowed}.

Cut‐off values in guidelines

Guideline committees have established cut‐off values for clinical relevance for some outcome measures. The following questions regard your opinion on these cut‐off values for HbA1c and mortality.

20. For HbA1c, the Dutch guideline T2DM in primary care (2018) considers a difference of 5 mmol/mol clinically relevant. What do you think of this value?

• Too low (I only consider a difference clinically relevant if it is greater than 5 mmol/mol)

• Good

• Too high (I consider differences less than 5 mmol/mol already clinically relevant)

21. For mortality, the Dutch guideline T2DM in primary care (2018) considers a relative risk reduction of 10% clinically relevant. What do you think of this value?

• Too low (I only consider a difference clinically relevant if it is greater than 10%)

• Good

• Too high (I consider differences less than 10% already clinically relevant)

Final questions

You have reached the end of the questionnaire. Every 10th participant in the questionnaire receives a free accredited online course on the treatment of T2DM. If you are interested in this free course, please enter your e‐mail address below.

22. Do you have any comments or questions regarding this questionnaire? {open question, not obligatory}.

23. What is your email address? {open question, not obligatory}.

Thank you for your cooperation! Click on ‘end survey’ to send your answers.

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available from the corresponding author upon reasonable request.