Abstract
Background:
The golimumab safety awareness study commenced in 2010 to measure, periodically, the awareness of golimumab prescriber healthcare professionals (HCPs) of specific risks associated with golimumab as well as awareness of the requirement to provide a patient alert card to each patient treated with golimumab, as described in the European golimumab educational program. The aim of this study was to measure the awareness of HCPs who prescribe or who intend to prescribe SIMPONI® (golimumab) of the risks potentially related to golimumab and of the requirements for distributing the patient alert card, as described in the golimumab educational program.
Methods:
A structured, quantitative Web-based survey was conducted in 2010, 2012, 2014, and 2016 in eight European countries among HCPs who were at that time current or future prescribers of golimumab for patients with rheumatoid arthritis, psoriatic arthritis, ankylosing spondylitis, nonradiographic axial spondyloarthritis, or ulcerative colitis.
Results:
The overall golimumab risk awareness was high for golimumab prescriber HCPs across the risk statement categories (median awareness in 2016 across categories: 91%). The awareness of the golimumab risks was generally slightly higher among rheumatologists (75–98%) and gastroenterologists (73–97%) than among dermatologists (67–94%). Overall, the awareness of the requirements for handing out the patient alert card to golimumab-treated patients remained steady or increased slightly in 2016 relative to the other surveys.
Conclusions:
The results of this study show that the awareness of risks associated with golimumab by golimumab prescriber HCPs is high. The information made available to golimumab prescriber HCPs appears to have been sufficient with respect to golimumab risk awareness education.
Keywords: educational program, golimumab, healthcare professionals, risk awareness, safety awareness study
Introduction
As defined in the 2012 European Medicines Agency (EMA) Good Pharmacovigilance Guidance (GVP) guidance, Module V, the overall aim of conducting a risk management plan (RMP) is to ensure that the benefits of a particular medicinal product exceed the risks by the greatest achievable margin.1 To this end, each RMP may include risk minimization measures (RMMs), interventions that are intended to prevent or reduce the impact of the occurrence of adverse reactions associated with the exposure to a medicinal product.2 Besides routine RMMs done for all approved products, such as the inclusion of safety information in the Summary of Product Characteristics, additional RMMs (aRMMs) may be implemented on a case-by-case basis for select products. Additional RMMs can include educational programs (EPs), controlled access programs, direct healthcare professional (HCP) communications, pregnancy prevention programs, controlled distribution systems, and patient alert cards (PACs).2 In addition to detailing these aRMMs, the EMA requires that RMPs include a plan for the periodic monitoring of RMMs/aRMMs to ensure that risk information is being communicated effectively.2 The assessment of the effectiveness of RMMs/aRMMs during the post-marketing phase can be challenging. The guidance provided in GVP Module XVI concerning the methodology for evaluating the effectiveness of RMMs/aRMMs lacks detail.2 For example, the guidance does not recommend any target threshold or cut-off point in relation to what is considered to be an adequate risk awareness level. In 2012, Prieto and colleagues proposed a framework for the measurement of the effectiveness of RMMs/aRMMs and their implementation, including an assessment of tool delivery and clinical knowledge, as well as final clinical outcomes.3 Banerjee and colleagues further enhanced this framework (i.e. five-level evidence) for the evaluation of the impact of RMMs/aRMMs during the post-marketing phase: tool coverage, tool awareness and usage, risk knowledge and comprehension, behavioral modification and incidence of safety outcomes.4
Version 2 of Module V of the GVP released in 2017 further specified that an RMP should be modified and updated continually throughout the lifetime of a medicinal product as new information on the product becomes available, possibly including the adaptation or discontinuation of aRMMs or the assessment of their effectiveness.1 Educational materials related to the EP required at the time of launch of a new medicinal product may no longer be necessary or relevant once they have been made available for a number of years.2 Conversely, a one-time distribution of educational tools may be insufficient to ensure that all potential prescribers are reached and additional periodic re-distribution of the tools might be necessary.2
Golimumab (GLM; SIMPONI®) is an anti-tumor necrosis factor (TNF)-α antibody that was approved for the treatment of rheumatoid arthritis (RA), ankylosing spondylitis (AS), and psoriatic arthritis (PsA) in the European Union (EU) in October 2009. Additional indications were added, including ulcerative colitis (UC) in September 2013, nonradiographic axial spondyloarthritis (nr-axSpA) in June 2015, and polyarticular juvenile idiopathic arthritis in June 2016.5 As part of the EU GLM marketing authorization in 2009, a GLM RMP was submitted that included the execution of an EP (GLM EP), which involved an HCP handout designed to inform current and future GLM prescribers about certain risks associated with GLM use, as well as the proper technique to administer GLM (Figure 1). This HCP handout was periodically updated as new key GLM risk information became available. The GLM RMP also details the need for GLM prescribers to hand out and explain the PAC to each patient treated with GLM (Figure 1). The GLM PAC describes certain risks to the patient associated with the use of GLM and serves as a wallet-sized reference card that they can share with their other (non-GLM prescribing) HCPs. In accordance with EMA guidance, the GLM EP has been updated as new indications have been approved and risk information concerning GLM use has emerged.
The GLM Safety Awareness Study was conducted to periodically evaluate the effectiveness of the GLM EP. The study also evaluated whether the GLM EP was used by GLM prescribers as a source of information on risks associated with GLM. This paper details the results of these surveys and discusses learnings on how the risk management approach could be further improved in view of emerging EMA recommendations.
Materials and methods
A structured, quantitative, Web-based questionnaire used to survey HCPs for this study was designed to ensure ease of use and accuracy of response, keeping in mind cultural/local language and cross-country consistency needs. Translations of the questionnaire in the local language were validated by an external translator. These translations were then sent to be programmed and fielded in a representative, random sample of GLM prescriber HCPs from eight countries in the EU (i.e. Belgium, France, Germany, Italy, Portugal, Spain, Sweden, and the United Kingdom). Comprehensive testing of the entire survey was conducted prior to fieldwork launch using the links that HCPs participating in the survey were to be provided. Database validity checks were conducted to ensure that all responses were being captured in their appropriate fields within the database.
Study participants
Identification of potential participants for this study was done using HCP databases for each of the eight selected survey countries. A sample of GLM prescriber specialists (rheumatologists, dermatologists, and gastroenterologists) was randomly selected from the HCP database for each of the eight survey countries. Each HCP in this randomly selected sample had been systematically pre-verified by the fieldwork agency for credentials including education, specialty, and address. Pre-verified HCPs were invited by the fieldwork agencies using email and phone calls to participate in the Web-based surveys. The fieldwork agencies offered HCPs a small payment, compliant with each survey country’s guidelines, for participation in the study. Those who accepted were sent an email message containing an individualized password and Web link for the study. In alignment with the indications in the GLM SmPC at the time of the surveys, the 2010 and 2012 surveys did not include gastroenterologists as a target, only rheumatologists and dermatologists. After the indication for UC was approved in late 2013, gastroenterologists were added as a target to the 2014 and 2016 surveys. The selection of HCPs who received invitations to participate in each of the four surveys was independent of one another.
In order to ensure the validity of the study, strict participant screening criteria were used for each of the four Web-based surveys. Upon logging in to the survey, HCPs were first asked to identify themselves as rheumatologists, gastroenterologists, or dermatologists, and then were asked to identify the number of years they had been in clinical practice. HCPs were allowed to participate in the study if they had prescribed GLM in the past 12 months or intended to prescribe it in the next 12 months to treat RA, AS, PsA, nr-axSpA (rheumatologists), UC (gastroenterologists), or PsA (dermatologists).
Study design
A baseline survey was conducted in 2010, 1 year after the initial EU approval of GLM for RA, AS, and PsA, with a follow-up survey being conducted every 2 years thereafter (2012, 2014, and 2016). The questionnaire used in these surveys was updated regularly in alignment with the above-mentioned updating of the GLM EP or the approval of a new indication.
In each survey, HCPs were asked to respond ‘true’, ‘false’ or ‘not sure’ on statements related to certain risks associated with GLM as described in the GLM EP (i.e. tuberculosis, severe infections, hepatitis B virus reactivation, congestive heart failure, injection site and hypersensitivity reactions, administration errors) and on the requirement for handing out the PAC to each patient treated with GLM, as shown in Figure 1. In 2012, a question was added concerning a risk of hypersensitivity in patients at the first dose of GLM reflecting the addition of that risk as an important identified risk in the GLM RMP. In 2014, a question was added concerning the recommendation of a periodic skin examination for patients using GLM, reflecting the addition of melanoma and Merkel cell carcinoma as important identified and potential risks respectively in the GLM RMP.
A specific question was also included to enquire about the data sources the HCPs used to learn about the risks associated with GLM. In 2016, a question was added to investigate which specific office personnel were responsible for distributing the GLM PAC to patients.
Statistical analysis
Data analysis was conducted using IBM SPSS Statistics (2015, International Business Machines Corporation [IBM], Armonk, NY, USA). Variables were checked for validity and consistency, and then used to perform univariate descriptive analyses yielding frequencies of responses as absolute numbers, percentages, and proportions.
Results
Table 1 shows the numbers of GLM potential or future prescribers participating in the surveys of the GLM Safety Awareness Study, presented by survey year, country, and specialty. In the initial 2010 and 2012 surveys, an average of 219 HCPs (174 rheumatologists and 45 dermatologists) participated, while for the 2014 and 2016 surveys, an average of 416 HCPs (200 rheumatologists, 135 gastroenterologists, and 81 dermatologists) participated.
Table 1.
Country | Rheumatologists |
Gastroenterologists |
Dermatologists |
All specialties |
||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
2010 | 2012 | 2014 | 2016 | 2010 | 2012 | 2014 | 2016 | 2010 | 2012 | 2014 | 2016 | 2010 | 2012 | 2014 | 2016 | |
Belgium | 28 | 30 | 17 | 30 | n/a* | n/a | 16 | 20 | 0 | 0 | 17 | 3 | 28 | 30 | 50 | 53 |
France | 22 | 19 | 31 | 26 | n/a | n/a | 10 | 25 | 2 | 4 | 7 | 9 | 24 | 23 | 48 | 60 |
Germany | 26 | 23 | 13 | 27 | n/a | n/a | 20 | 21 | 11 | 25 | 15 | 18 | 37 | 48 | 48 | 66 |
Italy | 17 | 25 | 30 | 28 | n/a | n/a | 12 | 17 | 6 | 12 | 21 | 20 | 23 | 37 | 63 | 65 |
Portugal | 9 | 13 | 14 | 15 | n/a | n/a | 6 | 9 | 5 | 2 | 7 | 3 | 14 | 15 | 27 | 27 |
Spain | 24 | 24 | 30 | 27 | n/a | n/a | 21 | 26 | 4 | 6 | 9 | 10 | 28 | 30 | 60 | 63 |
Sweden | 22 | 25 | 28 | 28 | n/a | n/a | 20 | 16 | 4 | 3 | 9 | 4 | 26 | 28 | 57 | 48 |
United Kingdom | 13 | 28 | 26 | 30 | n/a | n/a | 11 | 20 | 2 | 4 | 3 | 7 | 15 | 32 | 40 | 57 |
All countries | 161 | 187 | 189 | 211 | n/a | n/a | 116 | 154 | 34 | 56 | 88 | 74 | 195 | 243 | 393 | 439 |
n/a = not applicable.
Figure 2 shows the different risk statements assessed in the survey and the percentages of HCPs indicating ‘true’ to those statements, by year of survey. In the initial ‘baseline’ survey in 2010, levels of awareness of 70% or above were noted for all risk statements. Even for the more recently added risk statements concerning the risk of hypersensitivity in patients at the first dose of GLM (67%) and the need for periodic risk examination (73%), baseline awareness levels (2012 and 2014, respectively) were high. For all risk statements, a further increasing trend in awareness was noted across the survey years, yielding median overall HCP awareness across all risk statement categories in the 2016 survey of 91% (range: 74–97%; Figure 2).
Focusing on the 2016 survey, analysis of the awareness levels of the risk statements by specialty (Figure 3) showed a general tendency towards slightly lower awareness among dermatologists (67–94%) compared with rheumatologists (75–98%) and gastroenterologists (73–97%; Figure 3). In interpreting these data, consideration should be given to the relatively small sample of dermatologists relative to the other specialties included in this analysis (74 dermatologists versus 211 rheumatologists and 154 gastroenterologists).
With regard to the requirement for HCPs to provide patients with the PAC, awareness levels across the different relevant risk statements ranged from 86% to 91% in 2016, 82% to 90% in 2014, 80% to 85% in 2012, and 86% to 89% in 2010 (Figure 2). Again, a slightly lower knowledge of the requirement was noted for dermatologists relative to the other specialties. Based on the additional question asked in the 2016 survey, across all countries, 84% of GLM prescriber respondents indicated that their office was actively distributing the PAC (76%, 85%, and 86% of dermatologists, rheumatologists, and gastroenterologists, respectively; Table 2). The physicians held themselves accountable for handing out and explaining the PAC to patients (29%), followed by nurses at the infusion center (20%) or general ward nurses (18%).
Table 2.
Personnel distributing patient alert cards | Percent of physicians giving each ranking |
||||||
---|---|---|---|---|---|---|---|
Rank 1 | Rank 2 | Rank 3 | Rank 4 | Rank 5 | I do not know | Our office does not distribute patient alert cards | |
Physician | 29% | 14% | 19% | 7% | 0.3% | 15% | 16% |
Nurse or nurse practitioner at infusion centre | 20% | 20% | 18% | 11% | 0.2% | 15% | 16% |
Ward nurse or nurse practitioner | 18% | 22% | 15% | 14% | 0.2% | 15% | 16% |
Physician’s assistant | 2% | 13% | 16% | 35% | 2% | 15% | 16% |
Other | 0.2% | 0.2% | 1% | 2% | 67% |
Clinical journals (80%), symposia/conferences (78%), the Summary of Product Characteristics for GLM (60%), and interaction with company personnel and medical colleagues (57%), were the most cited sources of information accessed by GLM prescriber HCPs to acquire knowledge about the risks of GLM (Figure 4).
Discussion
The EMA guideline on GVP Module XVI recommends the use of EPs as aRMM interventions to inform target HCPs and patients about the specific risks associated with medicinal products.2 The EMA also recommends that an assessment of target HCPs’ knowledge of the risk information in an EP be done periodically using scientifically rigorous methods.2 As part of the GLM EU-RMP, an EP was implemented for GLM in 2009. The GLM safety awareness study was commenced in 2010 to measure, periodically, GLM prescriber HCPs’ awareness of specific risks associated with GLM as well as awareness of the requirement to provide a PAC to each patient treated with GLM, as described in the GLM EP.
The study was comprised of four Web-based surveys of rheumatologists, gastroenterologists, and dermatologists conducted over a 7-year period among current or future prescribers of GLM in eight countries in the EU. A baseline survey was conducted in 2010, 1 year after the initial EU approval of GLM, with a follow-up survey being conducted every 2 years thereafter (2012, 2014, and 2016). There was a similar high level of awareness of GLM risks among GLM prescriber HCPs across all four surveys in this study. This finding could be attributable, at least in part, to the fact that the GLM EP was initiated with GLM prescriber HCPs before any launch of GLM in the EU and well prior to the initiation of the baseline (2010) survey. However, it is also likely that the other key, routine sources of GLM risk information listed in Figure 4 had significant contributions with respect to establishing high GLM risk awareness among GLM prescribing HCPs prior to the GLM safety awareness study being conducted. In addition, GLM was the fifth anti-TNF-α antibody to be marketed in the EU, hence target HCPs had already been educated with regard to the risks generally associated with drugs in this class.
Based on the GLM surveys, a high level of awareness of all risk statements (median of 91% based on the 2016 survey) was noted, with a stable or slightly increasing level of awareness for those risks added later to the EP (linked to an expansion of indication or the identification of an additional safety concern in the RMP), and a slightly lower awareness of risks with those specialties for which the indication was added to the label at later stages of the product`s life cycle. Although these trends may provide evidence for the effectiveness of the EP, it is difficult to disentangle the specific contribution of the EP compared with other data sources such as peer-to-peer interactions, routine RMMs such as the SmPC, and information shared at scientific events. To investigate this, a question was included in the surveys regarding the sources of information HCPs used to learn about the risks associated with GLM. The latter data sources were clearly more frequently identified than the GLM EP. In line with this observation, in a previous survey study in the United States (US), physicians examining their preferences for and use of sources of medical information reported peer-reviewed journal articles and continuing medical education as being the most preferred and useful sources of medical information.6 Non-CME promotional meetings, pharmaceutical sales representatives, and managed care organizations were identified as being least useful or influential. The three previous reviews also concluded that the added benefit of distribution of printed educational materials on HCPs’ knowledge, clinical practices, and outcomes was uncertain.7–9 Previous studies have reported HCP recall of receipt of any printed materials distributed by pharmaceutical companies concerning drug risks as being generally low.10–12 Nevertheless, the same HCP survey studies reported high levels of HCP risk awareness or an acceptable level of appropriate prescribing behavior.
The high overall level of risk awareness observed in the present study was comparable to that reported for similar studies surveying HCP risk awareness for other drugs for which additional educational materials had been distributed.10–16 All of these studies reported risk awareness levels that generally ranged above 80%.10–16 The two previous publications of HCP survey studies examining risk awareness for other anti-TNF-α antibodies, for which EPs had been implemented, reported very similar risk awareness levels to those observed in this study.13–14 The question arises as to what level of risk awareness is considered adequate from a sponsor`s as well as a regulator`s perspective. Current EMA and US Food and Drug Administration regulatory guidelines concerning the monitoring of effectiveness of aRMMs do not recommend any specific target threshold or cut-off point in relation to an adequate risk awareness level. The present study also did not prespecify a threshold for adequate risk awareness, nor did the vast majority of previous HCP risk awareness survey studies. Nonetheless, as in the present study, most of the previously reported HCP risk awareness survey studies characterized the reported awareness levels as being ‘high’ or ‘very high’,10–14,16 with one study simply stating that the ‘majority’ of HCPs responded as being aware of the risk surveyed.15 Defining a target risk awareness threshold that can be considered adequate is a relevant topic for future research.
To achieve this, a correlation should be sought between the risk awareness levels measured in these surveys and higher-level information corresponding to the fourth and fifth levels in the five-level framework for the evaluation of aRMMs proposed by Banerjee and colleagues (i.e. behavioral modification and the occurrence of safety outcomes).4 Such a correlation requires more rigorous methodology including drug utilization studies, or even the use of patient registries. The EMA recommends comparisons of frequencies of post-marketing safety outcomes pre- and post- implementation of the RMM(s).2 When a pre–post design is infeasible (e.g. when RMMs are put in place at the time of initial marketing authorization, as was the case with GLM), the EMA has indicated that the comparison of an outcome frequency indicator obtained post-intervention against a predefined reference value obtained from a literature review, historical data, or expected frequency in the general population, would be acceptable.2 In the present study, examination of the rates of spontaneous post-marketing reports of adverse events linked to the risks in the GLM EP prior to the introduction of these risks in the EP compared with post-introduction provided no proof that the inclusion of the risks in the EP in any way influenced the reporting rates of the associated adverse events (data not shown). Given the multiple factors that affect safety reporting, this finding is not surprising.
Although from a safety perspective, the relative contribution of the different risk information sources is irrelevant (as long as a high level of risk awareness is achieved), from a risk management perspective, the use of the most efficient tool is relevant. Additional RMMs should be effective and not cause an excessive or undue burden on patients or the healthcare system.1
The updated GVP Module V also calls for the periodic reevaluation of aRMMs to assess the need for their continuation.1 Stopping or modifying an EP could be based on the fact that recommendations for specific clinical measures to address RMP risks have become part of routine clinical practice (e.g. inclusion into a standard treatment protocol or guideline), or in response to either a satisfactory or a substandard finding from an evaluation of the effectiveness of the aRMM. Given the high HCP awareness of GLM risks, the limited reported contribution of the GLM EP materials to this high awareness, the low utility described by HCPs of printed materials from pharmaceutical companies,9–11 as well as consideration of the contribution of the aRMM itself to overall risk awareness and the extent of the burden imposed by the aRMM on HCPs, it was determined by the sponsor that distribution of the GLM EP materials was, in fact, no longer warranted. The sponsor requested permission from the EMA to terminate the GLM EP, which was recently granted.
This study has several limitations. Participation in the study was voluntary. GLM prescriber HCPs who participated in the study may have been more compliant with the GLM EP than those who did not participate. Another limitation was that the questionnaire used in each of the surveys was not validated (using psychometric approaches), which may have resulted in bias due to unanticipated communication barriers.17,18 The fact that GLM prescriber HCPs were offered small financial incentives to participate in the study may also have introduced bias.19,20 The independent, random nature of the selection of HCPs invited to participate in each survey meant that there was no possibility to assess the longitudinal effects of the GLM EP on risk awareness in the same group of HCPs over time. The lack of a comparator HCP group that had not been exposed to the GLM EP also limited the assessment of the effectiveness of the program.
Conclusion
Overall, the results of the GLM safety awareness study demonstrated that the awareness of risks associated with GLM by GLM prescriber HCPs (rheumatologists, gastroenterologists and dermatologists) is high. Although the GLM EP itself does not appear to be a major source of risk information for GLM prescriber HCPs, the collective sources of information that have been made available to, and accessed by, GLM prescriber HCPs appear to have been sufficient with respect to GLM risk awareness education. In line with updated GVP guidance Module 5,1 it remains to be evaluated whether the GLM EP places undue and disproportionate burden to the HCP.
Acknowledgments
The authors wish to thank Alan Meehan (Merck & Co., Inc., Kenilworth, NJ, USA) for writing assistance and Jennifer Pawlowski and Jennifer Rotonda (Merck & Co., Inc., Kenilworth, NJ, USA) for assistance in preparing this paper for publication.
Footnotes
Funding: This study was supported by funding from Janssen Biologics BV (Leiden, The Netherlands) and Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc. (Kenilworth, NJ, USA).
Conflict of interest statement: L. Felo is an employee of Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc., Kenilworth, NJ, USA and holds stock/stock options in the company. M. Otero-Lobato and A. Geldhof are employees of Janssen Biologics BV, Leiden, The Netherlands, and W. Noël is an employee of Janssen Cilag, Belgium; all three of these employees hold stock/stock options in Johnson & Johnson. All of the authors are responsible for the work described in this manuscript. All authors were involved in at least one of the following: (1) conception, design, acquisition, analysis, statistical analysis, or interpretation of data and (2) drafting the manuscript or revising it for important intellectual content. All authors provided final approval of the version to be published.
ORCID iD: Lauren Felo https://orcid.org/0000-0002-1549-2189
Contributor Information
Lauren Felo, Merck & Co., Inc., 351 N. Sumneytown Pike, P.O. Box 1000, Mailstop UG3C-54, N. Wales, PA 19454.
Marijo Otero-Lobato, Medical Affairs, Janssen Biologics BV, Leiden, The Netherlands.
Anja Geldhof, Medical Affairs, Janssen Biologics BV, Leiden, The Netherlands.
Wim Noël, Medical Affairs, Janssen Biologics BV, Leiden, The Netherlands.
References
- 1. European Medicines Agency. Guideline on good pharmacovigilance practices (GVP) Module V – Risk management systems (Rev 2). EMA/838713/2011 Rev 2*, http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2012/06/WC500129134.pdf (2017, accessed 3 May 2017).
- 2. European Medicines Agency. Guideline on good pharmacovigilance practices (GVP) Module XVI – Risk minimisation measures: selection of tools and effectiveness indicators (Rev 2). EMA/204715/2012 Rev 2*, http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2014/02/WC500162051.pdf (2017, accessed 3 May 2017).
- 3. Prieto L, Spooner A, Hidalgo-Ramos A, et al. Evaluation of effectiveness of risk minimization measures. Pharmacoepi Drug Saf 2012; 21: 896–899. [DOI] [PubMed] [Google Scholar]
- 4. Banerjee AK, Zomerdijk IM, Wooder S, et al. Post-approval evaluation of effectiveness of risk minimization: methods, challenges and interpretation. Drug Saf 2014; 37: 33–42. [DOI] [PubMed] [Google Scholar]
- 5. European Medicines Agency. SIMPONI summary of product characteristics, http://www.ema.europa.eu/docs/en_GB/document_library/EPAR_-_Product_Information/human/000992/WC500052368.pdf (2019, accessed 5 May 2017).
- 6. Salinas GD. Trends in physician preferences for and use of sources of medical information in response to questions arising at the point of care: 2009–2013. J Cont Ed Health Prof 2014; 34: S11–S16. [DOI] [PubMed] [Google Scholar]
- 7. Giguere A, Legare F, Grimshaw J, et al. Printed educational materials: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012; 10: CD004398. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Grudniewicz A, Kealy R, Rodseth RN, et al. What is the effectiveness of printed educational materials on primary care physician knowledge, behaviour, and patient outcomes: a systematic review and meta-analyses. Implement Sci 2015; 10: 164. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Farmer AP1, Légaré F, Turcot L, et al. Printed educational materials: effects on professional practice and health care outcomes. Cochrane Database Syst Rev 2008; 3: CD004398. [DOI] [PubMed] [Google Scholar]
- 10. Toussi M. Evaluation of the effectiveness of risk minimisation measures: a survey among health care professionals to assess their knowledge and attitudes on prescribing conditions of Instanyl® in France and the Netherlands, http://www.encepp.eu/encepp/openAttachment/studyResult/16235. (2016, accessed 29 September 2017).
- 11. Lem J. Evaluation of the effectiveness of additional risk minimisation measures (aRMMs) that aim to reduce the risks of photoxicity, squamous cell carcinoma (SCC) of the skin and hepatic toxicity in patients receiving voriconazole in the European Union (EU), http://www.encepp.eu/encepp/openAttachment/studyResult/13665 (2016, accessed 29 September 2017).
- 12. Brody RS, Liss CL, Wray H, et al. Effectiveness of a risk-minimization activity involving physician education on metabolic monitoring of patients receiving quetiapine: results from two postauthorization safety studies. Int Clin Psychopharmacol 2016; 31: 34–41. [DOI] [PubMed] [Google Scholar]
- 13. Arana A, Allen S, Burkowitz J, et al. Infliximab paediatric Crohn’s Disease educational plan A European, cross-sectional, multicentre evaluation. Drug Saf 2010; 33: 489–501. [DOI] [PubMed] [Google Scholar]
- 14. Smith MY, Attig B, McNamee L, et al. Tuberculosis screening in prescribers of anti-tumor necrosis factor therapy in the European Union. Int J Tuberc Lung Dis 2012; 16: 1168–1173. [DOI] [PubMed] [Google Scholar]
- 15. Kellier N. Physician survey to assess effectiveness of Strattera risk minimisation activities in prescribers treating adult patients with ADHD, http://www.encepp.eu/encepp/openAttachment/studyResult/8934 (2014, accessed 29 September 2017).
- 16. Huang K, Madison T. A cross-sectional study to evaluate the effectiveness of XALKORI Therapeutic Management Guide among physician prescribing XALKORI in Europe, http://www.encepp.eu/encepp/openAttachment/studyResult/18663 (2017, accessed 29 September 2017).
- 17. Scott I. You can’t believe all that you’re told: the issue of unvalidated questionnaires. Inj Prev 1997; 3: 5–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Choi BCK, Pak AWP. A catalog of biases in questionnaires. Prev Chronic Dis 2005; 2: 1–13. [PMC free article] [PubMed] [Google Scholar]
- 19. Cannell CF, Henson R. Incentives, motives, and response bias. Ann Econ Soc Meas 1974; 3: 307–317. [Google Scholar]
- 20. Schweitzer M, Asch DA. Timing payments to subjects of mail surveys: cost-effectiveness and bias. J Clin Epidemiol 1995; 48: 1325–1329. [DOI] [PubMed] [Google Scholar]