Skip to main content
NCBI home page
PLOS Medicine logoLink to PLOS Medicine
. 2022 Nov 16;19(11):e1004133. doi: 10.1371/journal.pmed.1004133

Scaling-up a pharmacist-led information technology intervention (PINCER) to reduce hazardous prescribing in general practices: Multiple interrupted time series study

Sarah Rodgers 1,2,‡, Amelia C Taylor 1,‡,*, Stephen A Roberts 3, Thomas Allen 4,5, Darren M Ashcroft 6,7, James Barrett 2, Matthew J Boyd 8, Rachel A Elliott 4, Kamlesh Khunti 9, Aziz Sheikh 10, Despina Laparidou 11, Aloysius Niroshan Siriwardena 11, Anthony J Avery 1,7
PMCID: PMC9718399  PMID: 36383560

Abstract

Background

We previously reported on a randomised trial demonstrating the effectiveness and cost-effectiveness of a pharmacist-led information technology intervention (PINCER). We sought to investigate whether PINCER was effective in reducing hazardous prescribing when rolled out at scale in UK general practices.

Methods and findings

We used a multiple interrupted time series design whereby successive groups of general practices received the PINCER intervention between September 2015 and April 2017. We used 11 prescribing safety indicators to identify potentially hazardous prescribing and collected data over a maximum of 16 quarterly time periods. The primary outcome was a composite of all the indicators; a composite for indicators associated with gastrointestinal (GI) bleeding was also reported, along with 11 individual indicators of hazardous prescribing. Data were analysed using logistic mixed models for the quarterly event numbers with the appropriate denominator, and calendar time included as a covariate.

PINCER was implemented in 370 (94.1%) of 393 general practices covering a population of almost 3 million patients in the East Midlands region of England; data were successfully extracted from 343 (92.7%) of these practices. For the primary composite outcome, the PINCER intervention was associated with a decrease in the rate of hazardous prescribing of 16.7% (adjusted odds ratio (aOR) 0.83, 95% confidence interval (CI) 0.80 to 0.86) at 6 months and 15.3% (aOR 0.85, 95% CI 0.80 to 0.90) at 12 months postintervention. The unadjusted rate of hazardous prescribing reduced from 26.4% (22,503 patients in the numerator/853,631 patients in the denominator) to 20.1% (11,901 patients in the numerator/591,364 patients in the denominator) at 6 months and 19.1% (3,868 patients in the numerator/201,992 patients in the denominator). The greatest reduction in hazardous prescribing associated with the intervention was observed for the indicators associated with GI bleeding; for the GI composite indicator, there was a decrease of 23.9% at both 6 months (aOR 0.76, 95% CI 0.73 to 0.80) and 12 months (aOR 0.76, 95% CI 0.70 to 0.82) postintervention. The unadjusted rate of hazardous prescribing reduced from 31.4 (16,185 patients in the numerator/515,879 patients in the denominator) to 21.2% (7,607 patients in the numerator/358,349 patients in the denominator) at 6 months and 19.5% (2,369 patients in the numerator/121,534 patients in the denominator). We adjusted for calendar time and practice, but since this was an observational study, the findings may have been influenced by unknown confounding factors or behavioural changes unrelated to the PINCER intervention. Data were also not collected for all practices at 6 months and 12 months postintervention.

Conclusions

The PINCER intervention, when rolled out at scale in routine clinical practice, was associated with a reduction in hazardous prescribing by 17% and 15% at 6 and 12 months postintervention. The greatest reductions in hazardous prescribing were for indicators associated with risk of GI bleeding. These findings support the wider national rollout of PINCER in England.

In a multiple interrupted time series study, Dr. Amelia C. Taylor and colleagues explore scaling-up a pharmacist-led information technology intervention (PINCER) to reduce hazardous prescribing in general practices.

Author summary

Why was this study done?

  • Prescribing errors in general practice are a preventable cause of patient morbidity, hospitalisations, and deaths.

  • The pharmacist-led information technology intervention (PINCER) aims to reduce hazardous prescribing by searching general practice clinical computer systems to identify patients at risk from hazardous prescribing and pharmacists working with the practices to ameliorate these.

  • In a large cluster randomised trial, the PINCER intervention was found to be an effective and cost-effective method for reducing hazardous prescribing in general practice.

  • The current study was done to evaluate whether the intervention would be effective when rolled out at scale.

What did the researchers do and find?

  • PINCER was implemented in 370 (94.1%) of 393 general practices in the East Midlands of England between September 2015 and April 2017; data were successfully extracted from 343 (92.7%) of these practices.

  • We used 11 prescribing safety indicators to identify potentially hazardous prescribing and collected data over a maximum of 16 quarterly time periods.

  • We used a multiple interrupted time series design, where the rate of hazardous prescribing before the intervention was compared to 6 months and 12 months postintervention. The data was adjusted for calendar time and general practice.

  • The PINCER intervention was associated with an overall decrease in the rate of hazardous prescribing of 16.7% at 6 months and 15.3% at 12 months postintervention.

  • We were not able to collect 6- and 12-month follow-up data on all practices, and this is the main limitation of the study.

What do these findings mean?

  • The findings suggest that the PINCER intervention was effective when rolled out at scale.

  • These findings support the wider national rollout of PINCER in England and may help to inform policy makers when considering implementation of similar interventions.

Introduction

Medication errors in general practice are an important and expensive preventable cause of patient safety incidents associated with morbidity, hospitalisations, and deaths. A study in English general practices identified errors in 5% of prescription items, with one in 550 items containing a potentially life-threatening error [1]. Further studies have shown hazardous prescribing in general practices to be a contributory cause of around one in 25 hospital admissions [2]. Preventable adverse drug events leading to a hospital admission are estimated to cost £83.7 million and to cause 627 deaths in England each year [3].

The World Health Organization (WHO) has identified “Medication Without Harm” as the theme for their Third Global Patient Safety Challenge, which aims to reduce severe avoidable medication-related harm by 50% globally by targeting healthcare providers’ behaviour, systems and practices of medication, medicines, and the public [4]. In response to this challenge, the Department of Health and Social Care in England commissioned a report on the prevalence and cost of medication errors, which estimated that 66 million potentially clinically significant errors occur per year, 71% of which are in primary care [3]. There is therefore a need to develop and implement interventions to reduce medication error associated with avoidable harm.

The PINCER intervention (a pharmacist-led information technology intervention for medication errors in general practice) involves searching general practitioner (GP) clinical systems using automated computerised hazardous prescribing indicators to identify patients at risk from their prescriptions, and then acting to correct the problems and to minimise future risk with pharmacist support. Our cluster randomised controlled trial found the PINCER intervention to be acceptable, effective, and cost-effective in reducing rates of hazardous prescribing [5]. At six months’ follow-up, the general practices receiving computerised feedback and pharmacist support had significantly less hazardous prescribing than those that received computerised feedback alone.

Since the original PINCER study, work has been undertaken to further refine the intervention and to implement the intervention at scale across the East Midlands region of England. The hazardous prescribing indicators were updated, based on a systematic review that identified 12 drug groups accounting for 80% of medication-related and preventable hospital admissions [2]. Three drug groups, anticoagulants, antiplatelets, and nonsteroidal anti-inflammatory drugs (NSAIDs) (which all cause gastrointestinal (GI) bleeding), were found to be responsible for over one-third of these admissions. An important implication from this review is that reducing hazardous prescribing in general practice associated with specific groups of drugs could prevent most medication-related hospital admissions.

Therefore, we developed a set of prescribing safety indicators [6] to identify patients exposed to medication errors in general practice, and the PINCER intervention is designed to ameliorate risk from the most common and important of these errors. We sought to assess whether PINCER was effective at reducing rates of hazardous prescribing when rolled out at scale across the East Midlands in the real-world setting. Our hypothesis was that the intervention would result in clinically important, sustained reductions in such hazardous prescribing.

Methods

Study design and participants

We used a multiple interrupted time series design whereby successive groups of general practices received the PINCER intervention.

Across the East Midlands of England, 12 Clinical Commissioning Groups (CCGs), which are National Health Service (NHS) bodies responsible for the commissioning of healthcare services in local areas, agreed to participate. All general practices in these CCGs were invited to take part in the knowledge that they used electronic health records with embedded electronic prescribing capability. Eleven general practices were not eligible for inclusion in the analysis as they had previously piloted the hazardous prescribing indicators to be used in the study.

The intervention

The PINCER intervention [5] comprises three components. First, the computer systems of general practices are searched to identify patients at risk of potentially hazardous prescribing using a set of prescribing safety indicators (Table 1). Second, pharmacists, specifically trained to deliver the intervention, provide an educational outreach intervention where they meet with GPs and other practice staff to:

Table 1. The prescribing indicators grouped by serious harm outcome.

Indicator Description of Indicator Group at risk (Denominator) Group exposed to hazardous prescribing (Numerator)
Hazardous prescribing indicators associated with GI bleeding
A Prescription of an oral NSAID, without coprescription of an ulcer healing drug, to a patient aged ≥65 years Patients aged ≥65 years without coprescription of an ulcer-healing drug (PPI or H2 antagonist) in the 3 months leading up to the audit date Patients prescribed an oral NSAID in the 3 months leading up to the audit date
B Prescription of an oral NSAID, without coprescription of an ulcer-healing drug, to a patient with a history of peptic ulceration Patients aged ≥18 years with a Read code for peptic ulcer or upper GI bleed at least 3 months before audit date and not prescribed an ulcer-healing drug (PPI or H2 antagonist) within the 3 months leading up to the audit date Patients prescribed an oral NSAID within the 3 months leading up to the audit date
C Prescription of an antiplatelet drug, without coprescription of an ulcer-healing drug, to a patient with a history of peptic ulceration Patients aged ≥18 years with a Read code for peptic ulcer or GI bleed at least 3 months before audit date and not prescribed an ulcer-healing drug (PPI or H2 antagonist) within the 3 months leading up to the audit date Patients prescribed an antiplatelet drug (aspirin or clopidogrel or prasugrel or ticagrelor) within the 3 months leading up to the audit date
D Prescription of warfarin or DOAC in combination with an oral NSAID Patients aged ≥18 years prescribed warfarin or a DOAC (apixaban or dabigatran or rivaroxaban) within the 3 months leading up to the audit date Patients prescribed an oral NSAID within the 3 months leading up to the audit date
E Prescription of warfarin or DOAC and an antiplatelet drug in combination without coprescription of an ulcer-healing drug Patients aged ≥18 years prescribed warfarin or DOAC without coprescription of an ulcer-healing drug (PPI or H2 antagonist) within the 3 months leading up to the audit date Patients prescribed an antiplatelet drug (aspirin or clopidogrel or prasugrel or ticagrelor) within the 3 months leading up to the audit date and within 28 days of the warfarin/DOAC prescription
F Prescription of aspirin in combination with another antiplatelet drug (without coprescription of an ulcer-healing drug) Patients aged ≥18 years prescribed aspirin without coprescription of an ulcer-healing drug (PPI or H2 antagonist) within the 3 months leading up to the audit date Patients prescribed another antiplatelet drug (clopidogrel or prasugrel or ticagrelor) within the 3 months leading up to the audit date and within 28 days of the aspirin prescription
Hazardous prescribing indicators associated with asthma
G Prescription of a nonselective β-blocker to a patient with asthma Patients aged ≥18 years with a Read code for asthma at least 3 months before audit date and no subsequent asthma resolved code during that time period Patients prescribed a nonselective β-blocker within the 3 months leading up to the audit date
H Prescription of a long-acting beta-2 agonist inhaler (excluding combination products with inhaled corticosteroid) to a patient with asthma who is not also prescribed an inhaled corticosteroid Patients aged ≥18 years with a Read code for asthma at least 3 months before audit date (and no subsequent asthma resolved code during that time period) who have been prescribed a long-acting beta-2 agonist inhaler (excluding combination products with inhaled corticosteroid) within the last 3 months Patients not prescribed an inhaled corticosteroid within the 3 months leading up to the audit date
Hazardous prescribing indicators associated with heart failure
I Prescription of an oral NSAID to a patient with heart failure Patients aged ≥18 years who have a diagnosis of heart failure at least 3 months before the audit date Patients prescribed an oral NSAID within the 3 months leading up to the audit date
Hazardous prescribing indicators associated with cardiovascular events, including stroke
J Prescription of antipsychotics for >6 weeks in a patient aged ≥65 years with dementia but not psychosis Patients aged ≥65 years with a Read code for dementia at least 3 months before the audit date and no Read code for psychosis (or have a psychosis Read code and a subsequent psychosis resolved Read code) at least 3 months before the audit date Patients prescribed antipsychotic drugs at least once within the 3 months leading up to the audit date
Hazardous prescribing indicators associated with acute kidney injury
K Prescription of an oral NSAID to a patient with eGFR <45 mL/min Patients aged ≥18 years with chronic renal failure:
eGFR <45 mL/min at least 3 months before the audit date		 Patients prescribed an oral NSAID within the 3 months leading up to the audit date
Composite of all indicators (A-K) Composite indicator (all 11 indicators) Number of unique patients in any of the above “at risk” groups (denominators) Number of patients exposed to a high-risk prescription (sum of the number of patients included in indicators A-K)
Composite of GI bleed indicators (A-F) Composite of GI bleed indicator (all GI bleed indicators) Number of unique patients in any of the above “at risk” groups (denominators associated with a GI bleed) Number of patients exposed to a high-risk prescription (sum of the number of patients included in indicators A-F)
Open in a new tab

DOAC, direct oral anticoagulant; eGFR, estimated glomerular filtration rate; GI, gastrointestinal; NSAID, nonsteroidal anti-inflammatory drug; PPI, proton pump inhibitor.

  • Discuss the search results and highlight the importance of the hazardous prescribing identified using brief educational materials. These feedback sessions were to be held straight after running the searches and then at regular intervals. The educational materials included training on root cause analysis using well-established techniques such as the fishbone diagram and the 5 whys.

  • Agree on an action plan, retained within the practice, for reviewing patients identified as high risk and improving prescribing and medication monitoring systems using root cause analysis to minimise future risk.

Third, the pharmacists (sometimes supported by pharmacy technicians) work with, and support, general practice staff to implement the agreed action plan, sometimes making the necessary changes themselves.

The East Midlands rollout was a pragmatic implementation study, and the PINCER intervention was delivered by the pharmacists employed in each of the CCG Medicines Optimisation teams. Training on the PINCER intervention and CHART software was delivered to members of the Medicines Optimisation Teams in all 12 CCGs that implemented the PINCER intervention. The time that the pharmacists spent varied by CCG, depending on the resourcing level of each of the CCG Medicines Optimisation Team. For example, we know some localities were very well resourced in terms of numbers of CCG pharmacists, whereas other localities had very little CCG pharmacist resource.

Since the original PINCER study [5], further changes were made to scale the intervention including updating training materials. PRIMIS at the University of Nottingham wrote computerised queries using Morbidity Information and Query Export Syntax (MIQUEST) software to identify patients at risk of hazardous prescribing for each of the prescribing safety indicators. The pharmacists working in the general practices received training on how to run the MIQUEST queries using the general practice clinical information system and send aggregate summative data to PRIMIS via secure transfer for inclusion in a comparative analysis service (CHART and CHART Online software; [7]). To evaluate the intervention, quarterly retrospective data were collected towards the end of the study using a modified set of MIQUEST queries. Retrospective data collection was undertaken either by the pharmacists working in the general practices or by PRIMIS (who, with permission from the CCGs and practices, remotely extracted the data). Data extracted from the general practices were transferred into an electronic database and were processed (using similar approaches to those that we adopted in the PINCER trial; [5]) in readiness for analysis. This included collation of numerators and denominators for each of the indicators for each of the time points, and calculation of composite indicators.

Outcome measures

We prespecified 11 prescribing safety indicators to identify potentially hazardous prescribing, described in Table 1. These 11 indicators are associated with the following serious harm outcomes: GI bleeding, asthma, heart failure, stroke, and acute kidney injury. The number of patients in each practice exposed to each of the 11 indicators and the number of patients in groups classified as being at risk of being exposed were collected. Data were collected retrospectively at quarterly time points over the period of 4 years (30 November 2013 and 31 August 2017). However, not all practices contributed data at all 16 time points. The implementation quarter for each practice are presented in the Supporting information (S1 Appendix). For each practice, the implementation quarter was assigned Quarter 0, postimplementation quarters were assigned Q1, Q2, Q3, etc., and preimplementation quarters were assigned Q-1, Q-2, Q-3, etc.

This yielded numerator and denominator data for each practice for each of the prescribing safety indicators for each quarter. The numerator and denominator data were used to calculate the proportion of patients at risk who were exposed to hazardous prescribing for each indicator in each GP practice over a maximum of 16 quarters (4 years). Composite measures were defined for patients exposed to any of the indicators and for the combined GI indicators (Table 1). For the numerator, the sum of the number for patients exposed to each type of hazardous prescribing included in the composite was used, and for the denominator, the number of patients at risk of exposure to the hazardous prescribing indicators was used. This allowed a proportion of patients exposed to be calculated. The primary outcome was the composite of all the indicators. Secondary outcomes were the composite of the indicators associated with GI bleeds, and each of the 11 indicators. The GI indicators were singled out as a group because there was a relatively large number of these, and other studies have suggested these may be particularly sensitive to intervention (reference DQIP [7] and SMASH [8]).

Statistical methods

Although we used a multiple interrupted time series design, we did not conduct a classic interrupted time series analysis. Instead, the model we used includes a linear temporal effect over the preintervention period and estimates a preintervention effect from this; the postintervention effects are the differences from this extrapolated trend. We were not expecting a sustained effect from the intervention, so we did not use the classic linear postintervention effect (step plus change in slope) but rather parameterised the model to allow for a nonlinear intervention effect.

Analyses were conducted on an intention-to-treat basis, with the assumption that practices became exposed to the intervention at the time the intervention was introduced (defined as the time when the first computer search was conducted in each practice to identify patients at risk). The assumption was also made that the intervention start date was the date that practices uploaded their data to CHART Online as part of the rollout of the intervention. This date was then used to indicate the implementation date for the retrospective data collection. Practices were included in the analysis irrespective of the degree to which they engaged in the PINCER intervention.

Where there were quarters at the extremes of the data collection period with very few practices (<10), a pragmatic decision was taken prior to any formal analysis to exclude these from the analysis. The number of practices included in each quarter aligned by implementation quarter are presented in the Supporting information (S1 Appendix).

Prior to analysis, outcome data were summarised graphically by calendar quarter and by time since the intervention, presented as a mean rate across all practices.

Formal analysis of hazardous prescribing for each of the single and the composite indicators utilised a mixed model approach, with logistic mixed models for the quarterly event numbers with the appropriate denominator. As the intervention was expected to be delivered over the course of a few weeks and the effects of this to diminish over time [5], a priori we wished to allow for a time-dependent intervention effect, rather than the usual segmented linear assumptions. Therefore, we represented the time postintervention as a categorical variable coding each postintervention quarter; all preintervention quarters were assigned a single reference level for the treatment effect (noting that the fitted outcomes will still vary according to the fitted secular trend). Preimplementation base rates and numbers at risk are presented as the mean of the four quarters prior to the implementation at each site. The quarter during which the intervention began was treated as the first postintervention quarter. The intervention will have occurred at some point during this quarter. Calendar time (to account for secular trends) was included as a covariate, modelled as a linear function of the calendar quarter. Random intercept terms for GP practice allowed for within practice correlations. Estimation of confidence intervals (CIs) utilised the robust standard errors approach.

Specific hypotheses of improvements over temporal trends at 6 months (primary) and 12 months (secondary) corresponded to the assessment times in the previous cluster randomised trial [5]. These were tested by constructing appropriate contrasts comparing the second and fourth quarters, where the quarter at zero is the quarter during which the implementation was introduced, with the preimplementation level. Effect sizes are presented as (adjusted) odds ratios (aORs) compared with the preintervention period. The model is described explicitly in the Supporting information (S2 Appendix).

Data were analysed using the statistical package Stata SE 16.

This study is reported as per the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline (S3 Appendix).

Ethical approval

The study received approval from the University of Nottingham Research Ethics Committee on 23 January 2017 (reference number A16012017).

Patient and public involvement

In developing our funding application for this project, we took account of the views of patients and members of the public, and the East Midlands Patient and Public Involvement Senate was a partner on the application. Three Patient and Public Involvement representatives were involved throughout the project and actively contributed to ongoing discussions about the conduct of the study. The Patient and Public Involvement representatives attended the quarterly Project Steering Committee meetings and Evaluation Advisory Committee meetings to provide oversight for the project. They supported and checked project information sheets prior to submission for ethical review and contributed to the development of the interview and focus group guides used in the qualitative study and the interpretation of the findings.

Results

Of the 393 general practices in the 12 participating CCGs in the East Midlands (October 2015), PINCER was implemented in 370 (94.1%) between September 2015 and April 2017. Ten practices had closed by the end of the implementation period, and 11 practices from one CCG were involved in piloting the PINCER indicators and were excluded from analysis. Therefore, 349 GP practices were eligible for retrospective data collection of which data were collected for 343 (92.7% of those that implemented PINCER and 98.0% of those eligible for data collection). To be included, a practice must have run the intervention and uploaded at least one quarter of data. Therefore, not all practices contributed data at all 16 time points. We had data for all 343 practices for between three and seven quarters before the start of the intervention, but due to some practices uploading their own data to CHART Online, there were some gaps in the data in the postintervention quarters, including quarter 0. The reduction in the number of practices at follow-up was not believed to be related to the practices’ engagement with PINCER, but instead whether they uploaded the data to CHART Online, to be shared with the research team. The duration of the postintervention data collected for each practice was limited for those practices that implemented the intervention towards the end of the study period and where practices stopped uploading data during the follow-up period. No postimplementation data were collected in seven practices. At 6 months follow-up, data for 212 practices were collected, and by 12 months follow-up, this had reduced to 70 (S4 Appendix). The missing data at follow-up does not seem to be related to the practice characteristics (Table 2). We looked at the characteristics of the 212 practices included at 6 months postintervention and the 70 practices included at 12 months postintervention and found these to be comparable with the 343 participating practices. As a sensitivity analysis, we also repeated the analysis including only practices with at least 6 months of follow-up data. A similar effect size was noted at both 6 and 12 months postintervention (S5 Appendix). This was expected as the intervention effect estimates using the entire data set were derived from those practices that had sufficient follow-up. We also compared the raw and adjusted preimplementation rates for those practices with up to 6 months and up to 12 months follow-up (S6 Appendix). The rates for each group of practices were found to be similar.

Table 2. Characteristics of 343 general practices in the East Midlands that implemented PINCER and were included in the analysis compared with English general practices overall.

Characteristics Study practices (n = 343) Study practices included at 6 months (n = 212) Study practices included at 12 months (n = 70) All English practices
List size: Mean (SD) 8,096 (4,777.9) 8,516 (5,018.8) 9,205 (6,477.1) 7,586a
Clinical computer system: Number (%) TPP 263 (76.7) 152 (71.6) 54 (77.1) (30)b
EMIS 80 (23.3) 60 (28.3) 16 (22.9) (56)b
Index of multiple deprivation: Mean (SD) c 22.9 (11.1) 19.7 (10.0) 22.9 (11.0) 21.8
Percentage of patients from minority ethnic groups: Mean (SD) d 16.6 (21.3) 9.8 (13.3) 10.9 (14.1) 14.0
GP whole time equivalent (per 1,000 patients): Mean (SD) e 4.65 (3.10) 4.83 (3.12) 5.3 (3.9) 4.16
Quality and Outcomes Framework score 2016/2017: Mean (SD) out of 559 points available f 537 (29.0) 539 (26.7) 536 (28.6) 534
CQC Safety Rating: percentage g Outstanding 12 (40 practices) 12 (26 practices) 9 (6 practices) 1
Good 78 (268 practices) 78 (165 practices) 71 (50 practices) 84
Requires improvement 7 (23 practices) 7 (14 practices) 14 (10 practices) 13
Inadequate 1 (4 practices) 2 (4 practices) 3 (2 practices) 2
Open in a new tab

CQC, Care Quality Commission; EMIS, Egton Medical Information Supplies; PINCER, pharmacist-led information technology intervention; SD, standard deviation; TPP, The Phoenix Partnership.

aList size accessed from Public Health England National General Practice Profiles for 2016. http://fingertips.phe.org.uk/profile/general-practice/data

bEstimated share of primary care (general practice) clinical computer system market 2017. https://www.emisgroupplc.com/media/1420/emis-group-plc-final-year-results-2017-presentation.pdf

cIndex of Multiple Deprivation for 2015. Accessed from Public Health England National General Practice Profiles. http://fingertips.phe.org.uk/profile/general-practice/data

dEthnicity: Taken from 2011 Census. Accessed from Public Health England National General Practice Profiles. http://fingertips.phe.org.uk/profile/general-practice/data

fQuality and Outcomes Framework score 2016/2017: Accessed from Public Health England National General Practice Profiles. http://fingertips.phe.org.uk/profile/general-practice/data

gCare Quality Commission Safety Rating May 2017; Taken from Care Quality Commission: http://www.cqc.org.uk/sites/default/files/20170921_state_of_care_in_general_practice2014-17.pdf

The rollout of the intervention was phased for each CCG (S1 Appendix). Characteristics of the participating general practices are shown in Table 2. In most respects, these were similar to English general practices as a whole, but study practices had higher mean list size and GP whole time equivalents, were more likely to use The Phoenix Partnership (TPP) SystmOne clinical system, and more likely to be rated “outstanding” by the Care Quality Commission (CQC).

At baseline (the quarter prior to the quarter when the intervention was implemented in each practice), a total of 2.97 million patient records were searched, and 22,105 instances of potentially hazardous prescribing were identified in 21,283 patients. Table 3 shows the number of patients in the numerator and denominator for each indicator at baseline, along with the rate of hazardous prescribing per 1,000 patients at risk.

Table 3. Number of “at-risk” patients identified at baseline (n = 370 practices).

Prescribing safety indicator At risk of hazardous prescribing (denominator) Exposed to hazardous prescribing (numerator) Rate of hazardous prescribing per 1,000 patients at risk
Outcome: GI bleed
A Patients aged ≥65 years prescribed an oral NSAID without coprescription of an ulcer-healing drug 369,090 8,281 22
B Patients aged ≥18 years with a history of peptic ulceration prescribed an oral NSAID without coprescription of an ulcer-healing drug 20,668 471 23
C Patients aged ≥18 years with a history of peptic ulceration prescribed an antiplatelet drug without coprescription of an ulcer-healing drug 20,668 1,638 79
D Patients aged ≥18 years prescribed warfarin or DOAC in combination with an oral NSAID 51,551 635 12
E Patients aged ≥18 years prescribed warfarin or DOAC and an antiplatelet drug in combination without coprescription of an ulcer healing drug 32,829 1,222 37
F Patients aged ≥18 years prescribed aspirin in combination with another antiplatelet drug without coprescription of an ulcer-healing drug 65,288 3,009 46
Outcome: Exacerbation of asthma
G Patients aged ≥18 years with a Read code for asthma prescribed a nonselective beta-blocker 284,523 2,692 9
H Patients aged ≥18 years with a Read code for asthma prescribed a long-acting beta-2 agonist inhaler but not also prescribed an inhaled corticosteroid 5,217 868 166
Outcome: Heart failure
I Patients aged ≥18 years who have a diagnosis of heart failure prescribed an oral NSAID 23,842 467 20
Outcome: Stroke
J Patients aged ≥65 years with a Read code for dementia but no Read code for psychosis prescribed antipsychotic drugs for >6 weeks 21,931 2,094 95
Outcome: Kidney Injury
K Patients aged ≥18 years with an eGFR <45 prescribed an oral NSAID 35,419 728 21
Composite of all indicators
(including all exposures to hazardous prescribing identified by indicators A-K)		 931,026 22,105 24
Composite of GI bleed indicators
(including all exposures to hazardous prescribing identified by indicators A-F)		 560,094 15,256 27
Open in a new tab

DOAC, direct oral anticoagulant; GI, gastrointestinal; NSAID, nonsteroidal anti-inflammatory drug; PPI, proton pump inhibitor.

A decrease in hazardous prescribing rates over time was observed across the participating practices for most indicators (Fig 1) for the overall composite outcome measure and the GI composite outcome measure.

Fig 1. Hazardous prescribing rate by calendar quarter for the composite prescribing indicators using data from all practices combined.

Fig 1

Open in a new tab

Table 4 shows the number of patients at risk preimplementation and the rate of potentially hazardous prescribing preintervention and at 6 and 12 months postimplementation. Odds ratios, adjusted for GP practice and calendar time, are given for differences in rates of hazardous prescribing at 6 and 12 months compared to preintervention. There was a reduction in hazardous prescribing for the composite outcomes and most of the individual outcomes at both 6 and 12 months postintervention. For the primary composite outcome, the PINCER intervention was associated with a decrease in the rate of hazardous prescribing of 16.7% (aOR 0.83, 95% CI 0.80 to 0.86) at 6 months and 15.3% (aOR 0.85, 95% CI 0.80 to 0.90) at 12 months postintervention. The unadjusted rate of hazardous prescribing reduced from 26.4 (22,503 patients in the numerator/853,631 patients in the denominator) to 20.1% (11,901 patients in the numerator/591,364 patients in the denominator) at 6 months and 19.1% (3,868 patients in the numerator/201,992 patients in the denominator).

Table 4. Comparing the rate of hazardous prescribing at 6 months and 12 months postintervention to preintervention, adjusted for general practice and calendar time.

Preimplementation 6 Months 12 Months
Outcome Numeratora/Denominatora
(Raw Rateb)		 Fitted rateb,c Numeratora/Denominatora
(Raw Rateb)		 Fitted rateb,c OR (95% CI)d Numeratora/Denominatora
(Raw Rateb)		 Fitted rateb,c OR (95% CI)d
Overall composite 22,503/853,631
(26.4)		 26.7
(26.6:26.7)		 11,901/591,364
(20.1)		 22.3
(22.3:22.4)		 0.83
(0.80:0.86)		 3,868/201,992
(19.1)		 22.7
(22.6:22.8)		 0.85
(0.8: 0.90)
GI composite 16,185/515,879
(31.4)		 32.4
(32.4:32.4)		 7,607/358,349
(21.2)		 24.8
(24.8:24.9)		 0.76
(0.73:0.80)		 2,369/121,534
(19.5)		 24.8
(24.7:24.9)		 0.76
(0.7: 0.82)
Indicator A 8,977/333,980
(26.9)		 26.3
(26.3:26.4)		 3,921/233,533
(16.8)		 19.3
(19.3:19.4)		 0.73
(0.69:0.77)		 1,015/78,187
(13.0)		 18.1
(17.9:18.2)		 0.68
(0.60:0.77)
Indicator B 573/22,484
(25.5)		 23.7
(23.6:23.7)		 274/14,700
(18.6		 19.0
(18.9:19.1)		 0.80
(0.69:0.92)		 76/5,158
(14.7)		 16.9
(16.7:17.2)		 0.71
(0.57:0.89)
Indicator C 1,947/22,484
(86.6)		 78.9
(78.8:79.0)		 954/14,700
(64.9)		 63.4
(63.3:63.5)		 0.79
(0.73:0.85)		 307/5,158
(59.5)		 59.4
(59.2:59.5)		 0.74
(0.66:0.83)
Indicator D 44,569/44,569
(13.7)		 12.0
(11.9:12.1)		 470/36,014
(13.1)		 11.9
(11.8:12.0)		 0.99
(0.88:1.11)		 180/12,700
(14.2)		 11.2
(11.0:11.4)		 0.93
(0.80:1.09)
Indicator E 11,114/28,507
(39.1)		 38.4
(38.4:38.5)		 578/22,289
(25.9)		 27.0
(26.9:27.1)		 0.70
(0.62:0.77)		 251/7,729
(32.5)		 30.4
(30.2:30.5)		 0.78
(0.66:0.92)
Indicator F 2,962/63,855
(46.4)		 45.4
(45.3:45.4)		 1,410/37,113
(38.0)		 36.7
(36.7:36.8)		 0.80
(0.75:0.86)		 540/12,602
(42.9)		 39.1
(39.0:39.3)		 0.86
(0.76:0.97)
Indicator G 2,311/260,050
(8.9)		 7.6
(7.5:7.6)		 1,750/178,020
(9.8)		 7.3
(7.2:7.4)		 0.96
(0.91:1.02)		 710/62,689
(11.3)		 7.7
(7.6:7.8)		 1.02
(0.94:1.11)
Indicator H 884/5,571
(158.7)		 184.6
(184.5:184.7)		 504/2,961
(170.2)		 194.0
(193.8:194.1)		 1.06
(0.95:1.19)		 120/688
(174.4)		 165.9
(165.7:166.2)		 0.88
(0.70:1.10)
Indicator I 498/22,019
(22.6)		 20.5
(20.4:20.5)		 285/16,120
(17.7)		 18.2
(18.1:18.4)		 0.89
(0.80:0.99)		 91/5,238
(17.4)		 16.8
(16.6:17.1)		 0.82
(0.66:1.02)
Indicator J 1,903/19,837
(95.9)		 81.6
(81.5:81.7)		 1,358/14,476
(93.8)		 86.6
(86.5:86.7)		 1.07
(0.99:1.15)		 461/4,843
(95.2)		 93.8
(93.7:93.9)		 1.16
(1.04:1.30)
Indicator K 722/30,275
(23.9)		 20.6
(20.5:20.7)		 397/21,438
(18.5)		 19.3
(19.1:19.4)		 0.93 (0.83:1.05) 117/7,000
(16.7)		 17.1
(16.9:17.3)		 0.83
(0.68:1.00)
Open in a new tab

aNumber at risk and preintervention rates estimated as mean over the 4 quarters prior to intervention at each site.

bRate per 1,000 patients at risk.

cFitted rates are adjusted for calendar time and general practice.

dOdds ratio relative to preimplementation.

For the GI composite indicator, there was a decrease of 23.9% at both 6 months (aOR 0.76, 95% CI 0.73 to 0.80) and 12 months (aOR 0.76, 95% CI 0.70 to 0.82) postintervention. The unadjusted rate of hazardous prescribing reduced from 31.4 (16,185 patients in the numerator/515,879 patients in the denominator) to 21.2% (7,607 patients in the numerator/358,349 patients in the denominator) at 6 months and 19.5% (2,369 patients in the numerator/121,534 patients in the denominator).

Fig 2 shows the ORs for the overall composite outcome and the GI composite outcome over time, with a drop over the first two quarters after implementation, which was sustained over the whole study period. The plots for the individual indicators are included in S7 Appendix. The 95% CIs are much larger postintervention than preintervention. This is due to the number of preintervention quarters that were combined to contribute to the preintervention data point, where only single quarters contributed to each of the postimplementation data points.

Fig 2. Odds ratio and 95% confidence intervals for the rate of hazardous prescribing at each quarter interval compared to preintervention for the composite outcomes.

Fig 2

Open in a new tab

No reduction in the rate of hazardous prescribing was found for indicators associated with asthma (G and H) or stroke (J).

Discussion

The PINCER intervention, when rolled out and evaluated at scale to 343 general practices, was associated with reductions of hazardous prescribing for the overall composite indicator of 17% and 15% at 6 and 12 months postintervention, respectively, and 24% for the GI bleed composite indicator at both 6 and 12 months postintervention. The observed reductions in hazardous prescribing for the composite indicators were sustained up until 12 months postintervention.

This is one of the largest studies that has evaluated the effectiveness of widespread implementation of a medication safety intervention in primary care. The greatest reductions were apparent for indicators targeting hazardous prescribing of NSAIDs or antiplatelets in patients with risk factors for GI bleeding who were not also prescribed an ulcer-healing drug (Indicators A, B, C, E, and F). The only other indicators where a statistically significant reduction in hazardous prescribing was observed to be associated with the intervention was for the prescription of an NSAID to a patient with heart failure (Indicator I) at 6 months (but not 12 months) postintervention and for the prescription of an NSAID to a patient with chronic kidney disease (Indicator K) at 12 months postintervention.

In contrast, no reduction in the rate of hazardous prescribing at 6 months postintervention was found for indicators D (NSAID and anticoagulant prescribed concurrently), G and H (asthma patients prescribed nonselective beta-blockers and asthma patients prescribed a long-acting beta-2 agonist inhaler but not also prescribed an inhaled corticosteroid), and J (dementia patients prescribed antipsychotics). We can only speculate on the reasons for this, with the most likely one being that pharmacists found it more difficult to make these changes.

In our original PINCER trial [5], a more limited set of prescribing safety indicators was used. For the composite of these prescribing indicators there was a reduction in hazardous prescribing of 29% and 22% at 6 and 12 months postintervention, respectively. There are two prescribing indicators that are comparable between the two studies. In the PINCER trial, there was a reduction in the prescribing of NSAIDs to patients with a history of peptic ulcer and no coprescription of a proton pump inhibitor (PPI) (indicator B in the current study) of 42% at 6 months postintervention and a nonstatistically significant 9% at 12 months. Also, in the PINCER trial, there was a reduction in the prescription of beta-blockers in patients with asthma (similar to Indicator G in the current study) of 27% and 22% at 6 and 12 months postintervention, respectively. Comparing the two studies, there was a greater reduction in hazardous prescribing of NSAIDs in patients with a history of peptic ulcer in the PINCER trial at 6 months postintervention, but, unlike the current study, this was not sustained at 12 months postintervention. In the PINCER trial, there was a sustained reduction in the prescription of beta-blockers in patients with asthma postintervention, but this was not seen in the current study.

The DQIP trial used a set of prescribing indicators with some similarities to those used in the current study [8]. In this study involving 33 general practices, there was a 37% reduction in hazardous prescribing and reductions were sustained over the 12-month intervention period. In common with our current study, the greatest reductions were observed for six of the nine indicators where an ulcer-healing drug had not been prescribed when there was a medication-related increased risk of GI bleeding. In addition, the DQIP intervention was effective at reducing the prescription of NSAIDs to patients with chronic kidney disease.

The Safety Medication Dashboard (SMASH) is a pharmacist-led electronic audit and feedback intervention that identifies patients “at risk” of serious adverse events and is based on the PINCER intervention to reduce hazardous prescribing in general practices [9]. In an evaluation undertaken in 43 general practices in Salford, UK using SMASH [10], overall reductions in hazardous prescribing were 28% at 24 weeks postintervention and 41% at 12 months postintervention. For individual indicators, the findings were remarkably consistent with those found in the current study (although reductions were generally greater with SMASH). One exception was that SMASH was associated with reductions in prescribing of NSAIDs to patients with chronic kidney disease.

Overall, these studies confirm that these complex but pragmatic interventions can be effective at reducing hazardous prescribing, but the effects are greatest where a straightforward action can be taken (such as prescribing an ulcer-healing drug to a patient at risk of medication-related GI bleed) compared with stopping (or changing) a medication that may be perceived to be providing benefit to a patient, despite the risk (such as prescribing a beta-blocker in a patient with asthma). Nevertheless, in the current study, SMASH [9] and DQIP [8] studies, there was (at least in the short term) a reduction in prescribing NSAIDs to patients with heart failure, and in the SMASH and DQIP studies, there was a reduction in prescribing NSAIDs to patients with chronic kidney disease. Apart from the original PINCER trial, neither the current study nor the findings from the SMASH evaluation showed that these interventions were effective at reducing the prescription of beta-blockers to patients with asthma. Anecdotal evidence suggests that this indicator is often challenging due to the indications for beta-blockers for conditions such as anxiety, and, therefore, many clinicians may opt to continue to prescribe the beta-blocker despite the risks.

The level of uptake of the intervention was very high with 94% (370/393) of eligible practices in the East Midlands of England participating in the implementation. The 370 study practices that implemented PINCER had similar characteristics to English practices as a whole, although there were some differences whereby study practices had a slightly higher list size and a higher proportion with an “outstanding” rating for safety by the CQC. Although the practices included in the study are limited to one region of England, given similar findings from the SMASH study [9] in Greater Manchester, it is likely that the intervention would be effective in other English practices.

Originally, we had planned to roll the intervention out using a randomised stepped-wedge design. However, it became clear that CCGs needed to include the rollout of PINCER in their work plans (which are set prior to each NHS financial year), and in some cases, the Medicines Optimisation Teams had to submit a business case to their CCG to enable them to repurpose their pharmacist resource. This made the process of randomisation impossible. We did, however, take account of secular trends by including the calendar reference date in the models. This was clearly important given the trend towards safer prescribing over time (see Fig 1). The model used was an extension of the conventional segmented regression, which allows explicitly for the intervention effect to be nonlinear. We expected a priori when we specified the analysis that the treatment effect would reduce over time and wanted a model that would capture the time trends here. We were fortunate in having a very large sample size to allow a rich representation (via a simple qualitative postintervention time parameter) of these trends. Nevertheless, it should be recognised that this was an observational study, and the findings may have been influenced by unknown confounding factors or behavioural changes unrelated to the PINCER intervention.

We were not able to collect 6- and 12-month follow-up data on all practices, due to incomplete transfer of the retrospective data in some instances, and this may have biased the findings. However, a comparison between the rates for the raw data preimplementation for the full data set and separately for those practices contributing and not contributing data at 6 and 12 months postintervention (S6 Appendix) showed similar rates.

We constructed our indicators so that the denominators contained “at-risk” groups of patients but acknowledge that, in some cases, taking an action such as adding a PPI removes a patient from both the numerator and denominator for subsequent data collections. Given that the denominator is always larger than the numerator, this still means that reductions in both leads to reduction in the proportion of patients exposed to hazardous prescribing. Nevertheless, a larger effect might have been demonstrated if we had put the term “without coprescription of an ulcer-healing drug” in the numerator.

The findings may underrepresent the effectiveness of the intervention because we defined the time that practices downloaded the computer searches to identify patients at risk of prescribing as the time that each practice started the intervention. We are, however, aware that some general practices downloaded the computer searches but, due to competing priorities, did not start the intervention for several months, which would cause a delay in the effect of the intervention.

In the interrupted time series model, we have assumed that there is no residual autocorrelation. This assumption is implicit and unfortunately hard to test within the modelling framework/software we have used. A priori we expected that autocorrelation within practices would not be a substantive issue as the events of concern are intrinsically independent. We did anticipate strong correlations within practices and accounted for these in the model specification and also by utilising robust SE estimates.

Since 2015, the PINCER intervention [5] has been incorporated into national guidelines to support medicines optimisation by the National Institute for Health and Care Excellence (NICE) [11], meaning that general practices throughout the country are encouraged to use the intervention. Since this study was undertaken, several further policy developments have occurred that are helpful when discussing the place of PINCER with general practices and CCGs in England.

In 2017, PRIMIS was funded by the Health Foundation to work with Spring Impact to implement their systematic five-stage process to design a replication model for the scale and spread of PINCER using a social franchising approach [12]. As a result of this work, in 2018, PINCER was selected by the Academic Health Science Network (AHSN) for national adoption whereby PRIMIS acted as “franchisor” and the 15 AHSNs in England acted as “franchisees.” Since 2018, PRIMIS has worked with all 15 AHSNs in England to roll out PINCER to over 2,800 (41%) of GP practices in England and have trained more than 2,250 healthcare professionals (including 1,713 primary care pharmacists) to deliver the PINCER intervention through a combination of eLearning tools, online resources, live webinars, and face-to-face action learning set sessions [13].

In terms of longer-term sustainability of the PINCER intervention, in 2019, NHS England set out in its long-term plan a commitment for pharmacists to take on an expanded role at the heart of local Primary Care Networks (PCNs) across the country. The new General Medical Services (GMS) contract set out the ambition for every PCN to have access to a pharmacist, thus ensuring a commitment to establishing and expanding the pharmacy workforce capable of carrying out the PINCER intervention in collaboration with CCG teams. Also, in 2019, NHS Improvement published the NHS Patient Safety Strategy, highlighting PINCER as a successful example of delivering improvement [14]. Pharmacists in general practice in England undertake a range of activities aimed at medicines optimisation, and, while delivery of the PINCER intervention is not necessarily an explicit part of the job description, there is ongoing policy drive towards the use of this approach. For example, in 2022–23, all general practices in England have been incentivised to use the PINCER approach with pharmacists undertaking structured medication reviews for patients identified from PINCER indicators [15].

Important policy questions remain in relation to the effectiveness of the intervention at translating reductions in hazardous prescribing to reducing serious harm to patients, and the cost-effectiveness of the intervention. The DQIP [8] intervention was associated with a reduction in hospital admissions for GI bleeding and heart failure, and our detailed economic evaluation of the original PINCER intervention suggested it produced marginal health gain at slight reduced overall cost [16]. These findings are reassuring in terms of the likely benefits of rolling out PINCER at scale, but in order to give a definitive answer to these questions, we are currently evaluating the effectiveness of the national rollout of the PINCER intervention in reducing serious patient harm, particularly from GI bleeding as part of our NIHR Programme Grant for Applied Research “Avoiding patient harm through the application of prescribing safety indicators in English general practices (PRoTeCT)”[17].

In conclusion, the PINCER intervention, when rolled out at scale in routine clinical practice, was associated with a reduction in hazardous prescribing by 17% and 15% at 6 and 12 months postintervention. The greatest reductions in hazardous prescribing were for indicators associated with risk of GI bleeding, particularly where prescription of an ulcer-healing drug would improve patient safety.

Supporting information

S1 Appendix. Rollout of the PINCER intervention across general practices in 11 CCGs.

(PDF)

Click here for additional data file. (253KB, pdf)
S2 Appendix. The Explicit Statistical Model.

(PDF)

Click here for additional data file. (755.9KB, pdf)
S3 Appendix. STROBE Checklist.

(PDF)

Click here for additional data file. (393.6KB, pdf)
S4 Appendix. The number of practices included at each quarter, by time since the intervention.

(PDF)

Click here for additional data file. (217.2KB, pdf)
S5 Appendix. Comparing the rate of hazardous prescribing at 6 months and 12 months postintervention to preintervention, adjusted for GP practice and calendar time (excluding practices with less than 6 months of data postintervention).

(PDF)

Click here for additional data file. (236.5KB, pdf)
S6 Appendix. Comparing the baseline (preimplementation year) rates of hazardous prescribing for practice with at least and without at least 6 months and 12 months postintervention data: Raw data and data adjusted for GP practice and calendar time.

(PDF)

Click here for additional data file. (241.9KB, pdf)
S7 Appendix. Plots for each Individual Indicator.

(PDF)

Click here for additional data file. (455.1KB, pdf)

Acknowledgments

We thank:

  • Members of the Project Steering and Evaluation Advisory Committees who provided extremely valuable advice and support throughout the course of the project.

  • The Clinical Commissioning Groups (CCG), medicines management teams and general practices that participated.

  • The pharmacists and pharmacy technicians who delivered the PINCER intervention.

  • CCG research and development leads, pharmacy leads, prescribing advisors and other key individuals who helped to facilitate the project including Rachel Illingworth, Harriet Murch, Mindy Bassi and David Gerrett.

  • Tony Panayiotidis and Lauren Fensome, PRIMIS, University of Nottingham for delivering training on CHART software.

  • PRIMIS colleagues for collection of retrospective data and support throughout the project

  • Gill Gookey for delivering training on the PINCER intervention.

  • The following members of the PINCER implementation team: Mindy Bassi, Dr Cheryl Crocker, Despina Laparidou, Tony Panayiotidis.

  • Lincolnshire Community Health Services NHS Trust for hosting the grant and for oversite of the project finances.

  • East Midlands Patient and Public Involvement senate for their valuable advice and ongoing help and support.

  • Antony Chuter, Chris Rye and Glen Swanwick for providing Patient and Public Involvement and Engagement.

  • Christina Sheehan for oversite of the evaluation finances.

  • Sarah Armstrong and Rajnikant Mehta, NIHR RDS for the East Midlands, University of Nottingham, for advice on statistical analysis.

Abbreviations

AHSN

Academic Health Science Network

aOR

adjusted odds ratio

CCG

Clinical Commissioning Group

CI

confidence interval

CQC

Care Quality Commission

DOAC

direct oral anticoagulant

eGFR

estimated glomerular filtration rate

GMS

General Medical Services

GI

gastrointestinal

GP

general practitioner

MIQUEST

Morbidity Information and Query Export Syntax

NHS

National Health Service

NICE

National Institute for Health and Care Excellence

NSAID

nonsteroidal anti-inflammatory drug

OR

odds ratio

PCN

Primary Care Network

PINCER

pharmacist-led information technology intervention

PPI

proton pump inhibitor

SD

standard deviation

SMASH

Safety Medication Dashboard

TPP

The Phoenix Partnership

UK

United Kingdom

WHO

World Health Organization

Data Availability

We are unable to share the practice level data as the data sharing agreement, provided to the practices involved, restricted the use of the data to PRIMIS, the Clinical Commissioning Group and members of the study team. It stated that data would not be shared with any person or organisation that does not appear in this list. For enquiries about the data, please contact the University of Nottingham via the following email address: 'sponsor@nottingham.ac.uk'.

Funding Statement

This study was funded by The Health Foundation (www.health.org.uk, Award Number: 7419) AJA, DMA, MJB, RAE, KK, SR, AS and ANS, and the East Midlands Academic Health Science Network (www.emahsn.org.uk, Award Number: 39701) AJA, SR. DMA was supported by the National Institute for Health and Care Research (NIHR) Greater Manchester Patient Safety Translational Research Centre (award number: PSTRC-2016-003, http://www.patientsafety.manchester.ac.uk). KK is supported by the National Institute for Health and Care Research (NIHR) Applied Research Collaboration East Midlands (https://arc-em.nihr.ac.uk) and the NIHR Leicester Biomedical Research Centre (https://www.leicesterbrc.nihr.ac.uk). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

References

Decision Letter 0

Beryne Odeny

21 Mar 2022

Dear Dr Taylor,

Thank you for submitting your manuscript entitled "Scaling-up a pharmacist-led information technology intervention (PINCER) to reduce hazardous prescribing in general practices: multiple interrupted time series study" for consideration by PLOS Medicine.

Your manuscript has now been evaluated by the PLOS Medicine editorial staff and I am writing to let you know that we would like to send your submission out for external peer review.

However, before we can send your manuscript to reviewers, we need you to complete your submission by providing the metadata that is required for full assessment. To this end, please login to Editorial Manager where you will find the paper in the 'Submissions Needing Revisions' folder on your homepage. Please click 'Revise Submission' from the Action Links and complete all additional questions in the submission questionnaire.

Please re-submit your manuscript within two working days, i.e. by Mar 23 2022 11:59PM.

Login to Editorial Manager here: https://www.editorialmanager.com/pmedicine

Once your full submission is complete, your paper will undergo a series of checks in preparation for peer review. Once your manuscript has passed all checks it will be sent out for review.

Feel free to email us at plosmedicine@plos.org if you have any queries relating to your submission.

Kind regards,

Beryne Odeny

PLOS Medicine

Decision Letter 1

Beryne Odeny

1 Jun 2022

Dear Dr. Taylor,

Thank you very much for submitting your manuscript "Scaling-up a pharmacist-led information technology intervention (PINCER) to reduce hazardous prescribing in general practices: multiple interrupted time series study" (PMEDICINE-D-22-00818R1) for consideration at PLOS Medicine.

Your paper was evaluated by a senior editor and discussed among all the editors here. It was also sent to independent reviewers, including a statistical reviewer. The reviews are appended at the bottom of this email and any accompanying reviewer attachments can be seen via the link below:

[LINK]

In light of these reviews, I am afraid that we will not be able to accept the manuscript for publication in the journal in its current form, but we would like to consider a revised version that addresses the reviewers' and editors' comments. Obviously we cannot make any decision about publication until we have seen the revised manuscript and your response, and we plan to seek re-review by one or more of the reviewers.

In revising the manuscript for further consideration, your revisions should address the specific points made by each reviewer and the editors. Please also check the guidelines for revised papers at http://journals.plos.org/plosmedicine/s/revising-your-manuscript for any that apply to your paper. In your rebuttal letter you should indicate your response to the reviewers' and editors' comments, the changes you have made in the manuscript, and include either an excerpt of the revised text or the location (eg: page and line number) where each change can be found. Please submit a clean version of the paper as the main article file; a version with changes marked should be uploaded as a marked up manuscript.

In addition, we request that you upload any figures associated with your paper as individual TIF or EPS files with 300dpi resolution at resubmission; please read our figure guidelines for more information on our requirements: http://journals.plos.org/plosmedicine/s/figures. While revising your submission, please upload your figure files to the PACE digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at PLOSMedicine@plos.org.

We expect to receive your revised manuscript by Jun 22 2022 11:59PM. Please email us (plosmedicine@plos.org) if you have any questions or concerns.

***Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.***

We ask every co-author listed on the manuscript to fill in a contributing author statement, making sure to declare all competing interests. If any of the co-authors have not filled in the statement, we will remind them to do so when the paper is revised. If all statements are not completed in a timely fashion this could hold up the re-review process. If new competing interests are declared later in the revision process, this may also hold up the submission. Should there be a problem getting one of your co-authors to fill in a statement we will be in contact. YOU MUST NOT ADD OR REMOVE AUTHORS UNLESS YOU HAVE ALERTED THE EDITOR HANDLING THE MANUSCRIPT TO THE CHANGE AND THEY SPECIFICALLY HAVE AGREED TO IT. You can see our competing interests policy here: http://journals.plos.org/plosmedicine/s/competing-interests.

Please use the following link to submit the revised manuscript:

https://www.editorialmanager.com/pmedicine/

Your article can be found in the "Submissions Needing Revision" folder.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Please ensure that the paper adheres to the PLOS Data Availability Policy (see http://journals.plos.org/plosmedicine/s/data-availability), which requires that all data underlying the study's findings be provided in a repository or as Supporting Information. For data residing with a third party, authors are required to provide instructions with contact information for obtaining the data. PLOS journals do not allow statements supported by "data not shown" or "unpublished results." For such statements, authors must provide supporting data or cite public sources that include it.

We look forward to receiving your revised manuscript.

Sincerely,

Beryne Odeny,

PLOS Medicine

plosmedicine.org

-----------------------------------------------------------

Requests from the editors:

1) The Data Availability Statement (DAS) requires revision. For each data source used in your study:

a) If the data are freely or publicly available, note this and state the location of the data: within the paper, in Supporting Information files, or in a public repository (include the DOI or accession number).

b) If the data are owned by a third party but freely available upon request, please note this and state the owner of the data set and contact information for data requests (web or email address). Note that a study author cannot be the contact person for the data.

c) If the data are not freely available, please describe briefly the ethical, legal, or contractual restriction that prevents you from sharing it. Please also include an appropriate contact (web or email address) for inquiries (again, this cannot be a study author).

2) Please remove the ‘Funding,” “competing interests” and “Data availability statement” after the title page. In the event of publication, this information will be published as metadata based on your responses to the submission form.

3) Abstract:

a) Please structure your abstract using the PLOS Medicine headings (Background, Methods and Findings, Conclusions). Please replace the subheading “Objectives” with “Background”

b) Please ensure that all numbers presented in the abstract are present and identical to numbers presented in the main manuscript text.

c) Please quantify the main results (with 95% CIs and p values).

d) Please provide the actual numbers of events for the outcomes (numerator and denominator), not just summary statistics or ORs.

e) In the last sentence of the Abstract Methods and Findings section, please describe the main limitation(s) of the study's methodology.

4) Author summary - At this stage, we ask that you reformat your non-technical Author Summary. The Author Summary should immediately follow the Abstract in your revised manuscript. This text is subject to editorial change and should be distinct from the scientific abstract. The summary should be accessible to a wide audience that includes both scientists and non-scientists. Please see our author guidelines for more information: https://journals.plos.org/plosmedicine/s/revising-your-manuscript#loc-author-summary.

5) Your study is observational and therefore causality cannot be inferred. In the abstract and discussion, please remove language that implies causality, such as “reduced hazardous prescribing” or similar. Refer to associations instead.

6) Please add the following statement, or similar, to the Methods: "This study is reported as per the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guideline (S1 Checklist)."

7) When completing the STROBE checklist, please use section and paragraph numbers, rather than page numbers.

8) In the methods and results please address the following:

a) In the main text, please provide the actual numbers (numerator and denominator) for the outcomes, not just summary statistics or ORs.

b) Please present both numerators and denominators for rates, in the Table 4

c) Please provide both adjusted analyses and unadjusted analyses, where appropriate

d) In the text, please provide p values in addition to 95% CIs when describing numeral results

e) When a p value is given, please specify the statistical test used to determine it.

9) In Figure 2 and S4, please show the Y axis beginning at zero. If this is not possible, please show a break in the axis.

10) In the tables, please define all abbreviations e.g., CCG, Q, GI

11) In the table S3 footnotes, please define what Indicator A, B, C… stand for.

12) Please remove subheadings in the Discussion section, and present and organize the Discussion as follows: a short, clear summary of the article's findings; what the study adds to existing research and where and why the results may differ from previous research; strengths and limitations of the study; implications and next steps for research, clinical practice, and/or public policy; one-paragraph conclusion.

13) References:

a) Please ensure that journal name abbreviations consistently match those found in the National Center for Biotechnology Information (NCBI) databases. https://journals.plos.org/plosmedicine/s/submission-guidelines#loc-references.

b) Please include access dates for all weblinks and ensure that all weblinks are current and accessible, e.g. ref #4

Comments from the reviewers:

Reviewer #1: 1. Summary of the research and your overall impression

This is an observational study of the implementation of a pharmacist-led information technology intervention for medication errors in 343 GP practices between September 2015 and August 2016.

Electronic health care records were analysed to identify pre-specified patients at risk of harm from prescribed treatments. This is was same for the intervention and the analysis.

This intervention had previously been studied as a multicentre, cluster randomised, controlled trial in 72 GP practices in 2006; 32 in the intervention arm. Other similar studies were implemented in 2011-12 (DQIP) and 2017-17 (SMASH).

The authors reported findings are a reduction in the number of patients with hazardous prescribing. This reduction is limited to hazardous prescribing associated with gastro-intestinal bleeding (5 out of 6 individual measures), individual measures for exacerbation of asthma, heart failure stoke kidney injury had no observed reduction.

It was pleasing to see patient and public involvement.

2. Discussion of specific areas for improvement

Issue 1

The authors state in line 143 that data was extracted retrospectively. That up to 16 quarters were extracted (Dec 2013 to Aug 2017). Then in 147 they state that not all practice contributed 16 quarters of data and that for most data was extracted in Oct or Nov 2017.

The number of practices included in the analysis changes with time and this is summarised in table S2. The quarter before the intervention period (quarter -1) there are 341 practices included and for the 7 quarters preceding this there are between 341 and 343 practices. For the period of time at the start of the intervention there were 343 practices, then 310 in the following quarter (quarter +1) and 212 in the next (quarter +2). By quarter +4 there were 70 practices and by quarter +7 there are only 27 practices.

This means the primary statistical analysis of improvement at 6 months (quarters +2) was based on approximately two thirds of practices and the secondary analysis at 12 months (quarters +4) in one fifth of practices.

It could be argued that the paper is redrafted to only include the 212 practices with sufficient data to analyse the primary end point with a sensitivity analysis conducted with the subset of practice (70) with sufficient data to analyse the secondary end point. The other 131 practices (343 less 212) are contributing data to the baseline analysis but not the post-intervention period and estimation of intervention effect size.

As a minimum the number of GP practices contributing data to the primary and secondary end points should be made clearer at the start of the results section (currently it is at the end - line 284). Adding the number of practices contributing data to each quarter to the Figure 2 and S4 plots would help make this clearer.

Issue 2

The analysis method is described in the title as an interrupted time series analysis (ITSA).

The statistical methodology estimates a single pre-implementation base rate (mean of the 4 quarters immediately pre-intervention) with post-intervention values estimated at 6 months and 12 months from temporal trends. However, this appears to mean that pre-intervention trends are not used to estimate the effect size from the counterfactual trend line and that the temporal trends estimate in the post intervention period for the primary outcome are based on two values (quarter +1 and quarter +2). The minimum number recommended for Cochrane systematic reviews of ITSA is three pre and three post intervention. It would appear that according to Cochrane criteria this study would not be included in an Effective Practice and Organisation of Care (EPOC) review https://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/inttime.pdf

A review of the statistical methodology by a statistician would be useful to assess the appropriateness of the statistical method and the use of the description "interrupted time series study".

Other minor issues

1. The table footnotes in Table S3 are missing (a,b,c,d are not defined)

2. CIs in Table S3 12months OR (last column) have additional spaces which do not exist for the other CIs in Table 4 or Table S3

Reviewer #2: This large study explored the effectiveness of the PINCER intervention (a pharmacist-led information technology intervention for medication errors ) to identify patients at risk from hazardous prescribing and to reduce this risk across over 90% of GP practices in one region of England. The original PINCER study found that the outcome in terms of hard reduction, was greater with pharmacist follow up that with computerised feedback alone. This paper answers an important question; are the findings from the PINCER study transferable in real-life setting. However, whilst the authors confirm that indeed this approach can and does reduce the rates of hazardous prescribing there are a few questions which need to be addressed before it can be accepted for publication. Firstly, whilst almost 350 practices were recruited, data from only 212 were analysed at 6 months and only 70 at 12 months. The authors have included a sensitivity analysis to determine whether including data from practices where they have full data at 6 months produces the same outcomes as the partial data analysis and have confirmed that this is the case. However, they have not provided any explanation as to why the drop out rate is so high nor have they considered that those practices with full-data at 6 months may be more engaged and therefore have improved outcomes compared to those who dropped out? What are the differences between the characteristics of the practices that data was obtained at 6 and 12 months compared to those that started the initiative? Was the difference in engagement due to level of pharmacist support between practices? No information is provided in the manuscript regarding this detail. Particularly given that the PINCER study demonstrated better results with pharmacist support this is important. Figure 2 shows that the 95% confidence intervals post intervention (for GI and all outcomes) are much greater than pre-intervention. Why is this? Does this suggest that the intervention becomes less effective over time for some practices? Why? - is the presence or otherwise of a pharmacist contributing to this variability or are those practices who are rated as poor/ needs improvement by the CQC less likely to be following up on the recommendations with time or is there another explanation? The authors also fail to comment on why the prescribing of long acting beta-2 agonists to patients with asthma who were not receiving inhaled steroids was so high nor why it failed to improve at all with the intervention. Please clarify why outcome J is labelled stroke - the prescription of antipsychotics to patients with dementia without a diagnostic code of psychosis.

Reviewer #3: Thank you for submitting this interesting study. I have a few comments and questions for the authors to consider and address.

Line 118: I appreciate these indicators are from the original PINCER trial - but could you provide a reference to how these indicators were informed.

Line 123: Please provide more information regarding the pharmacists involved in the intervention. Were they co-located in the general practice? How many hours per week were they working in the general practice? What experience/training did they have prior to the study? Were they experienced general practice pharmacists? Were they prescribers?

Line 126: How often did these discussions take place, where and when did they take place?

Line 127: Please provide an example of these educational materials.

Line 128: Was the action plan retained in the general practice?

Line 130-131: This section states that pharmacists supported by technicians implemented the action plan. Were these pharmacists/technicians in the community setting? If so, where was the action plan retained? How did the pharmacist actually "support general practice staff" in implementing the action plan?

Line 146: Although the authors have referenced the original paper, it would be good to have an idea of what this processing included.

Line 156-157: "For most practices, retrospective data extraction was carried out during October 2017 and November 2017." Does this mean that practices only carried out data extraction at two time points?

Line 149-198: My main comment about this section is that it is quite hard to follow when data were collected, extracted, action plans were implemented etc. Would suggest revising how this is reported and perhaps include a timeline or the similar.

Line 166: Why were the gastro-intestinal indicators singled out as a group?

Line 287: "We had complete data for all 343 practices." What does this mean? Does it mean that data were collected for all practices at all quarters? This links back to my comment that the timeline of data collection and analysis is difficult to follow.

Line 314: "The level of uptake of the intervention was very high with 94% (370/393) of eligible practices in the East Midlands of England participating in the implementation." In order to be counted as having participated, did the practice only have to submit one set of data at one quarter, i.e. they didn't have to continuously submit data at all quarters?

Line 322: What were these pragmatic reasons?

Line 323: What were these "secular trends"?

Line 389-393: Although HCPs have been trained in PINCER, do the authors have any idea as to how much it is actually used in day-to-day practice?

Line 396-399: Can the authors better explain the relationship between pharmacists being employed in general practice and increased use of PINCER - is use of PINCER an explicit part of their job description?

Reviewer #5: The study uses an interrupted time series approach to evaluate the rate of hazardous prescribing after implementing the PINCER intervention. I found this to be an interesting and generally well presented study. My comments mainly concern the detail given on the statistical model used and its assumptions.

- Although the statistical model is described in words, I think it would be beneficial to also provide the model explicitly, perhaps in the supplementary material. This is not a straightforward model and would be clearer written out in equation form with full details given.

- Is it assumed that there is no time-dependant trend prior to intervention? The definition of base rates as an average of four quarters prior to implementation suggests it may be, although, again, this would be clearer if the model was written out. Is this a reasonable assumption? Could the data be plotted where the x-axis is 'quarter relative to intervention' (possibly after accounting for secular trend)?

- Another assumption of the interrupted time series model is that there is no residual autocorrelation. Is this justifiable? Could this be commented on and/or tested?

- Is it possible that a patient was at risk of or exposed to more than one hazardous prescribing indicator? The descriptions in Table 1 and the data in Table 3 suggests that this may be the case. Has any overlap been accounted for when calculating the numerators and denominators of the composite indices?

Any attachments provided with reviews can be seen via the following link:

[LINK]

Attachment

Submitted filename: Review TD.docx

Click here for additional data file. (17.5KB, docx)

Decision Letter 2

Beryne Odeny

26 Sep 2022

Dear Dr. Taylor,

Thank you very much for re-submitting your manuscript "Scaling-up a pharmacist-led information technology intervention (PINCER) to reduce hazardous prescribing in general practices: multiple interrupted time series study" (PMEDICINE-D-22-00818R2) for review by PLOS Medicine.

I have discussed the paper with my colleagues and it was also seen again by three reviewers. I am pleased to say that provided the remaining editorial and production issues are dealt with we are planning to accept the paper for publication in the journal.

The remaining issues that need to be addressed are listed at the end of this email. Any accompanying reviewer attachments can be seen via the link below. Please take these into account before resubmitting your manuscript:

[LINK]

***Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.***

In revising the manuscript for further consideration here, please ensure you address the specific points made by each reviewer and the editors. In your rebuttal letter you should indicate your response to the reviewers' and editors' comments and the changes you have made in the manuscript. Please submit a clean version of the paper as the main article file. A version with changes marked must also be uploaded as a marked up manuscript file.

Please also check the guidelines for revised papers at http://journals.plos.org/plosmedicine/s/revising-your-manuscript for any that apply to your paper. If you haven't already, we ask that you provide a short, non-technical Author Summary of your research to make findings accessible to a wide audience that includes both scientists and non-scientists. The Author Summary should immediately follow the Abstract in your revised manuscript. This text is subject to editorial change and should be distinct from the scientific abstract.

We expect to receive your revised manuscript within 1 week. Please email us (plosmedicine@plos.org) if you have any questions or concerns.

We ask every co-author listed on the manuscript to fill in a contributing author statement. If any of the co-authors have not filled in the statement, we will remind them to do so when the paper is revised. If all statements are not completed in a timely fashion this could hold up the re-review process. Should there be a problem getting one of your co-authors to fill in a statement we will be in contact. YOU MUST NOT ADD OR REMOVE AUTHORS UNLESS YOU HAVE ALERTED THE EDITOR HANDLING THE MANUSCRIPT TO THE CHANGE AND THEY SPECIFICALLY HAVE AGREED TO IT.

Please ensure that the paper adheres to the PLOS Data Availability Policy (see http://journals.plos.org/plosmedicine/s/data-availability), which requires that all data underlying the study's findings be provided in a repository or as Supporting Information. For data residing with a third party, authors are required to provide instructions with contact information for obtaining the data. PLOS journals do not allow statements supported by "data not shown" or "unpublished results." For such statements, authors must provide supporting data or cite public sources that include it.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Please review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the rebuttal letter that accompanies your revised manuscript.

Please note, when your manuscript is accepted, an uncorrected proof of your manuscript will be published online ahead of the final version, unless you've already opted out via the online submission form. If, for any reason, you do not want an earlier version of your manuscript published online or are unsure if you have already indicated as such, please let the journal staff know immediately at plosmedicine@plos.org.

If you have any questions in the meantime, please contact me or the journal staff on plosmedicine@plos.org.  

We look forward to receiving the revised manuscript by Oct 03 2022 11:59PM.   

Sincerely,

Beryne Odeny,

PLOS Medicine

plosmedicine.org

------------------------------------------------------------

Requests from Editors:

1) Abstract:

a) In the last sentence of the Abstract “Methods and Findings section,” please describe the main limitation(s) of the study's methodology. You indicate that this has been added; please ensure that you have situated in the right place within the abstract. It looks like you added the statement to the Abstract’s Conclusions – please move it to the “Methods and findings” instead.

b) Please interpret the study based on the results presented in the abstract, emphasizing what is new without overstating your conclusions. In addition please state specific implications of your study

2) Author summary:

a) Please put the limitations statement under the subheading “What Did the Researchers Do and Find?”

b) Author summary - Please avoid vague statements such as "these results have major implications for policy/clinical care". Mention only specific implications substantiated by the results.

3) Thank you for providing a completed STROBE checklist. Please remove the Column, Line No, as this will likely change in the event of publication. Instead provide the paragraph number in addition to the section number.

Comments from Reviewers:

Reviewer #4: Dear editors,

I would like to express my compliments to the authors for their careful revision, detailed responses and additional analyses. My concerns regarding the statistical analyses and specification of indicators have been addressed. My concerns regarding loss to follow up have only been partially addressed but the authors have done what is in their power to address them. I therefore recommend publication combined with a request for a minor correction. In response to comment 54, the authors now state that: "Nevertheless, a larger effect might have been demonstrated if had put the term "without co-prescription of an ulcer healing drug in the denominator". The word denominator should be replaced by the word numerator.

Reviewer #5: I thank the authors for their responses to my comments, which have been addressed satisfactorily.

Any attachments provided with reviews can be seen via the following link:

[LINK]

Decision Letter 3

Philippa Dodd

21 Oct 2022

Dear Dr Taylor, 

On behalf of my colleagues and the Academic Editor, Professor Aaron Kesselheim, I am pleased to inform you that we have agreed to publish your manuscript "Scaling-up a pharmacist-led information technology intervention (PINCER) to reduce hazardous prescribing in general practices: multiple interrupted time series study" (PMEDICINE-D-22-00818R3) in PLOS Medicine.

Before your manuscript can be published you will need to address the final revision detailed below:

AUTHOR SUMMARY – What do these findings mean?

Line 95: “This has important implications for policy makers considering the roll out of similar interventions.” Please revise this section to read as follows:

What Do These Findings Mean?

• The findings suggest that the PINCER intervention was effective when rolled out at scale

• These findings support the wider national rollout of PINCER in England and may help to inform policy makers when considering implementation of similar interventions.

Before your manuscript can be formally accepted you will need to complete some formatting changes, which you will receive in a follow up email. Please be aware that it may take several days for you to receive this email; during this time no action is required by you. Once you have received these formatting requests, please note that your manuscript will not be scheduled for publication until you have made the required changes.

In the meantime, please log into Editorial Manager at http://www.editorialmanager.com/pmedicine/, click the "Update My Information" link at the top of the page, and update your user information to ensure an efficient production process. 

PRESS

We frequently collaborate with press offices. If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximise its impact. If the press office is planning to promote your findings, we would be grateful if they could coordinate with medicinepress@plos.org. If you have not yet opted out of the early version process, we ask that you notify us immediately of any press plans so that we may do so on your behalf.

We also ask that you take this opportunity to read our Embargo Policy regarding the discussion, promotion and media coverage of work that is yet to be published by PLOS. As your manuscript is not yet published, it is bound by the conditions of our Embargo Policy. Please be aware that this policy is in place both to ensure that any press coverage of your article is fully substantiated and to provide a direct link between such coverage and the published work. For full details of our Embargo Policy, please visit http://www.plos.org/about/media-inquiries/embargo-policy/.

To enhance the reproducibility of your results, we recommend that you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option to publish peer-reviewed clinical study protocols. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Thank you again for submitting to PLOS Medicine. We look forward to publishing your paper. 

Sincerely, 

Pippa

Philippa Dodd, MBBS MRCP PhD 

Editor 

PLOS Medicine

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Supplementary Materials

    S1 Appendix. Rollout of the PINCER intervention across general practices in 11 CCGs.

    (PDF)

    Click here for additional data file. (253KB, pdf)
    S2 Appendix. The Explicit Statistical Model.

    (PDF)

    Click here for additional data file. (755.9KB, pdf)
    S3 Appendix. STROBE Checklist.

    (PDF)

    Click here for additional data file. (393.6KB, pdf)
    S4 Appendix. The number of practices included at each quarter, by time since the intervention.

    (PDF)

    Click here for additional data file. (217.2KB, pdf)
    S5 Appendix. Comparing the rate of hazardous prescribing at 6 months and 12 months postintervention to preintervention, adjusted for GP practice and calendar time (excluding practices with less than 6 months of data postintervention).

    (PDF)

    Click here for additional data file. (236.5KB, pdf)
    S6 Appendix. Comparing the baseline (preimplementation year) rates of hazardous prescribing for practice with at least and without at least 6 months and 12 months postintervention data: Raw data and data adjusted for GP practice and calendar time.

    (PDF)

    Click here for additional data file. (241.9KB, pdf)
    S7 Appendix. Plots for each Individual Indicator.

    (PDF)

    Click here for additional data file. (455.1KB, pdf)
    Attachment

    Submitted filename: Review TD.docx

    Click here for additional data file. (17.5KB, docx)
    Attachment

    Submitted filename: PINCER HF paper reviewers comments 10 Aug 22.pdf

    Click here for additional data file. (590KB, pdf)
    Attachment

    Submitted filename: Response to reviewers Oct 2022.pdf

    Click here for additional data file. (233.9KB, pdf)

    Data Availability Statement

    We are unable to share the practice level data as the data sharing agreement, provided to the practices involved, restricted the use of the data to PRIMIS, the Clinical Commissioning Group and members of the study team. It stated that data would not be shared with any person or organisation that does not appear in this list. For enquiries about the data, please contact the University of Nottingham via the following email address: 'sponsor@nottingham.ac.uk'.

    Articles from PLOS Medicine are provided here courtesy of PLOS

    RESOURCES