Skip to main content
BMJ Open logoLink to BMJ Open
. 2019 Sep 17;9(9):e031548. doi: 10.1136/bmjopen-2019-031548

Supporting the provision of pharmacy medication reviews to marginalised (medically underserved) groups: a before/after questionnaire study investigating the impact of a patient–professional co-produced digital educational intervention

Asam Latif 1,, Justin Waring 2, Li-chia Chen 3, Kristian Pollock 1, Josie Solomon 4, Nargis Gulzar 5, Sulma Gulzar 6, Emma Anderson 7, Shahida Choudhary 1, Nasa Abbasi 8, Heather J Wharrad 1, Claire Anderson 9
PMCID: PMC6756439  PMID: 31530620

Abstract

Objectives

People who are marginalised (medically underserved) experience significant health disparities and their voices are often ‘seldom heard’. Interventions to improve professional awareness and engagement with these groups are urgently needed. This study uses a co-production approach to develop an online digital educational intervention in order to improve pharmacy staffs’ intention to offer a community pharmacy medication review service to medically underserved groups.

Design

Before/after (3 months) self-completion online questionnaire.

Setting

Community pharmacies in the Nottinghamshire (England) geographical area.

Participants

Community pharmacy staff.

Intervention

Online digital educational intervention.

Primary and secondary outcome measures

The primary outcome measure was ‘behaviour change intention’ using a validated 12-item survey measure. The secondary outcome measure was pharmacist self-reported recruitment of underserved groups to the medication review service.

Results

All pharmacies in the Nottinghamshire area (n=237) were approached in June 2017 and responses were received from 149 staff (from 122 pharmacies). At 3 months (after completing the baseline questionnaire), 96 participants (from 80 pharmacies) completed a follow-up questionnaire, of which two-thirds (n=62) reported completing the e-learning. A before/after comparison analysis found an improving trend in all the five constructs of behaviour change intention (intention, social influence, beliefs about capabilities, moral norms and beliefs about consequences), with a significant increase in mean score of participants’ ‘beliefs about capabilities’ (0.44; 95% CI 0.11 to 0.76, p=0.009). In the short-term, no significant change was detected in the number of patients being offered and the patient completing a medication review.

Conclusions

Although increases in the numbers of patients being offered a medication review was not detected, the intervention has the potential to significantly improve pharmacy professionals’ 'beliefs about capabilities' in the short-term. Wider organisational and policy barriers to engagement with marginasied groups may need to be addressed. Future research should focus on the interplay between digital learning and practice to better identify and understand effective practice change pathways.

Keywords: Pharmacy Medicines Use Review (MUR), Co-production, Digital educational intervention, medically underserved, Behaviour change intention


Strengths and limitations of this study.

  • To our knowledge, this is the first study investigating the impact of a co-produced digital educational intervention for pharmacy professionals to improve the offer of a pharmacy medicine management to marginalised groups.

  • Out of the original sample, 33 pharmacies declined to take part in the study. It is not known whether there were perceived barriers to e-learning or institutional indifference to addressing inequities in health.

  • It is unknown whether participants completed the e-learning as intended or to what extent the digital learning accommodated different learning preferences.

  • It is also unclear whether professional ‘belief about capabilities’ were sustained beyond 3 months.

  • This study relied on self-reports of intention measures, as well as the number of patients engaged, so the actual recruitment and engagement with the underserved communities are not known.

Introduction

The right to physical and mental health is a universal human right.1 However, in practice, vulnerable people, particularly those belonging to marginalised or ‘medically underserved’ groups (eg, people with disabilities; people from black, Asian and minority ethnic backgrounds; the homebound; the homeless; and people with mental illness), can often face significant obstacles to receiving even basic healthcare.2 3 Consequently, they experience higher rates of disease and ill-health and find accessing and navigating health and screening services more challenging when compared with the general population.3–6

Research indicates that people from medically underserved groups encounter poorer patient-professional communication7 and sometimes racism or cultural bias.8 9 They also have been found to hold strong beliefs that they cannot be helped,10 face discrimination or disempowerment because of their circumstance,11 and are disenfranchised from mainstream primary healthcare.12 Inequitable access to routine or preventative care risks higher rates of emergency admissions13 and consequently increases pressure on emergency care services.14 Addressing inequitable access to health services is a key priority for the UK National Health Service (NHS).15

Strategies using co-production to address health inequalities have received growing professional and political attention in light of a changing patient–provider relationship that sees patients more as ‘partners’ rather than passive recipients of healthcare.16 The co-production concept could be well placed to improve services to medically underserved groups as it acknowledges the value of people’s ‘lived experience’ and recognises that people are well placed to advise on how best services could make a positive difference to their lives.17 Alongside tailoring to patient preferences, the process also acknowledges the input and expertise of front-line healthcare staff.18 Involving patients in service improvement has been suggested to help improve the quality of services delivered and helps redress inequities in service accessibility.19 20 In addition, there is growing literature demonstrating the positive outcome of co-produced e-learning materials designed to improve health professionals’ awareness and confidence to deliver care to vulnerable groups.21 22

We hypothesised that a co-produced digital educational intervention might improve pharmacy professionals’ awareness and engagement with medically underserved groups. This study therefore aimed to evaluate whether such an intervention could improve staff intention to engage with marginalised groups. Our study also sought to measure any impact this may have on the actual provision (numbers of MURs) offered to people who are medically underserved.

The English Medicine Use Review (MUR) service

The ‘MUR’ service, funded by the NHS,23 was commissioned in England and Wales in 2005 and is free to patients. The service seeks to establish a picture of the patient’s medicine use, supports knowledge and understanding of therapy and adherence to treatment. Through this, it aims to resolve medicine-related problems and to address medicine-related concerns to reduce avoidable waste.24 The service is organised as a one-to-one consultation and offered to patients annually. Pharmacists therefore have opportunities to hold detailed, private ‘sit down’ discussions with the patient. These differ from the more routine, limited and brief ‘shop floor’ counselling encounters that occur at the time when medicines are handed to patients.

The NHS provides a fee of £28 for each review performed, with the total number each pharmacy contractor can claim to be subject to a cap of 400 annually. Pharmacists must be accredited through additional training, demonstrating nationally set competencies, before they offer the service.25 Pharmacy support staff are often involved with identifying eligible patients for MURs during the routine labelling and assembling of the patients’ medicines and at times offer the service to patients. However, they are not required and often do not receive formal training about the service.26

Early MUR evaluative work identified significant variability in its delivery, raising questions over their value in improving medicine use.27 28 The annual cap of 400 is often cited as a ‘target’ resulting in pharmacists reports of feelings pressurised due to a commercial ‘quantity driven’ culture.29 30 Consequently, pharmacists appear to invite patients who are either more amenable to the service or who are taking simpler medicine regimes. Although this may lead to patients being recruited easily and their MUR performed conveniently, those with more complex needs may be overlooked.29 30 As a result, some groups, particularly those who may already be medically underserved, may not be offered or may not have access to MURs as often as others.28–30

In attempts to improve the perceived value of MURs, service reforms were made in 2011. These changes specified ‘at-risk’ patient groups from which pharmacists should seek to recruit, for example, patients receiving high-risk medicines (eg, anticoagulants and diuretics), those with respiratory disease or if they have recently been discharged from the hospital.31 Despite moves to encourage the offer and uptake from these at-risk groups, there are still no requirements to recruit people who are marginalised and from medically underserved groups. Over three million MURs conducted in 2017–2018 and over 90% of all pharmacies in England now offering the service.32 Given recent moves to decommission the service, there is a strong case to develop interventions to reinforce a value-based healthservice, to focus NHS resources to those who need them the most and to make services more equitable.33

Methods

Patient and public involvement (PPI)

Several patient support groups were consulted, which shaped the study’s research questions, methodology and design. These included Healthwatch Nottingham (part of Healthwatch England, which promotes local voices in order to influence the delivery and design of health services); The Wayfarer Trust (education institute that runs courses and public events for the Southeast Asian Muslim community); and The Himma Institute (a grassroots community charity providing services to vulnerable people).

An advisory panel for the study was formed, comprising two PPI representatives, who themselves identified with being from an underserved community. The panel met regularly to advise and challenge where necessary, on the study protocol, information to participants and on the digital educational intervention. There was also extensive involvement with patients from marginalised groups in the development of the digital educational intervention.

Digital educational intervention

Through their ease of accessibility, cost-effectiveness and learner flexibility and interactivity, digital learning interventions have been identified for delivering health professional training and improving health-related behaviours.34 35 Learning policies and professional educators have identified digital learning interventions as an effective delivery mechanism to ensure healthcare professional training is relevant and flexible enough to take into account different learning preferences.36–38

The digital learning intervention used in this study took the form of a series of three ‘Reusable Learning Objects’ (RLOs). RLOs are small, pedagogically designed, 'bite-sized' chunks of learning that focus on a specific topic.22 The contents included both visual illustration (eg, videos), interactivities and self-assessments. The intervention was hosted on an online platform accessible online (https://www.nottingham.ac.uk/helmopen/rlos/pharmacy/practice/under-served/). The resource consists of three ROLs that focused on (1) Discovering underserved communities, (2) Exploring medicine experiences and developing empathy, and (3) Effectively interacting and engaging patients.

The resource was developed through a process of co-production. This involved informative semistructured interviews with patients (from marginalised groups) and pharmacy staff, alongside mixed patient–professional workshops. Interviews and workshops first sought to reveal patient stories of belonging to a marginalised community. Then experiences of how they manage medicines were elicited, and finally, strategies that pharmacy staff could use to further engage with them were discussed. To improve face validity, the digital learning resource and study questionnaire were piloted on 10 pharmacy students from the University of Nottingham. Only very minor changes to the questionnaire were made to improve readability. Full details of the development of the digital learning have been reported elsewhere.33 39

Study design

This interventional study used a before/after comparison design and was conducted in Nottinghamshire, England, commencing in June 2017. Full details of the proposed protocol have been published elsewhere.33

Study participants were recruited from community pharmacies and to reduce bias, all pharmacies in the Nottinghamshire geographical area were included. Community pharmacy professionals (pharmacists and supporting staff) who were willing to complete online questionnaires and who were deemed to have ‘active’ involvement with the MUR service were invited. ‘Active involvement’ was defined as being involved with the process of identifying, inviting or undertaking MURs on a day-to-day basis.

After permission was sought from head offices or pharmacy owner/managers, an information sheet and invitation letter were sent via post to each pharmacy that provided the MUR service. At baseline, the online questionnaire was open for 4 weeks. Participants who completed the baseline questionnaire were given access to the digital learning intervention. Participants were offered a £25 gift voucher as an inconvenience allowance for them to complete the surveys; consent to take part was implied through their completion. To assess the impact of the intervention alone, no incentives were provided for participants to complete the digital learning.

To encourage participation, in addition to the postal invitation, pharmacies were contacted up to three times in this period by email or phone to invite them to complete the questionnaire. A follow-up questionnaire was then sent to participants 3 months after they were able to access the digital learning materials. To maximise the response rate, three reminders were sent via email or by telephone.

Data collection

The questionnaire survey was conducted using the Bristol Online Survey system (https://www.onlinesurveys.ac.uk/). Characteristics of participants, including demographics, that is, age, gender, educational achievement and details of any previous MUR training received; characteristics of their pharmacies, such as ownership type (ie, large multiples/supermarket), were initially collected. They were then granted access to the digital learning. Kirkpatrick’s Evaluation Model was used as a framework to guide data collection.40 This model is based on four levels. The first two assess learner satisfaction and change in knowledge or attitude with the training resource. The study aimed to go beyond these to detect higher order changes in levels 3 and 4. The third level describes the changes in behaviour brought about by the training, with the final level measuring the application of the participants’ learning to their practice.

The extent of participants’ behaviour change intention (Kirkpatrick’s level 3) was assessed by a Continuing Professional Development (CPD) Reaction Questionnaire. The CPD Reaction Questionnaire is a 12-item theory-based instrument (online supplementary appendix 1) grounded in the theory of planned behaviour and Triandis theory.41 It is a validated and reliable tool with robust metric properties which has been used to assess the impact of CPD activities on health professionals' behaviour intentions in clinical practice. The 12-item measure was used to compute five constructs relating to behaviour change intention, namely, (1) intention, (2) social influence, (3) beliefs about capabilities, (4) moral norm and (5) beliefs about consequences. Data were also collected on the effect of the CPD on the number of MURs performed on medically underserved groups (Kirkpatrick’s fourth level) before and after the intervention.

Supplementary data

bmjopen-2019-031548supp001.pdf (47.5KB, pdf)

Pharmacists were also asked to record at baseline and at 3 months the number of patients they had recruited from medically underserved groups over the previous 4 weeks. To facilitate recording, the following list of common medically underserved groups was provided:

  • People with disability (eg, physical, visual, hearing and learning).

  • People who are who are homebound.

  • People from Black, Asian and Minority Ethnic (BAME) communities.

  • People with mental health illness.

  • People from the lesbian, gay, bisexual, transgender and queer (LGBTQ) communities.

  • People from the traveller community, people who are homeless, or people who are refugees or seeking asylum.

  • Older people with multiple illnesses and many (>12) medicines.

  • People from any other medically underserved community.

Sample size

A sample size of approximately 60 participants (paired before/after data) was needed to provide 80% power to detect a mean change of 0.44 (behaviour change intention) or greater, assuming the SD is 1.2.

Data analysis

Descriptive statistics were used to summarise demographics and characteristics. Continuous data were presented using means and SD if approximately normally distributed and medians and IQR if non-normally distributed. Categorical data were described using frequencies and percentages.

To assess the effect of the digital learning, baseline scores were compared with follow-up data, which were collected after the intervention period. The changes between the baseline and follow-up test scores were analysed using t-tests if they were found to be normally distributed, and the mean difference and 95% CI were presented. It has been suggested that t-tests are appropriate tests and can be used where responses are recorded on a Likert scale.42 Statistical significance was assessed at the 5% (two-sided) level. All statistical analyses were conducted using IBM Statistical Package for Social Sciences (SPSS) V.24.43

Results

Recruitment of pharmacies and participants

All of the 237 community pharmacies within the Nottinghamshire area were approached to take part in the study. Three pharmacies were ineligible as they were not providing the MUR service, and 33 pharmacies declined to take part due to them being ‘too busy’, being ‘short staffed’, not having ‘enough time’ to complete the learning or questionnaire, or were ‘not interested’. One supermarket chain declined to give permission to contact their pharmacies because they had ‘other educational priorities’ for their pharmacies.

Baseline questionnaire responses (n=149 participants) were received from 122 out of the 201 pharmacies who initially expressed an interest in taking part in the study (response rate=61%). Further to this, 100 participants (67%) from 80 pharmacies completed the follow-up questionnaire. After excluding two of the 100 participants who had no baseline survey (follow-up questionnaire link mistakenly forwarded by a work colleague) and a further two that were found to have duplicate records of the follow-up survey, the adjusted follow-up participant response rate was 64% (96/149) (figure 1).

Figure 1.

Figure 1

Flowchart of the study process.

Characteristics of recruited pharmacies and participants

The profile of pharmacies that completed the follow-up questionnaire was found to be similar to those completing the baseline survey, suggesting there was no influence by pharmacy type (table 1). Most pharmacies reported being part of a multiple or classed as an independent, with almost a fifth being co-located within a general practice. Approximately three-quarters of the pharmacies had a single pharmacist on duty with most dispensing between 100 and 400 prescriptions items per day.

Table 1.

Comparing the characterises of pharmacies between the baseline survey with the follow-up survey

Characteristics Baseline (n=122 pharmacies) (%) Follow-up (n=80 pharmacies) (%)
Type of pharmacy
 A multiple/part of a chain 75 (61.5) 48 (60.0)
 An independent 40 (32.8) 26 (32.5)
 A supermarket 6 (4.9) 5 (6.3)
 Other 1 (0.8) 1 (1.3)
Co-location within a general practice
 No 100 (82.0) 64 (80.0)
 Yes 20 (16.4) 15 (18.8)
 Don’t know/unsure 2 (1.6) 1 (1.3)
Pharmacists on duty in the dispensary on a typical day
 A single pharmacist 91 (74.6) 56 (70.0)
 Two pharmacists 27 (22.1) 20 (25.0)
 Three pharmacists 4 (3.3) 4 (5.0)
Average number of prescription items dispensed on a typical day
 Less than 100 3 (2.5) 3 (3.8)
 101–200 26 (21.3) 20 (25.0)
 201–300 22 (18.0) 13 (16.3)
 301–400 28 (23.0) 18 (22.5)
 401–500 13 (10.7) 8 (10.0)
 500 plus 5 (4.1) 2 (2.5)
 I don’t know/unsure
 Not reported
8 (6.6)
17 (13.9)
6 (7.5)
10 (12.5)

Approximately half of the 149 participants were female (47%), with a mean age of 37 years. The majority (81%) of participants were pharmacists, with the remainder being dispensers, medicine counter assistants, non-pharmacist managers and preregistration pharmacists (table 2). Most of the participants reported having been in their role for 1–5 years (34%) and having worked on average 31–40 hours per week (44%). Two-thirds of respondents (65%) reported having attempted part of or all the digital learning. Of those who did not complete the resource, the most common reason provided was that they did not have enough time or that the topic of the learning material was not perceived as necessary (table 2).

Table 2.

Comparison of the characteristics of participants who completed the baseline survey with those completing the follow-up survey

Characteristics Category Baseline (n=149) (%) Follow-up (n=96) (%)
Gender Male 78 (52.3) 53 (55.2)
Female 70 (47.0) 43 (44.8)
Transgender 1 (0.7)
Mean age Mean (SD) (years) 36.7 (11.0) 37.6 (11.4)
Not reported 3 1
Job role Non-pharmacist manager 4 (2.7) 4 (4.2)
Preregistration pharmacist 5 (3.4) 1 (1.0)
Medicines counter assistant/dispenser 20 (13.4) 13 (13.5)
Pharmacists 120 (80.5) 78 (81.3)
Average number of hours worked Up to 20 hours a week 6 (4.0) 4 (4.2)
21–30 hours a week 19 (12.8) 14 (14.6)
31–40 hours a week 65 (43.6) 38 (39.6)
Over 41 hours a week 59 (39.6) 39 (40.6)
Not reported 1 (1.0)
How long the participant has worked at the pharmacy Less than a year 27 (18.1) 9 (9.4)
1–5 years 51 (34.2) 41 (42.7)
6–9 years 30 (20.1) 18 (18.8)
10 or more years 33 (22.1) 26 (27.1)
Don’t know/not sure 1 (1.0)
Not reported 8 (5.4) 1 (1.0)
Approximate number of years in current role Less than a year 23 (15.4) 12 (12.5)
1–5 years 58 (38.9) 35 (36.5)
6–9 years 29 (19.5) 24 (25.0)
10 or more years 39 (26.2) 25 (26.0)
Highest educational achievement Masters pharmacy degree/other Masters’ degree 90 (60.4) 51 (53.1)
Undergraduate degree 25 (16.8) 22 (22.9)
General Certificate of Secondary Education/National Vocational Qualification/Advanced Level (GCSE/NVQ/A level) 19 (12.8) 11 (11.5)
Postgraduate clinical diploma 12 (8.1) 11 (11.5)
PhD 3 (2.0) 1 (1.0)
Attempted part of or all the digital learning Yes 62 (64.6)
No 34 (35.4)
Reason for not attempting the digital learning Did not have enough time to do the learning 18 (52.9)
Did not feel the learning was necessary 6 (17.6)
Do not know/unsure 5 (14.7)
Did not receive or could not access the learning 3 (8.8)
Other (change to a non-customer facing role) 2 (5.9)

Concerning the 120 pharmacists, most respondents were either the pharmacy manager (32%) or the branch manager (29%) or described themselves as an employee pharmacist (27%). Almost half (43%) reported being qualified as a pharmacist for 10 years or less and a quarter between 11 and 20 years. Forty-one per cent of pharmacists had been accredited to perform MURs for more than 10 years with most (72%) not having undertaken any further MUR training since accreditation (table 3).

Table 3.

Subgroup comparison of the characteristics of pharmacist participants who completed the baseline survey with those completing the follow-up survey

Pharmacists Category Baseline (n=120) (%) Follow-up (n=78) (%)
Details of pharmacists Pharmacist and pharmacy manager 38 (31.7) 26 (33.3)
Pharmacist and branch manager 35 (29.2) 16 (20.5)
Employee pharmacist 32 (26.7) 28 (35.9)
Locum pharmacist 14 (11.7) 6 (7.7)
Superintendent pharmacist 1 (0.8) 2 (2.6)
Number of years qualified as a pharmacist 1–10 years 51 (42.5) 33 (42.3)
11–20 years 30 (25.0) 19 (24.4)
21–30 years 17 (14.2) 13 (16.7)
31 to 40 years 11 (9.2) 7 (9.0)
41 to 50 years 2 (1.7) 2 (2.6)
Other 9 (7.5) 4 (5.1)
Year accredited to perform MURs 2005–2007 49 (40.8) 36 (46.2)
2008–2010 17 (14.2) 13 (16.7)
2011–2013 31 (25.8) 14 (17.9)
2014–2016 19 (15.8) 10 (12.8)
2017 2 (1.7) 2 (2.6)
Not reported 2 (1.7) 3 (3.8)
Further MUR training undertaken* No 86 (71.7) 65 (83.3)
Yes 32 (26.7) 11 (14.1)
Not reported 2 (1.7) 2 (2.6)
Type of further MUR training undertaken Centre for Pharmacy Postgraduate Education/The National Pharmacy Association/Company Chemists’ Association 14 (43.8) 2 (18.2)
Self-directed learning (ie, pharmacy articles, diploma and specific training days) 14 (43.8) 4 (36.4)
Employer training 4 (12.5) 2 (18.2)
Not specified 3 (27.3)

*At the baseline, the participants were asked about any MUR training undertaken since their MUR accreditation; at the follow-up survey, participants were asked about any MUR training taken in the past 3 months.

MUR, Medicines Use Review.

Primary outcome: behaviour change intention

When scores were compared between the five constructs of ‘behaviour change intention’ (baseline vs the end of follow-up), it was found that from the total of 96 participants who had completed both questionnaires, the mean difference of scores improved. However, there was no statistically significant results found between the differences of mean scores before and after the intervention period (table 4).

Table 4.

Before/after subgroup comparison of the mean score for the five constructs of ‘behaviour change intention’ between the baseline and the end of follow-up

Construct variable Summary measure Baseline Follow-up Mean difference (95% CI) P value
Participants completed baseline and follow-up survey (n=96)
 Intention (n=96) Median (IQR) 5.50 (2.00) 5.75 (2.00) 0.22 (−0.07 to 0.51) 0.140
 Social influence (n=96) Mean (SD) 4.21 (1.26) 4.28 (1.26) 0.07 (−0.19 to 0.33) 0.578
 Beliefs about capabilities (n=95*) Mean (SD) 5.39 (1.20) 5.61 (1.02) 0.22 (−0.02 to 0.47) 0.067
 Moral norm (n=92*) Median (IQR) 6.50 (1.50) 6.50 (1.38) 0.21 (−0.04 to 0.45) 0.101
 Beliefs about consequences (n=96) Mean (SD) 5.96 (1.09) 6.08 (0.99) 0.13 (−0.10 to 0.35) 0.279
Subgroup participants: completed baseline and follow-up survey and the digital learning (n=62)
 Intention (n=62) Median (IQR) 6 (2.50) 6 (2.00) 0.36 (−0.26 to 0.75) 0.067
 Social influence (n=62) Mean (SD) 4.30 (1.28) 4.56 (1.12) 0.26 (−0.04 to 0.56) 0.085
 Beliefs about capabilities (n=61*) Mean (SD) 5.39 (1.30) 5.83 (1.03) 0.44 (0.11 to 0.76) 0.009†
 Moral norm (n=60*) Median (IQR) 6.5 (1.50) 7.0 (1.00) 0.29 (−0.02 to 0.61) 0.070
 Beliefs about consequences (n=62) Mean (SD) 6.05 (1.12) 6.21 (0.93) 0.16 (−0.13 to 0.45) 0.271

Means are reported where the data are normally distributed.

*Missing construct values.

†Statistically significant difference.

In the subgroup of the 62 participants who reported completing the digital learning, the improvement of mean difference of the five constructs before and after the intervention period was found to be higher than that of the total 96 participants (table 4). There was a statically significant increase in the construct relating to beliefs about capabilities; the mean difference was 0.44 (95% CI 0.11 to 0.76, p=0.009). No statistically significant differences were observed on the other four constructs.

Secondary outcome: self-reported recruitment of medically underserved groups to the MUR service

Pharmacists were asked to report the total number of MURs undertaken within the last 4 weeks and how many of these patients were from medically underserved groups. The before/after comparison analysis did not detect a statistical significance difference between both within the total sample and the subsample who had completed the digital learning (table 5).

Table 5.

A subsample before/after comparison of the pharmacist-reported number of MURs with medically underserved groups

Number of MURs undertaken over the last 4 weeks Summary measure Baseline Follow-up Mean difference (95% CI) P value
Respondents from participants completed baseline and follow-up survey
Total number ( n = 74 ) Mean (SD) 20.8 (15.35) 21.3 (17.18) 0.49 (−2.66 to 3.65) 0.756
With disability (physical, visual, hearing and learning) Median (IQR) 1.0 (2.0) 2.0 (2.0) −0.12 (−0.50 to 0.47) 0.956
Who are homebound Median (IQR) 1.0 (0.0) 1.0 (0.0) 0.14 (−0.10 to 0.37) 0.254
From black, Asian and minority ethnic communities Median (IQR) 2.0 (3.0) 2.5 (2.0) 0.35 (−0.09 to 0.79) 0.114
With mental health illness Median (IQR) 2.0 (2.0) 2.0 (3.0) 0.03 (−0.51 to 0.56) 0.920
From the lesbian, gay, bisexual, transgender and queer communities Median (IQR) 1.0 (0.0) 1.0 (1.0) 0.23 (−0.01 to 0.47) 0.065
The traveller community/homeless/refugees or seeking asylum Median (IQR) 1.0 (0.0) 1.0 (0.0) 0.14 (−0.12 to 0.39) 0.289
Who are older with multiple illnesses and many (>12) medicines Median (IQR) 4.0 (6.0) 4.5 (7.0) 0.39 (−0.31 to 1.09) 0.267
From any other medically underserved community Median (IQR) 1.0 (0.0) 1.0 (1.0) −0.01 (−0.49 to 0.47) 0.955
espondents from participants completed the baseline and follow-up surveys, and the digital learning
Total number ( n = 44 ) Mean (SD) 25.2 (16.50) 25.2 (19.25) −0.02 (−4.97 to 4.93) 0.993
With disability (physical, visual, hearing and learning) Median (IQR) 2.0 (2.0) 2.0 (2.0) 0 (−0.71 to 0.71) 1.000
Who are homebound Median (IQR) 1.0 (0.0) 1.0 (0.0) 0.05 (−0.11 to 0.21) 0.570
From black, Asian and minority ethnic communities Median (IQR) 3.0 (4.0) 3.0 (4.0) 0.43 (-0.23 to 1.09) 0.194
With mental health illness Median (IQR) 2.0 (3.0) 2.0 (4.0) 0.20 (−0.58 to 0.98) 0.600
From the lesbian, gay, bisexual, transgender and queer communities Median (IQR) 1.0 (0.0) 1.0 (1.0) 0.23 (−0.14 to 0.60) 0.222
The traveller community/homeless/refugees or seeking asylum Median (IQR) 1.0 (0.0) 1.0 (0.0) 0.32 (−0.06 to 0.70) 0.099
Who are older with multiple illnesses and many (>12) medicines Median (IQR) 4.5 (7.0) 4.5 (7.0) −0.02 (−1.03 to 0.98) 0.964
From any other medically underserved community Median (IQR) 1.0 (1.0) 1.0 (1.0) −0.09 (−0.77 to 0.59) 0.789

Only responses included where there are paired data.

MUR, Medicines Use Review.

Discussion

Main findings

This study found that a digital educational intervention has the potential to positively influence pharmacy professionals’ ‘beliefs about capabilities’ construct of behaviour change intention. This construct achieved statistical significance, suggesting a possible improvement in professionals’ confidence to interact and engage people from marginalised communities. There was a lack of statistical significance observed for the other four constructs.

The reported high baseline scores for ‘intention’, ‘moral norms’ and ‘beliefs about consequences’ may have been influenced by social desirability or approval bias.44 This may have resulted in a ceiling effect, which meant differences in scores before and after the intervention could not be detected. Given the behaviour being measured was ‘actively engaging with or inviting patients who are underserved for an MUR’, respondents may have been inclined to score themselves highly at baseline because they felt this was the right or ethical thing to do. The final construct ‘social influence’ relates to how the respondents’ peers are perceived. It is well known that pharmacists often feel a notion that there is frequently organisational pressure to undertake MURs resulting in many feeling the service is consequently inefficacious or suboptimal.29 30 This view may explain the lack of change with this construct.

When recruitment from medically underserved groups was examined, statistical differences were not detected at 3 months. There are several practice-based barriers that may have made recruitment from these groups more challenging. First, there is currently no financial incentives to offer the service to such groups that are deemed ‘hard to reach’. For example, there may have been a lack of available resources to provide translation services or a lack of staffing to allow pharmacists to undertake visits to people who are homebound.45 Second, there were low numbers of front-line pharmacy support staff participating in the study, suggesting either lack of interest with the subject or difficulty for the pharmacist-in-charge to adequately engage staff or provide them time to complete the training. With support staff routinely being involved in offering and recruiting patients for MURs, it may have been challenging for pharmacists to enable change without them also completing the digital learning. Third, from a patient perspective, there is evidence that pharmacy services are generally poorly understood and that engagement and understanding of their value are varied or conflicting, with consequences for framing and sense-making.46 These challenges may have been greater given medically underserved groups are not routinely approached or engaged with such services.39 Time may be needed for such marginalised groups to become accustomed to be offered services from the pharmacy.

Other more methodological reasons why there was no detected increase in numbers may have been because of the challenges in identifying patients belonging to these groups (eg, people from the LGBTQ community) or the infrequency in which they present at the pharmacy (eg, people who are homeless). A large-scale trial is needed to ensure sufficient people are recruited to detect differences. It is also not known whether a different outcome may have resulted if respondents could choose to undertake the digital educational intervention at a time that was convenient to them.

Implications

Medication review services has seen significant reconfiguration and investment in light of patient widespread problems taking medicines47 attributed to concerns about dependency, tolerance and side effects.48 49 The community pharmacist workforce is increasingly being deployed, predominantly in developed countries, to promote effective use of medicines.50 51 However, there remains significant challenges to ensure equitable access to these services.

This study has several important implications for pharmacy organisations, educationalists and policymakers. To target more vulnerable groups (who may benefit most from an intervention), there is a need to redress concerns over the lack of perceived effectiveness and cost-effectiveness of MURsand this could represent an opportunity to enhance professional status.52 For educationalists and those offering CPD courses, the co-production approach could ensure greater patient involvement and promote their unique perspective and preferences. Alongside this educational intervention, policymakers may wish to reconsider more funding to encourage scope for future development. In one evaluative study by Latif and colleagues, making additional service costs available to pharmacies to deliver MURs to vulnerable homebound patients potentially led to a reduction in hospital admissions.53

Future research should consider the broader influence that public engagement and health promotion campaigns have at engaging marginalised communities in health services. Strategies are also needed to develop policy reforms to ensure people have more equitable access to care and how system-wide quality improvements can be achieved to proactively increase engagement with communities perceived as being ‘hard to reach’.54

Strengths and limitations

To our knowledge, this is the first study attempting to triangulate findings from self-reported pharmacy professionals’ behaviour change intention and to assess this alongside actual changes in practice. The co-produced intervention sought to include views of both patients and professionals, and the digital format was a feasible means of delivery to pharmacy professionals. There were good baseline responses from pharmacies (61%), which may have been due to the £25 gift voucher incentive, which was made available to participants. However, there were approximately one-third of respondents who did not complete the follow-up questionnaire, and this could have potentially skewed the results. Out of the original sample, 33 pharmacies declined to take part in the study, and out of the 149 participants who completed the baseline questionnaire, just over 40% reported completing the digital learning. The extent to which these numbers reflect barriers to learning or institutional indifference to the issue is not known. Furthermore, this study was conducted in one geographical region. It is unknown to what extent the sample population is representative of other areas or those pharmacies in that are in more remote locations.

There are several other limitations to this study. Regarding study design, this was a before and after study. As Grimshaw et al notes,55 although such designs are relatively easy to conduct and are superior to observational studies, they can be intrinsically weak evaluative designs with non-randomisation making it difficult to attribute observed changes to the intervention. It is also unknown whether participants completed the digital learning as intended, to what extent the digital learning accommodated different learning preferences or whether any behaviour change intention was sustained beyond 3 months.

Regarding aspects of the digital resource which may have influenced participants beliefs about capabilities, we speculate that the final RLO called ‘effectively interacting and engaging patients’ may have had the greatest impact, although we cannot be certain. Lastly, this study relied on self-reports of intention measures as well as the number of patients engaged, so the actual recruitment and engagement with the underserved communities is not known.

Conclusion

The co-produced digital educational intervention was shown to be effective at improving the perceived capabilities of pharmacy professionals towards engaging with the medically underserved. However, this resource alone may not be sufficient to bring about actual practice change. Changes in the organisations and to the policy may also be needed to achieve meaningful engagement and traction with patients who belong to medically underserved groups. This study adds to the continued debate on how best we can make services more equitable in order to tackle health inequalities. In this respect, the study provides a novel approach and information on the feasibility of delivering the educational intervention to pharmacies. Future research should focus on the interplay between the agency, organisation and policy constructs to better identify and understand the barriers to effective practice change.

Supplementary Material

Reviewer comments
Author's manuscript

Acknowledgments

The authors would like to acknowledge and thank all the pharmacy organisations, pharmacists and pharmacy support staff who were involved with the study. We also extend our thanks to Andrea Venn (School of Health Sciences, the University of Nottingham) for her help with the statistical analysis. We are also grateful to our patient and public involvement representatives Mahomed Khatri and Abida Malik. The research was supported by Health Education England (HEE) and National Institute for Health Research (NIHR). The views expressed are those of the authors and not necessarily those of HEE or NIHR.

Footnotes

Contributors: AL is the Chief Investigator (CI) and made substantial contributions to the overall conception, development and design of the study. KP, CA, JW, JS, LC and HW provided guidance on the methodology and direction of the study, specifically on qualitative aspects of the survey (KP, CA and JW), statistical analyses (LC) and pedagogical advice developing the digital learning educational intervention (HW and JS). NG and SC were responsible for data input and data analysis and contributed to the write-up. EA, SG and NA contributed to the logistical aspects of recruitment, including advice on access to underserved communities, study administration and conduct. All named authors contributed to editing and approved the final manuscript.

Funding: This study is funded by the Department of Health through the Health Education England and National Institute for Health Research Integrated Clinical Academic Programme (grant number ICA-CL-2015-01-008).

Competing interests: None declared.

Patient consent for publication: Not required.

Ethics approval: The study received ethical approval from East Midlands Research Ethics Committee (REC ref: Derby 16/EM/0237) on 15 July 2016, along with full governance clearance through the NHS Health Research Authority.

Provenance and peer review: Not commissioned; externally peer reviewed.

Data availability statement: All unpublished data related to this research project are available and can be requested by emailing the Chief Investigator Dr Asam Latif (asam.latif@nottingham.ac.uk).

References

  • 1. World Health Organization Who global strategy on people-centred and integrated health services: interim report. World Health Organization, 2015. [Google Scholar]
  • 2. Aspinall PJ. Hidden needs: identifying key vulnerable groups in data collections: vulnerable migrants, gypsies and travellers, homeless people, and sex workers. Centre for Health Services Studies, University of Kent, 2014. [Google Scholar]
  • 3. Marmot M. The health gap: the challenge of an unequal world: the argument. Int J Epidemiol 2017;46:1312–8. 10.1093/ije/dyx163 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4. Evandrou M, Falkingham J, Feng Z, et al. . Ethnic inequalities in limiting health and self-reported health in later life revisited. J Epidemiol Community Health 2016;70:653–62. 10.1136/jech-2015-206074 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5. Jackson C, Dyson L, Bedford H, et al. . Understanding uptake of Immunisations in travelling aNd gypsy communities (UNITING): a qualitative interview study. Health Technol Assess 2016;20:1–176. 10.3310/hta20720 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. The Disability Partnership Evaluation report of the, 2015-16, Mencap-led pharmacy project. Available: https://www.mencap.org.uk/sites/default/files/2016-07/Mencap_DP_Pharmacy_report_v1.pdf [Accessed 8 May 2019].
  • 7. Garden R. Who’s Teaching Whom? Disability and Deaf Studies Approaches to the Health Humanities. Teaching Health Humanities 2019;23 10.1093/med/9780190636890.003.0013 [DOI] [Google Scholar]
  • 8. Hicken MT, Kravitz-Wirtz N, Durkee M, et al. . Racial inequalities in health: framing future research. Soc Sci Med 2018;199:11–18. 10.1016/j.socscimed.2017.12.027 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9. Meade MA, Mahmoudi E, Lee S-Y. The intersection of disability and healthcare disparities: a conceptual framework. Disabil Rehabil 2015;37:632–41. 10.3109/09638288.2014.938176 [DOI] [PubMed] [Google Scholar]
  • 10. O'Donnell P, Tierney E, O'Carroll A, et al. . Exploring levers and barriers to accessing primary care for marginalised groups and identifying their priorities for primary care provision: a participatory learning and action research study. Int J Equity Health 2016;15:197 10.1186/s12939-016-0487-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Schiffer K, Marginalisation SE. Social inclusion and health. Amsterdam: Foundation Regenboog AMOC & Correlation Network, 2008. [Google Scholar]
  • 12. Kapilashrami A, Hill S, Meer N. What can health inequalities researchers learn from an intersectionality perspective? understanding social dynamics with an inter-categorical approach? Soc Theory Health 2015;13:288–307. 10.1057/sth.2015.16 [DOI] [Google Scholar]
  • 13. Maringe C, Rachet B, Lyratzopoulos G, et al. . Persistent inequalities in unplanned hospitalisation among colon cancer patients across critical phases of their care pathway, England, 2011-13. Br J Cancer 2018;119:551–7. 10.1038/s41416-018-0170-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14. Care Quality Commission The state of health care and adult social care in England 2017-18 11 Oct 2018 Available: https://www.cqc.org.uk/publications/major-report/state-care [Accessed 8th May 2019].
  • 15. Alderwick H, Dixon J. The NHS long term plan, 2019: l84 10.1136/bmj.l84 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 16. England NHS. Five year forward view, 2014. [Google Scholar]
  • 17. Bovaird T, Loeffler E. The role of co-production for better health and wellbeing: why we need to change : Loeffler E, Power G, Bovaird T, Co-production of health and wellbeing in Scotland. UK: Governance International, Birmingham, 2013: 20–8. [Google Scholar]
  • 18. Batalden M, Batalden P, Margolis P, et al. . Coproduction of healthcare service. BMJ Qual Saf 2015;25 10.1136/bmjqs-2015-004315 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19. O’Mara-Eves A, Brunton G, McDaid D, et al. . Community engagement to reduce inequalities in health: a systematic review, meta-analysis and economic analysis. Public Health Res 2013;1:1–526. 10.3310/phr01040 [DOI] [PubMed] [Google Scholar]
  • 20. Marmot M, Allen J, Goldblatt P, et al. . Fair society, healthy lives In: The Marmot review. 14, 2010. [Google Scholar]
  • 21. Manning JC, Carter T, Latif A, et al. . 'Our care through our eyes'. impact of a co-produced digital educational programme on nurses' knowledge, confidence and attitudes in providing care for children and young people who have self-harmed: a mixed-methods study in the UK. BMJ Open 2017;7:e014750 10.1136/bmjopen-2016-014750 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22. Windle R, Wharrad H. Reusable learning objects in health care education in organizational learning and knowledge: concepts, methodologies. Tools and Applications: IGI Global, 2012: 966–81. [Google Scholar]
  • 23. Pharmaceutical Services Negotiating Committee The new contract for community pharmacy. Aylesbury: PSNC, 2004. [Google Scholar]
  • 24. Pharmaceutical Serv ices Negotiating Committee (PSNC)/NHS Employers Guidance on the Medicines Use Review Service, 2013. Available: http://www.nhsemployers.org/~/media/Employers/Documents/Primary%20care%20contracts/Pharmacy/MUR%20Guidance.pdf [Accessed Cited 18 October 2018].
  • 25. Pharmaceutical Services Negotiating Committee Competency framework for the assessment of pharmacists providing the medicine use review (Mur) and prescription intervention service. Available: https://psnc.org.uk/wp-content/uploads/2013/07/advanced_service_competency_framework.pdf
  • 26. Latif A, Boardman HF, Pollock K. A qualitative study exploring the impact and consequence of the medicines use review service on pharmacy support-staff. Pharm Pract 2013;11:118–24. 10.4321/S1886-36552013000200009 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27. Department of Health Pharmacy in England: building on strengths–delivering the future, 2008. [Google Scholar]
  • 28. Blenkinsopp A, Bond C, Celino G, et al. . Medicines use review: adoption and spread of a service innovation. Int J Pharm Pract 2008;16:271–6. 10.1211/ijpp.16.4.0010 [DOI] [Google Scholar]
  • 29. Bradley F, Wagner AC, Elvey R, et al. . Determinants of the uptake of medicines use reviews (MURs) by community pharmacies in England: a multi-method study. Health Policy 2008;88:258–68. 10.1016/j.healthpol.2008.03.013 [DOI] [PubMed] [Google Scholar]
  • 30. Latif A, Pollock K, Boardman HF. The contribution of the medicines use review (Mur) consultation to counseling practice in community pharmacies. Patient Educ Couns 2011;83:336–44. 10.1016/j.pec.2011.05.007 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31. Pharmaceutical Services Negotiating Committee National target groups for MURs. Available: https://psnc.org.uk/services-commissioning/advanced-services/murs/national-target-groups-for-murs/ [Accessed 8 May 2019].
  • 32. Pharmaceutical Services Negotiating Committee (PSNC) Mur statistics. Available: https://psnc.org.uk/funding-and-statistics/nhs-statistics/mur-statistics/ [Accessed 8 May 2019].
  • 33. Latif A, Pollock K, Anderson C, et al. . Supporting underserved patients with their medicines: a study protocol for a patient/professional coproduced education intervention for community pharmacy staff to improve the provision and delivery of medicine use reviews (MURs). BMJ Open 2016;6:e013500 10.1136/bmjopen-2016-013500 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34. Cook DA, Levinson AJ, Garside S, et al. . Internet-Based learning in the health professions: a meta-analysis. JAMA 2008;300:1181–96. 10.1001/jama.300.10.1181 [DOI] [PubMed] [Google Scholar]
  • 35. Sinclair PM, Kable A, Levett-Jones T, et al. . The effectiveness of Internet-based e-learning on clinician behaviour and patient outcomes: a systematic review. Int J Nurs Stud 2016;57:70–81. 10.1016/j.ijnurstu.2016.01.011 [DOI] [PubMed] [Google Scholar]
  • 36. McCutcheon K, Lohan M, Traynor M, et al. . A systematic review evaluating the impact of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse education. J Adv Nurs 2015;71:255–70. 10.1111/jan.12509 [DOI] [PubMed] [Google Scholar]
  • 37. Ruggeri K, Farrington C, Brayne C. A global model for effective use and evaluation of e-learning in health. Telemed J E Health 2013;19:312–21. 10.1089/tmj.2012.0175 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38. Ruiz JG, Mintzer MJ, Leipzig RM. The impact of e-learning in medical education. Acad Med 2006;81:207–12. 10.1097/00001888-200603000-00002 [DOI] [PubMed] [Google Scholar]
  • 39. Latif A, Tariq S, Abbasi N, et al. . Giving voice to the medically under-served: a qualitative co-production approach to explore patient medicine experiences and improve services to marginalized communities. Pharmacy 2018;6. doi: 10.3390/pharmacy6010013. [Epub ahead of print: 27 Jan 2018]. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40. Kirkpatrick D, Kirkpatrick J. Evaluating training programs: the four levels. Berrett-Koehler publishers, 2006. [Google Scholar]
  • 41. Légaré F, Borduas F, Freitas A, et al. . Development of a simple 12-Item theory-based instrument to assess the impact of continuing professional development on clinical behavioral intentions. PLoS One 2014;9:e91013 10.1371/journal.pone.0091013 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42. Sullivan GM, Artino AR. Analyzing and interpreting data from likert-type scales. J Grad Med Educ 2013;5:541–2. 10.4300/JGME-5-4-18 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43. Corp IBM. IBM SPSS statistics for windows. 24.0, 2016. [Google Scholar]
  • 44. Althubaiti A. Information bias in health research: definition, pitfalls, and adjustment methods. J Multidiscip Healthc 2016;9:211 10.2147/JMDH.S104807 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45. Lea VM, Corlett SA, Rodgers RM. Workload and its impact on community pharmacists' job satisfaction and stress: a review of the literature. Int J Pharm Pract 2012;20:259–71. 10.1111/j.2042-7174.2012.00192.x [DOI] [PubMed] [Google Scholar]
  • 46. Latif A, Waring J, Watmough D, et al. . 'I expected just to walk in, get my tablets and then walk out': on framing new community pharmacy services in the English healthcare system. Sociol Health Illn 2018;40:1019–36. 10.1111/1467-9566.12739 [DOI] [PubMed] [Google Scholar]
  • 47. Britten N, Stevenson F, Gafaranga J, et al. . The expression of aversion to medicines in general practice consultations. Soc Sci Med 2004;59:1495–503. 10.1016/j.socscimed.2004.01.019 [DOI] [PubMed] [Google Scholar]
  • 48. Pound P, Britten N, Morgan M, et al. . Resisting medicines: a synthesis of qualitative studies of medicine taking. Soc Sci Med 2005;61:133–55. 10.1016/j.socscimed.2004.11.063 [DOI] [PubMed] [Google Scholar]
  • 49. Mohammed MA, Moles RJ, Chen TF. Medication-Related burden and patients' lived experience with medicine: a systematic review and metasynthesis of qualitative studies. BMJ Open 2016;6:e010035 10.1136/bmjopen-2015-010035 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50. Bulajeva A, Labberton L, Leikola S, et al. . Medication review practices in European countries. Res Social Adm Pharm 2014;10:731–40. 10.1016/j.sapharm.2014.02.005 [DOI] [PubMed] [Google Scholar]
  • 51. Hatah E, Braund R, Tordoff J, et al. . A systematic review and meta-analysis of pharmacist-led fee-for-services medication review. Br J Clin Pharmacol 2014;77:102–15. 10.1111/bcp.12140 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 52. McDonald R, Cheraghi-Sohi S, Sanders C, et al. . Professional status in a changing world: the case of medicines use reviews in English community pharmacy. Soc Sci Med 2010;71:451–8. 10.1016/j.socscimed.2010.04.021 [DOI] [PubMed] [Google Scholar]
  • 53. Latif A, Mandane B, Anderson E, et al. . Optimizing medicine use for people who are homebound: an evaluation of a pilot domiciliary medicine use review (dMUR) service in England. Integr Pharm Res Pract 2018;7:33–40. 10.2147/IPRP.S160149 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 54. Höög E, Lysholm J, Garvare R, et al. . Quality improvement in large healthcare organizations. J Health Organ Manag 2016;30:133–53. 10.1108/JHOM-10-2013-0209 [DOI] [PubMed] [Google Scholar]
  • 55. Grimshaw J, Campbell M, Eccles M. Experimental and quasi-experimental designs for evaluating guideline implementation strategies. Fam Pract 2000;17:11S–16. 10.1093/fampra/17.suppl_1.S11 [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Supplementary data

bmjopen-2019-031548supp001.pdf (47.5KB, pdf)

Reviewer comments
Author's manuscript

Articles from BMJ Open are provided here courtesy of BMJ Publishing Group

RESOURCES