European Public Assessment Report (EPAR) summaries for the public: are they fit for purpose? A user-testing study

Abstract

Objectives

Apply ‘user testing’ methodology to test the readability of a European Public Assessment Report (EPAR) summary—which describes how the decision was made by the European Medicines Agency to approve a medicine.

Design

User testing uses mixed methods (questionnaire and semistructured interview), applied iteratively, to assess document performance—can people find and understand key points of information.

Setting and participants

Testing was undertaken with 40 members of the public in four consecutive rounds of 10. Inclusion criteria, matched across rounds, included range of ages and educational attainment.

Tested documents

In round 1 we tested 19 key points of information in a printed version of the EPAR summary for Bondronat (a cancer medicine). This was then revised to address the findings, and tested in round 2. In round 3 we tested the summary on-screen, and in round 4, tested a revised on-screen version, after addressing findings from both rounds 1 and 3.

Primary outcome measure

The target followed European guidance for medicine leaflets: for each point of information 90% of participants should be able to find, and of those, 90% able to show understanding of the point.

Results

For the original EPAR summary, 6 of the 19 points of information reached the target (both paper-based and on-screen). After revisions to format and content, using good practice in information writing and design, 14 and 16 points, respectively, met the target. The problems related to both finding (dependent on layout, headings and design) and understanding (words and sentences used, as well as design). We devised a new heading structure, increased use of bullet points, replaced difficult and technical words and divided long sentences.

Conclusions

People had difficulty finding and understanding key messages in the summary, but user testing identified the problems, and application of good practice resulted in a revised format which performed well.

Article summary.

Strengths and limitations of this study

• This is the first study using a performance-based method to assess the readability of these documents—determining whether people can find and understand the information they need.

• Only one document was tested, but it is typical of the nature of EPAR summaries, and most improvements identified were generic and could be applied across all such documents.

Introduction

European Public Assessment Report (EPAR) summaries are designed to inform members of the general public about how the European Medicines Agency (EMA) assess the risks and benefits of a new medicine, before deciding to grant a licence.¹ They are developed with input from patient and consumer organisations, with each summary being reviewed for readability and public-friendly language by a relevant patient prior to publication, but no testing of their readability is undertaken with members of the general public.

The summaries are based on the full European Public Assessment Report, prepared by the EMA for all medicines licensed by the Agency. This full report is said to “Reflect the scientific conclusion reached … at the end of the centralised evaluation process.” The full report is lengthy and written for professionals. Hence, the decision to produce a summary, lay version “written in manner understandable to the public.”¹ The EPAR summaries are available on the EMA website, but relatively little is known about the usefulness of these documents for members of the public, for whom they are designed.

To test their usefulness, the performance-based process for assessing the readability of documents called ‘user testing’ was employed. This method is routinely used for the mandatory leaflets for patients produced by manufacturers and included in every medicine pack.^{2 3} The use of the same testing approach for EPAR summaries is supported by the fact that they have significant similarities to patient leaflets.⁴ They are both relatively short documents, and based on long complex documents designed for professionals (the Summary of Product Characteristics or SmPC in the case of patient leaflets)—but revised to meet the needs of lay people. User testing determines whether potential readers can find and understand key points of information, as well as generating general feedback on the extent to which the information meets people's needs.⁵ The process has been successfully used and reported for a variety of health information for patients, including clinical trial patient information sheets,⁶ medicine label wordings⁷ and nationally produced medicines information booklets.⁵

This work was undertaken in the wider context of health literacy, which is not just the ability to read and understand health information, but is now acknowledged to have a wider scope. Essentially, it has three aspects—the ability to read and understand health information, the removal by healthcare systems of unnecessary complexity and barriers to patient understanding and involvement and a wider ability to engage with the healthcare process.⁸ In terms of EPAR summaries, the three aspects relate to whether people read and understand EPAR summaries, whether they contain barriers to patient understanding (both are the focus of this study) and do these documents enable people to better engage in decisions about medicines (for future study).

The objective was to test, then revise and retest, an EPAR summary (in both hard copy and on-screen formats) using user-testing methodology to assess whether the target audience, members of the public, can find and understand the key messages.

Methods

We chose to test the EPAR summary for Bondronat, which contains ibandronic acid, a ‘bis-phosphonate’ medicine. It is used to treat cancers when there are high levels of calcium in the blood, and breast cancer in particular, when it has spread to the bone. The Bondronat summary was chosen because the data were based on more than one trial and its size is representative of most EPAR summaries—between one and two pages long.

User testing is a process which uses mixed methods to identify problems readers have with written information. Good practice is then applied, designed to address the problems identified. There are many texts on good practice in information writing and design, and two publications have pulled together such good practice for people writing medicines information. First, a UK review of medicines information for patients⁹ (commissioned by the Department of Health) included an information design review, informed by five experts in information design, which was subsequently published in the form of key principles to guide the development of consumer medicine information.¹⁰ Second, the European Union (EU) guideline on readability for package leaflets¹¹ was written to ensure such information was accessible and understandable, and is widely used in the domain. These two complementary sources were used to guide the revisions of the EPAR summaries—each time followed by retesting to assess the effect of the revisions made. The process has both a quantitative component, using an administered questionnaire, and a qualitative component, using a semistructured interview, both of which generate feedback on how the information performs—used to revise the document, prior to retesting.³

Participants

The method used was in line with regulatory guidance on user testing for package leaflets,^{11 12} that is, undertaken with people from the target group for the document—the general public. They were included if they could speak English to native standard. They were recruited from the database of Luto Research, the university spin out company which undertook the user-testing interviews. The database draws on people in the Leeds area of the North of England, and comprises people who have volunteered to take part in the testing of health information materials. People were excluded if they were current or retired healthcare professionals, took the medicine or had the indicated condition personally (or related conditions) or were carers of such people—as such people would have prior knowledge, and so it would not be a fair test of the information. In line with common practice in user testing, there were 10 participants in each round with

• At least three participants of each gender.

• Participants across the age range for the medicine. As the medicine is used more often by middle-aged and elderly people were recruited, at least one person in each of the 40s, 50s, 60s and 70-plus age groups, with no more than three people under the age of 40.

• No more than two higher education graduates.

• At least two participants who either do not use written documents as part of their work, or who are currently not working or are retired.

A new set of participants was used in each round, to prevent a learning effect. Participants were recruited in order that age, gender, education and use of literature (people who used written documents regularly at work vs those who did not, were not working or were retired) were equally matched across the four rounds of testing.

Materials tested

EPAR summaries are available on the EMA website in each of the official languages of the EU—we tested the English version.

Round 1

The first document tested was the printed version of the Bondronat EPAR summary, downloaded directly from the EMA website on 23 August 2010 (see figure 1).

Figure 1.

The original EPAR summary on paper (round 1).

Round 2

A revised hard copy version was tested, revised according to the short-comings identified in the first round (see figure 2). Both documents were printed in black and white on two single-sided A4 sheets and stapled in the top left corner.

Figure 2.

The revised EPAR summary on paper (round 2).

Round 3

The EPAR summary was shown to participants directly on a computer screen in its original format from the EMA website between 19 and 27 May 2011 (see figure 3).

Figure 3.

The original EPAR summary on screen (round 3).

Round 4

After revision, subsequent to the outcome of round 3, a revised version was tested, as a webpage mock up equivalent to the original website, opened in an equivalent Internet browser (see figure 4).

Figure 4.

The revised EPAR summary on screen (round 4).

Procedure

The document was examined by two pharmacists in the team (DKR, DB) to identify the most important points of information in the document. There were 19 points considered to be the most critical information (see figure 5), under the categories:

• Purpose of the report and its origins (2 points);

• Nature of the medicine and its use (3 points);

• Studies that have been undertaken (5 points);

• Specific issues regarding the risks of Bondronat (4 points);

• Decision to grant a licence (3 points);

• Finding more information (2 points).

Figure 5.

Key points of information tested.

A questionnaire was then designed to assess whether people could find and understand each key point of information. The questions were set in the questionnaire in an order that did not follow the sequence in the document.

The objective of the user test in line with current European guidance for patient leaflets,¹¹ is for 90% of participants to find the information in the document, and of those, 90% to be able to show that they have understood it.

Participants were interviewed individually by one of the three experienced trained interviewers in purpose-built interview rooms, and guidance for interviewers in the questionnaire ensured consistent conduct of the interviews. Consistent scoring of the responses was guided by ‘indicative answers’ for each question; that is, the information from the leaflet that the participant was required to provide for the answer to be scored as understood. For example, for the question “What two things were weighed up before the decision was made about whether Bondronat should be approved,” the indicative answer was “(Bondronat's/its) benefits AND risks.”

The responses to the short semistructured interviews were audio-recorded and transcribed verbatim. We looked for recurring patterns of comments and selected key quotes which illustrated these points.

The questionnaire and original EPAR summary were pilot tested with three people from a convenience sample to determine whether the questions worked in practice.

Round 1 (original; paper)

The questionnaire was administered to 10 people from the target group, interviewed individually, with each participant first given time to read the EPAR summary at their own pace. Then, using the EPAR summary supplied, they were asked to find the information relevant to each of the 19 questions and then to explain it in their own words (see table 1).

Table 1.

European Public Assessment Report summary paper version: original (round 1) and revised (round 2)

	Original (round 1)		Revised (round 2)
Questions	Number of points found/understood	Target met?	Number of points found/understood	Target met?
1  Who has this report been written for?	7 / 7	No	8 / 8	No
2  When is Bondronat used in breast cancer?	9 / 4	No	10 / 7	No
3  What is the purpose of this report?	7 / 0	No	9 / 9	Yes
4  What can the tablets do to the food pipe?	10 / 0	No	10 / 10	Yes
5  Who made the decision about Bondronat described in this report?	10 / 1	No	9 / 9	Yes
6  What group of medicines does Bondronat belong to?	7 / 7	No	10 / 10	Yes
7  What did Committee for Medicinal Products for Human Use (CHMP) decide about Bondronat?	10 / 10	Yes	10 / 10	Yes
8  The tablets can irritate the food pipe. Which people cannot have the tablets as a result of this?	9 / 8	Yes	10 / 8	No
9  What two things were weighed up before decision made about whether Bondronat should be approved?	7 / 7	No	10 / 10	Yes
10  How does Bondronat prevent fractures, or broken bones?	10 / 9	Yes	8 / 7	No
11  In the studies looking into bone problems or complications, what was looked at to see if Bondronat was working?	5 / 4	No	10 / 9	Yes
12  During the studies into high calcium levels, how many people did the 4 mg dose work in?	9 / 9	Yes	10 / 10	Yes
13  How many people receiving a Bondronat infusion got a fever?	10 / 3	No	10 / 8	No
14  During the study of bone complications, what was Bondronat compared with?	9 / 9	Yes	10 / 10	Yes
15  Suppose you want more information on how the decision was made about Bondronat, what should you do?	9 / 1	No	10 / 9	Yes
16  How many people taking the Bondronat tablet got an inflamed food pipe?	10 / 5	No	10 / 9	Yes
17  In the research studies of high calcium levels in the blood, what was Bondronat compared against?	9 / 9	Yes	9 / 9	Yes
18  When Bondronat was studied in the treatment of bone complications, which group of patients got a new bone problem first?	8 / 8	No	9 / 9	Yes
19  If you are a patient and want more information about Bondronat, what should you do?	8 / 7	No	10 / 10	Yes
Total number of passes		6		14

For each question, the interviewer noted any comments or particular behaviour of the participants. At the end of the interview, participants were asked for feedback about the EPAR summary.

Round 2 (revised; paper)

The original EPAR summary was then refined by an experienced health information writer (DKR), focusing on the difficulties identified during the testing, and participants’ general comments about the usefulness of the document. These revisions were made using best practice in information writing and design (see above). This revised EPAR summary was tested on 10 more people from the target group for this product.

Rounds 3 and 4 (original; electronic and revised; electronic)

As EPAR summary documents are available in an electronic format, two further rounds of testing were conducted on-screen, to test both the original and then a revised EPAR summary. For each version, 10 people from the target group were interviewed, this time referring to the information on a computer screen rather than in printed form. Inclusion criteria for these rounds additionally included:

• Five people who use computers often (at least once a day).

• Five people who use computers occasionally (not every day).

We anticipated applying the learnings from the hard copy testing to the subsequent on-screen testing.

Results

Forty eligible participants were interviewed in four rounds of 10; their characteristics in terms of gender, age range, educational status and use of literature at work were matched across all four rounds and are described in table 2.

Table 2.

Demographics of participants in each round

	Female (F) or male (M)	Age range	Educational status*	Use of literature at work †
Round 1 (10)	7F, 3M	29–74	1=4, 2=4, 3=2	Y=3, N=7
Round 2 (10)	7F, 3M	22–71	1=4, 2=4, 3=2	Y=3, N=7
Round 3 (10)	7F, 3M	26–72	1=4, 2=4, 3=2	Y=3, n=7
Round 4 (10)	7F, 3M	21–73	1=4, 2=4, 3=2	Y=3, N=7

*Educational status: 1=education complete by 16 years; 2=A-level or equivalent; 3=higher educational graduate.

†Use of literature at work: Y=yes: uses written documents regularly at work; N=no: written document not used regularly at work, not working or retired.

Round 1: original paper version

Quantitative findings

In the first round of testing, with the original document in paper format, 6 of the 19 items of information met the target of 90% finding and 90% of those understanding (see table 1).

Qualitative findings

Participants’ comments on the layout of the information were mixed; some criticised it

It's not user friendly for a start. It's more like something from a lecture (P1)

I found it quite difficult to find things …they weren't where I‘d expect them to be under the headings (P8)

One participant gave a more specific comment about the box at the top of the page

The little box at the top with italics…people miss out things like that, they'll think ‘Oh it's the boring bit’ (P3)

Others were more content with the layout:

Well set out (P 10)

Well laid out with the titles (P3)

Most participants said they did not have a problem with the language used in the leaflet. Participants described the language as fine, okay and straight forward among other things. However, one participant (P1) said it was too technical and another said

There's all these big words and big sentences (P5)

Revising the EPAR summary

The revision took account of the results from the first round of testing—both quantitative and qualitative, where it was shown that improvements were needed. The revision also took account of good practice in information writing and design (see above).¹⁰

There were eight questions where, of the 10 participants, 5 or less could find and understand the information. Specific changes were made to address the issues with these points of information and are described in table 3.

Table 3.

Changes made in round 1 related to the responses to specific questions

	Question	Changes made
2	When is Bondronat used in breast cancer?	Original long sentence with multiple bracketing incorporated into 2 new subheadings, separated into dedicated bullet points
3	What is the purpose of this report?	Original text in italics in box at beginning became part of main document under new heading ‘Who is this report for’
4	What can the tablets do to the oesophagus or food pipe?	In original, mentioned both in the ‘How is Bondronat used’ section and ‘What is the risk’ section in isolation. Changed so clear in both places what problem was, and how it related to necessary actions
5	Who made the decision about Bondronat described in this report?	Original did not make clear the membership of CHMP—remedied in revised version under the heading ‘Who made the decision?’ in first section of the document
11	In the studies looking into bone problems or complications, what was looked at to see if Bondronat was working?	Answer was number of new bone complications—in original in middle of paragraph containing variety of pieces of information about the study. Revision included new subheading ‘What did the studies look for’
13	How many people receiving a Bondronat infusion got a fever?	Original listed side effects in long paragraph with long sentences. Revision bulleted the side effects, and separated out those relating to ‘drip’ and tablets
15	Suppose you want more information on how the decision was made about Bondronat, what should you do?	Information moved from original boxed italicised text into a new subsection titled ‘Where to get more information’
16	How many people taking the Bondronat tablet got an inflamed food pipe?	Information became part of bulleted list of side effects (see above), and frequency simplified from ‘seen in between 1 and 10 patients in a hundred’ to ‘affects less than 1 in 10 people’

As well as the changes in response to specific difficulties with answering the questions, other general changes made in line with good practice in information writing and design^{10 11} were

• Information split over two columns—some readers struggle with long lines of text.

• Boxes surrounding text removed—some readers ‘read round’ boxed information.

• Text and headings justified to the left (‘ragged right’)—text justified to left and right leads to unequal gaps between words which can hinder ease of reading.

• Bold text used as it is effective in giving emphasis—replacing underlined, italicised or capitalised text (which can make reading harder).

• Some paragraphs organised into bullet points—helps to organise text for readers, and aids finding as well as making document look more approachable.

• Technical or medical terms replaced with lay language—for example, ‘hypercalcaemia’ became ‘high calcium levels’.

Importantly, the structure was revised with a focus on making the document easier to navigate. It was split into six main headings, section numbers were introduced (see figure 6) and an index (‘In this leaflet’) was added to aid navigation. The main headings for each of the six sections of the leaflet were written as white text on a grey band which went across the column. This clearly demarcates the sections, helping readers to find the section they are looking for. Equally, the insertion of subheadings aids navigation, such as ‘Who is this report for’ and ‘What type of medicine is Bondronat’.

Figure 6.

Original and revised heading and subheading structure.

Finally, instead of separating the information from the two types of studies (into how studied and what has been shown), the studies of bone complications and high calcium levels were given their own subheadings, and became self-contained.

As a result, revisions were made to the original EPAR summary to produce the revised version tested in round 2 (see figure 4).

Round 2: revised paper version

Quantitative findings

The objective remained that 90% of participants should be able to find the information in the leaflet and of those 90% should be able to understand it. As can be seen from table 1, 14 of the 19 points of information now met this target.

Qualitative findings

Participants were generally positive about the layout of the revised EPAR summary. Comments included it was fine how it's laid out (P17) and it's logical (P20). Several participants made specific reference to the headings and the ‘In this leaflet’ index as positive features of the document, examples of comments were

I like how the headings are set out, which are set into sections which are easier to find (P18)

It's good the way you've got it laid out. ‘In this leaflet’ and it tells you which section's which (P13)

It is in bullet points and its easier to read than paragraph after paragraph of information (P14).

The language was well received and described as straight forward (P12) and very easy to understand (P18). Participants commented on the use of lay language, for example, participant 15 described the language used as everyday language.

Round 3: original electronic version

Quantitative findings

The objective that 90% of participants should be able to find the information in the leaflet and of those 90% should be able to understand it was met for 6 of the 19 points (see table 4).

Table 4.

European Public Assessment Report summary screen versions: original (round 3) and revised (round 4)

		Original (round 3)		Revised (round 4)
Questions		Number of points found /understood	Target met?	Number found/understood	Target met?
1	Who has this report been written for?	4 / 0	No	10 / 9	Yes
2	When is Bondronat used in breast cancer?	9 / 1	No	10 / 9	Yes
3	What is the purpose of this report?	6 / 0	No	9 / 9	Yes
4	What can the tablets do to the food pipe?	10 / 0	No	10 / 10	Yes
5	Who made the decision about Bondronat described in this report?	9 / 0	No	10 / 9	Yes
6	What group of medicines does Bondronat belong to?	5 / 5	No	10 / 10	Yes
7	What did the Committee for Medicinal Products for Human Use (CHMP) decide about Bondronat?	9 / 8	Yes	10 / 9	Yes
8	The tablets can irritate the food pipe. Which people cannot have the tablets as a result of this?	10 / 9	Yes	9 / 8	Yes
9	What two things were weighed up before the decision was made about whether Bondronat should be approved?	3 / 3	No	10 / 10	Yes
10	How does Bondronat prevent fractures, or broken bones?	10 / 9	Yes	7 / 6	No
11	In the studies looking into bone problems or complications, what was looked at to see if Bondronat was working?	8 / 8	No	7 / 7	No
12	During the studies into high calcium levels, how many people did the 4 mg dose work in?	7 / 7	No	10 / 9	Yes
13	How many people receiving a Bondronat infusion got a fever?	10 / 9	Yes	10 / 9	Yes
14	During the study of bone complications, what was Bondronat compared with?	10 / 8	No	10 / 10	Yes
15	Suppose you want more information on how the decision was made about Bondronat, what should you do?	7 / 2	No	10 / 9	Yes
16	How many people taking the Bondronat tablet got an inflamed food pipe?	9 / 8	Yes	10 / 10	Yes
17	In the research studies of high calcium levels in the blood, what was Bondronat compared against?	7 / 7	No	9 / 9	Yes
18	When Bondronat was studied in the treatment of bone complications, which group of patients got a new bone problem first?	6 / 5	No	8 / 5	No
19	If you are a patient and want more information about Bondronat, what should you do?	9 / 9	Yes	10 / 10	Yes
	Total number of passes		6		16

Qualitative findings

As in round 1, comments on the original EPAR summary were mixed with regard to the layout; however, the layout of the electronic version seemed to be better received than the paper version. Positive comments included it looked straight forward (P23) and It was easy to scroll down to whichever section your question applied to (P29). Participant 28 was more critical and suggested more care and attention to what you present under these various headings.

About half of the participants thought that the language was fine or okay. However, other participants were more negative. Comments included, unnecessarily formal and technical (P28) and it used a lot of big words (P23). One participant anticipated the team's revisions after round 1 by saying Why don't they put the easier words down instead of the big words and then in brackets put the smaller words (P22).

Two participants also made reference to the Package Leaflet, participant 28 noted that Package Leaflet, although mentioned, is not immediately available on the website. Participant 30 also commented to say that I'm realising this is an extra leaflet, it would be nice to see the proper leaflet because that's probably got it a lot clearer, how to take it.

Revising the EPAR summary

The changes made after the first round for the hard copy version were largely replicated in the revised on-screen version, as most of the problems identified were similar. However, a single column format was retained as this worked well in an on-screen format.

There were nine questions where of the 10 participants, 5 or less could find and understand the information. Of these, five questions (2–5 and 15) were addressed, as a result of problems with the hard copy original. This left four additional points to address, which are described in table 5.

Table 5.

Changes made in round 3 related to the responses to specific questions

	Questions	Changes made
1	Who has this report been written for?	Dedicated subheading ‘Who is this report for” introduced
6	What group of medicines does Bondronat belong to?	This information, formerly in the section ‘How does Bondronat work’ repositioned in the new subsection ‘What type of medicine is Bondronat’?
9	What two things were weighed up before the decision was made about whether Bondronat should be approved?	Information bulleted in revised version, to aid finding and understanding
18	When Bondronat was studied in the treatment of bone complications, which group of patients got a new bone problem first?	Simplified through new subheading of ‘What did the studies show?’

Round 4: revised electronic version

Quantitative findings

The objective that 90% of participants should be able to find the information in the leaflet and of those 90% should be able to understand it was met for 16 of the 19 points of information (see table 4).

Qualitative findings

The layout was generally well received by participants in this round. It was described as nice and clear (P31) and simple (P32). Participant 40 also mentioned the different sections of the leaflet and said that it was well structured and in different categories which all make sense.

In general, comments received about the language were positive. It was described as easy to understand (P31), and not complicated (P39).

There were varying views on the electronic presentation. Participant 32 said I love being able to do it on the screen. Whereas another participant thought that there is a lot of scrolling through the screen (P40). Other comments included

It wouldn't be very much use to me reading it on a computer because I haven't got a computer (P34)

Qualitative feedback on EPAR summaries in general

Participants from all four rounds of testing were asked to provide feedback in terms of their opinions on EPAR summaries in general.

The majority (32/40) of the participants interviewed thought that the information in the EPAR summary report would be useful.

If they could fight their way through it, it would be quite useful (P3)

It was good how they gave the background … explain the benefits … so people can see a reason for taking the drug (P33)

I think it's good to know that somebody in authority has checked it out (P20)

Those who did not think it would be useful made comments about the complexity of the information. When asked whether people who were not patients would find it useful, the most common response was that carers or people who know someone with the indicated conditions might find it useful to read the document. A few participants listed some groups of people who might utilise the information, for example, healthcare professionals (n=4) and researchers (n=2).

Discussion

The principal finding of this research is that the current EPAR summary format could be improved, with only 6 of the 19 items of information found and understood (to the level required for package leaflets) for both the paper-based and screen formats. However, the revised format, guided by user testing with the target population, better met people's needs, where 14 and 16 points, respectively, met the target.

As with most documents that are user tested, there were problems related to both finding information and understanding the information. Importantly, the heading structure did not work well, and points of information need to be placed where lay people would expect to find them (rather than where professionals are used to placing them). The addition of new numbered headings and subheadings (along with the inclusion of an ‘In this leaflet’ listing near the beginning) led to improvements in performance. Important points relating to the content were the need to clarify the purpose of the document at the beginning: saying what it is, and who it is for. The key recommendations are described in box 1.

Box 1. Recommendations.

General information design

• Appropriate use of bullets, no italics or boxes.

• Two columns for paper version.

• Clear signposting with main headings (numbered) and subheadings which work for lay people (see headings structure in figure 2).

• Still a need for both hard copy and on-screen versions. The printable version from the website needs to be designed to work well on paper—not just replicate what is on the screen.

Key messages

• Clarify the purpose of the document up front: what it is and who it is for.

• Keep the information about each study (or set of studies) together, that is, do not separate out ‘How has been studied’ and ‘What benefit has been shown in studies’.

• The general public may not be the right audience? It may be more usefully aimed at people who are take or are considering taking the medicine concerned and people from relevant patient groups, rather than the general public.

Future developments

• Clarify relationship with Package Leaflet—presenting both together would allow removal of most of the duplicated information, and allow the European Public Assessment Report summary to focus on the studies, their findings and the decision-making process.

A previous study of EPAR Summaries in Denmark by Askehave and Zethsen⁴ focused on their nature as a ‘mandatory genre’. The study involved a questionnaire on lay respondents’ opinion of the text, alongside an exercise in which participants marked problem words, sentences or paragraphs in the document. The authors concluded that most participants did not think the summaries fulfil their purpose of providing information which is understandable and useful to lay persons. Our findings with a UK population concur with those in Denmark, and the addition of a performance-based testing method also pointed the way to improvements.

Limitations

The study was based on one EPAR summary, which was of a representative size for such summaries. However, an EPAR summary for a medicine for cancer may be substantially different from one on a medicine for epilepsy or schizophrenia in terms of what the public need to know. This work would be enhanced by similar studies on different types of medicine. However, in the meantime, most of the improvements identified were generic, and not specific to the type of medicine, and would be applicable to most summaries.

We refer later in this section to studies which show that patients want more balanced information, including benefit information. However, we did not include such a heading in the final version of the EPAR summary, for example, ‘What are the benefits of Bondronat?’ The inclusion of such a heading may need to be studied in future research on these documents.

Relationship with wider literature

Previous studies examining health literacy have also found that many people have difficulty in understanding health information in the UK,¹³ Australia¹⁴ and the USA,¹⁵ including statutory medicines information.¹⁶ Furthermore, a review of drug information for consumers and patients with a focus on the EU concluded that ‘drug authorities’ should see themselves more as a source of medicine information for patients.¹⁷ This study has also demonstrated the concept that unnecessary complexity can act as a barrier to patient understanding²—shown here in the fact that a minority of the items of information could be found and understood. What this work does not show is whether the revised summaries enable a wider ability to become more empowered to engage with the healthcare process—part of the wider nature of health literacy¹⁸—this needs further study.

What is the target group?

Most stakeholders would agree that people should be able to find out how the decision to license a medicine was made. However, the full EPAR is never going to be a document which is going to be useful to lay people, and EPAR summaries are a laudable attempt to open up the medicines licensing process in the EU. Although the intention was that they should be understandable to the public, their effectiveness has not been previously tested. Owing to the target group, as stated by the EMA, being ‘the public’, the participants in this series of tests were representative of the general population, weighted to more older people and including people of lower educational attainment. However, one finding of the study is that it was not certain for some participants as to whether this type of information would be useful to them personally. Although not stated by participants, it could be argued that the target group should be people with the condition for which the medicine is used, who are actively involved in their healthcare, and people from relevant patient groups (rather than members of the public generally). If so, then future testing should be undertaken with people from these target groups.

Relationship with package leaflet

Some participants raised the question of the relationship between the EPAR summary and the Package Leaflet. It is clear that there is considerable overlap between the two documents. The EPAR summary has general information, also contained in the leaflet (about what the drug is, what it is used for and how it is used, along with some side effects). The rest of the document, detailing the studies and the decision process would not make sense without that background information. However, the background information does appear to be problematic, and takes the focus away from the key points relating to the research studies and the decision process. One option would be to incorporate the patient leaflet (or parts of it) into an EPAR summary ‘package’. This could comprise an introduction, which describes the purpose of the ‘package’, and explain that the reader should first read the leaflet, and then the EPAR summary. The latter could then focus only on the studies and the decision-making process, that is, the information under the current headings ‘How Bondronat was studied’ and ‘Why Bondronat was approved’.

A notable difference between the EPAR summary and the package leaflet is that the former explicitly mentions the likelihood of benefit. This is generally not present in patient leaflets, certainly not with any numerical information about the benefit. However, there is increasing support to include more benefit information in package leaflets,¹⁹ to answer the call from patients that they want more balanced information on which they can make decisions about whether a medicine is right for them.²⁰ At present, including this information is a problem, because of the current requirement that the leaflet is based on the SmPC. A second problem is that the information needs to be presented in a way that is understandable to the public. It could be argued that including benefit information based on the EPAR summary (in the revised format developed in this study) would solve both problems.

This study is further evidence to support the applicability and usability of the user-testing method to improve a variety of health-related information. It also shows that paper-based EPAR summaries as well as on-screen versions continue to be needed by different groups of the population. Importantly, the information needs to be appropriately formatted for each medium—as used in this study.

Finally, this study has shown that there is support for the idea of a document which describes in patient friendly terms, the studies on which the decision was made to make the medicine available in the EU. However, the EPAR summary document did not perform well in user testing, but the testing highlighted the problems, and application of good practice resulted in revised formats which performed well.