Development and validity of a method for the evaluation of printed education material

Abstract

Objectives

To develop and study the validity of an instrument for evaluation of Printed Education Materials (PEM); to evaluate the use of acceptability indices; to identify possible influences of professional aspects.

Methods

An instrument for PEM evaluation was developed which included tree steps: domain identification, item generation and instrument design. A reading to easy PEM was developed for education of patient with systemic hypertension and its treatment with hydrochlorothiazide. Construct validity was measured based on previously established errors purposively introduced into the PEM, which served as extreme groups. An acceptability index was applied taking into account the rate of professionals who should approve each item. Participants were 10 physicians (9 men) and 5 nurses (all women).

Results

Many professionals identified intentional errors of crude character. Few participants identified errors that needed more careful evaluation, and no one detected the intentional error that required literature analysis. Physicians considered as acceptable 95.8% of the items of the PEM, and nurses 29.2%. The differences between the scoring were statistically significant in 27% of the items. In the overall evaluation, 66.6% were considered as acceptable. The analysis of each item revealed a behavioral pattern for each professional group.

Conclusions

The use of instruments for evaluation of printed education materials is required and may improve the quality of the PEM available for the patients. Not always are the acceptability indices totally correct or represent high quality of information. The professional experience, the practice pattern, and perhaps the gendre of the reviewers may influence their evaluation. An analysis of the PEM by professionals in communication, in drug information, and patients should be carried out to improve the quality of the proposed material.

INTRODUCTION

Printed education material (PEM) is one of the most frequently used resources for the education of patients with chronic diseases. Different authors have established the most important steps in developing PEM1,2,3,4,5,6,7,8,9,10 for literate patients. Some articles in the current literature have reported the poor quality of information of such materials11,12,13,14,15,16, as well as the lack of scientific accuracy to promote the education of patients.11,12,16,17,18,19,20,21,22,23

Printed education materials have been used routinely in Brazil for patients from the public and private institutions, but with no defined validation process. Two reports on evaluation of such materials for Brazilian patients were found.24,25 The first focuses on criteria for evaluation of textbooks for elementary education24 and the second evaluates the acceptability of one piece of education material produced in the form of a game.25 The lack of assessment of PEM produced in the Portuguese language and the new requirements for the development of PEM17,23 demonstrate that it is necessary to develop further research in the educational area. In this report, the development and testing of an instrument for evaluation of PEM (EVALPEM) for health professionals and the performance of the acceptability index of the evaluated items, and the influence of some professional aspects in this process are presented.

METHODS

Development of the evaluation tool

The steps proposed by Lynn2 were used for identification of domains, generation of items and tool design.

Phase 1: Identification of domains

Domains were those identified in the literature review or regarded as important for determining the quality of a PEM, according to expert evaluation.3,17,23,26

Phase 2: Items Generation

The items were established for each domain (described in Table 1), specifying the basic principles pertinent to each item.3,16,17,23,26

Table 1.

- Part 1: Principles and criteria for evaluation of Printed Educational Materials.

Principle
1 – Scientific Accuracy
a) contents are in agreement with the current knowledge
b) recommendations are necessary and are correctly approached
2 – Content
a) objectives are evident
b) recommendation about the desired behavior is satisfactory
c) there is no unnecessary information
d) important points are reviewed
3 – Literary Presentation
a) language is neutral (no comparative adjectives, promotion or false appeals)
b) language is explanatory
c) language is conversational and, in at least 50% of the material, written in the active voice
d) material promotes and encourages treatment adherence following evaluation of the benefits and risks
e) majority of the vocabulary is composed of common words
f) context of each issue is communicated before the new information
g) identification of headings and subheadings help in the learning process
h) vocabulary is composed of simple words
i) language is adequate for outpatients
j) ideas are concisely expressed
k) text allows for interaction with verbal counseling
l) text allows for interaction with logical linkage of the therapeutic plan
m) planning and sequence of information is consistent, making it easier for the patient to predict its flow
n) material is reader-friendly

Table 1.

- Part 2: Principles and criteria for evaluation of Printed Educational Materials.

Principle
4 – Illustrations
a) illustrations are simple, appropriate and present an easily understandable outline
b) they are familiar to the readers
c) they are related to the text (express the desired purpose)
d) they are integrated with the text (easily located)
e) lists, tables and graphs are self explanatory
f) headings in e) are adequate
5 –Material is sufficiently specific and understandable
a) material promotes the correct use of the medication
b) provides maximum benefit with minimization of complications
c) instructions for administration of treatment are clear
d) recommendations on how to prevent complications are understandable
e) it is clearly explained how to identify the action of drug
f) material enables reader to recognize when they have to urgently look for a physician assistance
g) enables the patient to determine whether he or she is facing a serious problem
h) technical terminology is adequately defined
i) headings and subheadings are clear and informative
j) use of words or expressions with double meanings does not occur in the text
k) content is written in a patient-centered style; that is, the patient is the focus of importance

Table 1.

- Part 3: Principles and criteria for evaluation of Printed Educational Materials.

Principle

6 – Legibility and Printing Characteristics

a) size of the letters is adequate

b) style of the letters is adequate

c) spacing between letters is adequate

d) length of the lines is adequate

e) spacing between lines is adequate

f) use of bold characters and bullet points draw attention to specific points or key content

g) adequate use of blank space reduces overcrowded appearance

h) good contrast between the printed content and the paper

i) paper used makes it easier to read

j) subheadings or the inner margins make reading and memorization easier

k) spacing between paragraphs is adequate

l) format of the material is adequate

7 – Quality of Information

a) it is integrated to the local culture

b) information is updated

c) it is adapted to the current culture

d) material enables the patient to undertake the desired actions

e) material helps patient to prevent potential problems

f) material allows patient to achieve the maximum benefits possible

Phase 3: Design of the Instrument

Based on the previously established domains and their respective items, a tool for the evaluation of PEM was developed. Each item was presented in the form of an affirmation, followed by an assessment scale with 3 categories: totally disagree, corresponding to 0 points; partially agree, corresponding to 1 point; totally agree, corresponding to 2 points. This scale was adapted from the scale proposed in The Bernier Instructional Design Scale (BIDS).3 After the scoring scale, there was a blank space for comments. In the instructions, the evaluators were asked to report any problems faced in this field while setting up the PEM or to state “not applicable” when appropriate. At the end of the EVALPEM, four questions were asked, aiming an integral evaluation of the PEM: 1) What did you like in the PEM? 2) What did you not like in the PEM? 3) What should be added? 4) What should be reviewed?

Construct validity

It was not possible to validate EVALPEM as proposed by Bernier3 and Coulter et al.17 since it was not possible to find a sufficient number of experts in patient education in Brazil. Therefore, we decided to evaluate the validity of the designed instrument using an alternative method. The introduction of errors previously established and introduced in a PEM constructed for this purpose was the artifice used, thus providing correct and incorrect elements that acted as extreme groups.27 Through the analysis of the “extreme groups,” we intended to verify whether the evaluators would be capable of detecting errors using the proposed instrument, as well as identifying possible professional influences when applying a tool like EVALPEM.

The PEM consisted of two blocks, one related to systemic arterial hypertension and the other to the use of hydrochlorothiazide. These themes were chosen due to the high prevalence of hypertension in our city28,29 and the use of hydrochlorothiazide as one of the first choice medications for the management of this condition.29,30,31 A version which required less than 6 years of schooling from patients was utilized.32 The basic structure of the PEM was previously presented.17,33

Procedure for application of the EVALPEM

Health professionals that provided outpatient services in the Hospital de Clínicas de Porto Alegre and who were involved in the care of patients with hypertension were invited to evaluate the PEM. All of them agreed to participate of the study, not taking into account their real objectives or the existence of extreme groups.

For analytical purposes, they were divided into 2 groups. The first (Group 1) consisted of 10 physicians (9 men) from the Division of Cardiology. The second (Group 2) consisted of 5 nurses (all women) of the Nursing Service in Public Health. They received a copy of the PEM and the evaluation tool, as well as written instructions for completing it. After one week, the materials were returned with the questionnaires completed. The data from the two groups of professionals were analyzed separately and jointly in order to identify differences associated with their background.

Evaluation of the acceptability index of the proposed material

To evaluate the contents of the PEM, the parameters described by Lynn2 were adopted. These parameters state that the acceptability of a given criterion depends on the approval of a certain proportion of experts. This proportion varies according to the numbers of evaluators. In Group 1, an item was considered acceptable when at least 80% of the possible points were obtained; in Group 2, 100% of the points were required to approve the item. The score for both groups combined was set in 80%.

RESULTS

In the process of construct validity, the physician and the nursing groups identified, respectively, 6 and 7 from 9 intentional errors. For two items only one professional identified the errors: 1) the absence of report concerning the adequate arm position in the measurement of blood pressure, and 2) the inadequate spacing between lines. The intentionally false contraindication of hydrochlorothiazide in patients with rheumatoid arthritis was not identified by any professional.

Based on the previously established parameters, it was observed that for the physician group 95.8% of the items in the PEM were considered acceptable. Six items were considered unacceptable. Four of them corresponded to the errors deliberately introduced in the PEM. The 2 other corresponded to the items that physicians did not either totally or partially agree with. The description of the mechanism of the action of the drug was considered difficult for patients to understand, and the description of adverse effects of hydrochlorothiazide was considered unnecessary.

For the nurse group, just 29.2% of the items were considered as acceptable. Forty items were considered as unacceptable, 7 of them referring to the deliberate errors that were adequately identified. In relation to the others, a behavior similar to the physicians was observed in regard to the amount of information to be provided to the patient. Moreover, nurses were more critical, indicating inadequacies in the PEM that were not reported by the physicians, such as double meaning expressions and usage of terminology considered difficult to understand by the patients.

The differences between the scoring of the 2 groups were statistically significant in 27% of the items (P < 0.05, Mann-Whitney U test). In the overall evaluation, 66.6% of the items were considered as acceptable.

The analysis of each item revealed a behavioral pattern for each professional group. Among the physicians, 7 suggested modifications in the PEM, such as to use this opportunity to better explain the risks for arteriosclerosis, to teach how to recognize angina and stroke. One physician suggested an investigation of patients in order to determine the best way to orient them about the administration of hydrochlorothiazide. Five physicians were against at least one of the following items: to inform patients about the stages of hypertension; to measure BP outside the medical office; to inform about potential adverse drug reactions; to require the control of blood glucose before starting the use of hydrochlorothiazide.

Four nurses gave suggestions: to extend the concept of hypertension; to highlight titles in a different color from the printed text; to identify the producers and reviewers in the PEM; and to better explain the relationship between loss of potassium and cramps. They also suggested that some points should be directly investigated with the patients. Four nurses were against at least one of 8 points, for example, the explanation of the classification of hypertension and the orientation on how to prevent intercurrences, since they might scare patients and reduce concordance of the treatment.

Some comments from the nurses were in disagreement with the recommendations from the literature, such as (1) disagreement regarding to diet orientation; (2) the possibility of controlled alcohol and salt consumption, opting for prohibiting; (3) the explanation on how to conduct in face of lack of one dose of the drug, since it could lead to confusion, and (4) the risk of photo-sensitivity associated with the use of hydrochlorothiazide.

Contrasting ideas on the same topic were also observed. A physician considered the material too simple while 3 nurses stated that the material was too long. Three physicians and a nurse reported their concern about the participation of pharmacists in the education of patients. Three physicians and four nurses did not agree to warn patients about potential adverse reactions to hydrochlorothiazide and deemed unnecessary for patients to know the stages of hypertension.

DISCUSSION

The quality of information provided to patients unequivocally contributes to the prevention of illnesses and recovery of health.34,35,36 The quality of such materials should be evaluated. Previously to their use by patients, PEM needs to be evaluated by different groups of professionals involved with their care. Therefore, the development of instruments for this evaluation is required, such as the presented and tested in this report.

The questionnaire applied in this study was detailed and some items were considered irrelevant by the professionals. Both groups identified correctly the most evident deliberate errors. However, just a few professionals were able to identify the errors that would require a deeper analysis of the sources of information or about legibility and printing characteristics. These findings suggest that, in the domain of scientific accuracy, experts in the analysis of scientific literature should also review PEM information, as recommended by Coulter.17 Additionally, professionals of the communication area should be consulted about the illustrations, legibility and printing characteristics of the material.

The EVALPEM, which was tested in the evaluation of the printed educational material on hypertension and the use of hydrochlorothiazide, allowed identifying the opinions of physicians and nurses about the PEM. The physicians accepted most part of what was reported in the PEM, while the nurses did not accept many items.

The items that have discordant opinions from these two groups should be studied. Part of the different evaluation by physicians and nurses may be attributed to the different knowledge about the theoretical background topic by the professionals. Physicians were in charge of an outpatient clinic of hypertension, while the nurses that were invited to participate came from the Nursing Service in Public Health Hospital. We observed that much of the disagreement regarding the proposed PEM derive from their models of professional practice (The Medical Model versus The Helping Process)34 and from the clinical decision making.17 Professionals tend to establish a relation of dependence rather than partnership with their patients. Oliveira reported similar results37 in a study about the communication process between patients and physicians when a serious prognosis should be given to patients. Differences related to the gender could not be also discarded, since all nurses were women and most physicians were male.

We suggest that isolated remarks should not be discarded. Sometimes just one individual perceives the problems regarding to one topic, while all other evaluators approve it. These data reveal the limitation of using isolated acceptability indices in the PEM evaluation.

On the other hand, the fact of having consensus among the professionals on a particular item does not mean necessarily adequacy. Both groups, for example, disagreed on presenting potential drug adverse effects, a recommendation not supported by the best evidence (38,39,40,41,42,43). The PEM improved on the basis of the evaluation by physicians and nurses should be sequentially evaluated by professionals of communication and by the patients themselves.

The reaction to the participation of pharmacists in patient education expressed by some physicians and nurses probably reflects the model of professional pharmaceutical practice adopted in Brazil, which is primarily oriented to drug distribution. The adoption of a new paradigm – pharmaceutical care – one of the World Health Organization recommendations44,45, might contribute to solving health issues related to the use of medicines.

CONCLUSION

The present study demonstrated that printed education materials require the use of an evaluation process to improve their quality. Variables such as the evaluation instrument, the profile of reviewers, and the acceptability index should be used appropriately.