Abstract
Objectives
The development, implementation, and evaluation of an educational intervention to facilitate specialized asthma care provision by community pharmacists.
Design
Formative evaluation and a parallel group repeated measures design were used to test the effect of an educational intervention on pharmacist satisfaction and practice behavior as well as patient outcomes. The educational intervention was based on practitioner needs and principles of adult learning using flexible delivery formats.
Assessment
In the intervention area, 15 pharmacists were trained with the educational intervention, and they provided specialized asthma care to 52 patients over 6 months, while in the control area, 12 pharmacists provided “usual care” to 50 patients. The intervention pharmacists were highly satisfied with the education received and rated most aspects highly. Improvements in patient clinical, humanistic, and economic outcomes in the intervention area were obtained.
Conclusion
The positive results of the educational intervention demonstrate the effectiveness of an educational approach grounded in the theory that inducing behavioral changes in pharmacy practitioners results in improved patient outcomes.
Keywords: community pharmacists, continuing education, asthma care, patient outcomes
INTRODUCTION
Asthma is a significant public health issue worldwide.1 Australia has one of the highest prevalence rates for asthma in the world2 and asthma is ranked as one of the top 10 reasons for hospital and general practitioner visits.3 To combat this, the National Asthma Council (NAC) in Australia developed consensus-based guidelines for the “best practice” management of asthma and much effort was made to disseminate these guidelines to health professionals and the public.4 Despite the efforts of the NAC in propagating their guidelines, research conducted within Australian settings has shown that asthma management is not ideal,5 indicating that some aspects of the NAC guidelines are still not in widespread practice. Guidelines are often not implemented, unless well-designed educational strategies are used to encourage health professionals to understand and adopt them into everyday practice.6 Developing educational strategies that incorporate best practices, the needs of various health care practitioners, and benefit asthma patients is crucial.
Community pharmacists are in a unique position to help patients manage chronic illness in view of their expertise, their regular contact with patients, and their accessibility. In a survey conducted with a sample of 1610 Australian pharmacists by the Victorian College of Pharmacy and the National Asthma Council, however, practicing pharmacists’ knowledge about asthma was not adequate.7 In a qualitative study conducted with pharmacy practitioners in New South Wales, Australia, one of the main needs identified by the respondents was training in asthma care provision.8 Pharmacists in this sample identified areas in which they felt they needed extra training. Most indicated a preference for a mix of self-study and face-to-face training modes. Pharmacists also demonstrated typical characteristics of adult learners by emphasizing that any such training would be effective if it were “practical” and “hands on.” Another earlier study in the same region had also identified pharmacists’ needs for education on asthma.9
Training provided to pharmacists has a long-term effect on their knowledge and attitudes10 and can improve participant's knowledge and the service they in turn provide to their patients.11-13 While many studies record the improvement in knowledge, skills, and attitudes after provision of asthma education to pharmacists, there are none that show the impact of asthma training on pharmacists’ practice change or patient outcomes. In Australia, the NAC has established various recommendations and practice guidelines on the subject of asthma management4; however, there are still gaps between the ideal and reality of what is achieved in community pharmacies. Also, pharmacists have expressed a need for training in asthma to be able to provide specialized care to patients with asthma. This paper describes a study designed to bridge this gap and address the training needs of community pharmacists.
The aim of this study was to develop, implement, and evaluate an educational intervention designed to facilitate the delivery of a specialized asthma care service by community pharmacists. The specific objectives of this study were to:
Examine the training needs, and preferred training format of community pharmacists in order to provide specialized care for people with asthma.
Design an educational intervention that addresses pharmacists’ needs and incorporates principles of good instructional design.
Pilot test the delivery of the educational intervention.
Use the results to finalize an educational intervention for pharmacists planning to deliver a specialized asthma care program
Evaluate the effectiveness of the education intervention in terms of patient outcomes
DESIGN
Consent for the study was obtained from the Human Ethics Committee of the University of Sydney, Australia. All protocols and materials used in the study were developed by the authors, 2 of whom had considerable experience in pharmacy practice research, and 1 who was a doctoral candidate. As per the ethics committee requirements, none of the participants (pharmacists or patients) were compensated in any way for their time or involvement in the study.
To address the first objective, qualitative methods were used. Semi-structured open-ended interviews were conducted with a convenience sample of 17 community pharmacists. The interviews were transcribed verbatim and thematically analyzed. The methods and results of this phase have been described elsewhere.8 Briefly, all the respondents indicated that some sort of training would be necessary for pharmacists to be able to deliver improved asthma services. There were mixed responses to the ideal format of this training, most respondents indicated a desire for face-to-face training as well as some training material that they could use to inform themselves. The knowledge of the latest devices and practical demonstration as to their usage, lung function testing, and latest medications were common concerns.8
A program planning model was utilized step by step to fulfill objectives 2 and 3 (Figure 1). The approach used was to conceptualize planning as a process that consists of a set of points. This type of model allows the planner to address a number of the components simultaneously, to rearrange components, delete unneeded parts, and move directly to relevant ones.14
As the need for designing the educational intervention was evident and none of the available programs were suitable for the purpose of this study, designing the educational intervention was the first action taken (Figure 1). A continuing education (CE) approach was considered suitable for this adult population of learners. The intervention developed was called the Asthma Care Model Training (ACMT) and was registered for continuing education points with the Pharmaceutical Society of Australia (New South Wales Branch). For individual learning, the 2 formats used were self-directed learning and onsite training. For small group learning, workshops were used as the method of choice.
As relevance to practice was a critical issue in the design of the intervention, the Australian 6-step asthma management plan,4 which represented consensus guidelines for current best practice in asthma management by all health professionals and patients with asthma in Australia, was used for all formats of the ACMT.
Bloom's Taxonomy of learning domains was used as the basis for formulating the learning objectives.15 For the self-study component, the objectives were to facilitate basic learning, refresh concepts, and start the process of reflection and critical thinking. Sequentially, the learning objectives of the workshop and onsite training focused on enhanced thinking and analysis, development of psychomotor skills, and changes in beliefs and attitudes. This staged process of setting progressively difficult objectives for participants to achieve through the intervention was used to facilitate behavioral changes in pharmacists.
Available material on asthma and its management was read and reviewed and compiled into a self-study manual that was relevant to community pharmacy practice. The research team ensured that the manuscripts included followed some basic principles of instructional message design and optimal format. Each section of the manual was reviewed by either academic clinicians including 2 respiratory physicians, a pharmacologist, an immunologist, and a psychologist; or by practitioners such as a pharmacist and a medical scientist. Finally, indices such as the Flesch score were used to assess readability.16
The workshops aimed to reinforce the knowledge obtained through the self-study manuals and to use this knowledge to change attitudes and behavioral patterns of people with asthma in everyday practice, in accordance with the 6 steps proposed by the National Asthma Campaign (NAC). The activities included case studies that sequentially addressed all 6 steps of the NAC guidelines, video viewing, role-play, games, skills practice, critical incident sharing, and group discussion.
This was planned as an “ideal asthma service”demonstration in the pharmacy by a trained researcher with a real life patient. After this training session, pharmacists would deliver the service to a patient while they were observed and evaluated by a trained researcher. Afterward, they were given feedback and further training if needed. Project protocols were designed to assist standard service provision to asthma patients and a service evaluation checklist was designed to help researchers assess pharmacist performance.
The ACMT was evaluated using the “levels of evaluation” method proposed by Kirkpatrick (1987).17 This method measures 4 different areas: participant reactions, learning, behavior change, and results or outcomes. For the purposes of the intervention, it was decided that Level 1, 3, and 4 (Figure 1) would be evaluated. To evaluate participant reaction, a questionnaire was developed with a semantic differential scale for various criteria and a section for open-ended comments. An onsite evaluation checklist was developed to evaluate appropriate behavior changes made by participants for delivery of specialized asthma care. The ACMT program was delivered to a “convenience” sample of 6 pharmacists and 3 volunteer patients within their pharmacies (pilot pharmacists and patients). To develop the final version of the ACMT, the results of the questionnaire items, the onsite evaluation checklist, and the informal feedback from the pilot pharmacists were used.
Formative evaluation of the ACMT was carried out using the same questionnaires developed to evaluate the pilot. The onsite evaluation checklist from above was used to assess pharmacists’ practice behaviors when providing a specialized service to asthma patients.
A parallel controlled repeated measures design was used to implement the ACMT program. Two geographically distinct but demographically similar areas in New South Wales, Australia, were selected to serve as an intervention and control area. In the intervention area, pharmacists were trained using the ACMT program designed above and delivered a service modeled on the ACMT program. The service was delivered to recruited patients at baseline, and at 1, 3, and 6 months after baseline (pharmacists and patients in this arm will be referred to as the intervention pharmacists and patients). In the control area, pharmacists were not provided with any training and simply collected data and provided “usual care” at baseline and 6 months after baseline to patients (pharmacists and patients in this arm will be referred to as the control pharmacists and patients).
This model was evaluated based on clinical, humanistic, and economic outcomes compared between and within groups. Details of outcomes measured over the 6-month study included (1) clinical (asthma severity, peak flow indices, mean daily dose of medications, risk of non-adherence, ownership of a written asthma management/action plan); (2) humanistic (asthma-related quality of life, asthma knowledge, perceived control over asthma); and (3) economic outcomes, and these have been published elsewhere.18 A debrief meeting with intervention pharmacists was sought after the project and opinions on the usefulness of the ACMT were obtained. Proceedings of the meeting were tape recorded and transcribed verbatim.
Four self-study manuals were written, reviewed by experts and modified accordingly. The 4 manuals represented specific steps of the Australian Six Step Asthma Management Plan. The distribution of learning outcomes (cognitive, affective and psychomotor) through the 4 manuals is shown in Table 1. The workshops were planned over 2 consecutive days, and a workshop manual containing relevant material was prepared. The workshops had a mix of expert lectures, discussion, and planned activities (Table 2). Workshop trainers consisted of expert academic clinicians, medical practitioners, medical scientists, and the 2 experienced members of the research team. All trainers worked in research and teaching institutions or had a training role in their professional bodies. The workshops also introduced participants to specific tools and documentation sheets for use in service delivery. A service model based on the ACMT was outlined and resources required for this service delivery were provided during the workshops.
Table 1.
*Flow from easy to harder learning objectives
Table 2.
To achieve the third specific aim of this study, a convenience sample of community pharmacists (n = 6) was invited to pilot the training manual and the workshops for the educational intervention. Initially, they were sent the 4 self-study manuals and given 3 weeks to study these manuals. On the basis of the pilot test results the ACMT was finalized. They were then invited to attend the 2-day weekend workshop.
The workshop was held at the Faculty of Pharmacy, University of Sydney in May 1999. The Illawarra and Blue Mountains areas in New South Wales, were selected as the intervention and test areas for the final testing. In the intervention site, 15 pharmacists were recruited over a period of 2 months. Once the pharmacists received the self-study manuals, the workshops were organized to take place 6 weeks later (June 2000). All participants were asked to complete a questionnaire regarding the self-study manuals before the workshops.
ASSESSMENT
For evaluating the self-study manuals, participant reactions were ordered in terms of section content and process, relevance to practice, presenter skills, and logistic arrangements (Scale of 1-7), using a questionnaire. The self-study manuals scored well on most criteria. For each category most of the respondents rated the workshop close to the maximum.7 In the case of length, a mean score of 4.5 ± 1.6 (1= too short, 7= too long) suggested that perhaps a little less information was required. In the case of the workshops, the feedback was ordered in terms of relevance of workshop activities, excellence of talks and resources, structure and length of sessions, and overall experience. The length of the sessions was the only aspect that did not score optimally (mean score 5.1 ± 1.1).
In response to the open-ended questions which focused on what participants liked most or least about the workshop, pharmacists indicated a number of factors that they had liked about the workshops. These included the simplicity of the formats, relevance of all the activities, the talk on medications by the invited physician, the practice session with the devices, the session on communication and counseling skills, and the relaxed atmosphere in which the workshops had been conducted. The length of the sessions was the issue most participating pharmacists reported as having liked the least about the workshops.
Of the 6 participants, one was “very confident,” 4 were “confident” and one was “okay” in terms of being able to provide specialized care for future asthma patients. Participants also indicated that facilities provided to them at the workshop had been very good and had felt “well looked after.” Hence, following on from the pilot, the only change made was to shorten the workshop sessions. In addition, the workshops were divided into 3 sessions, ie, 1 full day and 2 half days, to make it less tedious for participants.
The 15 intervention pharmacists attended the workshops. Two pharmacists missed one session and others missed portions of the workshops because they were delayed at work, but all 15 attended the last day of the workshop. The results from the feedback questionnaire (Table 3) show that the intervention pharmacists rated the self-study manuals quite highly on all criteria. The criterion with the highest score was readability or reader friendliness, followed by content relevance and writing style. There were no significant differences in these scores compared with the scores from the pilot pharmacists.
Table 3.
In the case of the workshop evaluation, the only difference in scores that was significant between the intervention and the pilot pharmacists was that of session length (p = 0.03). Session length score improved in the intervention pharmacists from 5.1 ± 1.1 to 3.9 ± 1.0 (with 3.5 being the ideal score between 1= too short and 7 = too long) (Figure 2).
When the intervention pharmacists were asked what they had liked least or most and how further improvements could be made, they described the self-study manuals as “well presented” and “easy to understand.” The case studies were found to be “real” and “relevant” by most pharmacists who commented on these. A couple of pharmacists wrote that the self-study manuals were too long. One pharmacist found the manuals “plain.” Provision of more diagrams and illustrations was a suggestion for improvement made by one pharmacist, while another pharmacist suggested provision of more self-assessment exercises.
Regarding the workshops, all pharmacists mentioned the practical session on devices as one of the workshop sessions they liked most. “Practical” and “hands on” were the attributes cited by some pharmacists for the device section. The presentation of advantages and disadvantages of available devices was “useful information” as written by one pharmacist. Role-plays, activities, and group work were liked by most pharmacists. Some comments included “The workshops were a jewel of intense education.” “The whole process was a refresher checkpoint. The information provided was simple, straightforward and most importantly applicable to everyday situations.” “Up to date information on all aspects of asthma therapy, summarized well.” Suggestions for improvement included having more practical work sessions and communication skill role-plays, information on alternative treatments for asthma, and advanced communication skill techniques.
Many of the pharmacists commented that it would be useful to provide training to pharmacy assistant staff as they would be the frontline people recruiting patients for the asthma care model. This suggestion was taken up immediately by arranging a 2-hour “asthma information evening” session for pharmacy assistant staff members. Five out of 10 pharmacies sent pharmacy assistants to the evening; 17 pharmacy assistants attended the asthma information evening. In developing this session, the same principles were followed as with the pharmacist workshops. A workshop manual was prepared for each pharmacy assistant and the information evening was structured around the 6 steps of the asthma management plan. The pharmacy assistants’ information session was not formally evaluated.
In case of the onsite evaluation, the 15 intervention pharmacists who had undergone the training belonged to 10 pharmacies. Two of the pharmacies proceeded no further with the project as they failed to recruit any patients. Of the other 8 pharmacies, each nominated 1 pharmacist member who would undertake main responsibility for the project and set up the onsite training appointments with the researchers. This was done so that the other pharmacist could continue the regular activities of the pharmacy while their colleague undertook onsite training. Of these 8, 6 pharmacists felt that they did not require onsite training and would proceed to arrange for the researchers to “sit in” on a patient interview. Two pharmacists made “onsite training appointments” for the first patient they recruited. The main researcher evaluated the patient interview and pharmacist's service delivery using an evaluation checklist. Satisfactory performance on each criterion was considered if the pharmacist under evaluation actually performed that step.
All pharmacists, with the exception of one, were rated satisfactory on most criteria on this checklist (Table 4). This pharmacist, who did not follow essential service protocols (such as checking patient inhaler technique, counseling patients, documenting patient assessment, etc) even after feedback was dropped from the study.
Table 4.
The issues that arose from the service evaluation were that documentation and collaborative goal setting were procedures with which pharmacists had difficulty. Immediate feedback about possible ways to overcome this was provided. The researcher discussed difficulties encountered by the pharmacists around the 2 issues of goal setting and documentation. Possible solutions were jointly investigated. Documentation seemed to be more of a problem with lack of time. However, the importance of documentation was reinforced to pharmacists.
The intervention (n = 15) and control pharmacists (n = 12) recruited 52 and 50 patients, respectively, with asthma. In the intervention area, pharmacists were trained using the ACMT, and delivered an asthma care model, with 3 follow-up visits over 6 months, while control pharmacists provided usual care without the benefit of receiving the ACMT.
The results of the outcomes measured have been reported in depth elsewhere,18 but in summary, there was a significant reduction in asthma severity in the intervention group, 2.6 ± 0.5 to 1.6 ± 0.7 (mean ± SD; p < 0.001) versus the control group, 2.3 ± 0.7 to 2.4 ± 0.5. In the intervention group, peak flow indices improved from 82.7% ± 8.2% at baseline to 87.4% ± 8.9% (p < 0.001) at the final visit, and there was a significant reduction in the defined daily dose of albuterol used by patients, from 374.8 ± 314.8 mcg at baseline to 198.4 ± 196.9 mcg at the final visit (p < 0.015). There was also a significant improvement in perceived control of asthma and asthma-related knowledge scores in the intervention group compared with the control group between baseline and the final visit. In the intervention group, annual savings of $132.84(AU) in medication costs per patient and savings of $100,801.20 based on overall severity reduction for the whole group were demonstrated. Thus, the ACMT may be considered as having been successful in training community pharmacists to improve patient outcomes in asthma.
Seven pharmacists attended this meeting, which was facilitated by a researcher with whom they were not familiar. The meeting was held in November 2001. Once again, pharmacists reported that the educational intervention had been very useful in helping them deliver specialized asthma care to their patients.
DISCUSSION
In this study, we developed and successfully implemented an educational intervention to facilitate pharmacists to deliver specialized care for people with asthma. The education intervention designed was positively evaluated by the pharmacists and pharmacists changed their practice behavior from being nonproviders of specialized asthma care to providers of such services. In the case of asthma, this is the first reported pharmacy educational intervention that has been designed and implemented according to practitioner needs and national asthma management guidelines, evaluated by practitioners, and shown to be effective in terms of improving patient outcomes.
The ACMT was developed on the basis of practitioner need and educational principles. The researchers paid keen attention to the development of learning objectives, ensuring that strategies were in place to align the structure of the ACMT and the objectives and that the instructional materials used were learner friendly and contextual. The framing of the ACMT and the specialized service delivery subsequent to the ACMT around National Guidelines is a unique strength, as it allowed pharmacist practitioners to comprehensively understand and provide “best practice” asthma care.
According to the principles of adult learning, learning should be self-directed, collegial, and reflective19,20; none of these alone is sufficient. The ACMT allowed all 3 processes by using self-study manuals, interactive workshops, and onsite training to allow reflection on practice. The activities subsequent to undertaking training consisted of pharmacists actually providing asthma care to recruited patients which allowed learning to continue in context.
Health professionals are highly motivated to learn if the learning seems useful, particularly in the short term, and if it occurs within practice settings.19 Interestingly, within the workshops, a two-way learning process occurred. Participants within the group discussion shared valuable practice experience. At some stages, “live problem solving” occurred when one practitioner would say “I have a patient who…” and outline a problem scenario. Other pharmacists in the group would respond to this problem posed by their peer and come up with possible solutions, ie, “What I would do in your case…” Some of these suggestions were innovative and provided the research team with deeper insights into the realities of community practice. Some of the counseling tips shared by the pharmacists were actually used by the facilitating researcher at later stages when dealing with patients. The role-plays and group discussion also proved to be an excellent medium for instant feedback.
The principles of continuous quality improvement were followed21 as the formative evaluation with the pilot pharmacists was used to improve aspects of the workshop. The research team made a deliberate effort to act on feedback; this is reflected in the improvement in scores given to the length of self-study manuals by the intervention pharmacists compared with the pilot pharmacists. Using the continuous quality approach, the improvements to the educational intervention were not ceased after the formative evaluation, but continued throughout. At the last session, intervention pharmacists suggested that it would have been useful to provide training for pharmacy assistant staff members. While this had not been planned by the research team, it was immediately put in place and a date fixed with the pharmacists to send their staff members. Pharmacists were then called and asked what they would like to have included in the pharmacy assistant training. This was a productive exercise as researchers later realized that the pharmacy assistant staff members were involved in the recruitment of patients.
The decision to use the “participant reaction” and “on site observation” type of evaluation rather than a pre- and post-intervention knowledge/skill test was also crucial. The philosophical foundation of the educational intervention was not merely to enhance knowledge or skills, but to make the learning experience satisfactory so that the participants would become lifelong learners. In keeping with this philosophy, it would be inappropriate to measure knowledge at one point, as this would not be a measure of whether the participants maintained this knowledge and interest. Participation in group activities would serve as a better indication of learning that had occurred before and during the workshop and the impact of the application of this learning in practice would be demonstrable through any impact on patient health outcomes that would be monitored at later stages of the study.
The study has described in depth the learning objectives, content, format and design issues that were used in the ACMT. One of the drawbacks of educational intervention studies is often that the actual course or program development is not described. In addition, while many continuing education programs in the medical field have shown the benefits of physician training on patient outcomes,22 and also specifically for asthma,23,24 not many pharmacy-based studies on the design of continuing education and the effect of pharmacist training on patient outcomes have yielded such data. A few studies show the impact of lengthy pharmaceutical care certificate programs,25,26 but CE programs have not been assessed often in terms of practice change or patient outcomes. It is perhaps time for pharmacy educators to consider this issue and address it. The ACMT represents a way forward in this direction.
Limitations of this study include a small number of pharmacists and absence of measures of pretest-posttest increase in knowledge, skills, and attitudes. The intervention pharmacists’ activities were also not monitored after the project; hence, maintenance of practice behavior change in these pharmacists is also not known.
While it is not feasible that all community pharmacists choose to provide specialized asthma services, for those that express interest in asthma, an evidence-based educational program should be provided. Undergraduate students may also be provided with specialized training that equips them to improve asthma outcomes when they commence as pharmacy practitioners. In Australia, the onus of providing specialized asthma education is handled by state-based bodies known as the Asthma Educators Associations. These Associations have competency requirements for the performance of those allied health professionals who choose to engage in asthma education. It may bepertinent to consider a special asthma education certificate course for pharmacists that uses the elements of the educational intervention described in this paper and competencies required by the Asthma Educators Associations.
CONCLUSION
The positive effect of this asthma educational training on pharmacists practice behavior and asthma patients’ outcomes indicates that careful design of educational training can have a substantial impact. The training should be tested in a larger sample and for sustainability of results.
Acknowledgments
Funding for this project was provided by the Pharmacy Research Trust of NSW, Australia, and the Pharmacy Board of NSW, Australia.
REFERENCES
- 1. The Global Initiative for Asthma. Workshop Report, Global Strategy for Asthma Management and Prevention. October 2005. Available at: http://www.ginasthma.com/GuidelinesResources. Accessed January 12, 2006.
- 2. The Australian Institute of Health and Welfare (AIHW), Australian Centre for asthma monitoring 2005. Asthma in Australia 2005. Asthma series 2. AIHW Catalog No ACM 6. Canberra:AIHW:13, 41-3.
- 3.Britt H, Miller GC, Knox S, et al. Canberra: AIHW; General practice activity in Australia 2003-2004 AIHW Catalog No GEP 16. [Google Scholar]
- 4.Woolcock AJ, Rubinfeld AR, Seale P. Asthma management plan-1989. Med J Aust. 1989;151:650–3. [PubMed] [Google Scholar]
- 5.Cerveri I, Locatelli F, Zoia MC, Corsico A, Accordini S, de Marco R. International variations in asthma treatment compliance. The Results of the European Community Respiratory Health Survey (ECRHS) Eur Respir J. 1999;14:288–94. doi: 10.1034/j.1399-3003.1999.14b09.x. [DOI] [PubMed] [Google Scholar]
- 6.Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993;342:1317–22. doi: 10.1016/0140-6736(93)92244-n. [DOI] [PubMed] [Google Scholar]
- 7.Gattera J, Abramson M, Roller L. Knowledge of asthma symptoms, preventer medications and asthma counseling practices of Australian community pharmacists. Part 2. Aus J Pharm. 1998;79:976–82. [Google Scholar]
- 8.Saini B, Krass I, Armour C. Specialisation in asthma: Current practice and future roles. J Soc Admin Pharm. 2001;18:169–77. [Google Scholar]
- 9.Comino EJ, Ruffin R, Bauman A. Asthma management survey of Australian community pharmacists. Aus J Pharm. 1992;73:59–61. [Google Scholar]
- 10.Sinclair HK, Bond CM, Lennox SA. The long term learning effect of training in stage of change for smoking cessation: a three year follow up of community pharmacy staff's knowledge and attitudes. Int J Pharm Pract. 1999;7:1–11. [Google Scholar]
- 11.Monaghan MS, Turner PD, Skrabal MZ. Evaluating the format and effectiveness of a disease state management training program for diabetes. Am J Pharm Educ. 2000;64:181–96. [Google Scholar]
- 12.Osman LM, Bond CM, Mackenzie J, Williams S. Asthma advice giving by community pharmacists. Int J Pharm Prac. 1999;7:12–7. [Google Scholar]
- 13.Shen SH. Assessing pharmacists' interests in receiving continuing education (CE) programs in Tao-Yuan, Taiwan. Am Soc Hosp Pharm. 2005;62:P46D. [Google Scholar]
- 14.Cafarella R. Planning Programs for Adult Learners: A Practical Guide for Educators, Trainers, and Staff Developers. 2nd edition. San Francisco, Calif: Jossey-Bass; 2002. [Google Scholar]
- 15.Krathwohl DR, Bloom BS, Bertram BM. Taxonomy of Educational Objectives, the Classification of Educational Goals. Handbook II: Affective Domain. New York: David McKay Co., Inc; 1973. [Google Scholar]
- 16.Graham W. Readability and science textbooks. School Sci Rev. 1978;208:545–50. [Google Scholar]
- 17.Terry K. Grabbing the bandwagon of change. Med Econ. 1994;70:120–5. [PubMed] [Google Scholar]
- 18.Saini B, Krass I, Armour C. Development, implementation and evaluation of a community pharmacy based asthma care model. Ann Pharmacother. 2004;38:1954–60. doi: 10.1345/aph.1E045. [DOI] [PubMed] [Google Scholar]
- 19.Frankford DM, Patterson MA, Konrad TR. Transforming practice organisations to foster lifelong learning and commitment to medical professionalism. Acad Med. 2000;75:708–17. doi: 10.1097/00001888-200007000-00012. [DOI] [PubMed] [Google Scholar]
- 20.Knowles M. Andragogy: an emerging technology for adult learning. In: Edwards R, Hansen A, Raggatt P, editors. Boundaries of adult learning. New York: Routledge; 1996. pp. 82–98. [Google Scholar]
- 21.Bounds G, Yorks L, Adams M, et al. McGraw Hill International Editions; 1995. Beyond Total Quality Management - Towards the Emerging Paradigm; pp. 29–30. [Google Scholar]
- 22.Davis D. Does CME work? An analysis of the effect of educational activities on physician performance or health care outcomes. Intl J Psychiatr Med. 1998;28:21–39. doi: 10.2190/UA3R-JX9W-MHR5-RC81. [DOI] [PubMed] [Google Scholar]
- 23.Veninga CCM, Lagerlov P, Whalstrom R. Evaluating an educational intervention to improve the treatment of asthma in four European countries. Am J Resp Crit Care Med. 1999;160:1254–62. doi: 10.1164/ajrccm.160.4.9812136. [DOI] [PubMed] [Google Scholar]
- 24.Clark NM, Gong M, Schork A. Impact of education for physicians on patient outcomes. Pediatrics. 1998;101:831–6. doi: 10.1542/peds.101.5.831. [DOI] [PubMed] [Google Scholar]
- 25.Patterspn BD. Distance education in a rural state. Assessing change in pharmacy practice as a result of the pharmaceutical care certificate program. Am J Pharm Educ. 1999;63:56–63. [Google Scholar]
- 26.Barner JC, Bennett RW. Pharmaceutical care certificate program. Assessment of pharmacist's implementation into practice. J Am Pharm Assoc. 1999;39:362–7. doi: 10.1016/s1086-5802(16)30452-1. [DOI] [PubMed] [Google Scholar]