Abstract
Background:
Drugs with fiscalized substances without a correct prescription may lead to undesirable side effects. Pharmacy staff needs to improve their competencies (knowledge, skills, and attitudes) to contribute to providing ambulatory pharmacy services and minimizing medication errors. Continuing education programs (CEP) could favor access to relevant and quality information on health promotion, disease prevention, and the rational use of drugs.
Objective:
To evaluate the effectiveness of a continuing education program to improve pharmacy staff competencies to enhance the use of drugs with fiscalized substances.
Methods:
A multicenter, prospective, parallel-group, cluster-randomized, controlled clinical trial was conducted in drugstores and pharmacies in Colombia (ambulatory retail establishments). The intervention group (IG) received a CEP: a web-based social networking site, a virtual course, a dispensing information system, and face-to-face training. The control group (CG) received general written material on the correct use of drugs. We measured pharmacy staff’s skills, attitudes, and knowledge self-reported scores, and the simulated patient technique was used to assess the participant skills and attitudes in real practice. We used a questionnaire designed for this study, which was evaluated by a group of experts and piloted and showed a Cronbach’s alpha of 0.96.
Results:
Three hundred five drugstores and pharmacies were enrolled in two groups: IG (n = 153) and CG (n = 152). Out of the 750 potential participants, 88% (n=659) agreed to participate. The pharmacy staff’s skills, attitudes, and knowledge self-reported scores post-intervention were higher than baseline in both groups; however, the IG had statistically significantly higher scores than the CG. Post-intervention, the self-efficacy skills and attitudes in the IG improved by 88% (22 of 25) and in six of the seven assessed knowledge components (p<0.001). However, the dispensing criteria evaluated with simulated patient methodology showed no statistically significant differences between groups in the pharmacy staff’s skills and attitudes in real practice.
Conclusions:
Providing a continuing education program using different educational strategies improved the pharmacy staff’s competencies (assessed knowledge and self-reported skills and attitudes) to enhance the use of drugs with fiscalized substances. However, there were no improvements in skills and attitudes in real practice. These findings could show that pharmacy staff needs additional and continuous training/sustainability.
Keywords: education pharmacy continuing, pharmacies, pharmacist, pharmacy technicians, pharmaceutical services
INTRODUCTION
Problems in the drugs processes associated with drug utilization and results in the patient´s health are usual challenges faced by pharmacy staff during their usual practice. Therefore, pharmacy staff requires training, qualification, and continuing education programs focused on their competencies to improve the rational use of drugs, minimize medication errors, and help outpatients use their drug therapy.1 Competencies are the knowledge, skills, and attitudes that health care human talent requires in the different areas of practice to solve health problems efficiently and effectively. It could be achieved through education, training, and experience.2 In this context, continuing education and communication with other health professionals are key to drug rational use in all steps involve the stages in the medication use process.3
In the Colombian context, drugstores and pharmacies are private healthcare ambulatory retail establishments dedicated to sell allopathic, herbal, homeopathic products, cosmetics, personal hygiene products, medical devices, and dietary supplements. In 2003, there were 14,208 drugstores and around 1,000 pharmacies; however, by 2014, this number had increased to more than 20,000, mainly by an increase of drugstores. The ambulatory pharmacy staff is frequently the first healthcare provider with whom a patient comes into contact before using a medication, principally because the drugstores and pharmacies are more accessible to the public and are an essential component of primary health care. In addition, in Colombia, the pharmacy staff can work in these establishments with any of the following training levels: university formation (professional and technician) and without university formation (technical/auxiliary or with and without a certificate). Although the profiles, training areas, and activities of each pharmacy staff has been established in the regulations, only for the technicians there are specific labor competencies standards. So, they could be considered as the minimum competencies for all pharmacy staff.4
Drugs with fiscalized substances, such as benzodiazepines, tricyclic antidepressants, anxiolytics, antipsychotics, and opioid analgesics, are essential for the pharmacotherapy of pain, obstetric emergencies, mental disorders, and drug-dependent substances. Fiscalized substances are regulated by international conventions on drug control, recognizing their potential to cause harm, such as dependency syndrome and health disorders.5 In addition, there are failures in the surveillance and the control of these drugs in healthcare practice; for instance, in Colombia, some fiscalized drugs are dispensed without medical prescriptions.4,6 Nevertheless, there is evidence of the effectiveness and relevance of pharmaceutical interventions combined with Information and Communication Technologies (ICT), continuing education programs, and network support to improve the use of drugs with fiscalized substances by pharmacy staff and other health professionals.7-9
The impact of continuing education on pharmacy staff has been described, focusing efforts and strategies to promote rational use of drugs, including opioids, antidepressants, anxiolytics, and even preventing these medications abuse. These studies were carried out in high-income countries, combined face-to-face intervention delivery methodologies, and used live media methods.10-14 Differences between study designs, educational interventions, diseases treated, and working and academic conditions of pharmacy staff do not allow results extrapolation from high-income countries to low-middle countries. In addition, there is a need to standardize the terminologies of continuing education programs, the techniques and methods for evaluating the effectiveness/impact of educational programs and pharmacist interventions, and the comparison of different methods and strategies. In most studies, the competencies are among the most evaluated outcomes and other aspects such as satisfaction with the intervention and the trust generated in them.15 Also, some authors have recommended evaluating continuing education programs to compare and identify the most effective strategies; for this task in the intervened groups, some issues such as competencies, satisfaction, learning results, changes in performance, and behaviors could be assessed.16
Worldwide, pharmacy staff utilizes several options to update their competencies, mainly through congresses, seminars, conferences, short courses, workshops outside of work hours, and continuous education programs.17 It has been shown both, self-directed learning programs are commonly used and that formal continuing education itself does not facilitate changes in the practice. Currently, some countries are implementing mandatory continuing education programs to improve healthcare services for pharmacists.16 However, to our knowledge there is no information available regarding studies related to the competencies that pharmacy staff should have in providing ambulatory pharmacy services of drugs with fiscalized substances. Therefore, the objective of this study was to evaluate the effectiveness of a continuing education program to improve pharmacy staff competencies to enhance the ambulatory pharmacy services: dispensation, health education, and pharmacovigilance of drugs with fiscalized substances.
METHODS
The trial protocol was registered on ClinicalTrials.gov (Identifier NCT03388567), and the study methods were detailed in a previous publication of the study protocol.18
Study design and setting
A multicenter, prospective, parallel-group, cluster-randomized, controlled clinical trial was conducted in Medellin and the Metropolitan Area (Colombia) drugstores and pharmacies (Figure 1). We developed a 12-month intervention delivery and follow-up (September 2018 to September 2019). Drugstores and pharmacies (the clusters) were enrolled and randomized in a 1:1 ratio in the intervention or control group. This trial was reported based on the CONSORT statement checklist.
Figure 1.
The continuing education program process and methodology
Study population and recruitment
A cross-sectional epidemiological study on utilizing drugs with fiscalized substances was conducted to identify potential drugstores or pharmacies for this study.19 Pharmacy staff was invited those who were already working in drugstores or pharmacies to participate in this trial, including at least one member from each drugstore or pharmacy.
Sample size
The continuing education program was expected to improve the pharmacy staff competencies in the intervention group by at least 30% and in the control group by 15%. The educational intervention findings increase 37% of the competencies.20 Therefore, with a confidence level of 95% and a power of 80%, at least 121 drugstores and pharmacies were calculated in each group. Furthermore, the sample was added on by 10% to provide losses to follow-up or exclusions from analysis, increasing the sensitivity. Finally, the sample size was 135 drugstores and pharmacies in each group.
Allocation and blinding
The neighborhoods where the drugstores and pharmacies were geographically located (clusters) were randomly stratified. Drugstores and pharmacies were randomly selected from a sample frame previously constructed as a result of a previously conducted descriptive observational study.19 The allocation of the drugstores and pharmacies to the respective group (intervention or control) was blinded, and at least one person from each establishment entered the study. Due to the nature of the trial, it was not possible to blind pharmacy staff at drugstores and pharmacies (the clusters). The simulated patients did not know the allocation of the pharmacy or drugstore that they were visiting, thus controlling for selection bias. Also, to control information bias, at the end of the respective visit to the pharmacy or drugstore, the simulated patients immediately filled out the used checklist for the evaluation. One of the researchers transported all the simulated patients and supervised the immediate completion of the information. The stratified neighborhoods, groups (intervention and control), and simulated patients were randomized using a computer-generated random number table and placed in numbers, designed by a person external to the study using Microsoft Excel® (version 2010, Microsoft®). The statistical analysis was also performed unblended.
Intervention group
Continuing education program
The continuing education program was made with different strategies described in Table 1. These actions were executed transversely during their implementation and follow-up (12 months: September 2018 to September 2019).
Table 1.
Continuing education program content
| Strategy | Web-based social networking site | Virtual course | Drug information and dispensing software | Pharmacy staff training |
|---|---|---|---|---|
| Actions | Pharmacy staff built social relationships between them and communicated with the researchers. | Update course on the proper utilization of drugs with fiscalized substances (seven modules). | A software with three specific components: dispensing, pharmacovigilance, and pharmaceutical care. | Five face-to-face training sessions. |
Web-based social networking site and access to a virtual classroom
The Web-based social networking sites’ access and interactivity were continuous 24 hours a day, seven days a week, and the pharmacy staff had the option to access via computer (desktop and laptop) and phone (smartphone or tablet).
Virtual course
The virtual course was designed using Moodle 3.3.2+ and composed of 7 modules: 1) Pharmacy terminology and basic pharmacology, 2) Pharmacy legislation, 3) Administrative and technical activities in the pharmacy, 4) Communication and health education from the pharmacy, 5) Good practices of dispensation and correct use of medicines, 6) Pharmacovigilance from the pharmacy, and 7) Pharmacist actuation in depression, anxiety, sleep disorders, and pain management. Each module’s study time required an average of 20 hours (10 hours of synchronous and asynchronous online learning) for a time total of 140 hours. Pharmacy staff that completed all virtual course requirements received an Universidad de Antioquia certification.
Drug information and dispensing software
The software had three specific components: dispensing, pharmacovigilance, and pharmaceutical care. Once the pharmacy staff entered the patients’ information attended to the pharmacies and drugstores, the software provided information related to the correct use of medicines.
Pharmacy staff training
All the intervention group pharmacy staff were invited to participate in five face-to-face training sessions to complement the virtual course contents. Each training session required about 4 hours, for a total of 20 hours.
Control group
Written material
The control group (CG) received general written material on the correct use of drugs.
Strategies to improve adherence to the protocol
The pharmacy staff received regular phone calls from one of the researchers to encourage compliance with the protocol, solve questions, and provide technical support.
Outcomes
Pharmacy staff competencies (knowledge, skills, and attitudes) related to using drugs with fiscalized substances.
Data collection instruments
The pharmacy staff self-efficacy competencies assessment was done with two self-applied questionnaires. One questionnaire of 50 multiple-choice questions for the knowledge, and another questionnaire related to the skills and attitudes of 100 questions with four possible options (never, sometimes, usually, or always). The 100 skills and attitudes performance indicators were categorized into 25 different competencies. Outcome measures were conducted at baseline (T0) and 12 months (1-year) after completing the follow-up (T12).
To identified research instruments for measuring the pharmacy staff competencies we conducted a literature review in PubMed until 2019, using “Pharmacies” OR “Community pharmacy” OR “Pharmaceutical Services” OR “Pharmacist” OR “Pharmacy Staff” AND “Education, Pharmacy, Continuing” OR “Competence development” OR “Training” OR “Postgraduate education” OR “Recurrent education” OR “Adult learning” OR “Training and education” OR “Competence” OR “Continuing education” OR “Continuing Professional Development” OR “Life-long Learning” OR “Practical work training” OR “Health Knowledge, Attitudes, Practice”, as MeSH terms and title or abstract.21 However, the review did not identify an adequate research instrument for this goal, and therefore it was necessary to develop the instruments following a methodology previously published: a) bibliographic review, b) design of questions, c) evaluation by experts, d) pilot test, and reliability test.22 Initially, we performed a bibliographic review in PubMed/Medline using MeSH terms related to pharmacies and competencies in the measurement and evaluation of competencies in pharmacy staff.15 Additionally, it was complemented with a free search in Academic Google to review and select the functions, tasks, activities, elements, and criteria of performance of the competencies. The websites of organizations, associations, and working groups worldwide that guide and provide guidelines for the pharmaceutical profession were consulted. Then, the competencies found in the search were grouped according to domains (functions), elements, performance criteria, and competency frameworks. Next, the questions were designed in an appropriate language leaving out necessary technical terms. The measurement instrument was reviewed, and adjusted by a group of experts finally, the questionnaire was subjected to a pilot test with ten pharmacy staff. After the pilot test, the observations, suggestions, and necessary adjustments to the questionnaire were reviewed and approved by the same group of experts.21 Reliability was assessed using Cronbach’s alpha, whose calculated value was 0.96.
Simulated patient technique
The simulated patient methodology was implemented to assess the pharmacy staff’s skills and attitudes in the dispensation and the information provided for the correct use of tramadol.23 A simulated patient scenario of back pain requiring tramadol treatment was used to assess the actual dispensing practice following the Good Dispensing Practices.24 The ideal dispensing process or gold standard was related to the recommendations for the correct use of the drugs (dose, reconstitution, and frequency, with the information and education provided (minimum precautions, recommendations, side effects, and other common risks, proper storage, and treatment adherence). If the final result of the simulated patient achieved between 70-100% of these guidelines, the process was considered good; if it achieved between 40-70%, it was considered fair and <40% bad.25
Statistical methods
Baseline and demographic characteristics were analyzed descriptively. Wilcoxon sign rank test was used to compare pre-post changes knowledge scores. Mann-Whitney U test assessed whether the distribution of skills and attitudes medians in each group was statistically significant, and McNemar’s test evaluated differences between pre-and post-intervention. All statistical tests were computed using the Statistical Package for the Social Sciences (version 23; SPSS, Chicago, Ill., USA) with significance defined as a p<0.05.
Ethics approval and consent to participate
This study was approved by the Ethics Committee of the National School of Public Health at the University of Antioquia, Colombia, at the 123rd session. It was conducted following the principles of the Declaration of Helsinki. Written informed consent was take from all the participants (pharmacy staff, patients or caregivers, and simulated patients) before beginning the trial.
RESULTS
Baseline characteristics
Three hundred five drugstores and pharmacies were randomly selected and assigned to each group (153 in the intervention group and 152 in the control group). Among the 305 drugstores and pharmacies, 83% (253) were sited in low-income neighborhoods. Among the 749-pharmacy staff, 88% (659) agreed to participate; then, 379 were assigned to the intervention group and 280 to the control group. The mean age of participants was 35.6 years (SD 11.7), 65.3% (430) were women, the mean work experience as pharmacy staff was 10.9 years (SD 10.14), and 70% (462) did not have university formation. No differences were found in the sociodemographic variables between the two groups (Table 2).
Table 2.
Pharmacy staff sociodemographic characteristics
| Total (n=659) | Intervention (n=379) | Control (n=280) | P-value | |
|---|---|---|---|---|
| Gender, n (%) | ||||
| Female | 430 (65.3) | 244 (64.4) | 186 (66.4) | 0.585 |
| Male | 229 (34.7) | 135 (35.6) | 94 (33.6) | |
| Age, mean (SD) | 35.6 (11.7) | 35.1 (11.6) | 36.3 (11.9) | 0.191 |
| Categories by age group, n (%) | ||||
| 18 to 24 | 127 (19.3) | 79 (20.8) | 48 (17.1) | 0.620 |
| 25 to 34 | 218 (33.1) | 127 (33.5) | 91 (32.5) | |
| 35 to 44 | 148 (22.5) | 83 (21.9) | 65 (23.2) | |
| 45 to 54 | 112 (17) | 63 (16.6) | 49 (17.5) | |
| 55 to 74 | 54 (8.2) | 27 (7.1) | 27 (9.6) | |
| Pharmacy role, n (%) | ||||
| Employee | 457 (69.3) | 272 (71.8) | 185 (66.1) | 0.117 |
| Owner | 202 (30.7) | 107 (28.2) | 95 (33.9) | |
| Educational level, n (%) | ||||
| University formation: professional (pharmacist) or technician degree | 197 (30) | 116 (30.6) | 81 (29) | 0.667 |
| Without university formation: Technical/auxiliary or with and without a certificate | 462 (70) | 263 (69.4) | 199 (71) | |
| Working experience time (years), average (SD) | 10.9 (10.14) | 10.3 (9.62) | 11.7 (10.76) | 0.207 |
| Hours a day worked, average (SD) | 8.9 (2.2) | 8.7 (2.2) | 9.1 (2.5) | 0.065 |
| Time working at PE (years), average (SD) | 4.74 (60.4) | 4.13 (5.09) | 5.56 (7.04) | 0.067 |
| Lives in the same neighborhood of PE, n (%) | 289 (43.9) | 171 (45.1) | 118 (42.1) | 0.475 |
| Continuing education in life, n (%) | 149 (22.6) | 94 (24.5) | 55 (19.6) | 0.118 |
| Continuing education last year, n (%) | 208 (31.6) | 122 (32.2) | 86 (30.7) | 0.735 |
PE: Pharmaceutical establishment; SD: Standard deviation.
At baseline, regarding the skills, attitudes, and knowledge scores for every group, the results showed no statistically significant differences between the study groups in 96% (24) of the skills and attitudes competencies (Table 3). In addition, in 86% (43) of the 50 knowledge scores, the correct answers were not statistically different between the control and intervention groups (Table 4). Participants were excluded from the analysis for three main reasons: voluntary retirement, resignation or non-continuation of work activities in the establishments, non-compliance with the seven modules of the virtual course, which consequently, did not allow completion of the final evaluation (Figure 1).
Table 3.
Evaluation of pharmacy staff competencies (skills and attitudes)
| Initial measurement | Total (before and after) | Intervention (before and after) | Control (before and after | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Intervention (n=379) (Median – IQ) | Control (n=280) (Median – IQ) | P- value* | Intervention (n=193) (Median – IQ) | Control (n=179) (Median – IQ) | P- value* | Dif. Median | P-Value** | Dif. Medium | P-Value** | Dif. Medium | P-Value** | |
| Domain 1. Provision of patient care. | ||||||||||||
| Initial contact with the patient | 2.9 (2.6-3.1) | 3 (2.5-3.4) | 0.089 | 3.4 (3-3.6) | 3.1 (2.8-3.5) | 0.002 | 0.4 | 0.000 | 0.5 | 0.000 | 0.1 | 0.000 |
| Prescription (medical formula) | 3.8 (3.3-4) | 3.8 (3.15-4) | 0.243 | 4 (3.8-4) | 4 (3.5-4) | 0.060 | 0.2 | 0.000 | 0.2 | 0.000 | 0.2 | 0.000 |
| Need for patient medications | 3.5 (3-4) | 3.5 (3-4) | 0.518 | 3.5 (3-4) | 3.5 (3-4) | 0.616 | 0.0 | 0.000 | 0.0 | 0.000 | 0.0 | 0.001 |
| Drug reconciliation | 2.5 (2-3) | 2.5 (2-3) | 0.149 | 3 (2.5-4) | 3 (2.5-3.5) | 0.196 | 0.5 | 0.000 | 0.5 | 0.000 | 0.5 | 0.000 |
| Supply of medicines | 3.8 (3.5-4) | 4 (3.5-4) | 0.087 | 4 (3.8-4) | 4 (3.8-4) | 0.199 | 0.2 | 0.000 | 0.2 | 0.000 | 0.0 | 0.062 |
| Using guides and protocols | 2.5 (3-4) | 3 (3-4) | 0.019 | 3.5 (3-4) | 3 (3-4) | 0.741 | 1.0 | 0.000 | 0.0 | 0.000 | 0.0 | 0.609 |
| Drug specifications | 3.4 (3-3.9) | 3.65 (3-3.9) | 0.158 | 3.9 (3.6-4) | 3.7 (3.3-4) | 0.063 | 0.3 | 0.000 | 0.5 | 0.000 | 0.0 | 0.003 |
| Drug interactions | 3 (2-3) | 3 (2-3) | 0.654 | 3 (2-4) | 3 (2-4) | 0.448 | 0.0 | 0.000 | 0.0 | 0.001 | 0.0 | 0.000 |
| Drug information and patient education | 2.5 (2.2-2.8) | 2.5 (2.1-2.8) | 0.238 | 2.9 (2.6-3.3) | 2.6 (2.3-3.1) | 0.000 | 0.3 | 0.000 | 0.4 | 0.000 | 0.1 | 0.001 |
| Identifying drug-related problems | 3 (2-4) | 3 (3-4) | 0.335 | 4 (3-4) | 4 (3-4) | 0.238 | 1.0 | 0.000 | 1.0 | 0.000 | 1.0 | 0.002 |
| Patient care | 2 (2-3) | 2 (2-3) | 0.730 | 3 (2.5-4) | 2.5 (2-3) | 0.000 | 1.0 | 0.000 | 1.0 | 0.000 | 0.5 | 0.001 |
| Results assessment | 1 (1-2) | 1 (1-2) | 0.555 | 2 (1-3) | 1.5 (1-2.5) | 0.000 | 1.0 | 0.000 | 1.0 | 0.000 | 0.5 | 0.337 |
| Domain 2. Personal | ||||||||||||
| Personal organization | 3.7 (3.3-4) | 3.7 (3.3-4) | 0.208 | 3.8 (3.5-4) | 3.8 (3.5-4) | 0.686 | 0.1 | 0.001 | 0.1 | 0.004 | 0.1 | 0.058 |
| Effective communication skills | 3.5 (3-4) | 3.7 (3.2-4) | 0.154 | 3.8 (3.3-4) | 3.7 (3.3-4) | 0.224 | 0.1 | 0.000 | 0.3 | 0.000 | 0.0 | 0.003 |
| Team | 3.8 (3.4-4) | 3.8 (3.6-4) | 0.114 | 4 (3.6-4) | 3.8 (3.6-4) | 0.197 | 0.0 | 0.001 | 0.2 | 0.001 | 0.0 | 0.227 |
| Professionalism | 3.4 (3.1-3.8) | 3.5 (3.2-3.8) | 0.480 | 3.7 (3.5-3.9) | 3.6 (3.2-3.8) | 0.000 | 0.2 | 0.000 | 0.3 | 0.000 | 0.1 | 0.950 |
| Domain 3. Troubleshootin g | ||||||||||||
| On the information | 3 (2.3-3.7) | 3 (2.3-3.7) | 0.218 | 3.7 (3-4) | 3.3 (2.7-3.7) | 0.000 | 0.3 | 0.000 | 0.7 | 0.000 | 0.3 | 0.004 |
| Knowledge | 3 (2.5-3.5) | 3 (2.5-3.5) | 0.059 | 3 (3-4) | 3 (2.5-4) | 0.264 | 0.0 | 0.000 | 0.0 | 0.000 | 0.0 | 0.071 |
| Analysis of information | 2.5 (2-3.2) | 2.6 (2.2-3) | 0.431 | 3 (2.7-3.7) | 2.8 (2.3-3.3) | 0.001 | 0.5 | 0.000 | 0.5 | 0.000 | 0.2 | 0.001 |
| Provides information | 3.7 (3.7-4) | 3.7 (3.7-4) | 0.717 | 4 (3.7-4) | 4 (3.3-4) | 0.000 | 0.3 | 0.000 | 0.3 | 0.000 | 0.3 | 0.083 |
| Tracking | 2 (2-3) | 2 (2-3) | 0.667 | 3 (2-4) | 2 (2-3) | 0.016 | 1.0 | 0.000 | 1.0 | 0.000 | 0.0 | 0.002 |
| Domain 4. Management and organization | ||||||||||||
| Clinical management and pharmaco-security | 2.3 (2-2.7) | 2.5 (2-3) | 0.257 | 2.7 (2-3.3) | 2.7 (2-3.3) | 0.837 | 0.4 | 0.000 | 0.4 | 0.000 | 0.2 | 0.002 |
| Provision of service | 3 (3-4) | 3 (3-4) | 0.187 | 4 (3-4) | 3 (3-4) | 0.006 | 1.0 | 0.000 | 1.0 | 0.000 | 1.0 | 0.347 |
| Acquisition | 3.5 (3.3-4) | 3.8 (3.3-4) | 0.161 | 3.8 (3.5-4) | 3.8 (3.3-4) | 0.258 | 0.3 | 0.003 | 0.3 | 0.000 | 0.0 | 0.660 |
| Satisfaction of the services offered | 1.5 (1-2.5) | 1.5 (1-2.5) | 0.256 | 2.5 (1.5-3) | 2.5 (1.5-3) | 0.740 | 1.0 | 0.000 | 1.0 | 0.000 | 1.0 | 0.000 |
Mann-Whitney U test;
Wilcoxon test; IQ: Inter quarterly range.
Table 4.
Evaluation of knowledge in pharmacy staff
| Initial measurement | Final measurement | |||||||
|---|---|---|---|---|---|---|---|---|
| Total (n=659) | Intervention (n=379) | Control (280) | P-value* | Total (n=372) | Intervention (n=193) | Control (n=179) | P-value* | |
| Block 1: Basic pharmacology and pharmaceutical terminology, n (%) | 437 (66.3) | 261 (68.9) | 176 (62.9) | 0.107 | 261 (70.2) | 148 (76.7) | 113 (63.1) | 0.004 |
| Block 2: Pharmaceutical legislation, n (%) | 546 (82.9) | 324 (85.5) | 222 (79.3) | 0.037 | 328 (88.2) | 182 (94.3) | 146 (81.6) | 0.000 |
| Block 3: Technical-administrative processes of the drugstores, n (%) | 243 (36.9) | 150 (39.6) | 93 (33.2) | 0.094 | 223 (59.9) | 140 (72.5) | 83 (46.4) | 0.000 |
| Block 4: Information on the proper use of medicines and health education, n (%) | 355 (53.9) | 195 (51.5) | 160 (57.1) | 0.147 | 243 (65.3) | 138 (71.5) | 105 (58.7) | 0.009 |
| Block 5: Good Dispensation Practices, n (%) | 479 (72.7) | 285 (75.2) | 194 (69.3) | 0.092 | 327 (87.9) | 183 (94.8) | 144 (80.4) | 0.000 |
| Block 6: Pharmacovigilance of Pharmacies, n (%) | 114 (17.3) | 73 (19.3) | 41 (14.6) | 0.121 | 124 (33.3) | 80 (41.5) | 44 (24.6) | 0.000 |
| Block 7: Pharmaceutical actuation in common diseases using drugs with fiscalized substances, n (%) | 560 (85) | 320 (84.43) | 228 (81.4) | 0.078 | 345 (92.7) | 187 (96.9) | 158 (88.3) | 0.001 |
| Total, n (%) | 305 (46.3) | 180 (47.5) | 125 (44.6) | 0.468 | 266 (71.5) | 163 (84.5) | 103 (57.5) | 0.000 |
Chi-squared test
Effect of continuing education program
Overall, 56.4% (372) of pharmacy staff completed the continuing education program, and skills, attitudes, and knowledge self-administered questionnaire. The pharmacy staff’s skills, attitudes, and knowledge scores post-intervention were statistically significantly higher than the pre-intervention, and the intervention group had higher scores than the control group (Table 3). Post-intervention, the self-efficacy skills and attitudes in the intervention group improved by 88% (22), and knowledge rose in six of the seven topics of the questionnaire. Although the measure of knowledge increased in the control group, it was not statistically significant. Dispensing criteria evaluated with simulated patient methodology showed no statistically significant differences between groups. Of the 585 total tramadol dispensations, 98% (575) were evaluated as poor.
DISCUSSION
The ambulatory pharmacy staff plays a key role in improving pharmacotherapy and patient outcomes, promoting rational use of medicines, and optimizing healthcare costs.26 To our knowledge, this is the first controlled and multicenter trial that evaluates a continuing education program’s effectiveness to improve ambulatory pharmacy staff’s competencies in providing dispensing, health education, and pharmacovigilance services of drugs with fiscalized substances (including controlled drugs). The results showed that the implementation of a continuing education program significantly improved the competencies of the pharmacy staff, with a significant difference between the intervention group and the control group; however, there were no improvements during the dispensing of tramadol using the simulated patient methodology.
A systematic review summarized the effects of the media methods used in continuing education activities on pharmacy staff. This shows their impact on continuing education programs that provide ambulatory pharmacy services and assessment methods. According to this review, which included 19 articles mainly conducted in high-income countries, only two studies found that the continuing education program was ineffective or partially effective. However, most studies used non-robust, non-validated, and no standardized methods to measure the effectiveness of the interventions. The authors concluded that there is insufficient evidence to support which media methods are the most effective. Nevertheless, there are programs of continuing education, evaluation, and continued implementation, which have innovative approaches supported by ICT.16
A strength of the study was the identification and initial characterization of the pharmacy staff’s training and continuing education needs of the same population. This characterization identified the pharmacy staff’s perception of the risk of fiscalized drugs and the lack of technical processes and care services in drugstores and pharmacies.19 Also, another strength of this study was the design, construction, and evaluation of different tools supported by ICTs: the social network-like web platform, the virtual course, and the system of information and dispensing of drugs. It is important to denote that studies that combine such multimedia strategies into the same intervention were not found. In addition, it is the first web platform-type social networking aimed exclusively at ambulatory pharmacy staff and a virtual course focused on updating the proper utilization of drugs with fiscalized substances.
The participation of 659 pharmacy staff who worked in 305 different drugstores and pharmacies located in different cities gives this trial the characteristics of a multicenter study. The variability of the interventions becomes a strength of the study since it is in line with actual ambulatory practice. According to the outcomes, it was observed that only 30% of the pharmacy staff who work in these establishments have university formation (professional or technician degree). This supports the need for this type of study, favoring continuing education strategies in the remaining population. Similar to our findings, another study justifies the need for this type of interventions.27 They evaluated the profile and practices of drug dispensers in drugstores and pharmacies, reporting that about 67% of pharmacy staff did not have professional or technician degrees and that dispensing was carried out inadequately in 55% of the cases. Therefore, these results showed a critical scenery mainly characterized by a pharmacy staff of nearly 70% without a university education and training necessary to work as a pharmacy staff in pharmacies and drugstores.
An advantage of the study was the complementing of the virtual educational process of the pharmacy staff with face-to-face training cycles. The total training time was 160 hours: 140 hours of virtual intervention and 20 hours of face-to-face training. The most similar findings in this context were an analytical observational study of a virtual educational program with an intervention time of 120 hours, which evaluated knowledge, skills, and performance attitudes (self-informed changes and considered in standard practice).28 To our knowledge, it is the study with the highest number of virtual and face-to-face training hours during the 12 months of follow-up.
Self-directed learning, continuing education programs, and other traditional activities of education alone do not facilitate change in the practice.29-30 In our study, despite substantial increases in knowledge and self-reported skills and attitudes by the pharmacy staff these improvements were not reflected in the evaluation of the dispensing process because pharmacy staff showed limitations both in dispensing and providing advice and support for the use correct of tramadol. These outcomes can generate future intervention studies focused on the effect evaluation of improving skills and attitudes in dispensing and providing information for the correct use of drugs. Also, these findings could show that pharmacy staff needs additional and continuous training/sustainability in pharmacy oriented to improve dispensing and provide advice and support for the correct use of pharmacotherapy.9
As part of the contributions of this trial, it is necessary to the development of new multimedia tools that improve the effectiveness and timeliness of ambulatory pharmacy services. In addition, identify and develop complementary activities to the educational intervention to deepen skills and attitudes in real conditions, such as simulations, supervised practices, and role-playing. Finally, continuing education and professional development programs could be structured and offered as part of a certification process of knowledge and practices as is done in developed countries.
Limitations and potential bias
Despite the fact that no instruments were found in the literature that would allow the evaluation of competencies (knowledge, skills, and attitudes), we develop research instruments, using a methodology previously proposed by other authors,22 which included the calculation of the reliability of the design instrument. Although the simulated patient technique has shown validity and reliability in pharmacy practice research,19 the lack of audio or video recording may have caused potential bias regarding information and assessment carried out by simulated patients. However, this limitation could be minimized by continuous training of these persons. Therefore, to minimize this bias, we sought to ensure that all simulated patients completed the evaluation questionnaire immediately after each visit with the principal investigator, who was the same person who trained and transported the simulated patients on all visits to the pharmacies. Also, the simulated patient technique itself may include subjectivity on the part of the evaluator; however, these simulated patients were trained, and additionally, categories were established that allowed identifying whether the dispensing process was correct.
CONCLUSIONS
Continuing education program using virtual tools plus additional personalized training is a feasible and practical educational intervention strategy for pharmacy staff. This showed positive results in the pharmacy staff’s competencies (assessed knowledge and self-reported skills and attitudes) related to the use of drugs with fiscalized substances. However, the intervention did not improve skills and attitudes to dispensing and providing advice and support for the rational utilization of tramadol. The educational intervention, integrated with a social network-type web platform, a virtual course for updating pharmacotherapeutic, and drug information and dispensing software of drugs, added to a personalized training focused on the proper utilization of drugs with fiscalized substances could be enhanced ambulatory pharmacy services, such as dispensing, health education, and pharmacovigilance.
AUTHORS’ CONTRIBUTIONS
MC was the principal investigator and developed the original idea for this research. The design of the study was done by PA, DSH, MC, and ASO. All authors contributed to the creation of the manuscript. Finally, all authors approved the final version of this manuscript and agreed to the publication.
ACKNOWLEDGMENTS
The authors thank the participating pharmacy staff and simulated patients.
Contributor Information
Ceballos Mauricio, Research Group on Pharmaceutical Promotion and Prevention, Research Group on Pharmacy Regency Technology. University of Antioquia, Medellin, Colombia. javier.ceballos@udea.edu.co.
Salazar-Ospina Andrea, Research Group on Pharmaceutical Promotion and Prevention, Research Group on Pharmacy Regency Technology. University of Antioquia, Medellin, Colombia. andrea.salazaro@udea.edu.co.
Sabater-Hernández Daniel, Pharmaceutical Care Research Group, Faculty of Pharmacy, University of Granada, Granada, Spain. dsabater@gmail.com.
Amariles Pedro, Research Group on Pharmaceutical Promotion and Prevention. University of Antioquia, Medellin, Colombia; Pharmaceutical Care Research Group, Faculty of Pharmacy, University of Granada, Granada, Spain. pedro.amariles@udea.edu.co.
FUNDING
The Pharmaceutical Promotion and Prevention Group received financial support from the Committee for Development Research and the sustainability program (2018-2019) of the University of Antioquia. Also, this research was financed partially by Laproff Laboratories, providing the multimedia resources used in this study.
CONFLICTS OF INTEREST
The authors declare no conflicts of interest.
References
- 1.1. Ceballos M, Amariles P. Needs and opportunities for outpatient pharmacies in Colombia:The case of continuing education programs for pharmacists. Vitae. 2017;24(1):10-11. https://doi.org/10.17533/udea.vitae.v24n1a01; Ceballos M, Amariles P. Needs and opportunities for outpatient pharmacies in Colombia:The case of continuing education programs for pharmacists. Vitae. 2017;24(1):10–11. doi: 10.17533/udea.vitae.v24n1a01. [DOI] [Google Scholar]
- 2.2. Svetlana S, Ivana T, Tatjana C, et al. Evaluation of Competences at the Community Pharmacy Settings. Indian J Pharm Educ Res. 2014;48(4):22-30. https://doi.org/10.5530/ijper.48.4.4; Svetlana S, Ivana T, Tatjana C, et al. Evaluation of Competences at the Community Pharmacy Settings. Indian J Pharm Educ Res. 2014;48(4):22–30. doi: 10.5530/ijper.48.4.4. [DOI] [Google Scholar]
- 3.3. Schostak J, Davis M, Hanson J, et al. Effectiveness of continuing professional development project:A summary of findings. Med Teach. 2010;32(7):586-592. https://doi.org/10.3109/0142159x.2010.489129 [DOI] [PubMed]; Schostak J, Davis M, Hanson J, et al. Effectiveness of continuing professional development project:A summary of findings. Med Teach. 2010;32(7):586–592. doi: 10.3109/0142159x.2010.489129. [DOI] [PubMed] [Google Scholar]
- 4.4. Amariles P, Ceballos M, González-Giraldo C. Primary health care policy and vision for community pharmacy and pharmacists in Colombia. Pharm Pract (Granada). 2020;18(4):1-7. https://doi.org/10.18549/pharmpract.2020.4.2159 [DOI] [PMC free article] [PubMed]; Amariles P, Ceballos M, González-Giraldo C. Primary health care policy and vision for community pharmacy and pharmacists in Colombia. Pharm Pract (Granada) 2020;18(4):1–7. doi: 10.18549/pharmpract.2020.4.2159. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.5. Rigg KK, March SJ, Inciardi JA. Prescription Drug Abuse &Diversion:Role of the Pain Clinic. J Drug Issues. 2010;40(3):681-702. https://doi.org/10.1177/002204261004000307 [DOI] [PMC free article] [PubMed]; Rigg KK, March SJ, Inciardi JA. Prescription Drug Abuse &Diversion:Role of the Pain Clinic. J Drug Issues. 2010;40(3):681–702. doi: 10.1177/002204261004000307. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.6. Kouladjian L, Chen TF, Gnjidic D, et al. Education and Assessment of Pharmacists on the Use of the Drug Burden Index in Older Adults Using a Continuing Professional Development Education Method. Am J Pharm Educ. 2016;80(4):63. https://doi.org/10.5688/ajpe80463 [DOI] [PMC free article] [PubMed]; Kouladjian L, Chen TF, Gnjidic D, et al. Education and Assessment of Pharmacists on the Use of the Drug Burden Index in Older Adults Using a Continuing Professional Development Education Method. Am J Pharm Educ. 2016;80(4):63. doi: 10.5688/ajpe80463. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.7. Sullivan MD, Gaster B, Russo J, et al. Randomized Trial of Web-based Training About Opioid Therapy for Chronic Pain. Clin J Pain. 2010;26(6):512-517. https://doi.org/10.1097/ajp.0b013e3181dc7adc [DOI] [PubMed]; Sullivan MD, Gaster B, Russo J, et al. Randomized Trial of Web-based Training About Opioid Therapy for Chronic Pain. Clin J Pain. 2010;26(6):512–517. doi: 10.1097/ajp.0b013e3181dc7adc. [DOI] [PubMed] [Google Scholar]
- 8.8. Kahan M, Wilson L, Wenghofer EF, et al. Pharmacists'experiences with dispensing opioids:provincial survey. Can Fam Physician. 2011;57(11):e448-454. [PMC free article] [PubMed]; Kahan M, Wilson L, Wenghofer EF, et al. Pharmacists'experiences with dispensing opioids:provincial survey. Can Fam Physician. 2011;57(11):e448–454. [PMC free article] [PubMed] [Google Scholar]
- 9.9. Wilsey BL, Fishman SM, Casamalhuapa C, et al. Computerized progress notes for chronic pain patients receiving opioids;the Prescription Opioid Documentation System (PODS). Pain Med. 2010;11(11):1707-1717. https://doi.org/10.1111/j.1526-4637.2010.00977.x [DOI] [PMC free article] [PubMed]; Wilsey BL, Fishman SM, Casamalhuapa C, et al. Computerized progress notes for chronic pain patients receiving opioids;the Prescription Opioid Documentation System (PODS) Pain Med. 2010;11(11):1707–1717. doi: 10.1111/j.1526-4637.2010.00977.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.10. Fisher JE, Zhang Y, Sketris I, et al. The effect of an educational intervention on meperidine use in Nova Scotia, Canada:a time series analysis. Pharmacoepidemiol Drug Saf. 2012;21(2):177-183. https://doi.org/10.1002/pds.2259 [DOI] [PMC free article] [PubMed]; Fisher JE, Zhang Y, Sketris I, et al. The effect of an educational intervention on meperidine use in Nova Scotia, Canada:a time series analysis. Pharmacoepidemiol Drug Saf. 2012;21(2):177–183. doi: 10.1002/pds.2259. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.11. Buxton EC, De Muth JE. Pharmacists'perceptions of a live continuing education program comparing distance learning versus local learning. Res Social Adm Pharm. 2013;9(2):230-235. https://doi.org/10.1016/j.sapharm.2012.05.003 [DOI] [PubMed]; Buxton EC, De Muth JE. Pharmacists'perceptions of a live continuing education program comparing distance learning versus local learning. Res Social Adm Pharm. 2013;9(2):230–235. doi: 10.1016/j.sapharm.2012.05.003. [DOI] [PubMed] [Google Scholar]
- 12.12. Walters C, Raymont A, Galea S, et al. Evaluation of online training for the provision of opioid substitution treatment by community pharmacists in New Zealand. Drug Alcohol Rev. 2012;31(7):903-910. https://doi.org/10.1111/j.1465-3362.2012.00459.x [DOI] [PubMed]; Walters C, Raymont A, Galea S, et al. Evaluation of online training for the provision of opioid substitution treatment by community pharmacists in New Zealand. Drug Alcohol Rev. 2012;31(7):903–910. doi: 10.1111/j.1465-3362.2012.00459.x. [DOI] [PubMed] [Google Scholar]
- 13.13. Palmer E, Hart S, Freeman PR. Development and delivery of a pharmacist training program to increase naloxone access in Kentucky. J Am Pharm Assoc (2003). 2017;57(2S):S118-22. https://doi.org/10.1016/j.japh.2016.12.071 [DOI] [PubMed]; Palmer E, Hart S, Freeman PR. Development and delivery of a pharmacist training program to increase naloxone access in Kentucky. J Am Pharm Assoc (2003) 2017;57(2S):S118–22. doi: 10.1016/j.japh.2016.12.071. [DOI] [PubMed] [Google Scholar]
- 14.14. Wheeler A, Fowler J, Hattingh L. Using an intervention mapping framework to develop an online mental health continuing education program for pharmacy staff. J Contin Educ Health Prof. 2013;33(4):258-266. https://doi.org/10.1002/chp.21198 [DOI] [PubMed]; Wheeler A, Fowler J, Hattingh L. Using an intervention mapping framework to develop an online mental health continuing education program for pharmacy staff. J Contin Educ Health Prof. 2013;33(4):258–266. doi: 10.1002/chp.21198. [DOI] [PubMed] [Google Scholar]
- 15.15. Ceballos M, Madrigal-Cadavid J, Amariles P. Educational programs and interventions offered to pharmacy staff to improve the delivery of pharmaceutical assistance services:A structured review. Salud UIS. 2021;3:e21010. https://doi.org/10.18273/saluduis.53.e:21010; Ceballos M, Madrigal-Cadavid J, Amariles P. Educational programs and interventions offered to pharmacy staff to improve the delivery of pharmaceutical assistance services:A structured review. Salud UIS. 2021;3:e21010. doi: 10.18273/saluduis.53.e:21010. [DOI] [Google Scholar]
- 16.16. Obreli-Neto PR, Marques dos Reis T, Guidoni CM, et al. A Systematic review of the effects of continuing education programs on providing clinical community pharmacy services. Am J Pharm Educ. 2016;80(5):88. https://doi.org/10.5688/ajpe80588 [DOI] [PMC free article] [PubMed]; Obreli-Neto PR, Marques dos Reis T, Guidoni CM, et al. A Systematic review of the effects of continuing education programs on providing clinical community pharmacy services. Am J Pharm Educ. 2016;80(5):88. doi: 10.5688/ajpe80588. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.17. McConnell KJ, Delate T, Newlon CL. Impact of continuing professional development versus traditional continuing pharmacy education on learning behaviors. J Am Pharm Assoc. 2012;52(6):742-752. https://doi.org/10.1331/japha.2012.11080 [DOI] [PubMed]; McConnell KJ, Delate T, Newlon CL. Impact of continuing professional development versus traditional continuing pharmacy education on learning behaviors. J Am Pharm Assoc. 2012;52(6):742–752. doi: 10.1331/japha.2012.11080. [DOI] [PubMed] [Google Scholar]
- 18.18. Ceballos M, Salazar-Ospina A, Sabater-Hernández D, et al. Evaluation of the effects of a drug with fiscalized substance dispensation, health education, and pharmacovigilance continuing education program in Colombia drugstores and drugstores/pharmacies:study protocol of a multicenter, cluster-randomized control. Trials. 2020;21(1):545-559. https://doi.org/10.1186/s13063-020-04481-1 [DOI] [PMC free article] [PubMed]; Ceballos M, Salazar-Ospina A, Sabater-Hernández D, et al. Evaluation of the effects of a drug with fiscalized substance dispensation, health education, and pharmacovigilance continuing education program in Colombia drugstores and drugstores/pharmacies:study protocol of a multicenter, cluster-randomized control. Trials. 2020;21(1):545–559. doi: 10.1186/s13063-020-04481-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.19. Ceballos M, Giraldo JA, Marín VH, et al. Characterization of aspects related to the use of controlled drugs in Drugstores and pharmacies-drugstores in Medellin and the metropolitan area. Rev Univ Ind Santander Salud. 2018;50(1):27-36. https://dx.doi.org/10.18273/revsal.v50n1-2018003; Ceballos M, Giraldo JA, Marín VH, et al. Characterization of aspects related to the use of controlled drugs in Drugstores and pharmacies-drugstores in Medellin and the metropolitan area. Rev Univ Ind Santander Salud. 2018;50(1):27–36. doi: 10.18273/revsal.v50n1-2018003. [DOI] [Google Scholar]
- 20.20. García Corpas JP, Ocaña Arenas A, González García L, et al. Changes in knowledge after attending a community pharmacists asthma workshop. Pharm Pract. 2006;4(3):139-142. [PMC free article] [PubMed]; García Corpas JP, Ocaña Arenas A, González García L, et al. Changes in knowledge after attending a community pharmacists asthma workshop. Pharm Pract. 2006;4(3):139–142. [PMC free article] [PubMed] [Google Scholar]
- 21.21. Ceballos Rueda JM, Salazar Ospina A, et al. Review, design, and construction of a measuring instrument to assess the labor competencies of outpatient pharmacy staff. Salud UIS. 2021;53(e21027):1-14. https://doi.org/10.18273/saluduis.53.e:21027; Ceballos Rueda JM, Salazar Ospina A, et al. Review, design, and construction of a measuring instrument to assess the labor competencies of outpatient pharmacy staff. Salud UIS. 2021;53(e21027):1–14. doi: 10.18273/saluduis.53.e:21027. [DOI] [Google Scholar]
- 22.22. Pino-Marín D, Gómez L, Madrigal-Cadavid J, et al. Proposed procedure for the construction and validation of a psychometric instrument based on a knowledge quiz about cardiovascular disease in patients attending in Spanish community pharmacies. Pharm Care Esp. 2017;19(3):121-138. https://doi.org/10.1371/journal.pone.0180622; Pino-Marín D, Gómez L, Madrigal-Cadavid J, et al. Proposed procedure for the construction and validation of a psychometric instrument based on a knowledge quiz about cardiovascular disease in patients attending in Spanish community pharmacies. Pharm Care Esp. 2017;19(3):121–138. doi: 10.1371/journal.pone.0180622. [DOI] [Google Scholar]
- 23.23. Xu T, de Almeida Neto AC, Moles RJ. A systematic review of simulated-patient methods used in community pharmacy to assess the provision of non-prescription medicines. Int J Pharm Pract. 2012;20(5):307-319. https://doi.org/10.1111/j.2042-7174.2012.00201.x [DOI] [PubMed]; Xu T, de Almeida Neto AC, Moles RJ. A systematic review of simulated-patient methods used in community pharmacy to assess the provision of non-prescription medicines. Int J Pharm Pract. 2012;20(5):307–319. doi: 10.1111/j.2042-7174.2012.00201.x. [DOI] [PubMed] [Google Scholar]
- 24.24. World Health Organization. Chapter 30:Ensuring good dispensing practices. Management Science for Health - Part II:Pharmaceutical Management. 2012. https://msh.org/wp-content/uploads/2013/04/mds3-ch30-dispensing-mar2012.pdf. Accessed February 25, 2021.; World Health Organization. Chapter 30:Ensuring good dispensing practices. Management Science for Health - Part II:Pharmaceutical Management. 2012. [Accessed February 25, 2021]. https: //msh.org/wp-content/uploads/2013/04/mds3-ch30-dispensing-mar2012.pdf .
- 25.25. Ceballos M, Llano Y, Salazar-Ospina A, et al. Skills and practices of pharmacy staff for dispensing of drugs with fiscalized substances in drugstores and pharmacies. Rev Saude Publica. 2021;55:44. https://doi.org/10.11606/s1518-8787.2021055003103 [DOI] [PMC free article] [PubMed]; Ceballos M, Llano Y, Salazar-Ospina A, et al. Skills and practices of pharmacy staff for dispensing of drugs with fiscalized substances in drugstores and pharmacies. Rev Saude Publica. 2021;55:44. doi: 10.11606/s1518-8787.2021055003103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.26. Nkansah N, Mostovetsky O, Yu C, et al. Effect of outpatient pharmacists'non-dispensing roles on patient outcomes and prescribing patterns. Cochrane Database Syst Rev. 2010;2010(7). https://doi.org/10.1002/14651858.cd000336.pub2 [DOI] [PMC free article] [PubMed]; Nkansah N, Mostovetsky O, Yu C, et al. Effect of outpatient pharmacists'non-dispensing roles on patient outcomes and prescribing patterns. Cochrane Database Syst Rev. 2010;2010(7) doi: 10.1002/14651858.cd000336.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.27. Vacca C, Orozco J, Figueras A, et al. Assessment of Risks Related to Medicine Dispensing by Nonprofessionals in Colombia:Clinical Case Simulations. Ann Pharmacother. 2005;39(3):527-532. https://doi.org/10.1345/aph.1e420 [DOI] [PubMed]; Vacca C, Orozco J, Figueras A, et al. Assessment of Risks Related to Medicine Dispensing by Nonprofessionals in Colombia:Clinical Case Simulations. Ann Pharmacother. 2005;39(3):527–532. doi: 10.1345/aph.1e420. [DOI] [PubMed] [Google Scholar]
- 28.28. Moczygemba LR, Pierce AL, Dang A, et al. The ADAPT online education program:A tool for practicing pharmacists delivering patient-centered care. J Am Pharm Assoc (2003). 2017;57(5):601-607. https://doi.org/10.1016/j.japh.2017.05.007 [DOI] [PubMed]; Moczygemba LR, Pierce AL, Dang A, et al. The ADAPT online education program:A tool for practicing pharmacists delivering patient-centered care. J Am Pharm Assoc (2003) 2017;57(5):601–607. doi: 10.1016/j.japh.2017.05.007. [DOI] [PubMed] [Google Scholar]
- 29.29. McConnell KJ, Newlon CL, Delate T. The impact of continuing professional development versus traditional continuing pharmacy education on pharmacy practice. Ann Pharmacother. 2010;44(10):1585-1595. https://doi.org/10.1345/aph.1p161 [DOI] [PubMed]; McConnell KJ, Newlon CL, Delate T. The impact of continuing professional development versus traditional continuing pharmacy education on pharmacy practice. Ann Pharmacother. 2010;44(10):1585–1595. doi: 10.1345/aph.1p161. [DOI] [PubMed] [Google Scholar]
- 30.30. Davis D, O'Brien MAT, Freemantle N, et al. Impact of formal continuing medical education:Do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes?JAMA. 1999;282(9):867-874. https://doi.org/10.1001/jama.282.9.867 [DOI] [PubMed]; Davis D, O'Brien MAT, Freemantle N, et al. Impact of formal continuing medical education:Do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA. 1999;282(9):867–874. doi: 10.1001/jama.282.9.867. [DOI] [PubMed] [Google Scholar]