Abstract
Objective
To determine the effect of a 5-week Career Explorers Program (CEP) on high school students' perceptions of pharmacists' characteristics, duties, and training.
Methods
A 16-item survey instrument with attitudinal, frequency, and relative quantity response options was completed by all CEP students on the first and last day of the program. The survey assessed students' attitudes concerning pharmacist characteristics, duties, and training.
Results
All students who participated in the CEP in 2003 completed the survey instrument (n = 50). Seventy percent of respondents' answers to the attitudinal subscale questions significantly changed from preassessment to postassessment.
Conclusion
A 5-week CEP provided high school students with more realistic perceptions of pharmacists' roles, duties, and training before the students entered the pharmacy program.
Keywords: experiential learning, career choice, pharmacists, high school students
INTRODUCTION
There is a pharmacist shortage in many areas of pharmacy in the United States.1,2 To encourage increased student knowledge of the pharmacy profession, the Walgreen Company and Midwestern University Chicago College of Pharmacy (MWU CCP) collaborated to develop the Career Explorers Program (CEP). The CEP gives exceptional high school students who are interested in the pharmacy profession the opportunity to receive pharmacy practice experience. Upon CEP foundation in 2000, an inaugural class of 25 students was admitted. In 2001, Albertson's/Osco joined the partnership and sponsored an additional 25 students. The short-term goals of the CEP were to increase the number of: students applying to pharmacy schools and pharmacy technicians practicing in the Chicago area. Long-term CEP goals included increasing the quality and number of pharmacists actively practicing in the United States.
A literature search was performed to determine whether similar programs had been described in the literature. This search resulted in the identification of several articles that described programs ranging from pharmacy summer camps to “mini-internships.”3-15 In addition, many programs were identified that provide similar programs for other health care professions (ie, nursing and medicine).11-25 There are also programs that target minority students to encourage careers in the health sciences.18-22 In Georgia, the Health Professions Partnership Initiative started a program for minority students from 2 high schools. 18 Students met with instructors for 3 hours on Saturday mornings throughout the academic year. Classes encompassed a variety of topics including Scholastic Aptitude Test (SAT) preparation, biology, algebra, English composition, etc. Thus far, the program has been successful in increasing the mean SAT score for participating students.
Additional programs were identified that promote health science research as a career opportunity.23-25 Researchers at Oregon Health and Science University established a course to educate high school students on research methodology.23 Students are required, with the assistance of an assigned mentor, to develop a “hypothesis driven research proposal.” Students also have the opportunity to participate in laboratory activities. Overall, 73% of surveyed participants remained interested in a career in health or science. In addition to encouraging health professions, Zavattieri and colleagues found that of 1,218 participating high school students in their longstanding Health Science Careers program, 95% perceived the program as valuable.25
None of the programs described in the literature included a report on program evaluation, specifically program effect on participant attitudes. Of the programs identified, the CEP seems to be the most competitive, comprehensive program that focuses on pharmacy. It further distinguishes itself by providing a student stipend for participation and by helping to prepare participants for the Pharmacy Technician Certification examination.
This paper will describe the CEP and present results from an analysis of CEP students' perceptions of pharmacist characteristics, duties, and training.
METHODS
In January 2003, program information packets and CEP applications were mailed to 400 public and parochial high schools throughout the Chicago area to inform school counselors, faculty members, and students of program availability. An “open house” was held each January to provide interested students, parents, and faculty members with supplemental information.
To be considered for the CEP, students must have completed their junior or senior year of high school and had a grade point average of 3.0 (out of 4.0) or ranked in the top 30% of their class. Along with the application, students had to submit 2 letters of recommendation and a personal statement similar to that which pharmacy school applicants complete. MWU CCP faculty members then conducted telephone interviews with the students who met admission criteria in an attempt to quantify students' interest in the pharmacy profession and assess their communication skills. After 50 participants were chosen, 25 were assigned to a Walgreen's Pharmacy practice site and 25 to an Osco Pharmacy practice site in the Chicago area.
The CEP employed a combination of didactic and experiential components to educate students about the pharmacy profession. The didactic portion was held 2 to 3 days per week and the experiential component filled the remaining days. Classroom days included lectures, laboratories, and reading assignments that provided an introduction to basic pharmacy coursework. Examples of lecture topics included pediatric medication dosage forms, over-the-counter medications, novel drug dosage forms, and asthma pathology and treatment options. Laboratory sessions were incorporated into campus days. For instance, students participated in a laboratory session where basic compounding concepts were introduced and practiced. The curriculum was developed based on the knowledge and skills necessary for pharmacy technicians to practice in a retail environment and to provide an introduction to other careers in pharmacy. The students evaluated the topics on their level of difficulty as well as their relevance to the program. These evaluations were used to make changes to the program each year. A comprehensive list of CEP classroom and laboratory topics is included as Table 1.
Table 1.
On the first day of the CEP, students were paired up and randomly assigned a drug information (DI) question and a faculty mentor. Student pairs researched and formulated a response to their DI question over 5 weeks with the guidance of their faculty mentor. Each student pair presented their results during the last week of the CEP.
When at the pharmacy practice sites, students completed workbook assignments that reinforced lecture topics. In addition, students observed the duties of pharmacy staff members: an experience comparable to first-professional year pharmacy students' early community pharmacy practice experience. The students learned about the job responsibilities of a pharmacy technician as well as those of the pharmacist. Once students became comfortable with the pharmacy technician tasks, they practiced these duties under the guidance of a pharmacist.
After students successfully completed a midterm and final examination, they received a stipend for CEP participation. Some of the students were offered continuing employment as a pharmacy technician with their corresponding CEP sponsoring pharmacy corporation after completion of the CEP. On the final day of the CEP, a graduation ceremony was held. Students received participation certificates and plaques were awarded to the best student in the class, the best student at each site location, and the student pair with the best DI presentation. A best preceptor award was presented as well.
On the last day of the CEP class, students completed a printed 16-item survey instrument to determine their perceptions of pharmacist characteristics, duties, and training. The survey instrument was designed as a retrospective pretest-posttest, which can be defined as a “self-report during the course or at the end of treatment, which measured subjects' recall of how they were functioning before program outset.”30 Because preassessments and postassessments are both taken after a stimulus, retrospective pretest-posttests can limit response-shift bias or the change in the subject's self-defined criterion for assessing his or her ability in a dimension.30-32 Retrospective pretest-posttests are more valid and decrease response-shift bias when compared with traditional pretest-posttest methodology,30-36 and have been used to measure attitudes. Thus, the researchers chose to utilize the retrospective pretest-posttest method in lieu of a traditional pretest-posttest.
A pilot survey instrument had been developed and administered on the last day of the CEP the previous summer (2002, n = 49). That 10-item retrospective pretest-posttest pilot instrument utilized a 4-point Likert-type scale and items addressed student beliefs concerning pharmacist/patient counseling, pharmacists' detailed knowledge of medication, and the pharmacists' healthcare provider role. Data were analyzed using the Rasch rating scale method and the Wolf and Chiu procedure for pretest and posttest comparisons.37 Edits were made pursuant to these analyses and the resulting 16-item survey, instrument that was utilized in this investigation, contained 3 subscales: attitudinal, frequency, and relative quantity. The subscales, which contained a mixture of items assessing student perceptions of pharmacist characteristics, duties, and training, were named after the response scale each item set utilized. Due to the nature of the response options, student perceptions of pharmacist characteristics, duties, and training were distributed throughout the 3 scale formats.
The attitudinal subscale contained 8 items and utilized a 4-point Likert-type scale. A scale with a neutral point was not used to avoid problems in interpretation.38-40 Response options ranged from “agree” to “disagree.” The frequency subscale contained 6 items and utilized response options ranging from “never” to “always.” The relative quantity subscale utilized a 3-point scale with response options ranging from “too little” to “too much.” Subject demographic data were not collected in the pilot assessment or in the current assessment (summer 2003).
Data were entered into an Excel data file. This file was then imported into Rasch Winsteps, version 3.5 (Winsteps, Chicago, IL), for Rasch analysis. The Wolf and Chiu procedure was used for pretest and posttest comparisons.37 The Rasch rating scale model was selected for data analysis because it provides objective evidence of unidimensionality (ie, all items measure the same construct) and produces additivity of measures necessary for subsequent statistical analysis.41,42 That is, the Rasch model converts ordinal raw scores into interval level data on a log odds or logit measurement scale if the data fit the model. SPSS statistical analysis system version 11.5 for Windows (SPSS Inc, Chicago, IL) was used to calculate descriptive statistics as well as frequencies and traditional statistics. Paired t tests were performed on Rasch data when appropriate. An alpha of 0.05 was chosen a priori.
RESULTS
The retrospective pretest-posttest survey was completed by all 2003 CEP students (n = 50) for a 100% response rate. Students' responses are presented in Table 2. The 4-points of the Likert-type rating scale met the requirements for proper scale functioning.41 All 8 items met Rasch model requirements (ie, INFIT and OUTFIT MNSQ values were greater than 0.6 and less than 1.4). Thus, these data exhibited good fit and supported the unidimensionality and local independence requirements of the model, demonstrating that the 8 items measured the same construct and produced additivity of measures (ie, true interval level data) such that categorical data were converted to interval level data. The separation index for the 8-item survey translated to an item reliability of 0.88. The person reliability (analogous to Cronbach's Alpha) was 0.81.
Table 2.
Abbreviations: D = disagree; TD = tend to disagree; TA = tend to agree; Agr = agree; N = never; S = sometimes; Alw = always; NR = no response; TL = too little; JR = just right; TM = too much
The group means for student ability measures were 0.93 logits (±1.35) for the pretest and 4.17 logits (±1.35) for posttest. This 3.24 logit increase is significant (p < 0.001) and is illustrated in Figure 1 as the change between the mean scores pre and post. Individually, 70% of CEP students (n = 35) showed a significant improvement in their attitude on individual dependent t tests (p < 0.05). Student responses relative to each item were evaluated using the normative distribution provided in Figure 1. For example, the normative distribution shows that the average student would have responded “tend to disagree” to item 5, “Pharmacists improve patients' overall health,” on pretest. However, on posttest the average student's response was “agree.” Similarly, the results for the other 6 items would be interpreted using the normative distribution provided. Respondents indicated perceived improvement for all 7 items.
Figure 1 also depicts the hierarchical ordering of item endorsement and what can be expected from each person/item interaction based on the results of the survey. The right side of Figure 1 shows the item hierarchy, with items at the bottom of the hierarchy being the easiest with which to agree and items at the top being the most difficult with which to agree. For example, item 1, “Pharmacists are knowledgeable about Rx meds,” was the easiest item for students to indicate agreement with. Correspondingly, item 3, “Pharmacists are knowledgeable about herbal products,” was the most difficult of the 7 items for students to agree with.
Table 2 reports the number and percent of students who choose each option for the items which used the frequency scale. The 3-point Likert-type rating scale met the requirements for proper scale functioning.41 In addition, all 6 items met Rasch model requirements. The separation index for the 6 items was 2.91, which translates to an item reliability of 0.89. The person reliability (analogous to Cronbach's Alpha) was 0.68.
The mean ability measure was 1.24 logits (± 1.84) for pretest and 3.93 logits (± 1.55) on posttest, showing a significant improvement in the endorsement of the 6 items collectively (p < 0.001). This 2.69 logit change is illustrated in Figure 2 as the change between the mean scores pretest and posttest. Individually, 34% of respondents (n = 17) selected responses of significantly higher frequency on posttest (p < 0.05).
Student responses relative to each item were evaluated using the normative distribution provided in Figure 2. For example, the normative distribution shows that for item 11, “Pharmacists save lives by preventing medication problems,” the average student would have indicated “sometimes” on pretest and “always” on posttest. Similarly, the results for the other 5 items would be interpreted using the normative distribution provided. Respondents indicated perceived improvement for all 6 items.
Figure 2 also depicts the hierarchical ordering of item endorsement and what can be expected from each person/item interaction based on the results of the survey and can be interpreted in a manner similar to Figure 1. For example, item 10, “Pharmacists counsel patients on their medication,” was the most difficult item for students to rate as a high frequency of occurrence. The item hierarchy shows that item 14, “Pharmacists are trustworthy,” was the easiest of the 6 items for which to assess a high frequency of occurrence.
As seen in Table 2, students were more apt to assess higher relative quantities for each statement after attending the CEP. Because this subscale included only 2 items, results were not Rasch analyzed.
DISCUSSION
A large improvement in overall measures was observed on the attitudinal subscale (items 1-8), indicating that students' attitudes changed during the CEP. This change implies that many students held negative attitudes about pharmacists when entering the CEP, but these attitudes became more positive upon program completion. Because these students became more aware of the realities of pharmacy practice, they will be better able to make an informed career choice.
Items 1 and 2 were the easiest items for the CEP students to agree with. Both items were included to determine students' perceived attitude regarding pharmacists' knowledge about prescription and over-the-counter medications (items 1 and 2). Most CEP students agreed that pharmacists were knowledgeable about prescription medications as well as nonprescription medications. This is consistent with the students' pharmacy practice exposure during the CEP. Many community pharmacists spend a significant portion of their days counseling patients about a variety of prescription and nonprescription medication issues ranging from proper medication administration to more complex points such as minimizing drug-drug interactions. Because each CEP student spent many hours at the pharmacy practice site over the 5-week program, he/she had ample opportunity to observe these skills being practiced by pharmacists as well as the opportunity to observe patient reactions.
Items 6, 7, and 8 represent the midlevel difficulty items. Item 6 was designed to assess students' attitudes about pharmacists' prescription processing responsibilities (ie, filling and dispensing medications). Actually, none of the students' reported disagreement with the statement, “pharmacists have many responsibilities when filling a prescription” after completing the CEP. This again is a reasonable outcome due to the students' exposure. In community pharmacy, multitasking tends to be a daily ritual for pharmacists due to the large number of prescriptions to fill and verify, phones ringing, patients requiring assistance, etc.
Items 7 and 8 were designed to assess students' attitudes toward pharmacists' career options post pharmacy school. After completion of the CEP, all respondents reported agreement with the statement that pharmacists have career options that expand beyond retail and hospital (item 7). Additionally, the majority of students' agreed that pharmacists may receive further education in order to specialize within their pharmacy career (ie, residency or fellowship; item 8). The authors felt these were important concepts to address during the CEP due to the overwhelming anecdotal evidence that suggested prior to participation in the CEP, students felt community and hospital pharmacy were the only opportunities for pharmacists. This expanded knowledge of career options and opportunities may encourage students' who retain an interest in pharmacy to explore a variety of areas within the pharmacy profession early in their careers.
Items 3, 4, and 5 represent the high difficultly items. Items 4 and 5 were developed to assess students' attitudes towards pharmacists being considered medication experts (item 4) and the impact pharmacists have on patients' overall health (item 5). Again, these responses may be highly dependent upon each student's experience with pharmacists as well as the individual's personal definition of “expert.” Some pharmacists tend to go above and beyond when providing patient care, whereas others may do as little as possible. Additionally, pharmacists in the community rarely see their direct impact on patients' overall health because they usually do not have access to patient outcome data. These personal data may be disclosed by the patient and tend to only be offered when there is an ongoing relationship with the pharmacist or if the pharmacist specifically requests the information. Moreover, patients typically only report negative encounters to their pharmacists rather than positive ones. This may explain why these statements were more difficult items with which to express agreement for students' at this level.
Item 3, which was developed to assess students' attitudes towards pharmacists' knowledge of herbal products, was the most difficult item for students to agree with. The students' agreement level with this statement varied depending on several factors including how much herbal information the pharmacist preceptor provided to patients and how often the opportunity presented itself.
A significant improvement in overall measures was observed in the items in the frequency (items 9-14) portion of the scale indicating that students' perception of the frequency with which pharmacists perform the selected activities improved during the CEP. This change implies that some CEP students entered the program with the opinion that pharmacists did not often impact patient care by preventing problems, counseling patients, and making medication therapy recommendations. However, after CEP attendance, these items were rated with more frequency by students, perhaps because they had the opportunity to observe them in the community practice setting.
Items 9 and 14 were the easiest items for students to agree with in this section of the survey. Item 9 was developed to assess the students' opinion regarding pharmacists' accessibility to patients whereas Item 14 assessed student perception of the trustworthiness of pharmacists. These items were the easiest for CEP students to assess a high level of frequency. This is because the community pharmacists' role and location in the chain pharmacy is somewhat stationary and standardized (ie, in most chains there is a usual place for the pharmacy within the overall store) such that people know where to find the pharmacist. In addition, pharmacists' have in recent history been thought of as one of the most trusted health care professionals.43
Items 11 and 13 represent the midlevel difficulty items. Item 11 was designed to assess students' opinions concerning the frequency with which pharmacists' positively impact patient care by preventing medication problems. Item 13 was intended to assess student perceptions of how often pharmacists act in an ethical manner. These issues were more difficult for CEP students to observe and assess with a high degree of frequency. Although pharmacists do positively impact patient care by preventing adverse medication events, this is routinely performed through prospective drug utilization review with assistance from the pharmacy computer system. About 1% of medication orders received by community pharmacists require some type of intervention.44 Although these interventions are important and sometimes vital, they are usually performed without any recognition given to the pharmacist. Therefore, a CEP student located in the pharmacy may or may not have known an intervention was occurring unless the pharmacist took the opportunity to explain and treat the intervention as a teachable moment. In addition, assessment of the frequency of ethical behavior on behalf of the pharmacist was dependent on many factors such as the recognition that a behavior or action had an ethical component and the student's internal definition of ethical behavior.
Items 10 and 12 were the items of highest difficultly. Items 10 and 12 were developed to assess students' perception concerning how often pharmacists counsel patients on medication use and the frequency of medication therapy recommendations respectively. Again, students' ability to assess these items was highly dependent upon each student's experience with pharmacist preceptors. Although many in the pharmacy community believe it would be ideal if pharmacists made drug therapy recommendations to physicians with great frequency, the reality is such that drug therapy recommendations are made relatively infrequently in community practice. There are several reasons for this, not the least of which is that community pharmacists have poor access to additional patient information beyond medication therapy on which to base these recommendations.
CEP student responses to item 10 concerning the frequency of pharmacist consultation on medication use are a bit more disturbing. The patient counseling requirements of the Omnibus Budget Reconciliation Act of 1990 (OBRA '90) were implemented and added into pharmacy practice acts over a decade ago. Because of this, it is surprising to the authors that the CEP students have not reported observing a higher frequency of patient counseling in community pharmacy practice. As pharmacists, the authors hoped that this item would have been observed much more frequently.
The CEP also provides an opportunity for 2 current pharmacy students to play an active role in CEP administration and the education processes. In this capacity, the pharmacy students participate in lecture preparation and delivery, workbook and examination grading, laboratory session development and facilitation, examination development and administration, student mentoring, and program assessment. A research project concerning the impact of the CEP on participant attitude was begun due to anecdotal evidence which supported that some CEP participants had negative attitudes and beliefs about pharmacists upon program admission. In general, many students felt that pharmacy school is of a greater scope, breadth, and duration than is required considering the duties pharmacists perform. In addition, many expressed the belief that pharmacists earn a disproportionately large income for the services they provide.
As described above, anecdotal evidence indicates there are stereotypes associated with the pharmacy profession. Initially, CEP students voiced many of these negative attitudes about pharmacy, but as the program progressed, student attitudes seemed to change. It has also been documented in the literature that consumers have similar preconceived notions about pharmacists and the role of a pharmacist until further education or receipt of a new service demonstrates otherwise.26-29 Therefore, the objective of this research project was to assess the change in student attitudes concerning pharmacist characteristics, duties, and training during the CEP. This research does not measure the long-term goals of the CEP. Rather, it was meant to gain an immediate glimpse of how the program influenced student attitudes.
A small sample size is a limitation of this study. This survey was completed by CEP students who completed the program in the summer of 2003 and may not be representative of CEP students who completed the program in other years or students who completed similar programs elsewhere. In addition, each year the pharmacy students employed to be teaching assistants for this program have the opportunity to develop and implement their own research project; hence, the survey instrument used in this study has not been utilized in subsequent years.
Three hundred seventy-three students had successfully completed the CEP program as of August 2007. Each year beginning in 2001, a survey of past CEP students has been conducted to determine the number of students pursuing pharmacy as a career. This survey is conducted by phone, e-mail, or regular mail. As of this publication, 236 responses had been received of the 323 survey instruments sent (73% response rate). Of these, 2.5% (6) of the students have graduated from pharmacy school, and 81% (N = 192) of the respondents were pursuing pharmacy as a career. Currently, 39% of the respondents are in pharmacy school, while the remaining are in prepharmacy programs or are obtaining a 4-year degree prior to applying to pharmacy school. At least 36% (N = 86) have continued to be employed as pharmacy technicians, with most of the respondents working in the chain pharmacy setting.
In looking specifically at the 2003 class, which is the subject of the research conducted in this article, there was a response rate of 74% (n = 37). Fourteen of the respondents are not currently pursuing pharmacy as a career for various reasons. Of the remaining, 46% (n = 17) are currently enrolled in a pharmacy program, while 16% (n = 6) are in pre-pharmacy programs. Of the respondents 48.6% (n = 18) are still employed as pharmacy technicians.
Another limitation of this study is that all survey responses/respondents were anonymous. Therefore, student responses could not be correlated with CEP practice sites and/or preceptors. Students had varied experiences at their designated CEP sites, resulting in different responses on the survey instrument. Some pharmacist preceptors may practice pharmaceutical care, exposing the students to more counseling and interaction with the patients.
Finally, students were introduced to many career opportunities for pharmacists during the didactic campus days of the CEP. However, during the pharmacy practice site days, students were exposed to only community pharmacy practice. This is because the companies who sponsored the program were retail pharmacy chain stores. Expanding the program to allow for a variety of site visits would allow for a more balanced and complete experience for students.
In the future, the authors intend to explore the correlation between preceptor's attitude and activities with the attitude change in the student. This information can then be used to evaluate the CEP practice site. The authors are following Career Explorer students longitudinally as a cohort to determine how many pursue pharmacy as a career. Evaluating what type of pharmacy specialization these former CEP participants choose may also be beneficial and may influence future programmatic alterations.
The CEP is relatively new (ie, 7 classes of students have graduated at the time of publication) and as such, programmatic effects on the long-term goals have not been realized (ie, students from the first CEP classes who entered pharmacy school have only recently graduated or are completing their fourth year). As was described above, CEP graduates are being followed as a cohort to determine programmatic effect and future manuscripts will focus on these results.
CONCLUSION
CEP students' attitudes and opinions regarding several aspects of pharmacist characteristics, duties, and training changed during the course of the CEP program. Awareness of the profession increased among the students and many of the participants' negative stereotypes about pharmacy and pharmacists were dispelled.
Throughout high school, students attempt to determine what the future holds for them upon graduation but it is challenging to make life-altering decisions with limited awareness of available careers. The CEP provides students considering pharmacy as a career with the opportunity to experience the pharmacy profession, thus preparing CEP students to make a more informed career choice. The CEP may be used as a template for program creation at other schools of pharmacy.
ACKNOWLEDGEMENTS
The authors would like to formally recognize the contributions of Dr. Mary Lee, Dr. Janis MacKichan, Dr. Bhavina Jain, Dr. Aeman Choudhury, Dr. Beata Slomiany, Ms. Sunita Arora, Ms. Bonnie Pearsall, and Ms. Sue Yeater. In addition, the authors would like to thank Dr. Terrence R. Jackson for serving as measurement consultant for this project.
REFERENCES
- 1.Knapp KK, Quist RM, Walton SM, Miller LM. Update on the pharmacist shortage: national and state data through 2003. Am J Health-Syst Pharm. 2005;62:492–9. doi: 10.1093/ajhp/62.5.492. [DOI] [PubMed] [Google Scholar]
- 2.Walton SM, Knapp KK, Miller L, Schumock GT. Examination of state-level changes in the pharmacist labor market using Census data. J Am Pharm Assoc. 2003;47:348–57. doi: 10.1331/JAPhA.2007.06081. [DOI] [PubMed] [Google Scholar]
- 3.Lake Erie introduces h.s. students to pharmacy. American Association of Colleges of Pharmacy News. Vol. 33. 2002. p. 10. [Google Scholar]
- 4.Valdez C, Thompson J, Altiere R, Ortega C. Colorado pharmacy camp sparks interest among minority h.s. students. American Association of Colleges of Pharmacy News. 2004;57:7–8. [Google Scholar]
- 5.Rutgers University Ernest Mario School of Pharmacy: Pharmacy Education Summer Enrichment Program. Available at http://pharmacy.rutgers.edu/OSD/summer.html. Accessed on June 25, 2004
- 6.Levy S. NACDS offers mini-internship for high school students. Drug Topics. 2002;11:49. [Google Scholar]
- 7.Early JL. Toledo Pharmacy Camp Boosts Student Enrollment. American Association of Colleges of Pharmacy News. 2004;57:6. [Google Scholar]
- 8.Albany College of Pharmacy News. Troy Savings Bank Makes $11,000 Gift to Support Albany College of Pharmacy High School Research Program 2004. Available at http://www.acp.edu/news_troybank.html. Accessed on June 25, 2004.
- 9.North Dakota State University Summer Enrichment Program. Available at http://www.ndsu.nodak.edu/pharmacy/napp/summer_program.shtml. Accessed on June 25, 2004.
- 10.The University of Arizona Health Science Center Advances. AHSC Summer Programs Encourage Health-Care Careers. August 2006;22:2. Available at http://www.opa.medicine.arizona.edu//ahsnews/aug06/aug06.pdf. Accessed November 4, 2006.
- 11.Howard University Summer Enrichment Program and Recruitment Program Description and Application Materials. Available at http://www.cpnahs.howard.edu/ctr_excellence/summer_enrichment.htm Accessed on June 25, 2004.
- 12.The University of Tennessee Health Science Center Summer Program Descriptions. High School to Health Sciences Librarianship and Informatics Project. Available at http://bioinfo-career.org. Accessed on June 25, 2004.
- 13.Marshall EC. An experiment in health careers recruitment: a summer program at Indiana University. J Am Optom Assoc. 1975;46:1284–92. [PubMed] [Google Scholar]
- 14.The Univeristy of Mississippi 2004 Health Professions Program. Available at http://www.outreach.olemiss.edu/Summer_High_School/high_school_04/programs/healh.html. Accessed on June 25, 2004.
- 15.The University of Utah Area Health Education Centers: Health Professions Academy. Available at http://www.ahec.utah.edu/UTAH-AHEC/Program%20Office/Youth_Programs/Pre-College_Programs/pre-college_programs.html. Accessed on June 25, 2004
- 16.Crump R, Byrne M, Joshua M. The University of Louisville Medical School's Comprehensive Programs to Increase Its Percentage of Underrepresented-minority Students. Acad Med. 1999;74:315–7. doi: 10.1097/00001888-199904000-00011. [DOI] [PubMed] [Google Scholar]
- 17.Oak Park and River Forest High School Health Careers Opportunity Program 2004. Available at http://oprfhs.org/division/dean/SUMMER%201-04.htmARQUETTE. Accessed on June 25, 2004.
- 18.Fincher RME, Sykes-Brown MA, Allen-Noble R. Health Science Learning Academy: A Successful “Pipeline” Educational Program for High School Students. Acad Med. 2002;77:737–8. doi: 10.1097/00001888-200207000-00023. [DOI] [PubMed] [Google Scholar]
- 19.Fleming R, Berkowitz B, Cheadle AD. Increasing minority representation in the health professions. J Sch Nurs. 2005;21:31–9. [PubMed] [Google Scholar]
- 20.Patterson DG, Carline JD. Promoting minority access to health careers through health profession-public school partnerships: a review of the literature. Acad Med. 2006;81(6 Suppl):S5–S10. doi: 10.1097/01.ACM.0000225247.84379.9c. [DOI] [PubMed] [Google Scholar]
- 21.Cohen R, Burns K, Frank-Stromborg M, Flanagan J, Askins DL, Ehrlich-Jones L. The Kids Into Health Careers (KIHC) Initiative: innovative approaches to help solve the nursing shortage. J Nurs Educ. 2006;45:186–9. doi: 10.3928/01484834-20060501-09. [DOI] [PubMed] [Google Scholar]
- 22.Bumgarner SD, Means BH, Ford MJ. Building bridges: from high school to healthcare professional. J Nurs Staff Dev. 2003;19:18–22. doi: 10.1097/00124645-200301000-00003. [DOI] [PubMed] [Google Scholar]
- 23.Rosenbaum JT, Martin TM, Farris KH, Rosenbaum RB, Neuwelt EA. Can medical schools teach high school students to be scientists? Fed Am Soc Exp Biol J. 2007;21:1954–7. doi: 10.1096/fj.06-7845lsf. [DOI] [PubMed] [Google Scholar]
- 24.Gum A, Mueller K, Flink D, Siraj S, Batsche C, Boothroyd R, Stiles P. Evaluation of a summer research institute in behavioral health for undergraduate students. J Behav Serv Res. 2007;34:206–18. doi: 10.1007/s11414-007-9059-1. [DOI] [PubMed] [Google Scholar]
- 25.Zavattieri L, D'Anna S, O'Sullivan Maillet J. Evaluation of high-school health science careers program impact on student retention and careers. J Allied Health. 2007;36:81–7. [PubMed] [Google Scholar]
- 26.Amsler MR, Murray MD, Tierney WM, Brewer N, Harris LE, Marrero DG, Weinberger M. Pharmaceutical care in chain pharmacies: beliefs and attitudes of pharmacists and patients. J Am Pharm Assoc. 2001;41:850–5. doi: 10.1016/s1086-5802(16)31326-2. [DOI] [PubMed] [Google Scholar]
- 27.Reid LD, Wang F, Young H, Awiphan R. Patients' satisfaction and their perception of the pharmacist. J Am Pharm Assoc. 1999;39:835–42. [PubMed] [Google Scholar]
- 28.Stergachis A, Maine LL, Brown L. The 2001 National Pharmacy Consumer Survey. J Am Pharm Assoc. 2002;42:568–76. doi: 10.1331/108658002763029535. [DOI] [PubMed] [Google Scholar]
- 29.Youmans SL, Schillinger D, Mamary E, Stewart A. Older African Americans' perceptions of pharmacists. Ethn Dis. 2007;17:284–90. [PubMed] [Google Scholar]
- 30.Aiken LS, West SG. Invalidity of true experiments: self-report pretest biases. Eval Rev. 1991;14:374–90. [Google Scholar]
- 31.Howard GS. Response-shift bias: a problem in evaluating interventions with pre/post self-reports. Eval Rev. 1980;4:93–106. [Google Scholar]
- 32.Sprangers M, Hoogstraten J. Pretesting effects in retrospective pretest-posttest designs. J Appl Psychol. 1989;74:265–72. [Google Scholar]
- 33.Levinson W, Gordon G, Skeff Retrospective versus actual pre-course self-assessments. Eval Health Prof. 1990;13:445–52. [Google Scholar]
- 34.Skeff KM, Stratos GA, Bergen MR. Evaluation of a medical faculty development program: a comparison of traditional pre/post and retrospective pre/post self-assessment ratings. Eval Health Prof. 1992;15:350–66. [Google Scholar]
- 35.Rippey R, Geller L, et al. Retrospective pretesting in the cognitive domain. Eval Q. 1978;2:481–9. [Google Scholar]
- 36.Bray JH, Howard GS. Methodological considerations in the evaluation of a teacher-training program. J Educ Psychol. 1980;72:62–70. [Google Scholar]
- 37.Wolfe EW, Chiu CWT. Measuring pretest-posttest change with a Rasch rating scale model. J Outcome Meas. 1999;3:134–61. [PubMed] [Google Scholar]
- 38.Low G.D. The semantics of questionnaire rating scales. Eval Res Educ. 2003;2:69–79. [Google Scholar]
- 39.Smith E, Wakely M, et al. Optimizing rating scales for self- efficacy (and other) research. Educ Psychol Meas. 2002;63:369–91. [Google Scholar]
- 40.Stone M.H. Substantive Scale Construction. J Appl Meas. 2003;4:282–97. [PubMed] [Google Scholar]
- 41.Linacre J.M. Investigating rating scale category utility. J Outcome Meas. 1999;3:103–22. [PubMed] [Google Scholar]
- 42.Wright BD, Masters GN. Best Test Design: Rasch Measurement. Chicago, Illinois: MESA Press; 1982. [Google Scholar]
- 43.Institute for Global Ethics. Medical Practitioners Top List of Most Trusted Professions. Available at: http://www.globalethics.org/newsline/1999/11/22/medical-practitioners-top-list-of-most-trusted-professions/. Accessed on June 25, 2004.
- 44.Rupp MT, DeYoung M, Schondelmeyer S. Prescribing problems and pharmacist interventions in community practice. Med Care. 1992;30:926–40. doi: 10.1097/00005650-199210000-00005. [DOI] [PubMed] [Google Scholar]