Abstract
Objectives
To evaluate students’ frequency of use and degree of usefulness of NABPLaw Online, a pharmacy-specific, online, licensed resource produced by the National Association of Boards of Pharmacy (NABP).
Methods
Students usage of various information resources, including NABPLaw Online were evaluated through (1) usage statistics gathered by NABP, (2) students’ response to a questionnaire, and (3) citation analysis performed on students’ project reports.
Results
Students used NABPLaw Online less frequently than other online tools, partly related to the relevance of the tool to their projects, and partly related to ease of use in comparison to other tools.
Conclusions
Although it was not extensively used, NABPLaw Online represents a unique resource for students researching multistate aspects of pharmacy practice law.
Keywords: law, National Assocation of Boards of Pharmacy, pharmacy law education, information retrieval, citation analysis, web site evaluation
INTRODUCTION
University of Washington pharmacy students are required to complete the course Pharmacy Laws and Ethics during the fall quarter of their second-professional year. As part of the course, they complete a group project for which the students identify some aspect of laws regulating the practice of pharmacy that does not “work” and propose corrective action.1 Law resources that are readily available to students include printed materials, several web sites providing searchable access to Washington State laws (statutes and regulations), LexisNexis Academic (providing access to all 50 states' statutes but no regulations; each state's statutes must be searched individually), and such access as individual states make available through the Web, ie, through the Washington State Board of Pharmacy's or Legislature's web sites. One approach to evaluating the strengths and weaknesses of a law or regulation is to determine what laws or regulations other states have enacted or promulgated; however, previous students had been frustrated by difficulties in locating out-of-state resources.
High-quality online tools to support legal research are generally too expensive for most schools or individuals to license, or as in the case of “full” access to LexisNexis or Westlaw access, may only be available to law school students. This article describes the use of a trial subscription to NABPLaw Online (procured via grant funding) to support student research at the University of Washington School of Pharmacy.
A review of the past 10 years of library science literature reveals numerous studies evaluating online databases, assessing library service quality, or measuring database content and coverage.2-4 We took the unique approach of using grant money to license a database for 1 academic term, with the intent of having students evaluate the tool as they used it to complete their assignments.
The “Fix the Law” group project builds on research skills that the students developed in first-year courses, and which are supplemented through training sessions and other assistance from reference librarians in health sciences, government publications, and law libraries. Project groups also received feedback from faculty members on weekly progress submissions. Students were encouraged to use online resources but have been frustrated, historically, because state-specific information was difficult to access for states other than Washington. NABPLaw Online was proposed. NABPLaw Online, which is used by Washington's State Board of Pharmacy, is an online licensed resource produced by the National Association of Boards of Pharmacy and is a compilation of pharmacy practice acts and board of pharmacy regulations for all 50 states. The State Boards of Pharmacy provide frequent updates to NABP so that the reference will be current. It permits simultaneous searches for all 50 states (in contrast to the Lexis-Nexis Academic available through most research university libraries) using keywords or phrases with some advanced search features such as Boolean and “near word” capabilities.5 A requirement of the group project is a presentation and well-referenced report.
METHODS
We attempted to assess the types of information resources used by students in completing the “Fix the Law” project. We made the NABPLaw Online resource available to students via a password-protected link on the course web site. Students had received a general library resource orientation in a prior course. The availability of NABPLaw Online was mentioned in course introductory lectures, project instructions, and comments returned to students through weekly project-related exercises. The University of Washington contracted access to NABPLAW Online for student use for 1 quarter for a nominal fee, negotiated with NABP.
The enrollment for the Pharmacy Laws and Ethics course was 87 in the fall quarter 2004. Each student was allocated at random to 1 of 12 project groups. See Table 1 for demographic data for the University of Washington Class of 2004, all pharmacy students at the University in 2004, and comparable information for all US schools of pharmacy in 2004.
Table 1.
Comparison of Demographic Information for the University of Washington School of Pharmacy's Class of 2004
UW = University of Washington; SOP = school of pharmacy; COP = college of pharmacy; US = United States
*Identified as a nonpermanent resident or resident of another country
From American Association of Colleges of Pharmacy Data, accessed 12-April-2006 www.aacp.org/Docs/MainNavigation/InstitutionalData/6664_Enrollments.pdf
Approval was received from University of Washington's Human Subjects Division for exempt status for the study. In addition to respondent anonymity, the Human Subjects Division required that the student questionnaire and all data tabulation and analysis be conducted after course grades were submitted to the University's Registrar.
Patterns of information resource use were assessed as follows:
NAPB provided pattern use statistics for the specific, password-protected, IP addresses provided for the course.
A questionnaire was used to measure students’ frequency of use for several popular online resources to complete the course project, including Google and similar web search engines, LexisNexis Academic, the Washington State Board of Pharmacy web site and the Legislature's search engine, and NABPLaw Online.
The number and type of citations from submitted project reports conducted after grades for the course had been submitted were evaluated.
The online questionnaire tool was created using the University of Washington Center for Teaching, Learning and Technology's WebQ software, which is a questionnaire and survey tool that permits the development and administration of online quizzes or survey instruments. It provides a variety of question formats, gathers and stores responses (anonymous if desired), and permits downloading of results in a variety of formats. After the conclusion of the course, students were contacted by e-mail and encouraged to complete the questionnaire. The e-mail message provided a hotlink to the webQ tool. Questionnaire items were pilot-tested before distribution. The final questionnaire items are listed in Table 2.
Table 2.
Questionnaire Items Used in a Survey of Pharmacy Students Regarding a Pharmacy Laws Project
MSN = Microsoft Network; NABP = National Association of Boards of Pharmacy
*Likert Scale: 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree = didn't use
The WebQ system was programmed to generate follow-up e-mail reminders on a weekly basis for the month following completion of the Pharmacy Laws and Ethics course. The initial “announcement” e-mail requesting participation was sent on February 11, 2005, with reminders sent weekly through March 4, 2005.
RESULTS
The WebQ system permits exporting data in a spreadsheet-compatible format. The downloaded data were then imported into Microsoft Excel. Likert scale responses (Table 2) were aggregated using a “COUNTIF” function in Excel, and weighted averages were calculated as the sum of the products of “percent responding” times the Likert index (“strongly disagree” = 1 through “strongly agree” = 5).
During the period that the NABPLaw Online Internet protocol (IP) addresses were open for student use, September 2004 through December 2004, the online resource received 125 hits. As access to the web site was controlled by a password linked to registered students and faculty members, usage by nonstudents was unlikely. Data on same user hits were unavailable.
Sixty-one of 87 students (70%) responded to the questionnaire. Usable results varied per question. The weighted averages of the Likert scale responses are presented in Table 3. Thirty-two of 61 students responding to the questionnaire (52%) indicated that they used NABPLaw Online in researching their project. Their response to the “ease of use” question was lower than for the “helpful” or “would recommend” questions.
Table 3.
Pharmacy Students’ Responses Regarding The Usefulness of Resources Provided for Completing a Pharmacy Laws Project, N = 61
Likert scale: 1 = strongly disagree; 2 = disagree; 3 = neutral; 4 = agree; 5 = strongly agree
*Only 60 of 61 students responded to this statement
Twenty-three students responded to the open-ended questions (Table 4). Responses were analyzed and grouped into 6 categories. Approximately one third of the responses noted identification of relevant information or multi-state comparisons as useful. The remaining responses largely addressed difficulty of use, inconsistent performance between searches, or other difficulties with the web site.
Table 4.
Summary of Pharmacy Students’ Responses to an Open-Ended Question Regarding Their Use of an Online Pharmacy Law Database, N = 23*
Only 23 of the 61 students responded to the open-ended questions
Web site usage and student perception of accuracy responses are presented in Table 5. The most frequently used and seemingly accurate site was the Washington State Board of Pharmacy web site, which provides a current listing of Washington laws, as well as links to other state's web sites and other relevant information. Following that was the web site for the Washington State Legislature, which provides a search engine for the Revised Code of Washington (statutes), Washington Administrative Code (regulations), State Register, and bill information. The NABPLaw Online site ranked fifth for usage and forth for student perception of accuracy. There was little difference between usage and students perception of the online resources with the exception of Google, the information from which few respondents felt confident was accurate.
Table 5.
Pharmacy Students’ Responses to Questionnaire Items Regarding Their Usage of Online Web Sites and Services in Completing a Pharmacy Law Project
All students were inculcated with the need for adequate citations in scholarly writing, through University policy (plagiarism concerns) and in prior coursework.6,7 Each project report was reviewed, citation categories developed, and frequency of use evaluated. Projects were also assessed for potential use of the NABPLaw Online tool. Results of the citation analysis are presented in Table 6. Responses regarding general topics considered and the usefulness of NABPLaw Online in completing the projects are presented in Table 7.
Table 6.
Citation Types Identified in Project Reports Submitted by Twelve Pharmacy Student Groups Completing Pharmacy Laws Projects
Table 7.
Use of NABPLaw Online by Pharmacy Students Completing a Pharmacy Laws Project
The dominant sources of cited information were from various listings of laws regulating the practice of pharmacy, followed by scholarly and lay press publications and professional and advocacy web sites. NABPLaw Online and various proprietary information utilities available to the students (ie, LexisNexis Academic, Thompson Micromedex) were mentioned with equal frequency, though with much less frequently than citations from scholarly and lay press.
DISCUSSION
The purpose of this study was to evaluate the frequency, degree of use and usefulness of NABPLaw Online, a web site that allows users access to pharmacy practice-relevant laws and regulations throughout the United States. During the study period, the web site was accessed approximately 1.4 times per student in the course (125 hits/87 students). In general, students in the University of Washington School of Pharmacy's Laws and Ethics course moderately agreed that the web site was helpful for completing their course projects, that relevant information was easily found, and that they would recommend this site to someone needing pharmacy law information. Interestingly, while course instructors determined post hoc that the NABPLaw Online site was relevant to 10 of the 12 group project topics, students ranked the site fifth in use compared to other possible web sources for finding information to complete their projects. This was most likely due to some students’ opinions that: (1) the site was difficult to use, navigate, and/or categorize information, (2) information on the site was not relevant to their particular project topic, or that it was inconsistent with other searches, or (3) the site performed inconsistently. It is unclear why the NABPLaw Online site was ranked fourth in student perception of accuracy after Washington state-based resources, though perhaps it was due to students’ lack of familiarity with NABP or the lack of utility of the site for some students.
Some students seemed to think the NABPLaw Online web site was helpful in completing their course projects while others felt it was irrelevant or difficult to use. It might be anticipated that as the site improves over time and the course instructors better explain its utility to students that it might become more useful as a learning tool, depending on the type of information needed.
One possible reason for ranking this resource lower might be the different ways that the various State Legislatures and Boards of Pharmacy choose similar but not identical words to describe various requirements in laws and rules. For example, most states require that a licensed pharmacist be designated as responsible for the operations of a pharmacy. Depending upon the state, this person may be called the “pharmacist in charge,” “PIC,” “responsible pharmacist,” “responsible manager,” etc. Unless the student searched for each of these terms, the results would be incomplete.
CONCLUSIONS
Despite its shortcomings, NABPLaw Online represents a sufficiently unique resource for students researching multi-state aspects of pharmacy practice law to make it available for future classes.
Acknowledgments
This study was supported by funding from Elmer M. Plein Endowment Research Fund. The authors would like to thank the students in the 2004 Pharmacy 543 class for their participation in this project. They would also like to thank NABP staff for their efforts making the web site available for this study.
REFERENCES
- 1.Smith K, Hazlet TK, Hammer DP. “Fix the law” project: An innovation in students' learning to affect change. Am J Pharm Educ. 2004;68(1) Article 18. [Google Scholar]
- 2.Kupferberg N, Hartel LJ. Evaluation of five full-text drug databases by pharmacy students, faculty and librarians: do the groups agree? J Med Librarian Assoc. 2004;92(1):66–71. [PMC free article] [PubMed] [Google Scholar]
- 3.Lapidus M. Library services for pharmacy and health sciences students: results of a survey. J Acad Libr. 2003;29(4):237–44. [Google Scholar]
- 4.Blessinger K, Olle M. Content analysis of the leading general academic databases. Libr Collections Acquisitions Tech Serv. 2004;28:335–46. [Google Scholar]
- 5. NABPLaw Online. National Association of Boards of Pharmacy. Available from http://www.nabp.net/. Accessed July 20, 2005.
- 6.Cronin B. Bibliometrics and beyond: some thoughts on web-based citation analysis. J Info Sci. 2001;27(1):1–7. [Google Scholar]
- 7.Weale AR, Bailey M, Lear PA. The level of non-citation of articles within a journal as a measure of quality: a comparison to the impact factor. BMC Med Res Methodology. 2004;4(1):14. doi: 10.1186/1471-2288-4-14. [DOI] [PMC free article] [PubMed] [Google Scholar]