Developing health information literacy: a needs analysis from the perspective of preprofessional health students

Abstract

Objective:

The research identified the skills, if any, that health preprofessional students wished to develop after receiving feedback on skill gaps as well as any strategies they intended to use to address these gaps.

Methods:

A qualitative approach was used to elicit students' reflections on building health information literacy skills. First, the students took the Research Readiness Self-Assessment instrument, which measured their health information literacy, and then they received individually tailored feedback about their scores and skill gaps. Second, students completed a post-assessment survey asking how they intended to close identified gaps in their skills on these. Three trained coders analyzed qualitative comments by 181 students and grouped them into themes relating to “what skills to improve” and “how to improve them.”

Results:

Students intended to develop library skills (64% of respondents), Internet skills (63%), and information evaluation skills (63%). Most students reported that they would use library staff members' assistance (55%), but even more respondents (82%) planned to learn the skills by practicing on their own. Getting help from librarians was a much more popular learning strategy than getting assistance from peers (20%) or professors (17%).

Conclusions:

The study highlighted the importance of providing health preprofessional students with resources to improve skills on their own, remote access to library staff members, and instruction on the complexity of building health literacy skills, while also building relationships among students, librarians, and faculty.

Highlights.

• After receiving feedback on skill gaps, most preprofessional health students intend to develop their information literacy skills.

• Some students report that a trip to the library is a barrier to using library resources.

• Students see the need to build their information evaluation skills, knowledge of citations and plagiarism, and library skills, which they differentiate from Internet skills.

• Students are more likely to identify librarians as sources for assistance in finding information than faculty or peers after receiving individual feedback explaining the role of libraries and library staff members.

Implications.

• Students' health information competencies can be built through assessment and feedback that reveals skill gaps, highlights misconceptions, and offers ideas on how to improve.

• Access to professionally designed self-study resources is needed for students who intend to develop health information competencies on their own.

INTRODUCTION

Many students lack important competencies essential for finding and evaluating health information. However, their self-appraisals indicate that 84% of undergraduate students think favorably of their own information skills and rate them as good, very good, or excellent [1]. This study builds on the authors' previous research using an online health information assessment tool, Research Readiness Self-Assessment (RRSA), health version, that contains objective measures of information literacy skills related to finding and evaluating academic health information from library databases and the open access Internet [1]. Upon completing the RRSA, students are given feedback about their health information competencies intended to raise their awareness of their skill gaps and competency building needs. The authors were interested in learning how students translate this feedback into action to improve their health information literacy. The research questions were: From a student perspective, what is the best way to close a skill gap in health information competencies? Specifically, what skills would they like to develop? To whom would they reach out for assistance?

LITERATURE REVIEW

The growing number of publications on health information literacy is an indicator that this topic is of interest to professionals in a variety of health fields. A search for “health AND information AND literacy” in PubMed produces 364 documents published in 2000–2005, 865 documents published in 2005–2010, and 340 documents published in 2011 and the first 4 months of 2012. Students' competencies in finding and evaluating health information are often assessed using course assignments [2–5] or self-reports [6–10] and rarely using standardized tests [1]. Researchers who have measured both self-reports and actual skills have found that the 2 variables were only weakly related [1]. After controlling for education (undergraduate credit hours earned), researchers have found that self-reports of health information skills failed to explain a significant amount of variance in the score obtained from an objective skill assessment (beta = 0.08, P = 0.23). This finding indicates that self-reports may not be accurate predictors of students' actual health information competencies. Yet, both measures are important because they can be used to motivate students to build their competencies by giving feedback on perceived and actual skills. How students might take responsibility for their own learning after discovering skill gaps has yet to be fully explored.

In this study, the authors examined preprofessional students' intentions to apply the individualized feedback received from a health information literacy assessment, the RRSA [1]. Delivered online, the RRSA measures a variety of health information competencies and provides feedback. It offers multiple question sets and scales that can be turned on or off to manage the assessment length. Post-assessment feedback displays numerical results by question set or scale and their interpretations. Adaptable to the needs of an academic institution or an academic program, the RRSA includes questions that measure skills in finding and evaluating health information from electronic sources as well as students' assumptions (e.g., “The quality of health information found through web search engines, such as Google or Yahoo, is usually higher than health information in libraries”) and perceived skills (e.g., “My ability to judge the quality of health information is” poor to excellent). In addition, the RRSA contains demographic questions.

Thousands of students enrolled in higher education institutions completed the RRSA. Several past studies used the RRSA as a stand-alone assessment [1, 11–13] and as a pretest and posttest to measure the effectiveness of library instruction [14]. This line of research produced findings of interest to academic librarians and educators. First, indicators of health information competency—RRSA overall scores—varied widely among undergraduate students, even within the same education level (M = 36.78, SD = 6.35, score range 20–54) [1]. Students found it hard to narrow a search by using multiple search categories simultaneously. They also had trouble discriminating between primary and secondary sources of information or between references to journal articles and other published documents. When presented with websites on nutritional supplements, only half of undergraduates could identify the website with the most trustworthy features. In another study, a quarter of RRSA participants (graduate and undergraduate students) were blind to multiple signs of danger displayed in rogue Internet pharmacy websites and were willing to recommend them to others [13]. Responses to RRSA questions on ethical use of health information indicated that about 45% of students were unsure if they needed to provide references for ideas expressed in paraphrased sentences or in sentences whose structure they modified [1]. These findings indicated that students enrolled in higher education programs need to learn how to find, evaluate, and use health information.

Second, multiple studies showed that individuals with more education had better health information literacy skills than individuals with less education [1, 13]. When undergraduates enrolled in two communication disorders courses (introductory and mid-level) were compared to students enrolled in a graduate communication disorders program, their RRSA scores were significantly different. Group means were neatly aligned by education level [11]. Thus, even though there was much within-group variation, educated groups demonstrated higher levels of literacy than less educated groups.

Third, contrary to what might be predicted, individuals with lower information literacy, as indicated by the RRSA scores, reported sharing health information with others more frequently than their peers with higher information literacy [13]. Specifically, after controlling for the effects of education, health major, and age, respondents who supplied health information to others had significantly worse judgment of Internet pharmacies than those who did not act as information suppliers [13]. This alarming finding needs to be verified and explained.

Finally, there is some evidence that using the RRSA as a pretest can improve the effectiveness of library instruction. Students who got RRSA feedback prior to instruction reported more library and research experience and demonstrated more accurate assumptions about information sources at the end of the instruction session than their peers who did not get RRSA feedback [14]. Norm-referenced RRSA feedback is given in the form of a numerical report and a narrative that interprets the score, explains the value of academic libraries, and gives suggestions for skill improvement. The narrative also prompts students to compare their perceived skills with objectively measured skills. Because many students overestimate their skills [1], this comparison often reveals gaps, which were the focus of this study coupled with students' intentions to close these gaps.

METHODS

Participants and procedure

In this study, 348 students from an introductory health course were invited to complete the RRSA as an optional exercise to improve their research skills. The feedback consisted of numerical scores for each competency measured, written interpretations of numerical scores, and suggestions for improvement. RRSA measures are summarized in Table 1. Within a week after they received RRSA feedback, students were asked to complete an online survey. Not all students participated in the RRSA and completed a survey. Completion of the RRSA and of the follow-up survey was optional; extra credit points were used as an incentive to participate. The Internal Review Board at Central Michigan University approved the study.

Table 1.

Descriptive statistics for the Research Readiness Self-Assessment

Administered online, the survey consisted of a series of demographic questions (age, gender, education level, academic specialization, frequency of Internet use, grade point average) and two sets of open-ended questions:

1. If you were to retake Research Readiness Self-Assessment (RRSA) a year from now and wanted to earn a perfect score, what would you need to learn or improve upon?

2. How would you go about doing it? What resources do you need? Who could help you?

Coding

Survey question responses were coded according to themes as outlined in a codebook developed specifically for this study by one of three trained coders. To address the research questions, the codebook described themes relating to what students might wish to improve (knowledge, skills, and abilities) and how they intended to make improvements (on their own or with the help of others, which was further broken down into these subthemes: a librarian, a friend, a professor, or other). To calibrate coding, two coders first assigned themes to comments from fifty randomly chosen respondents. If a comment related to a theme, a code of one was used; otherwise, a code of zero was assigned. The coders then reviewed each other's codes to resolve inconsistencies and to refine the themes. They then proceeded to independently code the remaining comments. If a student's response fell into multiple themes, multiple codes were assigned. Inter-rater reliability was estimated by computing the percent of identical codes assigned by two independent coders. Post-hoc analyses examined relationships between the comments received and RRSA scores.

RESULTS

Participation rates

Of 348 students, 243 participated in the RRSA (70% participation rate). Of those who completed the assessment, 181 (74%) responded to the follow-up online survey. Email addresses were used to match RRSA scores and demographic information collected as part of the survey. Sixteen students used 2 different email accounts, and their data could not be matched, leading to a dataset with all study variables measured for 165 participants, whose quantitative RRSA results are shown in Table 1.

Participant characteristics

Most study participants were freshmen (51%), female (65%), and younger than 25 years old (98%). About one-third of the students, usually those in the second and third year of college, had declared their majors in a health discipline. Others were taking an introductory health course as the first step in exploring health-related areas of study. More than 80% reported using the Internet on a daily basis.

What skills would the students like to develop?

As can be seen in Table 2, the majority of students indicated that they wished to improve in one or more of the areas on which they were tested. The most common area for improvement, identified by 64% of respondents, was their library skills. Representative comments were: “I would need to take a library course so I knew how to better use what is available to me” and “I definitely would need to learn how to utilize the library resources better.” Students wanted to learn more about what the library and its website had to offer and to learn strategies for conducting library research. Utilization of library resources was often seen as a task that required a visit to the library building. A freshman commented, “I never use the library unless required. This will be easy to fix, because I don't mind going there. I just feel it [is] easier to stay here and find Internet [documents].” Similarly, a sophomore said, “I tend to neglect the library because of its size. In the past I have found it extremely difficult to locate a book after searching for it, and sometimes I was never even positive that it contained the information that I needed to get. So being more comfortable in the library is something that I would need to learn.” Other participants indicated that having further training on how to obtain scholarly journal articles from online library databases was a high priority.

Table 2.

Prevalence and inter-rater agreement for skills students would like to develop (n = 181)^*

The remaining three themes were mentioned less often by students. First, 63% of students wished to improve their Internet skills. A common goal was to improve skills for gathering information from the Internet and becoming better at differentiating between credible websites and non-credible websites. For example, a student felt a need to start “reading more carefully into the sources,…making sure that it is a worthy source.” Another freshman reflected, “I had a tough time distinguishing whether or not the three [websites displayed in a hands-on RRSA exercise] were valid” and concluded that she should “only look at professional websites with .edu or .gov [to] know that they are valid sites.” Multiple students commented not only on the need to build skills for finding and evaluating Internet resources, but also on their tendency to use the Internet as the only source of information. Reflecting on their excessive reliance on the open access Internet, several students expressed an idea that they “shouldn't always be so dependent on websites” because they “may not be as credible…as what you would be able to find in the library.” Thus, the students expressed the need to not only build their Internet skills, but also to improve their ability to choose high-quality sources of information for their academic assignments. This and other comments by 63% of students were assigned to the theme “information evaluation skills.” For one freshman, for example, building evaluation skills entailed learning how to identify scholarly sources of information: “[I need to get better at] finding the most scholarly resources available, and differentiating between scholarly resources and just ‘helpful’ resources.” For a junior student majoring in health sciences, building evaluation skills involved examining a document to make sure “it is research information and not based solely on opinions.” Finally, a small number of students (6%) wished to develop skills in using citations and/or avoiding plagiarism. A common concern by students was a lack of awareness of the proper way to cite a source in text and in a bibliography.

In post-hoc analyses, the authors used regression analyses to test for relationships between RRSA scores and demographics on the one hand and the themes shown in Table 2 on the other. No significant relationships were found. However, the number of skill areas students chose to improve (ranging from 0 to 4, as listed in Table 2) was significantly and positively related to their information literacy score (beta = 0.15, P = 0.03), after controlling for student assumptions about information sources (Table 1). A high score on assumptions, reflecting dependence on Google and Yahoo rather than the library for information, had a strong negative effect on the literacy score (beta = −0.49, P<0.001), explaining 23% of variance. The number of skills to be improved explained an additional 2% of variance in the health information literacy score. This positive relationship indicates that students with stronger skills were slightly more likely to indicate that they wished to improve multiple skills.

To whom would the students reach out for assistance?

In addition to reflecting on what skills to improve, the students indicated how they would go about making these improvements (Table 3). Asking someone for help—such as a friend, a professor, or a librarian—was one of the preferred strategies. Of those who would ask for help, 55% indicated they would use library staff members' assistance, as reflected, for example, in this comment: “There are many people that could help me with resource citing and looking for scholarly sources, but one that would be most beneficial would be a librarian.” A freshman commented:

Table 3.

Prevalence and inter-rater agreement for how students would build their skills (n = 181)^*

I would go about doing this by taking the time and energy to go to the library and search for sources and use the online catalog to find the sources I need. The type of sources I would need to access could be a librarian to aid my search, and be able to use some of their prior knowledge to speed up my searches. This would not be very hard at all, it would only take some time. I think after a few visits to the library I would be able to search and find all of the sources that I would need on a given project. People that could help me would be the librarian, or a friend that has had previous experience in finding sources.

As can be seen from the above comment, this student not only wanted to use librarians' expertise, but also intended to practice the skills individually. Seeking help from a librarian (or knowledgeable others) was a less frequently mentioned learning strategy than building skills on one's own. Specifically, 72% of students planned on seeking outside help versus 82% who relied on self-directed learning. This comment by a junior referred to self-learning: “To change my behavior, I would need more practice. By using the resources provided, and experiencing hands on what works and what does not will fine-tune these skills.” Practice was a commonly mentioned approach, frequently linked to improved efficiency as illustrated by a freshman student's comment: “Practicing [and] visiting [the library] website so my research becomes much quicker and easier when searching for certain topics.” A senior who was working on her health sciences degree thought that the library website was user friendly and “isn't too difficult to use and manage so I think that if I just practice with it, it could become very useful when I need to look up information on certain topics.” Twenty percent of study participants indicated they would engage in self-study without asking for help. Even fewer students (11%) would only ask for help. Most study participants (69%) chose both approaches to improve their skills.

Friends and professors were preferred as sources of help by less than one-fifth of respondents, who indicated that they would seek assistance from knowledgeable others. Often, students looked for knowledgeable others among their own peers, as did this freshman: “My roommate [is] a resource because she has done a lot of reference papers before. It would probably take a couple of hours for me to master the [document referencing] skills but if I took my time and am patient then I know I can do it.” Undergraduate students' peers were more frequently mentioned as a source of information than students' professors.

Finally, the third theme included other resources such as library, research, and computer technology classes offered through the university; publications on how to use different citation styles; books on writing research papers; exploration of the library website and its features; and so on. A freshman student commented, “To better understand the areas that I had trouble with I would have to take a couple college courses dealing with the library.”

Students also reflected on the need to change their behavior to become more mindful, deliberate, and purposeful information seekers, for example, by not “taking the easy way out of things,” by finding “the best information I can obtain,” or by “being persistent” and patient when working with journal articles. A freshman student commented that “after the librarian came…to talk to us, I feel like I can use [the] online library pretty well. I need to stop being so impatient, because that is what keeps me from looking long and hard for the best resource.” Moreover, some students acknowledged the complexity of the task of building health information competencies. A health sciences major freshman student wished to study on his own to learn style and citation rules: “I am betting that these methods of learning would end up taking up several months, and would take a good deal of effort.” Those who thought that it would be relatively easy to build their health information competencies commented on the fact that it would take time. A junior stated, “To improve on [how I cite sources] wouldn't be hard, but it would take time to improve.”

Post-hoc analyses examined relationships between demographic variables, RRSA scores, and the themes listed in Table 3. Grade point average (GPA) was significantly related to the selection of several themes, “on my own” (chi-square (2, n = 163) = 4.50, P = 0.03) and “seek help from a friend/other student” (chi-square (2, n = 165) = 5.07, P = 0.02). Specifically, students with GPAs below 3.3 were more likely to study on their own and ask for friends' help than students with GPAs between 3.3 and 4.0. Scores for health information literacy and for assumptions about information sources did not relate to any of the themes. Students who rated their skills in finding and evaluating health information higher were less likely to say they would “use other resources” than students who self-reported lower skills (t(164) = 2.40, P<0.05).

In sum, when prompted to engage in needs analysis, the students identified multiple gaps and ways to close them. The most common was a method involving self-study, self-control, and individual practice. The majority of respondents also planned to seek help from others. Library staff members were preferred over other helpers.

LIMITATIONS OF THE STUDY

First, the comments were contributed by university undergraduates enrolled in an introductory health course. Half of the participants were freshmen with limited library exposure. Second, although participation rates calculated separately for the completion of the RRSA and a follow-up survey were high, this two-step approach, in which the second step was contingent upon the completion of the RRSA, increased the number of nonparticipants. Therefore, the study findings should be carefully generalized to other settings and academic levels.

DISCUSSION

Similar to participants in Kipnis and Frisby's study [10], 82% of students in this study expected to learn information literacy skills through trial and error, while working on their own. Students expressed the need to spend more time practicing searching and evaluating documents, using both their library and the Internet. The study participants pointed out that this practice was likely to translate into greater efficiency in completing information management tasks, an awareness that might be based on their experiences in solving hands-on RRSA problems, some of which appeared simple but were hard to solve. Although an average student takes about 30 minutes to complete the RRSA, some take over an hour. It would take time to build their own health information competencies, according to some students, but their opinions were mixed on how hard or easy it would be. A few students thought it would be easy (“I would need to change my behavior of just going to Google and typing something in”), while other study participants recognized that this was a complex skill set that might require a structured learning environment (e.g., library classes) and/or assistance from competent others. Prior research has suggested that awareness of task complexity can help improve self-regulation of learning, leading to goal setting, planning, and selecting learning strategies [15]. One implication of this finding is that some students may need instruction on task complexity. That is, students in health disciplines may develop more comprehensive approaches to learning health information competencies if they view them as a set of complex skills, knowledge, and even beliefs and awareness (accurate estimation of one's skill gaps) that can be built over time using multiple strategies.

RRSA exercises and feedback call attention to the differences between popular and scholarly information, a fact that was reflected in students' comments on the need to rely less on the Internet as the sole source of information. This reliance is very strong as most students start their searches in a general-purpose search engine [16]. The desire to improve Internet and evaluation skills was noted as often as the need to improve library skills, suggesting that students might see these as two distinct skill sets. When students talked about building their library skills, they spoke about learning about what the library had to offer, becoming familiar with library websites, understanding library databases and print collections, and so on. Students thought that the library website was assumed to be a reliable source of information. At the same time, students were leery of using general or commercially oriented websites and recognized the need to spend more time to critically review Internet sources.

A strong emphasis on self-reliance in building health information competencies can be explained, at least in part, by students' perceptions that their skill gaps were due to a lack of self-discipline in how they sought information. The study participants reflected on the need to be more mindful, deliberate, and purposeful while searching for and screening documents for their academic assignments. Another explanation for self-reliance is the possibility that the RRSA assessment cued students to engage in self-directed study. First, it provided detailed feedback that showed how much and in which specific areas perceived skills were misaligned with actual skills. Second, it offered a list of external links to resources for building health information skills that students could explore on their own.

Contrary to participants in the Kipnis and Frisby study [10], students who participated in this study identified librarians, not peers, as their first line of assistance if they chose to ask for research help. Over half of all study participants (99 out of 181) mentioned contacts with librarians as a competency-building approach. Peers and professors were only identified as sources for help by 20% or fewer of the respondents. The importance of collaboration among librarians and faculty members is underscored by the fact that few students thought of reaching out for help to their course instructors, the very people who give them information-intensive research assignments.

The strong preference for seeking librarians' assistance may reflect the fact that the RRSA feedback emphasizes the relevance of today's academic libraries. RRSA measures a diverse set of competencies that includes not only objective measures of skills and knowledge, but also attitudes and beliefs that predict them. Students in this study who scored in any feedback category in the top one-third of their normative group received congratulatory feedback and encouragement for further competency development. In contrast, students with lower scores received feedback on gaps and ways of remedying them, accompanied by explanations of the value of using academic libraries and the role of library staff members as health information experts. For example, feedback is provided for a scale that assesses the strength of an assumption that general-purpose search engines always give better information than academic library databases. If this assumption is strong, students get a narrative that explains that many academic databases can deliver fewer but better screened results. It also states that library documents are likely to be scholarly, which is what most professors expect. Students saw their RRSA feedback and test scores prior to being asked the survey questions. As a result, they may have associated building information literacy skill sets with information most frequently available in an academic library from librarians. Educational researchers offer strong arguments that timely feedback that reveals gaps and offers ideas on how to close these gaps “can be one of the most critical influences on student learning” (102) [17]. There appears to be an opportunity for librarians to capitalize on this association and to build appropriate programming to address student information literacy needs.

In addition, the study revealed that some students might not be aware of extensive online resources offered through their academic library that could be accessed without going to the library building. A number of students spoke of visits to the library as a way to contact librarians, noting that the need to go to a different part of campus was a barrier. It was easier to browse the Internet from their study locations. One implication of this finding is the importance of raising students' awareness of the library website as a preferred gateway not only to scholarly databases, but also to online advice from librarians who can be accessed remotely, for example, through chats and electronic request forms.

CONCLUSION

This study provides evidence that individual feedback following self-assessment of health information competencies promotes reflection about personal learning strategies. In this study, many students reflected on the need to build their library skills, Internet skills, and information evaluation skills. The participants also reported the need to learn more about citations and plagiarism. The study highlighted the importance of providing students with resources for self study, while also building student-librarian and faculty-librarian connections.