Abstract
Engagement of the community through informal dialogue with researchers and physicians around health and science topics is an important avenue to build understanding and affect health and science literacy. Science Cafés are one model for this casual interchange; however the impact of this approach remains under researched. The Community Engagement Key Function of the Clinical and Translational Science Institute of Southeast Wisconsin hosted a series of Science Cafés in which topics were collaboratively decided upon by input from the community. Topics ranged from Personalized Medicine to Alzheimer's and Dementia to BioMedical Innovation. A systematic evaluation of the impact of Science Cafés on attendees' self‐confidence related to five health and scientific literacy concepts showed statistically significant increases across all items (Mean differences between mean retrospective pre‐scores and post‐scores, one tailed, paired samples t‐test, n = 141, p < .0001 for all items). The internal consistency of the five health and scientific literacy items was excellent (n = 126, α = 0.87). Thematic analysis of attendees' comments provides more nuance about positive experience and suggestions for possible improvements. The evaluation provides important evidence supporting the effectiveness of brief, casual dialogue as a way to increase the public's self‐rated confidence in health and science topics.
Keywords: translational research, community engagement, evaluation, literacy (science/health), scientists, measurement
Introduction
Patient engagement and personalized medicine demand a deeper dialogue among the public and institutions conducting translational health science research.1, 2, 3 Optimally, this would encourage transitions from scientist‐designed to citizen/scientist‐designed research from the outset.1, 2, 3 Involving the public in scientific research has produced varying levels of success.2, 4, 5 Controversy exists around scientists' understanding of the public and the public having the skill or scientific literacy to understand, and contribute to, advanced research.5, 6, 7 Historically, it has been a point of contention that scientists do not interact well with the public or sufficiently report their results.4, 6, 7 This paper discusses Science Cafés as a way to foster dialogue and evaluate personal disposition1 toward health and scientific literacy topics for attendees. Health literacy was defined as the degree to which individuals have the capacity to obtain, process, and understand basic health information needed to make appropriate health decisions,8 and scientific literacy as the knowledge and understanding of scientific concepts and processes required for personal decision making, and participation in civic, and cultural dialogue.9
Community engagement agenda and scientific citizen/citizen scientist ideal
The NIH Clinical and Translational Science Award program seeks to transform biomedical research processes in part by engaging communities in research.10 Community Engaged Research (CEnR) is central to the Clinical and Translational Science Institute of Southeast Wisconsin's (CTSI) Community Engagement Key Function, following the principles of developing trust, building capacity, equitable treatment of all partners, and shared goals for research and dissemination.2, 3, 10 CEnR is essential in working with segregated communities, like Milwaukee, Wisconsin. Milwaukee ranks number one as the most racially segregated city in the United States, with the highest Black/White dissimilarity index.11 Health disparities within geographic regions of differing socioeconomic status (SES) in Milwaukee have been well documented and historical factors often contribute to community distrust toward science and/or scientists who may not have previously worked in transparent and trustworthy ways.12, 13, 14 In this sociopolitical and economic environment, a CEnR approach presents a unique chance to engage in dialogue with the community on issues of mutual interest and importance.
Science Cafés embody the mission of creating bidirectional dialogue based on participatory action research.5, 15 The Cafés focused on health issues prioritized by a council of representatives from community‐based organizations and CTSI academics and is one way to approach the Citizen/Scientist ideal. Evaluating the impact of each Café on attendees' disposition toward health and scientific literacy topics was carried out and suggests that Cafés are a useful approach for creating dialogue between researchers and the community.
History of the Science Café approach
Open public forums discussing intellectual ideas have existed for thousands of years.15, 16 While names have changed, the intent remains the same; to provide an opportunity to learn, defend, and contribute thoughts on the latest in research. Science Cafés have been present nationally and internationally since the 1990s as, “a place where, for the price of a cup of coffee or a glass of wine, anyone can meet to discuss the latest ideas of science that are impacting society.”17 In the United States, Science Cafés vary in size, shape, and purpose.18
Few organizers of Cafés appear to conduct a formal evaluation, aside from attendee satisfaction or demographic information; a review found that none of these past efforts measured change in perceptions or knowledge.19 In health and scientific literacy evaluation, there are a few instances of evaluation of the public; typically focusing on students or patients.20, 21 The TALKDOC tool demonstrates the importance of attitude and confidence on health literacy measuring disposition toward health literacy to assess competencies. Disposition includes self‐confidence and attitudes toward health information and prevention necessary for individuals to make functional decisions regarding health care treatment.21 While debates exist for what qualifies as “literate” in this context, research suggests evaluations must also take into account the cultural history and value system of communities.12, 16
The primary goal in evaluating the CTSI Science Cafés was to understand their impact on attendees' understanding of health and scientific information.
We hypothesized that (1) postscores would be greater than retrospective prescores, demonstrating the impact of this brief intervention, (2) repeat attendees would have increased confidence compared to first time attendees, and (3) qualitative analysis of written comments on the evaluation would support the quantitative findings.
Methods
Research activities and the evaluation tool were approved by the Medical College of Wisconsin Institutional Review Board.
Procedures
Flyers were distributed to libraries, community‐based organizations, and e‐mail notifications were sent to CTSI members and previous Café attendees. The Cafés were free and open to anyone interested. After the Café, all participants were asked to complete an anonymous evaluation.
The Café involved a medical professional or researcher engaging with the public in an informal, nonacademic environment (i.e., public library). The speaker provided 15–20 minute background on the evening's topic, notably without PowerPoint, followed by 30–60 minutes of discussion among the audience and speaker.
Sample
Four Science Cafés were held using this evaluation tool; see Table 1 for topics and response rates (prior 12 Cafés used different tool; data not included2). Demographic characteristics are reported in Table 2. Ethnic information was not collected in the demographic questions. Zip codes where attendees live were collected, 60.3% from the City of Milwaukee, 37.6% from surrounding towns of Southeast Wisconsin, and 2.1% unreported. Throughout the series, there was a presence of higher, middle, and lower socioeconomic status (SES) zip codes from the city of Milwaukee.14 Over one‐third (38.2%) of the sample reported they were “first time attendees.” Based on analysis of registration lists from previous Cafés and the four studied in this paper, this is consistent with the history of the CTSI Cafés; there have been a total of 334 unique registrants and 234 are one‐time attendees.
Table 1.
Science Café topics and attendance
| Date | 2013 Science Café topic | Actually attended | Retrospective pre– postevaluations | Postresponse rate |
|---|---|---|---|---|
| March | Obesity and Lifestyle modifications | 41 | 34 | 82.9% |
| April | Alzheimer's and Dementia | 46 | 42 | 91.3% |
| May | Technology advancements | 33 | 25 | 75.8% |
| June | Breast cancer screening | 46 | 40 | 87.0% |
| Totals | 166 | 141 | 84.9% |
Table 2.
Demographic characteristics of participants
| Participants (n = 141) | ||
|---|---|---|
| n | % | |
| Gender | ||
| Male | 35 | 25.5 |
| Female | 102 | 74.5 |
| Age | ||
| 60+ | 43 | 31.2 |
| 50–59 | 23 | 16.7 |
| 40–49 | 46 | 33.3 |
| 30–39 | 13 | 9.4 |
| 20–29 | 12 | 8.7 |
| 19 or < | 1 | 0.7 |
| Education | ||
| Graduate degree | 65 | 46.4 |
| Bachelor's degree | 44 | 31.4 |
| Associate's degree | 10 | 7.1 |
| Some school | 17 | 12.1 |
| High school graduate | 4 | 2.9 |
| No response | 1 | 0.1 |
| Attendee status | ||
| First‐time | 52 | 38.2 |
| Repeat | 84 | 59.6 |
| No response | 5 | 2.2 |
The majority of attendees were highly educated which is consistent with research on health information seeking behavior.22 More females than men attended also consistent with research that women are more likely to seek out information related to their own health and their families, than men.22
Measures
Increasingly, evaluators advocate for replacing the pre‐postdesign3 with retrospective pretest for very short duration interventions, particularly if it is not feasible to administer a pretest, and if the goal is to measure more general knowledge of a topic.23 Both designs capture outcome data; self‐reported changes in knowledge, attitudes, and/or behaviors. Research suggests that incorporating a retrospective pretest may provide more valid results than a traditional pretest if, at the time of the traditional pretest, participants lack familiarity with an area or concept they are rating (i.e., experience limitation)23 or when a participant's frame of reference is likely to change over the course of an event (e.g., participants may enter thinking they have a very good understanding and confidence in their abilities to talk about breast cancer, only to discover that their understanding was not as extensive as they thought after attending). This response‐shift sometimes causes misleading results.24
Attendees completed a five‐item scale assessment of health and scientific literacy (Figure 1 ). Literature on measuring and defining health and scientific literacy,8, 9, 16 input from the Community Engagement Citizens Advisory Council, and data from previous Café evaluations informed the development of the evaluation tool. The measure used a seven‐point Likert‐type scale of confidence with (1) indicating low and (7) indicating high confidence.
Figure 1.
Five areas of scientific and health literacy.
The five‐item health and scientific literacy scale had an excellent internal consistency (n = 1264, α = 0.87). Item‐total correlation analyses show the alpha would not improve if any of the items were removed. Although Cronbach's alpha indicates good internal consistency of the items in the scale, dimensionality, reliability, and validity questions were not explored further.
Open‐ended comments were collected from the evaluation question “Please write any suggestions for future Science Café topics or general comments.”
Data analysis
Paired samples t‐test was used to test for statistical differences between retrospective pre‐ and postratings on the five areas of health and scientific literacy, using a one‐tailed test. De‐identified written comments were coded independently by two independent raters. After codes were generated and developed, comments were then assigned into groups. A total of 49 comments were coded, falling into five groups. Codes, frequencies, and exemplar comments are shown in Table 4.
Table 4.
Qualitative analysis of comments
| Frequency | Code | Example responses |
|---|---|---|
| 15 | Positive Comment | “Good presentation and Q&A”; “Brought information to my level of understanding” |
| 13 | Suggestion | “A handout or slides with the technical terms—proteins, drugs, etc.”; “The medical terms were hard to grasp and I know I won't remember. Pictures would be helpful for me.” |
| 10 | Topic | “Diabetes, Kidney, Heart cure, and prevention of complications”; “Exercise Metabolism, pollution affects health” |
| 5 | Negative Comment | “I would have liked to hear more of the science of diet and exercise”; “Too many questions, went on too long” |
| 4 | Opinion | “Food fed into problem of obesity: soda, starch, chips”; “I already work in science” |
Results
This study evaluated measures of health and scientific literacy from Science Café evaluation tool to understand the impact of attending the events.
Does attending a Science Café increase attendees' perceived confidence in health and scientific literacy on the topic?
When comparing postscores with retrospective prescores, significant differences were found on all five health and scientific literacy statements, showing that attending a Café increased attendees' perceived confidence in health and scientific literacy. Statements 2 and 3 (Figure 1 ) had the highest increases in confidence level following attending Café. The lowest increase in confidence was observed with statement 1. Table 3 outlines the means, standard deviations, and difference values between post‐Café ratings and retrospective preratings for each of the areas of health and scientific literacy.
Table 3.
Mean and standard deviations for scientific and health literacy areas
| Scientific and health literacy areas | Mean retrospective Prescores (s.d.) | Mean postscores (s.d) | Mean difference (95% CI) | DF | p † |
|---|---|---|---|---|---|
| I am confident in my… | |||||
| Ability to talk about [today's Science Café topic] with a healthcare provider | 5.5 (1.5) | 6.2* (0.9) | +0.70 (0.52–0.86) | 128 | <0.0001 |
| Ability to tell what information is trustworthy or not on [this Science Café's topic] | 5.4 (1.5) | 6.1* (1.1) | +0.65 (0.47–0.83) | 125 | <0.0001 |
| Ability to speak to a scientist or a health researcher | 5.6 (1.5) | 6.2* (1.1) | +0.60 (0.41–0.77) | 128 | <0.0001 |
| Ability to find other sources of information on [this Science Café's topic] | 5.7 (1.4) | 6.2* (1.1) | +0.52 (0.36–0.66) | 127 | <0.0001 |
| General understanding of the methods used by scientists | 5.5 (1.5) | 6.0* (1.1) | +0.50 (0.32–0.68) | 128 | <0.0001 |
* Statistically significant at p < 0.0001.
† One tailed, paired samples t‐test; n = 141 for both retrospective pre‐ and posttests; 1= low to 7 = high.
Do participants express comments that show they are learning at the Science Café?
Table 4 displays the major themes identified in open‐ended comments about the Cafés. The majority of the comments were positive, suggesting that attendees enjoyed the Café or had a good grasp of the information. Attendees also suggested the need for tools which might reinforce or expand the topics using other learning modalities. This request for tools could help to increase attendees' confidence further in the future in statement 1 which showed the lowest increase in confidence. Ten comments recommended topics for future Cafés. Few negative comments were received; however several comments suggest some attendees' preferred additional scientific evidence be presented. Opinion comments were limited and did not conflict with the quantitative findings.
Is there a difference in disposition toward health and scientific literacy between repeat attendees and new attendees?
Independent samples t‐test compared means postscores between those attending a Science Café for the first time (n = 74) and those attending a Café more than once (n = 51). No significant difference was found when comparing disposition toward health and scientific literacy between new and repeat attendees. Mean scores differed by no more than 0.29 on the seven‐point Likert response scale. This is contrary to what was expected, i.e., that repeat attendees would have greater increases in confidence toward health and scientific literacy topics given their prior attendance of Cafés. Since it is not known how many Cafés repeat attendees have come to there may be too diverse of a population to tell if there is any effect from attending multiple times.
Discussion
Science Cafés provide an opportunity to discuss scientific advances in a public setting for little to no cost with a high return for both community members and academics. Scientists have faced criticism for not interacting well or reporting results with the public.4, 6, 7 Cafés provide a comfortable venue and encourage dialogue that bridge areas of misunderstanding.
This paper presents initial findings suggesting attending a Science Café improves self‐rating of confidence on the five item scale and that there is no significant measured difference between repeat attendees and new attendees. While the current study does not fully validate the scale used, future efforts should focus on addressing this limitation. The series of Science Cafés was held in a central, public location; however the representation of genders, educational level, and SES was skewed toward well‐educated women consistent with observed behavioral patterns. Zip code analysis suggests the population that attends the Science Cafés is not only from the city of Milwaukee, but from surrounding areas in Southeastern Wisconsin. Together, these limit the extent to which generalizations can be made. In light of this limitation, we plan to access the program's Citizen Advisory Council to provide input on additional ways of positioning the Science Cafés as an attractive event for males and less educated individuals to attend and to bring in a different population. Another limitation is lack of long‐term evaluation of a Café's impact on self‐rated confidence. Evaluating future informal events and impact on long‐term literacy levels would benefit from a cohort of individuals with validated evaluation tools.
Qualitative analyses of comments suggest an overall positive review of the Science Cafés. Future studies should incorporate handouts or visual aids as learning tools—while still avoiding PowerPoint presentations. Qualitative comments indicate what resonates with attendees. Future focus groups and personal interviews from attendees and academic presenters in the Cafés would further highlight the individual impact.
Conclusions
Engaging scientists and communities through health and science dialogue using a Café format is an effective approach for increasing self‐rated levels of confidence in these topics. This, overall, leads to an opportunity for both scientists and the community to understand each other's perspective in a nonformal setting and provides an opportunity to increase in health and science literacy. Through Science Cafés and bidirectional dialogue, a pathway for translational researchers and citizen scientists is created with the vision of future engagement of the public to collaborate on mutually beneficial research projects.
Acknowledgments
This project is funded by the Clinical and Translational Science Institute of Southeast Wisconsin, a recipient of a National Clinical and Translational Science Award 8UL1TR000055 and by Advancing a Healthier Wisconsin Research and Education Initiative Fund, a component of the Advancing a Healthier Wisconsin endowment at the Medical College of Wisconsin.
Footnotes
For the purposes of this paper, disposition is defined as a tendency to think or act in a particular way.
The evaluation tool used previously did not provide substantial insight into the intervention, hence why the evaluation tool was changed.
Comparison of pre and retrospective prescores were also tested and no significant differences were found
Fifteen records excluded.
References
- 1. Hasnain‐Wynia R, Sheridan S. Conversations with the Lating/Hispanic community. 2013. Available at: http://www.pcori.org/blog/conversations‐with‐the‐latinohispanic‐community/. Accessed October 28, 2013.
- 2. Ahmed SM, Palermo AG. Community engagement in research: frameworks for education and peer review. Am J Public Health. Aug 2010; 100(8): 1380–1387. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Jones L, Wells K. Strategies for academic and clinician engagement in community‐participatory partnered research. JAMA. Jan 24 2007; 297(4): 407–410. [DOI] [PubMed] [Google Scholar]
- 4. Scheufele DA. Communicating science in social settings. Proc Natl Acad Sci U S A. Aug 20 2013; 110(Suppl 3): 14040–14047. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Bonney R, Ballard H, Jordan R, McCallie E, Phillips T, Shirk J, Wilderman C. Participation in Scientific Research: defining the Field and Assessing Its Potential for Informal Science Education. Washington DC: Center for Advancement of Informal Science Education; 2009. [Google Scholar]
- 6. Peters HP. Gap between science and media revisited: scientists as public communicators. Proc Natl Acad Sci U S A. Aug 20 2013; 110(Suppl 3): 14102–14109. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Condit C. Science reporting to the public: does the message get twisted? CMAJ. Apr 27 2004; 170(9): 1415–1416. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Ratzan SC, Parker RM. Introduction In: Selden CT ZM, Ratzan SC, Parker RM, eds. National Library of Medicine Current Bibliographies in Medicine: Health Literacy. NLM Pub No. CBM 2000‐1 Bethesda, MD: National Institutes of Health, U.S. Department of Health and Human Services; 2000. [Google Scholar]
- 9. National Science Education Standards . 2013. Available at: http://www.nap.edu/openbook.php?record_id=4962&page=22. Accessed October 23, 2013.
- 10. Michener L, Cook J, Ahmed SM, Yonas MA, Coyne‐Beasley T, Aguilar‐Gaxiola S. Aligning the goals of community‐engaged research: why and how academic health centers can successfully engage with communities to improve health. Acad Med. Mar 2012; 87(3): 285–291. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Jacobs H, Kiersz A, Lubin G. The 25 most segregated cities in America. 2013. Available at: http://www.businessinsider.com/most‐segregated‐cities‐in‐america‐2013–11?op = 1. Accessed January 22, 2014.
- 12. Roth WM, Lee S. Science education as/for participation in the community. Sci Educ. 2004; 88(2): 263–291. [Google Scholar]
- 13. Haynes EN, Beidler C, Wittberg R, Meloncon L, Parin M, Kopras EJ, Succop P, Dietrich KN. Developing a bidirectional academic‐community partnership with an Appalachian‐American community for environmental health research and risk communication. Environ Health Perspect. Oct 2011; 119(10): 1364–1372. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Chen H‐Y, Baumgardner DJ, Frazer DA, Kessler CL, Swain GR, Cisler RA. Milwaukee Health Report 2012: Health Disparities in Milwaukee by Socioeconomic Status. Milwaukee, WI: Center for Urban Population Health; 2012. [Google Scholar]
- 15. Parsons JA, Lavery JV. Brokered dialogue: a new research method for controversial health and social issues. BMC Med Res Methodol. 2012; 12: 92. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Holbrook J, Rannikmae M. The meaning of scientific literacy. Int J Environ Sci Educ. 2009; 4(3): 275–288. [Google Scholar]
- 17. Dallas D. Cafe Scientifique–deja vu. Cell. 2006; 126(2): 227–229. [DOI] [PubMed] [Google Scholar]
- 18. Loew T. Science pubs quench a thirst for knowledge. USA Today. January 21, 2011. [Google Scholar]
- 19. Bitgood S, Ahmann K. Science Cafe: summary of evaluations and lessons learned In: Social Design in Museums: The Psychology of Visitor Studies. Vol. 2 Edinburg, UK: Museum, Etc; 2012: 374–395. [Google Scholar]
- 20. Gormally C, Brickman P, Lutz A. Developing a Test of Scientific Literacy Skills (TOSLS): measuring undergraduates' evaluation of scientific information and arguments. CBE Life Sci Educ. Dec 2012; 11(4): 364–377. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Helitzer D, Hollis C, Sanders M, Roybal S. Addressing the “other” health literacy competencies—knowledge, dispositions, and oral/aural communication: development of TALKDOC, an intervention assessment tool. J Health Commun. 2012; 17(Suppl 3): 160–175. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22. Tu HT, Cohen GR. Striking jump in consumers seeking health care information. Track Rep. Aug 2008. (20): 1–8. [PubMed] [Google Scholar]
- 23. Nimon K, Zigarmi D, Allen J. Measures of program effectiveness based on restrospective pretest data: are all created equal? Am J Evaluat. 2011; 32(1): 8–28. [Google Scholar]
- 24. Klatt J, Taylor‐Powell E. Program Development and Evaluation. Using the Retrospective Post‐then‐Pre Design, Quick Tips #27. Madison, WI: University of Wisconsin‐Extension; 2005: 2pp. [Google Scholar]