Survey of Chinese Medicine Students to Determine Research and Evidence Based Medicine Perspectives at Pacific College of Oriental Medicine

Abstract

Context

Research literacy and the practice of evidence based medicine (EBM) are important initiatives in complementary and alternative medicine (CAM), which requires cultural change within educational institutions for successful implementation.

Objective

To determine the self-assessed research and EBM perspectives of Chinese medicine Masters degree students at Pacific College of Oriental Medicine, New York campus (PCOM-NY).

Design and Methods

A survey with seventeen close-ended questions and one open-ended question was administered through Survey Monkey to students at PCOM-NY.

Results

The survey was sent to 420 Masters students and 176 (41.9%) responded. Students in all 4 years of the Masters degree indicated a generally high degree interest in, and support for the value of research. However increasing years (1 – 4) in the program was associated with lower interest post-graduation in research participation and entering the doctoral program, and 4th year students reported low levels of interest in having greater research content and training in their Masters degree programs. Students who responded to the open-ended question (23% of respondents) expressed enthusiasm for research and concerns about the relevance of research to Chinese medicine.

Conclusions

Consistent with findings in similar studies at CAM colleges, interest in research and EBM of the PCOM-NY Masters students appeared to decline with increasing years in the program. Concerns around paradigm and epistemological issues associated with research and EBM among Chinese medicine students and practitioners warrants further investigation, and may be an important challenge for integrative medicine.

Introduction

Pacific College of Oriental Medicine (PCOM), in collaboration with Albert Einstein College of Medicine (Einstein), is currently funded by the NIH National Center for Complementary and Integrative Health (NCCIH) to develop faculty training and curriculum in evidence informed practice (EIP). This project leverages the previous NCCIH R25 grant program¹ entitled the CAM Practitioner Research Education Project Grant Partnership, aimed at increasing the quality and quantity of research education in CAM institutions^2,3.

Prior to grant submission, surveys were undertaken at PCOM (New York) to determine faculty and student research and EIP knowledge and interest. Outcomes from the faculty survey⁴ showed a generally high degree of interest and support for research, EIP, and institutional participation in research activities. Open-ended question responses also indicated concerns about the relevance of research to Chinese medicine, and the possibility of co-option by biomedicine. Similar concerns and resistance to research and EIP have been reported at other complementary and alternative medicine (CAM) institutions^5,6,7, and it is clear that significant cultural change is necessary for successful implementation of research and EIP training^8,9,3. Students are important stakeholders in determining the culture of an institution. This study describes the outcomes of the student survey investigating students’ self-reported research literacy, interest in research, attitudes and interest in EIP, and the impact of an introductory research course.

Materials and Methods

Survey Development and Implementation

The survey instrument was developed using previously published surveys^10,6,11,12. Following PCOM institutional review board approval, implementation of the survey was undertaken in Survey Monkey via emails sent by the PCOM-NY Academic Dean to 420 students undertaking the 4-year Masters degree at PCOM-NY. Survey participation was voluntary with no penalty for non-participation. An initial email and four reminder emails were sent between July 16^th and August 6^th, 2012. The survey consisted of demographic questions and eighteen questions addressing opinions regarding research value, literacy and interest; science and Chinese medicine; desire for additional research training and future research involvement; and attitudes and beliefs about evidence based medicine (EBM). This survey was a cross-sectional assessment of student cohorts enrolled during a single calendar year and did not assess longitudinal responses of a single entering cohort throughout the Masters program. Responses were in the form of a 5-point Likert scale (5 = Strongly agree; 4= Agree; 3 = Neutral; 2 = Disagree; 1 = Strongly disagree). There was one open-ended question at the end of the survey – ‘Please add any additional comments below’.

Of the 18 close-ended survey questions, 10 were answered exclusively by 4th year Masters degree students who had completed the introductory research course. This course provided students with an overview of research principles and methodologies, with a special focus on issues specific to undertaking research in complementary and alternative medicine.

Statistical Analysis

Demographic characteristics of respondents were described as proportions for the categorical variables. Responses to the Likert Scale were reported as proportion expressing agreement (Agree and Strongly Agree combined). To facilitate interpretation, broad survey question topic area scores were constructed by adding the Likert Scale value (1 through 5) for all questions related to that topic area. For example, an area with 5 questions could have a range of possible scores from 5 to 25 whereas an area with 3 questions could have a range from 3 to 15. There were seven main topic areas: three were presented to all students (years 1–4): Research Value, Science and Chinese Medicine, and Future Research Interest; and four were presented only to 4th year students who had taken the introductory research course: Research Course Value, Literacy, Further Research Training, and EBM Value. Bivariate associations between the topic area scores were assessed with Spearman rank correlations. A Spearman rho (ρ) and P value for that ρ were reported for each bivariate correlation. Statistical significance was ascribed based on a two-tailed alpha of 0.05. Statistical analyses were performed with SPSS for Windows (Version 20). Qualitative theme analysis¹³ was used to assess the open-ended question responses.

Results

Survey Response Rate

The survey was emailed to 420 PCOM-NY Masters degree students and completed by 176, a response rate 41.9%. Table 1 shows the numbers of Masters degree responders, and percentages of total Masters students, that completed the survey in the four years of the Masters program. PCOM has 3 terms per calendar year. The Masters degree at that time was a total of 11 terms that were unevenly divided among the year designation in the program. Year 1 was the first 4 terms (1.33 calendar years) up until the 1^st year comprehensive exams, Year 2 the next 4 terms (1.33 calendar years) up to the 2^nd year comprehensive exams. There was one remaining term in the 3^rd calendar year of the program, term 9, when students undertook the introductory research course. Fourth year students were defined as those who had passed the 2^nd year comprehensive exams and the introductory research course, and were in their final two terms of the Masters program.

Table 1.

Response rate and number of students in each year of the Masters program

Overall survey response rate (students in all degree programs)	41.9%*
Number of 1st year Masters students	64	36.4%
Number of 2nd year Masters students	70	39.8%
Number of 3rd year Masters students	11	6.3%
Number of 4th year Masters students (completed the introductory research course)	31	17.6%

^*Survey was sent to 420 Masters students and completed by 176 students

Demographics

Table 2 presents the sociodemographic data of the Masters degree student survey responders by gender, age, previous education, and years at the College. Nine percent of the students had not previously completed a degree program because the entrance requirement is 60 college credits. The length of time that students had been attending the College is affected by several factors including studying part-time and/or being in several programs simultaneously. It also cannot be directly compared to the year-by-year response rate because those classifications do not correspond directly to calendar years or the number of years that students had been at the College.

Table 2.

Student Demographics

Gender	75% female
Age (years)	18–25 6.3% 26–35 48.9% 36–45 24.4% 46–55 12.5% ≥56 7.9%
Education – highest degree	None: 8.5% Undergraduate: 64.8% Masters: 24.4 % Doctorate: 1.7% Medical Doctorate: 0.6%
Prior medical education	Yes: 23.8%
Ethnic Background	White: 61.4% Asian: 28.4% Hispanic: 8.0% African American or Black: 7.4% Other: 5.7%
Years already at the College	< 1 18.2% 1–2 27.3% 2–3 27.8% 3–4 19.3% >4 7.4%
Prior research education	Yes: 39.8%	Level: 68.6% Undergraduate  34.3% Masters  14.3% Doctoral
Prior research experience	Yes: 28.8%	Role: 14.9% PI/co-PI  61.7% Research assistant  36.2% Res. coordinator  31.9% Data entry  14.9% clinician
		Fields: 69.8% Biological sciences  36.5% Social sciences  15.9% Physical sciences
Current degree program	MSTOM - Acupuncture & Chinese herbs: 70.5% MSAc - Acupuncture: 29.5%

Research Interest Prior to Taking the Introductory Research Course

Students in the first three years of the Masters program, prior to taking the introductory research course, appeared interested (79% agreement) in biomedical research into acupuncture and Chinese herbal medicine (Table 3), and there was no significant difference between the years.

Table 3.

Research interest of first, second and third year students prior to taking the introductory research course

	% Agree + Strongly Agree
Research interest	All	1st yr	2nd yr	3rd yr
I am interested in the biomedical research investigating acupuncture and Chinese herbal medicine	79.2	74.6	84.3	72.7

Research Values, Interests and Beliefs of Masters Students in all Years of the Program

Figure 1 presents the proportions agreeing with statements aimed at gauging the value that students perceive research has in relation to the profession, their future research interests, and their perspective on science and Chinese medicine. Panel B shows that for all questions (listed in Panel A), except question 3, 4^th year students (who had completed the introductory research course) were less in agreement with statements about the value of research and future research activities, although only for questions 4 and 5 was this difference statistically significant. For statements about the value of research to the profession (Questions 1–3) there was a high degree of agreement ranging from 77–89%. Statements about their future research interests and interest in doing the postgraduate Doctoral Program (Questions 6 and 7) showed lower levels of agreement ranging from 55–73%, with the 4^th year students having markedly lower levels of agreement compared to years 1–3 for both statements. Statements about science and Chinese medicine (Questions 4 and 5) showed both low levels of agreement (28–61%) and statistically significant trends of declining agreement from years 1 to 4.

Figure 1.

Comparing students in the first, second, third and fourth years of the Masters degree program. Fourth year students have completed the introductory research course. A is a table listing the 7 survey questions, and B is a graph comparing the combined Agree + Strongly Agree percent of student responses to the survey questions

**P for trend = 0.003

*P for trend = 0.01

Fourth Year Student’s EBM Value, and Research Literacy, Values and Interest

Eighty percent of the 4^th year students agreed that the introductory research course, taken in the final year of their program, expanded their understanding of biomedical research and its application to Chinese medicine. Ninety-three percent of these students agreed they were comfortable finding research literature on the Internet, and 77% agreed they were comfortable critically appraising scientific literature. However much lower levels of agreement were seen for statements about further research training in their Masters degree. Only 37% agreed that they would have liked additional research training as part of their Masters degree, and 30% were in agreement with preferring a greater amount of research material or assignments in other courses in their Masters degree.

Students in their final year of the Masters program perceived value in EBM. Students were most likely to agree (93%) that the quality of patient care is improved when practitioners assess the outcomes of their treatments and use this knowledge to decide upon further treatment strategies. A high proportion of students also agreed (77%) with the concept that defining successful outcomes with patients allows practitioners and patients to agree on what successful treatments are and facilitates patient retention and satisfaction. Similarly 77% agreed that critical evaluation of biomedical research methodology and outcomes is necessary to be able to assess the value and application of biomedical research to the practice of CAM. Seventy-three percent agreed that the quality of patient care is improved when practitioners seek out evidence about effective approaches to treating patients. A much lower level of agreement (57%) was shown for the notion that evidence from well-designed biomedical research clinical trials in which the acupuncture intervention appears appropriate, is useful to an acupuncturist’s clinical practice.

Correlation Between Survey Responses and Prior Research Education and Experience

Correlations between the focus categories of the survey questions, as defined by the seven categories shown in Figure 1 (Panel A) and Table 4 (in the shaded areas) and prior research education and experience were examined with Spearman Rank Correlation Coefficients (Figure 2). Panel A of Figure 2 lists the seven categories of survey question focus, and Panel B shows the correlation coefficients (with P values) between the seven categories of survey question focus along with survey responses of students regarding prior research education and experience. For student in years 1–4, research value, future research interest, and agreeing that Chinese medicine is more of a science than an art, and that the scientific method is compatible with Chinese medicine were all significantly correlated. Prior research education was significantly correlated with future research interest. For 4^th year students, perceived value of the introductory research course was correlated with a desire for further research education and EBM value, and research education was correlated with EBM value.

Table 4.

Survey responses from students who had completed the introductory research course in the 4^th year of the Masters program

Value of Introductory Research Course	% Agree + Strongly Agree
The introductory research course expanded my understanding of biomedical research and the way it can be applied to acupuncture and Chinese herbal medicine research	80
Research Literacy
I am comfortable searching the web for relevant scientific literature	93.3
I am comfortable critically appraising scientific literature	76.7
Desire for Further Research Training
I would have liked to have additional research training as part of my Master’s degree	36.7
I would have preferred a greater amount of research material and or assignments in the other courses (biomedical and Oriental medicine) in my Master’s degree	30
Value of Evidence Based Medicine
Critical evaluation of biomedical research methodology and outcomes is necessary to be able to assess the value and application of biomedical research to the practice of CAM	56.7
Evidence from well-designed biomedical research clinical trails, where the acupuncture intervention appears appropriate, is useful to an acupuncturist’s clinical practice	76.7
Defining what successful outcomes look and feel like with patients allows practitioners and patients to agree on what ‘successful’ treatment is, and facilitates patient retention and satisfaction	76.7
The quality of patient care is improved when practitioners seek out evidence about effective approaches to treating patients	73.3
The quality of patient care is improved when practitioners assess the outcomes of their treatments and use this knowledge to decide on further treatment strategies	96.7

Figure 2.

Correlation between survey question categories and prior research education and experience. A: Survey question topic areas; B: Correlation coefficients and probability significance

**Correlation is significant at the 0.01 (2-tailed)

*Correlation is significant at the 0.05 (2-tailed)

Qualitative Outcomes

Of the 176 Masters students that completed the survey, 41 (23%) responded to the open-ended question. Figure 3 (Panel A) shows the number and percentage of respondents from each year of the Masters program. Of the 41 responses 29.5% were from 1^st year students, 34% were from 2^nd year, 7% were from 3^rd year, and 29.5% from 4^th year. Across all student comments three broad themes were apparent: 1. Research is important for the profession, 2. Biomedical research paradigms are not compatible with Chinese medicine, and 3. Biomedical research cannot account for practitioner intention. Figure 3, Panel B presents student comments divided between these three themes. Eleven of the 41 students made strong statements in support of the usefulness of research and evidence. However, 15 of the 41 students made statements that were less supportive of the value of research, and highlighted issues associated with paradigm differences, and concerns around the inability of current biomedical research methods to account for practitioner intention.

Figure 3.

Analysis of open-ended question - ‘Please add any additional comments below’. A is a table showing the number of Masters students that responded, and B presents student comments categorized into the three most common themes

Discussion

Trend of Declining Value Perception and Interest in Research as Students’ Progress Through CAM Degree Programs

Student’s responses to statements about the impact of research upon patient referrals, insurance coverage, and public perception were highly agreeable and consistently similar across all years of the masters program (Figure 1). Conversely, students were much less agreeable with statements in Figure 1 related to their personal beliefs and attitudes about research. This striking difference in beliefs about the relevance of research to those outside the profession compared to those practicing the medicine was also seen among the PCOM-NY faculty⁴, and has been reported at other Chinese medicine colleges⁶. Chinese medicine students and faculty appear keenly aware of the value and necessity of research to validate and ‘prove’ the effectiveness of Chinese medicine, and the impact this has upon public perception and insurance reimbursement. However this level of enthusiasm is not seen with regard to the value research and EBM may have in clinical practice.

The level of agreement to the statements soliciting student’s personal beliefs and attitudes about research declined from years 1 to 4 (Figure 1). Interest in research involvement after graduation, and in doing the Doctoral program increased slightly from year 1 through 3, then in 4^th year decreased to a level lower than that in 1^st year. This may reflect that the introductory research course had satisfied their research interest, or may suggest a more general overall decline in research interest in 4^th year.

Similar outcomes have been reported in other studies. Surveys of students at Oregon College of Oriental Medicine (OCOM) and the New England School of Acupuncture (NESA)⁶, who both have 3-year degree programs, showed a declining level of interest in both learning about, and participating in, research from 1^st through 3^rd years (75% to 37%, respectively). Conversely, in response to questions related to the perceived value of research to the Chinese medicine profession (public perception and insurance reimbursement), students were very agreeable (80–90%), and this did not decline from 1^st through 3^rd years, and support for the statement ‘Scientific method is inconsistent with Acupuncture and Oriental medicine principles’ increased from an average of 37% in 1^st year to 58% in 3^rd year. Surveys of OCOM students taking new curriculum that was developed through NIH funding aimed at increasing research and EBM knowledge, showed that 14% of students in year 3 of the new curriculum appeared skeptical of how research was conducted and reported, and 24% were of the opinion that research had little or nothing to offer⁵. Similarly, research undertaken to determine the impact of an EBM curriculum upon chiropractic students showed that those that had received the old curriculum, with significantly less EBM content, showed the most favorable attitude towards reading the research literature, however their attitude towards the randomized trial declined throughout their program, and students receiving the old and new curricula reported less support for the importance of EBM continuing education as they progressed through their programs.

The above studies suggest that interest and perceived value for EBM and research may decline as CAM student’s progress through their degree programs. Possible explanations that have been suggested are issues associated with students becoming more knowledgeable about the flaws associated with CAM research methodology⁷, and a possible tension between a growing interest and focus upon clinical activities, and less interest and time for research related activities⁶. Previous studies have shown that EBM educational programs can increase knowledge and skills, but do not have an impact on clinical behaviors¹⁴. Factors that improve student’s engagement are active participation in research activities, and application of newly learned research skills in areas of direct interest and/or relevance to clinical activities⁵. This strongly emphasizes the importance of the clinical component of research and EBM training programs.

Research Skills and EBM Attitudes of PCOM-NY Students and Faculty

Students who had taken the introductory research course generally agreed that it expanded their research knowledge, and were supportive of the value of EBM. These outcomes are mostly similar to those reported by the PCOM-NY faculty⁴, however there are some interesting differences. Students reported high levels of research literacy (average 85% agreeable for the two research literacy questions in Table 4), which were greater than those reported by the PCOM faculty (average 75% for the same questions). Given that 50% of the faculty, and 40% of the students had had prior research training, and 29% of both students and faculty had had prior research experience, this difference in research literacy may be due to the introductory research course taken by the students. Student agreement with the statement ‘Critical evaluation of biomedical research methodology and outcomes is necessary to be able to assess the value and application of biomedical research to the practice of CAM’ was 57%, whereas the faculty were 78% agreeable. A possible reason for this difference may be related to the improved quality of acupuncture research over the past decade. Faculty, having likely had greater exposure to the older poorer quality research, may feel a greater need for critical evaluation of research.

Very high levels of agreement were seen from both faculty (93%) and students (97%) for the statement ‘The quality of patient care is improved when practitioners assess the outcomes of their treatments and use this knowledge to decide on further treatment strategies’. This is very encouraging, as it suggests that there is a high level of awareness of the value of collecting and assessing patient outcomes to inform ongoing patient treatment.

Perceived Philosophical, Epistemological and Paradigm Issues are Challenges for CAM Practitioner Research and EBM Training

Greater proportions of later year students responded to the open-ended question possibly because they may have a better understanding of the research and EBM issues and limitations when applied to CAM. The open-ended comments expressed reservations and concerns about paradigm differences, the appropriateness of using the scientific method to examine Chinese medicine, and the inability of conventional research to take into consideration the role of intuition. These comments are very similar to those expressed by the PCOM faculty⁴, who also recognized the value of research and EBM, but also expressed concerns about paradigm differences, co-option of Chinese medicine by biomedicine, and the relevance of research to the practice of Chinese medicine. The faculty comments have a greater focus on clinical practice, as would be expected, and reflect a deeper consideration for the meaning and possible impact of biomedical research upon the way in which Chinese medicine is practiced.

Culture change has been identified through the NIH NCCIH R25 grant initiative to be a very important aspect of successful implementation of EBM faculty training and new EBM curriculum in CAM training programs^8,9,3. Responses from the PCOM-NY students and faculty, and those from the OCOM and NESA students (discussed above) suggest that Chinese medicine institutions might be more challenged in this arena. Such concerns have also been recognized more broadly as challenges for integrative medicine. Kaptchuk and Miller¹⁵ feel that the challenges to integration are so significant that they advocate for a pluralistic model instead of integration. Kligler and Weeks¹⁶ remind us of the fallibility of the randomized trial, and the fact that ‘evidence’ from randomized trials is frequently disproven. Many articles have been written criticizing the applicability of the randomized controlled trial (RCT) for studying acupuncture^17,18,19. The recently refuted²⁰ notion that placebo acupuncture is equally as efficacious as verum acupuncture came from RCT evidence, largely because of the poor fit of this research model for a therapy like acupuncture. Much of what is available to acupuncturists as research evidence are RCTs. Given these issues, is it really surprising that Chinese medicine students and faculty would be skeptical about using research evidence to inform their clinical practice?

Creating culture change in CAM institutions to support evidence based approaches to CAM clinical practice is, and will continue to be, a significant challenge. Philosophical, epistemological, and paradigm issues appear to be important barriers to EBM in the Chinese medicine community and require further investigation. This and other studies suggest that there may be fundamental worldview differences between biomedical and Chinese medicine practitioners. It is likely that an investigation of these worldviews, and their differences, would provide opportunities to more directly address important barriers to integration and EBM.

Limitations of the Study

This study has several limitations. The response rate was 41.9% and therefore the sample may not be representative of all students. Similarly, the response rate may not be the same for each year of the program and we cannot assess whether this influenced the associations observed with year in program. The response rate to the open-ended question was 23%, and opinions expressed may not accurately represent all students’ perspectives. Single cohorts were surveyed to represent the different years of the Masters program, and no longitudinal analyses of these cohorts were undertaken. Therefore differences between the years may be due to cohort differences rather than changes in attitudes over time. The survey used in this study represented a modification of previously used instruments and was not psychometrically tested. The results related to the association between survey topic areas and research education and experience should be interpreted with caution and do not imply causal relationships. Given that research training is not an accreditation requirement for Chinese medicine colleges, the outcomes at PCOM may not be typical for all Chinese medicine colleges in the US.

Conclusions

Despite these limitations, this study provides important preliminary data to inform future curriculum development in research and EBM in CAM colleges. Concerns around paradigm and epistemological issues associated with research and EBM among Chinese medicine students and practitioners warrants further investigation, and may be an important challenge for integrative medicine.