Development and use of an instrument adapted to assess the clinical skills learning environment in the pre-clinical years

Abstract

Background

The Communication, Curriculum, and Culture (C3) instrument is a well-established survey for measuring the professional learning climate or hidden curriculum in the clinical years of medical school. However, few instruments exist for assessing professionalism in the pre-clinical years. We adapted the C3 instrument and assessed its utility during the pre-clinical years at two U.S. medical schools.

Methods

The ten-item Pre-Clinical C3 survey was adapted from the C3 instrument. Surveys were administered at the conclusion of the first and second years of medical school using a repeated cross-sectional design. Factor analysis was performed and Cronbach’s alphas were calculated for emerging dimensions.

Results

The authors collected 458 and 564 surveys at two medical schools during AY06-07 and AY07-09 years, respectively. Factor analysis of the survey data revealed nine items in three dimensions: “Patients as Objects”, “Talking Respectfully of Colleagues”, and “Patient-Centered Behaviors”. Reliability measures (Cronbach’s alpha) for the Pre-Clinical C3 survey data were similar to those of the C3 survey for comparable dimensions for each school. Gender analysis revealed significant differences in all three dimensions.

Conclusions

The Pre-Clinical C3 instrument’s performance was similar to the C3 instrument in measuring dimensions of professionalism. As medical education moves toward earlier and more frequent clinical and inter-professional educational experiences, the Pre-Clinical C3 instrument may be especially useful in evaluating the impact of curricular revisions.

INTRODUCTION

Medical students are influenced by the actions of those around them [1]. Certain attitudes and behaviors, such as professionalism, are particularly linked to learners’ experiences of how others talk about patients and colleagues [2]. The informal educational environment has been termed the hidden curriculum [2] and the Liaison Committee of Medical Education (LCME) of the American Association of Medical Colleges and the American Medical Association requires the educational environment to be evaluated to maintain appropriate standards of professionalism [3].

While many instruments have been used to measure professionalism and organizational culture [4–6], the Communication, Curriculum, and Culture (C3) survey is one of the most carefully developed measures of professionalism and patient-centered care [7]. This 29-item instrument has constructs that include role-modeling, observed interactions, and perception of support for students’ patient-centeredness. In developing the C3 instrument, Haidet et al. [7] used a process of literature review, experts’ item refinement, and assessment of its use with factor and reliability analyses. By demonstrating differences between schools that external experts deemed as having a high or low likelihood of producing patient-centered physicians, discriminant validity was suggested. In a more comprehensive validation study from the same working group, the C3 instrument was shown to differentiate among training sites [8], and it has been applied, subsequently, to other cultural settings [9].

The C3 survey is generally given to third- or fourth-year medical students (MS3 and MS4) to assess their clinical experiences; however, tools to assess pre-clinical experiences are limited. Two factors argue for evaluating the learning environment of first- and second-year medical students (MS1 and MS2). First, the pre-clinical years are associated with changes in students’ self-reported patient-centeredness [10–11], idealism [12], and empathy, with “hardening of the heart” [13]. The precise factors contributing to those observations in the pre-clinical years--including the impact of the faculty teaching [14] and the learning environment on these observations, whether from a limited number of clinical experiences or from basic science training, are not well understood. Second, ongoing reform efforts are attempting to better integrate medical school curricula by adding clinical experiences to the pre-clinical years [15–17]. Understanding the impacts of curricular revisions requires assessing their effects on the learning environment. In anticipation of the reorganization of the pre-clinical medical student experience, and to further understand the factors influencing the learning environment and the changes, we modified and shortened the C3 instrument to measure the pre-clinical years’ hidden curriculum, resulting in development of the Pre-Clinical C3 survey (Pre-C3 survey). We report the development, reliability and initial use of the Pre-C3 survey at two medical schools.

METHODS

Survey Development

A committee of faculty members and education staff at Oregon Health & Science University (OHSU), working under a National Institutes of Health grant to enhance the curriculum in behavioral and social sciences, adapted the 29-item C3 survey into a ten-item Pre-C3 survey. The original C3 survey includes dimensions of “Role Modeling” (15 items), “Patients as Objects” (5 items), “Learning Relationships” (4 items), “Communicating Bad News” (2 items), and “Support for Students’ Patient-Centered Actions” (3 items). Because certain items on the C3 survey, such as “Communicating Bad News”, are a limited aspect of the pre-clinical experience, those dimensions were not included in the Pre-C3 survey items. Table 1 compares the derived Pre-C3 instrument and the corresponding C3 survey items. Since the clinical components of the MS1 and MS2 curriculum involve physicians from a range of disciplines, three items were added to capture the construct of how faculty talked about each other, “Talking Respectfully of Colleagues.” As with the original C3 instrument, students answer using either a five-point Likert scale for observed behaviors (1=Very Often to 5=Never) or a five-point scale of perceived support for their own behaviors (1=Discourage to 5=Completely Encourage).

Table 1.

Comparison Derived From Pre-C3 Survey Items and Corresponding Original C3 Survey Items

Pre-Clinical C3 Survey (Pre-C3 Survey) Items	Corresponding Original C3 Survey Items
Patients as Objects
When you hear an instructor discussing a patient’s case history, the patient is referred to as a diagnosis (such as, I saw an MI [myocardial infarction] …).	You overhear an attending physician discussing a patient’s case history with another attending or house officer. During the course of the conversation, the patient is referred to as a diagnosis (e.g., “I had a great pancreatitis on my team the other day”).
Social history information about a patient is not included or dismissed as not important when patients are discussed.	When you describe social history information about a patient (e.g., career, hobbies) during ward rounds, you notice that the rest of the team is not paying attention.
When students discuss patients, the histories they use refer to patients as diagnoses (such as, I saw an MI [myocardial infarction] …)	You hear students telling stories about patients. These stories tend to portray patients as diagnoses rather than unique human beings.
Learning Relationships
When you hear students telling stories about patients, the students include how their relationship with the patient personally affected them.	You hear students telling stories about patients. These stories tend to portray how the relationship with the patient affected the student(s) personally.
Talking Respectfully of Colleagues
My basic sciences instructors are respectful when they talk about practicing physicians.
My physician instructors and preceptors talk respectfully of complementary and alternative medicine providers.
My physician instructors and preceptors talk respectfully about the care provided by physicians of different specialties.
Support for Students’ Own Patient-Centered Behaviors
In general, when I made an effort to develop rapport with patients, my instructors____me	In general, when I made an effort to develop rapport with patients, my instructors______me.
In general, when I made an effort to get to know patients as unique persons, my instructors______me.	In general, when I made an effort to get to know patients as unique persons, my instructors______me.
In general, when I made an effort to legitimize patients’ concerns about their condition or care, my instructors______me.	In general, when I made an effort to legitimize patients’ concerns about their condition or care, my instructors______me.

Setting and Participants

The Pre-C3 survey was implemented at two medical schools, Indiana University (IU) and OHSU. At both IU and OHSU, medical students completed the survey near the end of their first and second years. The data analyzed includes surveys completed by IU students who entered medical school in 2006 and 2007 and by OHSU students who entered medical school in 2007, 2008, and 2009. At IU, surveys were distributed and collected in late May. At OHSU, first-year surveys were distributed and collected prior to a May required course lecture, and second-year surveys were distributed during the June required Transition to Clerkship course. For all years at both schools, participation in the survey was voluntary and responses were anonymous. Surveys included the Pre-C3 items as well as demographic information. The study protocol was approved and granted an exempt status by the institutional review boards at both IU and OHSU.

The curriculum at IU includes basic science courses in the first two years of training, along with a longitudinal Introduction to Clinical Medicine (ICM) course. ICM is taught in small groups and includes history taking, physical examination, and doctor-patient relationship skills development. Similarly, the OHSU curriculum had a traditional structure, with morning courses in basic science (MS1 year) and pathophysiology (MS2 year), paired with one weekly afternoon in a two-year Principles of Clinical Medicine course (lecture/discussion) and a clinical preceptorship. No major changes were made in the overall structure of either school’s curriculum during the study period.

Data Analysis

All Pre-C3 items were scored so that a higher score indicated appropriate behaviors (this required reverse scoring for some of the items). Using the combined data from the two IU cohorts and from all three OHSU cohorts, two factor analyses with Varimax rotation were performed to assess the construct validity of the survey results. Factors with eigenvalues greater than 1.0 were retained. The internal consistency of data was determined with Cronbach’s alpha testing. Subscale scores were calculated for each factor by summing the individual item scores, then dividing by the number of items for each subscale.

The internal consistencies of construct items for the two institutions were compared, and the Pre-C3 instrument was used to compare students when partitioned by gender. Comparison was by t-test. Statistical analysis was performed using STATA, version 12 (manufactured by StataCorp LP, College Station, TX).

RESULTS

Table 2 presents a matrix of the number of surveys assessed across the five cohorts at the two schools along with descriptive information about the five groups. At IU, a total of 458 surveys were assessed, with class sample sizes of 219 and 239 in AY06-07 and AY07-08, respectively. The response rate at IU was 91 percent, and the gender and race distributions were comparable across the two cohorts. A total of 564 surveys were assessed at OHSU. The OHSU class sample sizes ranged from 76 to 108 in AY07-08, AY08-09, AY09-10, with an average response rate of 79 percent. The gender and race distributions were comparable across the three cohorts for OHSU, as well.

Table 2.

Number of Pre-C3 Surveys Assessed Across the Five Study Cohorts at IU and OHSU

	Cohort
	Indiana University (Year of Entry)		OHSU (Year of Entry)
	2006	2007	2007	2008	2009
Number of Students in Cohort	219	239	121	120	121
Number of 1st Year Respondents		239	107	99	108
Number of 2nd Year Respondents	219		76	86	88
Average Age (range)	26 (22–45)	25 (21–41)	27 (21–42)	27 (23–39)	26 (22–47)
Male %/Female %	58/42	54/46	46/54	45/55	50/50
Black %	7.1	5.1	2.5	1.7	0
American Indian %	1.4	0.9	0.8	0.8	1.7
Asian %	8.6	15.3	14.9	22.5	21.5
Caucasian %	72.5	67.0	76.0	56.7	62.8
Hispanic %	7.3	6.3	0	0	0
Unknown %	3.0	5.5	5.8	18.3	14.0

Using data from the two IU and three OHSU cohorts produced three factors with three items each (Table 3). One of the ten survey items, measuring whether students discuss how their relationships with patients personally affected them, did not load at 0.45 or greater on any factor. Accordingly, that item was not included in the analysis. The three identified factors, “Patients as Objects”, “Talking Respectfully of Colleagues”, and “Support for Student’s Own Patient-Centered Behaviors”, explained 68.6% and 60.2% of the variance in the ratings for IU and OHSU, respectively.

Table 3.

Three Common Factors Surface From IU and OHSU Pre-C3 Survey Data Ten-Item Factor Analyses

	Factor Loadings
	IU			OHSU
	1	2	3	1	2	3
	Patients as Objects
When you hear an instructor discussing a patient’s case history, the patient is referred to as a diagnosis (such as, I saw an MI [myocardial infarction] …).			.882			.811
Social history information about a patient is not included or dismissed as not important when patients are discussed.			.689			.469
When students discuss patients, the histories they use refer to patients as diagnoses (such as, I saw an MI [myocardial infarction] …).			.856			.821
Learning Relationships *
When you hear students telling stories about patients, the students include how their relationship with the patient personally affected them.		.315				.365
Talking Respectfully of Colleagues
My basic sciences instructors are respectful when they talk about practicing physicians.		.804			.718
My physician instructors and preceptors talk respectfully about the care provided by physicians of different specialties		.842			.816
My physician instructors and preceptors talk respectfully of complementary and alternative medicine providers.		.732			.671
Patient-Centered Behaviors
In general, when I made an effort to develop rapport with patients, my instructors______me.	.923			.891
In general, when I made an effort to get to know patients as unique persons, my instructors______me.	.937			.921
In general, when I made an effort to legitimize patients’ concerns about their condition or care, my instructors______me.	.875			.827

^*Factors loading less than 0.40 not used

For the retained Pre-C3 survey items, the mean scores, ranges, and Cronbach’s alphas are shown in Table 4. Students used the entire response range and the mean scores were comparable to those reported for the original C3 survey items.[7] For the two dimensions derived from the C3 instrument, “Patients as Objects” and “Support for Students’ Own Patient-Centered Behaviors”, the Cronbach’s alphas for IU were 0.74 and 0.92, and for OHSU, 0.58 and 0.88, respectively, both comparable to the original C3 instrument.[7] Reliability for the “Talking Respectfully of Colleagues” dimension was 0.77 and 0.61 for IU and OHSU, respectively, and it accounted for 20.8 percent and 17.3 percent of the variance, respectively.

Table 4.

Mean Scores and Cronbach’s Alpha Test Results From the IU and OHSU Pre-C3 Survey Data

	IU			OHSU
	Mean Score	SD	Cronbach’s α	Mean Score	SD	Cronbach’s α
Patients as Objects a	3.55	0.94	0.74	3.42	0.71	0.58
1. When you hear an instructor discussing a patient’s case history, the patient is referred to as a diagnosis (such as, I saw an MI [myocardial infarction] …)	3.48	1.17		3.45	0.93
2. Social history information about a patient is not included or dismissed as not important when patients are discussed.	3.76	1.11		3.39	0.92
3. When students discuss patients, the histories they use refer to patients as diagnoses (such as, I saw an MI [myocardial infarction] …).	3.37	1.18		3.41	1.03
Talking Respectfully of Colleagues a b	4.07	0.71	0.77	3.92	0.60	0.61
4. My basic sciences instructors are respectful when they talk about practicing physicians.	4.36	0.79		4.30	0.70
5. My physician instructors and preceptors talk respectfully about the care provided by physicians of different specialties	4.28	0.80		4.15	0.70
6. My physician instructors and preceptors talk respectfully of complementary and alternative medicine providers.	3.59	0.96		3.31	0.96
Patient-Centered Behaviors c	4.26	0.80	0.92	4.00	0.87	0.88
7. In general, when I made an effort to develop rapport with patients, my instructors______me.	4.33	0.86		4.14	0.92
8. In general, when I made an effort to get to know patients as unique persons, my instructors______me.	4.25	0.86		3.94	0.99
9. In general, when I made an effort to legitimize patients’ concerns about their condition or care, my instructors______me.	4.21	0.88		3.90	0.98

^aScores are a five-point scale of 1=Very Often to 5=Never.

^bThese items were reverse-scored so that a higher score represents a more favorable behavior.

^cScores are on a five-point scale of 1=Discourage to 5=Completely Encourage.

As an initial assessment of discriminant validity, and because there is a suggestion that gender may influence perceptions of the learning environment [18], we compared scores for female and male students (Figure 1). On each dimension, female students within each school reported significantly higher scores for the observed experiences and reported feeling significantly more supported in their patient-centered behaviors than did male students. These scores had magnitudes comparable to construct differences observed when the C3 instrument was used to compare high and low performing schools.[7]

Fig. 1.

Comparison of Gender Differences within School for Mean Pre-C3 Survey Data Construct Scores

DISCUSSION

The derived Pre-C3 instrument is brief, easily administered, simple to score and a useful index of the pre-clinical learning environment. Given that it is a modification from a well-established instrument with extensive prior use, the face and content validities of the Pre-C3 instrument are high. The construct’s reliability was acceptable, and the values obtained were similar to those from the C3 instrument, suggesting the Pre-C3 instrument may be useful for monitoring the quality and changes of the professionalism climate in the pre-clinical experience (comparable to using the C3 survey for monitoring the hidden curriculum in the clinical years).

The study results for both IU and OHSU were similar; one explanation for the similarity may be the traditional curriculum structure that both institutions currently share. Application of the Pre-C3 instrument in more heterogeneous educational environments is needed to further measure the instrument’s ability to monitor changes in the pre-clinical environment. One opportunity to do so is IU’s implementation of a more integrated curriculum scheduled in 2015 and OHSU’s new curriculum, scheduled to begin in August of 2014. OHSU will transition from a traditional model to a competency-based model. The Pre-C3 instrument may be a useful tool to evaluate curriculum transformation efforts such as those occurring at both IU and OHSU.

The Pre-C3 survey measures three important dimensions of the learning environment during medical school training: viewing “Patients as Objects”, “Talking Respectfully of Colleagues”, and “Support for Students’ Own Patient-Centered Behaviors”. The LCME acknowledges that “the learning environment includes both formal learning activities and the attitudes, values, and informal ‘lessons’ conveyed by individuals who interact with the medical student” [19]. Using the Pre-C3 instrument to measure formal and informal experiences can guide curricular revisions which support the modeling of professional attributes in the educational environment.

The results of this study show a significant difference between male and female perceptions on all three dimensions. Both IU and OHSU have approximately equal ratios of male to female students. Because most of their pre-clinical curriculum is taught in lectures or small groups, the experiences of female and male student’s experiences should be comparable. The Pre-C3 instrument measures not only the environment, but also the learner’s perception of that environment, and perceptions may vary across genders. Research has found differences in observations of behaviors by gender. Specifically, Krupat, Pelletier, and Chernicky [20] found that females record observations of relationships, teams, and social climate twice as often as males, whereas males record observations focused on learning by doing, learning by observation and receiving instruction and feedback, more often than females. The difference in types of behaviors observed or noticed by females versus males, may contribute to the gender effects in this setting.

This study is limited by the inclusion of only two medical schools in the analyses; therefore, findings may not be representative of all U.S. schools. Additionally, we found that others have used different methods for evaluating the hidden curriculum such as qualitative interviews [21], reader’s theater [22], licensing exam scores [23], or surveys regarding the ethics of receiving gifts [24], however, we believe there is no other quantitative assessment instrument specifically designed to measure the pre-clinical hidden curriculum in medical school.

CONCLUSION

As hospitals and clinics move toward a patient-centered medical home model, it is important to expose students to patient-centered professional behaviors and attitudes as early as possible. The Pre-C3 instrument is a tool, which may be useful for evaluating the impact of curricular revisions, including earlier and more frequent inter-professional educational experiences in clinical environments emphasizing patient-centered care. While the C3 instrument measures these qualities in the clinical years, the Pre-C3 instrument is a complementary tool useful for measuring aspects of the training environment earlier in medical education.