Quantitative Study of the Characteristics of Effective Internal Medicine Noon Conference Presentations

Abstract

Background Increasing demands on residents' time have made it critically important to maximize the effectiveness of didactic activities and motivate independent study.

Objective Our aim was to correlate characteristics of noon conferences with internal medicine (IM) residents' ratings of perceived effectiveness and intent to pursue independent reading.

Methods We assessed characteristics of each noon conference by direct observation using predetermined metrics. We surveyed IM residents to assess their perception of the conference's effectiveness and their intention to pursue additional reading. A variety of modeling techniques were used to discern meaningful correlations of effectiveness and motivation.

Results A total of 649 evaluations of 29 conferences were submitted by 153 of 185 (83%) residents in the program. Median effectiveness score was 6 (on a scale of 1 to 7). Clinicopathological conferences had 0.55-point higher effectiveness scores than traditional conferences (P = .011). In multivariable analyses focusing on traditional conferences, summary statement inclusion was significantly associated with 0.43-point higher effectiveness scores (P = .016), and having resident speakers was associated with 0.50-point higher effectiveness scores than unfamiliar faculty (P = .045). Conferences with higher effectiveness scores had significantly higher proportions of respondents indicating intention to read.

Conclusions This is the first study to quantitatively assess correlations of high effectiveness ratings of noon conferences in a residency program. Intention to read improved with increasing effectiveness scores of conferences, suggesting residents are more inclined to pursue self-directed learning when topics are well presented. Considering these attributes in designs of didactic sessions may enhance their educational value.

What was known and gap

Didactic sessions are an important element of resident education, yet little is known about what constitutes effective didactics.

What is new

A study quantified attributes of didactic sessions that enhanced their effectiveness and promoted resident self-learning.

Limitations

Single site study reduces generalizability; lack of assessment of objective learning outcomes.

Bottom line

Residents are more inclined to pursue self-directed learning when topics are well presented, focus on clinical reasoning, and use a summary statement.

Editor's Note: The online version of this article contains the noon conference data collection sheet

Introduction

The Accreditation Council for Graduate Medical Education requires that internal medicine (IM) residency programs provide a didactic curriculum based on the core knowledge content.¹ The ability of this curriculum to motivate independent study among residents is not known.^2–9

A study in continuing medical education found that lecture effectiveness was correlated with the identification of important points, lecture clarity, and the ability to engage participants.⁷ An education task force made up of experts developed an instrument for peer assessment of faculty medical lectures and endorsed the following dimensions of effective didactics: clarity, enthusiasm, statement of importance and goals, summarization, subject command, audiovisual aids, audience interaction, and query of understanding.⁸ In a focus group study, IM residents reported that effective lectures were clinically relevant and practical, presented a balance of evidence, and included cases and questions.⁹

Characteristics of resident noon conferences have not been assessed with respect to lecture evaluations, and little is known about lecture characteristics that motivate independent study. Our aim was to evaluate the characteristics of noon conferences at a large academic teaching hospital, as well as to find correlations with IM residents' ratings of perceived effectiveness and intent to pursue independent reading.

Methods

Setting and Participants

This study was conducted at Massachusetts General Hospital, where the IM residency program offers 60-minute noon conference sessions covering a core set of clinical topics each weekday. While attendance is strongly encouraged, it is not required, and an average of 45 (out of a total of 185) residents attend each conference. Historically, 80% of the program's residents have entered a subspecialty fellowship.

Between October and December 2013, we assessed the characteristics of each noon conference by direct observation, and we surveyed residents about the educational value of each conference. Conferences were excluded if study staff were not available to observe, if the conference was not intended to be educational, or if resident survey data were not available.

Measures

Conference Characteristics

Guided by a review of the literature,^7–9 we hypothesized that the educational value of noon conferences could be affected by who presents the conference (speaker characteristics), what is presented (content characteristics), and how the material is delivered (presentation characteristics). Specific characteristics for study were selected within this framework and based on feasibility of measurement.

Study investigators or chief residents trained by study investigators attended each noon conference to assess the speaker, content, and presentation characteristics according to a checklist (provided as online supplemental material). When more than 1 observer assessed a conference, the authors discussed discordant results to obtain a consensus.

Speaker characteristics included sex, specialty area (general IM, IM subspecialty, or non-IM specialty), and academic profile (resident, familiar faculty, or unfamiliar faculty). A faculty speaker was considered familiar to residents if he or she spent at least 2 months as an attending physician on a resident service each year or had an administrative role in the residency program.

Content characteristics included the estimated frequency with which a topic is encountered on resident services (≥ weekly, < weekly but ≥ monthly, or < monthly), the use of a patient case for at least 5 minutes (yes or no), and the use of at least 3 multiple-choice questions (yes or no). Additionally, we distinguished between traditional conferences and clinicopathological conferences (CPCs). CPCs include a brief case presentation by a resident, an in-depth discussion of diagnostic clinical reasoning by a faculty member, and a summary of the patient's diagnosis by a clinical pathologist.¹⁰

Presentation characteristics include the following binary variables: having an early statement of objectives, interacting with the audience, stepping away from the podium for at least 5 minutes, using humor (audience laughter at least 3 times), making a specific suggestion for further reading, including a summary statement (brief emphasis of take-home points at the end), and ending late (at least 5 minutes overtime).

Resident Evaluations

To assess residents' perceptions of the educational quality of each noon conference, we conducted a voluntary, deidentified electronic survey of all residents after each noon conference. We requested that residents complete the survey only if they had attended at least 50% of the conference. Survey participants rated the conference effectiveness on a scale of 1 to 7 (with 7 being the most effective). Additionally, participants indicated their intention to read about the conference topic in the following week, their postgraduate training year, and their sex. Four IM residents reviewed the survey for clarity prior to its deployment.

This study was declared exempt by the Partners Human Research Committee.

Statistical Analysis

Our primary analytic objective was to determine which noon conference characteristics were most strongly associated with 2 outcome measures: (1) the resident-rated effectiveness score (range 1 to 7), and (2) the intention to pursue additional reading (yes versus no).

We used generalized linear mixed models to account for correlations in the data arising from each conference having multiple evaluations and each resident providing multiple conference evaluations over time. We specified a normal response distribution and used the identity link function to model the continuous outcome of effectiveness score. We also specified a binary response distribution and the logit link function to model the dichotomous outcome of intention to read. For both model types, we entered conference characteristics as fixed effects and estimated their associations with each outcome. We treated evaluations from multiple residents about the same conference as repeated measures with compound symmetric covariance structure. We modeled a resident identifier variable (based on usernames provided with each survey) as a random effect for each outcome to account for multiple conference evaluations arising from a single resident over time. Surveys without an identifier were treated as independent observations.

We first compared effectiveness scores and intention to read outcomes for traditional conferences versus CPCs. Because CPCs are predetermined and differ considerably from traditional conferences, we focused subsequent modeling on traditional conference formats only. In the absence of a specific predictor variable of interest, we took an exploratory approach to model building to determine which characteristics of speakers, content, and presentation were most strongly associated with resident effectiveness scores and intention to read. Prior to modeling, we assessed the intention to read outcome's association with effectiveness score.

We began the modeling process by examining the bivariate associations between the independent variables (conference characteristics) and dependent variables (effectiveness scores and intention to read). Independent variables associated with dependent variables at a significance level of P < .20 were then entered in multivariable models. We considered variables associated with each outcome in multivariable models to be statistically significant at P < .05. We conducted all analyses using SAS version 9.4 (SAS Institute Inc, Cary, NC).

Results

A total of 35 noon conferences occurred during the study period. From the analysis, we omitted 2 conferences because the survey was not sent to the residents, 1 conference because no designated observers were available, and 3 conferences because these were resident meetings without educational intent. The final analytic data set included 29 noon conferences, all of which were either traditional lectures or CPCs. Characteristics for 15 conferences (52%) were recorded by more than 1 person. In most cases, the majority of raters agreed and the majority opinion was used in the analysis.

Conference Characteristics

The characteristics of the 29 noon conferences are shown in table 1. A total of 86% (25 of 29) of noon conferences opened with a statement of objectives, and 62% (18 of 29) concluded with a summary statement (table 1). Most presenters were general or subspecialty internists, and more than two-thirds were men. On average, 22.4 (SD = 5.5) evaluations were submitted for each conference.

table 1.

Characteristics of Noon Conferences (N = 29)

Resident Respondents

Residents submitted a total of 649 evaluations. Of these, 577 evaluations were associated with 153 distinct resident usernames (median 2 evaluations per resident, range 1 to 19 per resident), comprising approximately 83% of the 185 residents in the program during the study period. Of those who provided identifying information, 37% (57 of 153) were postgraduate year (PGY) 1, 29% (45 of 153) were PGY-2, 33% (51 of 153) were PGY-3, and 56% (86 of 153) were men.

Effectiveness Scores

Across all conference evaluations, the mean effectiveness score was 6 out of 7 (Q1 = 5, Q3 = 7) , and 91% (591 of 649) of scores were ≥ 5. CPCs had 0.55-point higher effectiveness scores than traditional conferences (95% CI 0.13–0.98, P = .011).

In unadjusted analyses of traditional conferences, having a summary statement (P = .005) and the academic profile of the speaker (P = .032 for overall effect) were significantly associated with higher effectiveness scores (table 2). In adjusted multivariable analyses, a summary statement was significantly associated with 0.43-point higher effectiveness scores (95% CI 0.08–0.77, P = .016), and resident speakers had 0.50-point higher effectiveness scores than unfamiliar faculty (95% CI 0.01–0.99, P = .045).

table 2.

Regression Analysis of Lecture Characteristics With Effectiveness Scores of Residents

Intention to Read

Across all conference evaluations, 58% (376 of 649) indicated an intention to read about the topic the following week. Conferences with higher effectiveness scores had significantly higher proportions of respondents indicating an intention to read, such that for each 1-point increase in the effectiveness score, the odds of reading were 1.9-fold higher (95% CI 1.6–2.3). Unlike effectiveness scores, intention to read was not significantly different between CPC and traditional conference formats (OR = 0.81, 95% CI 0.40–1.62).

In unadjusted analyses of traditional conferences, having a summary statement (OR = 1.90, 95% CI 1.02–3.54) and speaker specialty (P = .007) were significantly associated with intention to read (table 2). In analyses mutually adjusted for these variables, only speaker specialty remained significant (P = .034), with conferences given by subspecialty internist speakers associated with 2-fold higher odds of intention to read relative to conferences given by general internists (OR = 2.04, 95% CI 1.11–3.75).

Discussion

To our knowledge, this is the first study to quantitatively assess the correlates of effective noon conferences in a residency training program. Conferences with a clinicopathological conference format were rated as more effective than traditional conferences, which is concordant with the findings of a prior qualitative study documenting residents' preferences for lectures that focus on clinical cases and questions.⁹ Despite this, the mere use of a patient case for at least 5 minutes was not significantly correlated with higher effectiveness scores among traditional conferences. While a possible advantage of CPCs may be the central role of the patient case throughout the conference, the emphasis on clinical reasoning rather than the features of a particular illness may be more important to the CPC format's perceived effectiveness.

Among traditional conferences, the use of a summary was a significant predictor of effectiveness scores. This finding supports the frequent suggestion by educational experts to summarize key points at the end of presentations.^11,12 Additionally, resident speakers received higher effectiveness scores than either familiar or unfamiliar faculty members, pointing toward a potential role for peer-based educational strategies. Peer teaching is increasingly used in medical schools,^13,14 and reports on its use in both surgical and radiology residency conferences have been positive.^15–18 Peer teaching may also foster mastery of the topic in residents who give the talks.

Intention to read was bolstered with increasing effectiveness scores, suggesting that residents are more inclined to pursue self-directed learning when topics are well presented. Although there were few significant correlations of intention to read, talks given by subspecialty IM faculty were associated with greater reading intentions than talks by general IM faculty. While the basis for this finding is uncertain, talks with a more specialized orientation may prompt learners to extend their own knowledge in less familiar topic areas.

Limitations of this study include that it was conducted at a single, large residency program, with a high proportion of graduates entering an IM subspecialty education. This may limit generalizability. The small number of lectures and the ceiling effect of ratings limit the ability to detect potentially important predictors of effective conferences. The outcome variables were subjective, and it is not known to what degree they correlate with subject mastery, actual independent reading, or standardized examination performance. Future research should include multi-institution studies that assess more distal educational outcomes.

Conclusion

Our findings suggest that residents are more inclined to pursue self-directed learning when topics are well presented. A noon conference curriculum that focuses on clinical reasoning skills, utilizes a summary statement, and incorporates resident speakers may have enhanced educational value.