Validation of content of clinical cases in obstetric medicine for a shared web-based educational tool

Short abstract

Subsequent to the validation of a Canadian Curriculum blueprint for Obstetric Medicine (OM), the Canadian Consensus for a Curriculum in Obstetric Medicine (CanCOM) research group was approached to develop 20 cases to address gaps in clinical exposure during clinical rotations in OM. Forty-nine Obstetric Internists were identified and 43 confirmed their affiliation to the group. Participants (N = 22) reviewed the content of the CanCOM blueprint and identified curriculum content that they considered essential for a rotation for senior General Internal Medicine residents. This survey led to the creation of the CanCOM II essential content blueprint for General Internal Medicine. Following this step, a second subgroup of participants (N = 21) participated in a Delphi survey to identify the content that should be addressed by a teaching case for senior General Internal Medicine residents. A high-level of consensus was obtained for 13 topics and a moderate level for the 7 subsequent topics resulting in the creation of the CanCOM II clinical cases available at http://gemoq.ca/cancom-ii-clinical-case-databank/

Introduction

Obstetric Medicine (OM) is a growing specialty at the intersection of Internal Medicine and Maternal–Fetal Medicine. It aims to provide specialized care for women with medical conditions entering a pregnancy or women who develop medical complications during gestation and/or the early postpartum period. This care is essential to ensure that both mother and child are protected from preventable morbidity and even mortality.^1,2 The recognition of a need for Obstetric Internists by early advocates³ has led to the formation of several significant societies such as the International Society of Obstetric Medicine,⁴ a dedicated peer-reviewed journal, international discussion groups and several OM curricula.^5–9

Concurrently, residency programs in Canada have undertaken to review and adapt their training to ensure that residents achieve necessary competency in several domains—including the care of women with medical conditions around pregnancy.¹⁰ The Royal College of Physicians and Surgeons of Canada has developed a model of competency-based education for residency training that will better define the level of competency that a post-graduate year 4–5 General Internal Medicine (GIM) resident would need to reach prior to entering independent practice. This model, referred to as Competence by Design, outlines a number of milestones and entrustable professional activities that define the stages of a developmental continuum towards competency. These stages take place during residency training, largely in the context of clinical rotations—which is appropriate, given that they represent authentic learning environments.¹¹ Cognitive models on medical expertise describe the transition of biomedical knowledge into integrated illness scripts.¹² Based on the fourth stage of this model, one may expect that repeated exposure to clinical cases leads to fine tuning of the expert’s illness scripts. Achievement of competency in the clinical setting, however, remains at the mercy of various elements such as the specific case mix encountered, along with other patient-related elements such as: the condition of the illness encountered, case numbers and clinical pace.^13–15 Given all these considerations, providing an additional tool for clinical exposure may support the final stage of transition-to-expertise with in-time exposure to informative cases with adequate time for learning and feedback.

Objective

The purpose of this study is two-fold. First, we wish to validate an OM curriculum blueprint that includes essential content to help define baseline competency in OM by post-graduate years 4 and 5 (PGY 4–5) GIM Residents in Canada. Second, we used this essential content blueprint to identify, by consensus, the content for 20 clinical cases to be used as a pedagogical tool to supplement gaps in clinical exposure to OM. The validation of the essential OM blueprint is based on prior work that led to the creation of the initial Canadian Consensus for a Curriculum in Obstetric Medicine (CanCOM) curriculum blueprint defining OM as a field of expertise.^5,16 The ultimate study objective is to publish an online, interactive databank of clinical cases to support clinical exposure and teaching in OM in Canada and beyond.

Methods

Study sample

Forty-eight eligible physicians were identified as specialists with a self-declared active interest in the practice and teaching of OM in Canada. These physicians were identified by means of: (a) participation in the CanCOM I curriculum content development study; (b) active publication and/or attendance at OM conferences; or (c) recruitment by a colleague. The only exclusion criterion for the study was not being available to complete the three rounds of Delphi over a specified period of time.

Phase 1: Validation of the essential content for clinical rotations for senior GIM residents

In the summer of 2014, members of the CanCOM research group who were available and agreeable to participate in the study were asked to review the CanCOM I curriculum content blueprint and identify the essential content that they would want post-graduate year 4–5 residents in GIM to be exposed to during their rotation(s) in OM. The CanCOM I content blueprint contains 487 separate content entries, organized in 21 sections. The first seven sections include sections on: CanMEDS roles for OM, maternal physiology, pharmacology, radiation, acute maternal care, and relevant surgical, anesthetic and gynecological conditions. CanMEDS is a framework by the Royal College and Physicians and Surgeons of Canada “that identifies and describes the abilities physicians require to effectively meet the health care needs of the people they serve.” The CanMEDS framework is built around seven roles including the central role of the physician as a medical expert but that has also developed competency as a scholar, health advocate, professional, communicator, collaborator, and leader. The subsequent sections of the CanCOM content blueprint are divided into classic Internal Medicine disciplines such as cardiology, respirology, etc.—and further subdivided in the principal conditions seen by Obstetric Internists (for example: preeclampsia, peripartum cardiomyopathy, asthma, etc.). For each medical condition, separate entries reference content related to evaluation, management, and complications. This content blueprint can be accessed in the English section of a Quebec Obstetric Medicine website at http://gemoq.ca/welcome-to-the-gemoq/cancom-research-group/

Participants were then asked to complete two additional tasks: (a) selection of blueprint content deemed “essential” for PGY4–5 resident training and (b) identification of a list of 20 “priority subjects” for which a web-based clinical case would be created.

A cut-off of 80% consensus agreement was selected as a significant level for inclusion, based on previous work⁵ and established standards.¹⁷ A free-marginal kappa was calculated as a measure of agreement that exceeds chance levels.

Phase 2: Delphi survey for clinical cases to supplement gaps in clinical exposure

A Delphi methodology was then used to obtain consensus on the 20 final case topics. Based on previous work and considerations of feasibility, we set the number of rounds at three.⁵ Thirty-three possible case topics, based on the CanCOM I blueprint, were selected by at least 20% of participants during phase 1 and were presented to participants at the beginning of round 1 of the Delphi process.

Each Delphi survey participant received instructions to rate the curriculum content using a Likert scale of 1–5, to state their level of agreement with the inclusion of the content as one of the 20 priority cases. Space was provided for justification of their rating, if desired. After each round, each participant received a new survey with their prior rating, the group’s mean rating, standard deviations, bar histograms, and a list of anonymous comments for each case topic. For transparency purposes, any item that obtained consensus remained in the survey with comments, means, standard deviations, and bar histograms with the indication that consensus had been achieved. A pre-determined three-round Delphi was agreed upon as sufficient to obtain a ranking of the 20 topics based on feasibility considerations and previous work (CanCOM I study).

Phase 3: Construction of clinical cases

The final phase of the study was to develop the 20 cases and the web platform to share these cases. We performed a search of the literature to identify existing templates on which we could base the development of our cases. There was no existing model for web-based clinical cases in OM, but other medical specialties have developed tools that served as a basis for our research and development. The principal model for our goal of creating a low-fidelity simulation for advanced learners and aimed at content rather than skills, was the New England Journal of Medicine Interactive Medical Cases at www.nejm/org/multimedia/interactive-medical-case.

Participants were asked to form dyads for the writing and reviewing of two OM clinical cases per dyad. Case topics were assigned or chosen as preferred. Case writers and reviewers used a standardized but not validated template developed with educational experts at the University of Sherbrooke, Quebec, Canada. This template was developed to meet prespecified educational principles such as the importance of reactivating prior learning (Gagné’s conditions of learning), anchor cases in realistic contexts, and address classic diagnostic or management errors. The template thus helped case writers to draft cases that unfold over time and are accompanied by high-quality questions that reactivate prior learning and stimulate further reflection.

Results

CanCOM II consensus group

Forty-three physicians were identified by their participation in the previous CanCOM I study or attendance at national OM conferences. Six additional names were added by colleagues, for a total of 49 names. Contact information was available for 48 physicians. Forty-three of these (90%) confirmed their interest as members of the CanCOM research group and constituted our sample of eligible participants (Figure 1). Of these, a subset of N = 22 (51%) completed the validation instrument, and a second subset of N = 21 (48%) were available to participate in a three-round Delphi process aimed at obtaining a consensus on the 20 cases to be developed as a pedagogical tool (phase 2). Finally, 20 of the Delphi group (90%) participated with the authors in the final phase of the study by writing and/or reviewing 1–2 of the clinical cases based on the Delphi consensus.

Figure 1.

Outline of recruitment process for the three phases of the study.

CanCOM: Canadian Consensus for a Curriculum in Obstetric Medicine.

Study participants came from 8 of Canada’s 10 provinces or territories with a wide range of experience in the practice and teaching of OM (Figure 2).

Figure 2.

Practice and teaching experience in Obstetric Medicine in study sample.

Phase 1: CanCOM II curriculum validation

Of the 487 entries in the CanCOM I content blueprint, 204 were selected by 80% or more of participants as essential content items for post-graduate year 4–5 residents in GIM (Figure 3).

Figure 3.

Number of items selected for each level of % agreement.

This results in a reduction of 58% in the content of the blueprint of OM. In other words, 204 items of content were retained as essential content to be addressed during clinical rotations for senior GIM residents.

Principal selections and cuts are summarized in Table 1. For clarity, the inter-rater level of agreement for each section is presented, but the actual decision to include/exclude was taken according to the level of agreement for each individual item.

Table 1.

Description of principal content selection with level of agreement per discipline.

Content and outcome	Content kept	Content removed	Overall % agreement;Kappa
2. Physiology of pregnancy	2.1 Cardiovascular changes2.2 Respiratory physiology2.3 Renal physiology2.4 Metabolic changes	Aspects pertaining to sleep ADH metabolism	78; .57
3. Pharmacology of pregnancy and lactation	3.1 General principles	Critical periodsEtiology of congenital anomaliesFactors that affect drug transfer into breast milk	65; .31
4. Diagnostic and therapeutic radiation in pregnancy	4.1 Diagnostic imaging	IV Contrasts during lactation 4.2 Therapeutic radiation	70; .41
5. Acute care and maternal resuscitation	5.1 Acute respiratory failure—concepts5.2 Cardiopulmonary resuscitation	5.1.2 Acute respiratory care—management of the intubated patient	63; .27
6. Surgical and anesthetic considerations	6.2.2 Indications for ANTB prophylaxis	6.1 Impact of pregnancy on surgical and anesthetic practices	58; .15
7. Relevant obstetric and gynecologic conditions	7.2 Amniotic fluid embolism—concepts	7.1 Septic vein thrombosis7.2.2 AFE—management7.3 Endometritis7.4 Postpartum hemorrhage7.5 Assisted reproductive technology	57; .15
8. Hypertensive disorders	8.1 Chronic hypertension8.2 Gestational hypertension/preeclampsia	8.1.2 Role of uteroplacental Doppler	92; .84
9. Cardiology	9.1 Evaluation of palpitations, presyncope and syncope9.2 Acquired and congenital heart disease— management of complications9.3 Cardiomyopathies	9.2 Acquired and congenital heart disease—concepts9.3 Cardiomyopathies—delivery plan	69; .38
10. Endocrinology	10.1 Pregestational diabetes10.2 Gestational diabetes10.3.1 Obesity and metabolic syndrome—counseling10.3.2 Obesity and metabolic syndrome—evaluation10.4 Thyroid disorders	10.3.1 Obesity and metabolic syndrome—physiology10.3.3 Obesity and metabolic syndrome—placental complications10.4.1 Thyroid disorders—subclinical hypothyroidism and fetal outcomes10.4.3 Thyroid disorders—Fetal surveillance 10.5 Pituitary disorders10.6 Adrenal disorders10.7 Diabetes insipidus	69; .38
11. Hematology	11.1 Anemia11.2 Thromboembolic disorders11.3 Thrombophilia and respective thrombotic risk11.4 Thrombocytopenia	11.3 Thrombophilia and obstetrical complications11.3 Thrombophilia—management11.5 Bleeding disorders11.5 Hemoglobinopathies	70; .40
12. Respirology	12.1 General principles12.2 Asthma	12.3 Cystic fibrosis	71; .42
13. Nephrology	13.1 Physiology13.2 Acute renal failure—concepts and management13.3 Chronic renal failure	13.2 Acute renal failure—complications13.3 Chronic renal failure—hemodialysis and complications	71; .42
14. Transplant medicine		14.1 General principles and management	60; .20
15. Gastroenterology	15.1 Hyperemesis gravidarum—evaluation15.2 Dyspepsia15.3 Hepatitis and liver dysfunction in pregnancy—concepts and management15.5 Inflammatory bowel disease—concepts	15.1 Hyperemesis gravidarum—management and complications15.3 Hepatitis and liver dysfunction—complications15.4 Cirrhosis15.5 Inflammatory bowel disease—evaluation, management and complications	62; .25
16. Neurology	16.1 Headaches16.2 Seizures—concepts	16.2 Seizures—management and complications16.3 Cerebrovascular disease16.4 Nerve compression syndromes16.5 Myasthenia gravis16.6 Multiple sclerosis	60; .20
17. Infectious diseases	17.1 Infections in general17.2 Urinary tract infections17.3 Respiratory infections	17.4 Tuberculosis17.5 Human immunodeficiency virus17.6 Immunizations	65; .29
18. Immunologic disorders	18.1 Systemic lupus erythematosus18.2 Antiphospholipid syndrome—concepts18.3 Rheumatoid arthritis—concepts	18.2 Antiphospholipid syndrome—evaluation and management18.3 Rheumatoid arthritis—management	64; .28
19. Neoplastic disorders		19.1 General principles	62; .24
20. Dermatology	20.1 Dermatoses related to pregnancy—concepts	20.1 Dermatoses related to pregnancy—management and complications	59; .18
21. Psychiatric illness	21.3 Substance abuse disorders—concepts	21.1 Chronic psychiatric conditions21.2 Acute psychiatric conditions21.3 Substance abuse disorders—management and complications	53; .05

ADH: antidiuretic hormone; ANTB: antibiotic; AFE: amniotic fluid embolism.

Delphi consensus

During phase 1 (content validation), 59 topics were proposed by the 22 participants as priorities for the development of teaching cases. A level of agreement greater than 80% was present for five cases, and therefore these topics were presented in the Delphi as “already selected” for case material (Table 2). Twenty-five of the 59 topics were excluded from the Delphi because the initial level of agreement was below 20%, leaving 29 topics to be rated during the first round of Delphi consensus.

Table 2.

Description of the 20 priority topics to be developed into a clinical case.

Case topic	Moment of selection	% agreement
#1 Chronic hypertension	Validation phase	86
#2 Cardiomyopathy		82
#3 Thromboembolic disorders		82
#4 Hepatitis and liver dysfunction		82
#5 Gestational hypertension		82
		Mean Likert score (SD)
#6 Cardiopulmonary resuscitation	Round 1	4.42 (1.14)
#7 Systemic lupus erythematosus and antiphospholipid syndrome		4.26 (0.87)
#8 Pregestational diabetes		4.12 (1.10)
#9 Headaches		4.12 (1.15)
#10 Thyroid dysfunction	Round 2	4.61 (0.61)
#11 Palpitations		4.33 (0.69)
#12 Thrombocytopenia		4.22 (0.91)
#13 Seizures		4.17 (0.85)
#14 Chronic renal failure	After Round 2	3.94 (1.21)
#15 Acute respiratory failure and respiratory infections		3.89 (1.18)
#16 Bleeding disorders		3.28 (1.02)
#17 Asthma		3.28 (1.27)
#18 Cerebrovascular disease		3.22 (0.94)
#19 Dermatoses of pregnancy		3.17 (1.14)
#20 Inflammatory bowel disease		2.89 (1.02)

After two rounds of Delphi, eight more topics were finalized based on a high level of agreement (mean Likert score above 4.0). Instead of performing a third round, it was decided to select the seven missing topics based on the rank order of the remaining items. Based on comments from participants, some compatible topics (for example, acute respiratory failure and respiratory infections) were merged after round 1 and rated as a combination topic thereafter.

Construction of clinical cases

Twenty subject-matter experts participated in the writing and reviewing of the 20 clinical cases based on the template provided. Cases were translated by the author (AC), who is bilingual in French and English, and uploaded into the web platform with the assistance of a certified programmer. They can be found on the web at http://gemoq.ca/cancomii/CanCOMII.html.

Discussion

This second initiative from the CanCOM Research Group aimed to answer the question, “what is the essential content in OM that senior GIM residents need to be exposed to during their clinical rotations?” A corollary question is “and how can we support exposure to this essential content in the face of a variable clinical case mix?” We posit that these two questions will help to support the development of specific competencies in a competency-based curriculum. The identification of a minimal level of proficiency when caring for pregnant patients should also be based on perceived needs.¹⁸ Such competency-based frameworks should decrease documented gaps between the importance of a subject-matter and preparation.¹⁹

The development of an “essential content” blueprint was seen as a priority—as the original OM curriculum blueprint was extensive in order to map out the relatively new domain of OM. The research group felt that having a more focused content blueprint would be useful to Program Directors and other curriculum experts who were faced with the task of deciding upon the minimal level of competency required of a GIM specialist—and with designing appropriate educational opportunities to reach this goal. The resulting document is more succinct, as the 80% inter-rater agreement cut-off permitted a 60% reduction in the blueprint content. It would have been difficult to reduce the competencies further, as we would have entered a zone with insufficient levels of agreement. Indeed, it was easy for subject-matter experts to agree on the inclusion of some content (e.g. hypertensive disorders) but difficult to exclude content even for conditions that are not usually seen as cornerstones of OM (for e.g. psychiatric illness). To overcome the reduction in inter-rater agreement created by “inclusive raters,” we could have implemented a forced-choice format to create an imposed reduction in content (for e.g. select 50% of the items only), but this approach would have resulted in other measurement-related biases. Further work will be required to validate subject-matter experts’ opinions with those of Program Directors (work underway in Canada) and those with subject-matter experts in other countries.

With regard to the objective of developing 20 priority cases for teaching, a high level of consensus was achieved for the first 13 case topics. The preset number of 20 cases permitted the inclusion of seven more cases which were deemed “lower priority” but which nonetheless provide a useful complement to clinical exposure. These cases are now accessible to anyone to utilize, in both French and English language. A multi-center observational trial is underway in Canada to measure the contribution of these cases to the senior GIM residents’ clinical exposure. These cases will require regular review and maintenance in order to maintain an up-to-date evidence-based status. The electronic format would permit interfaces to other languages (in addition to French and English) if the need were to become apparent. Consideration should also be given, however, to adaptations beyond language in order to reflect practice patterns in other lower and middle income practice settings. Other case topics may also be included in the future to better reflect educational needs of trainees and practitioners outside of Canada (i.e. including a case on malaria in pregnancy).

Further research will also be needed to measure whether such case-based learning has an impact on the achievement on preset developmental milestones and attainment such as those defined by the Competence by Design model of the Royal College of Physicians and Surgeons of Canada.

Conclusion

This article presents the results of the second initiative of a Canadian research group which aims to address gaps in exposure to essential content in OM during the senior GIM residency years. The result is a more focused curriculum blueprint which can be further validated and adapted to meet other research and curricular needs. Indeed, a blueprint of essential content can be used as further validity evidence to build a framework of essential competencies for a GIM specialist entering practice, or to construct an examination blueprint. We chose to seek further validity evidence by comparing with the opinions of Canadian Program Directors and to develop 20 cases as a teaching tool for residents during their clinical rotations. That work will be forthcoming shortly.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical approval

Ethics review was obtained from Sherbrooke University’s Comité d’éthique de la recherche en Éducation et Sciences Sociales (project 2014–1039, 2014–24-ESS). Written consent was obtained from study participants.

Guarantor

AC.

Contributorship

Both authors meet the criteria for authorship including contributing to design, data analysis, drafting and revision of manuscript.