Developing and Establishing Content Validity of Vignettes for Healthcare Education and Research

Abstract

The use of vignettes in research and education is well documented. Documentation of how vignettes are developed and content validity established, however, are rare. The purpose of this paper is to describe a method for development and establishing content validity of vignettes that can be used to assess student performance and research participant outcomes. This three-phase method was designed to validate seven vignettes for a future research project. Content validity index survey and expert panel interviews were used to inform improvements in validity. Findings showed that clinical vignettes were improved with respect to content relevance (from 0.93 to 0.96) and importance (from 0.68 to 0.93). Developing validated vignettes is essential when used to measure outcomes in education and in research. This three-phase method of vignette development and validation is feasible and effective in improving content validity of vignettes and can be used in other education and research projects.

Circumstances in clinical teaching and research often preclude real time evaluation and testing due to risk to the patient, situation sensitivity, lack of time, and availability of high-risk patients. One solution to this challenge is the use of vignettes. Vignettes are defined as short stories about a fictional person or scenario that are plausible enough to simulate real events and have been extrapolated into healthcare for the purposes of education and research (Jenkins et al., 2010; Lowcock et al., 2017). There is a reliance on vignettes in healthcare for teaching and assessing or measuring student learning, decision making, diagnosing, and intervening in complex situations or with pertinent clinical factors (Brauer et al., 2009; Leicher & Mulder, 2018; McCrow et al., 2013; Moll et al., 2017; Schigelone & Fitzgerald, 2004; Stacey et al., 2014). Using vignettes creates distance between the clinician and the experience, allowing the clinician to experience situations that may be variable and sensitive while avoiding harm or embarrassment to patients and clinicians (McCrow et al., 2013; Randhawa et al., 2015).

As a research tool, vignettes have been used in various disciplines beginning with anthropology and psychology in the 1950s (Nygren & Oltedal, 2015; Herskovitz, 1950). Traditionally, vignettes have been used for research in psychological and sensitive health issues such as vulnerable populations of mental health, pediatrics, and illicit substance users (Lowcock et al., 2017). Vignettes are widely utilized now particularly within health care and social work research (McCrow et al., 2013; Nygren & Oltedal, 2015). Most studies using vignettes focus on the study processes while using vignettes and validate the questions following the vignettes rather than the vignettes themselves (McCrow et al., 2013; Schigelone & Fitzgerald, 2004). There are few articles found that provide detailed descriptions of processes for development and validation of vignettes for research (Evans et al., 2015). These will be expanded upon as the process for development and validation of vignettes is presented in this paper.

Development of vignettes have been discussed in the literature with common agreements that vignettes are developed from personal clinical practice (McCrow et al., 2013) and are plausible (Hughes & Huby, 2002); are clear, relevant and real to the phenomenon of interest (LaFond et al., 2015); and are brief with language appropriate for the group tested (Paddam et al., 2010). Constructing vignettes include evaluating the appropriateness of vignettes in the context to the research question, establishing topics from scholarly literature and theoretical concepts, and formatting the vignettes in written or video formats (Jackson et al., 2015). Vignettes need relevant and realistic scenarios of the sample population that mirror real-life situations within the area of interest. Evaluation of vignettes occurs through obtaining feedback from experienced researchers, and assess timing and delivery of vignettes through pilot study (Jackson et al., 2015). Randhawa et al. (2015) stated when vignettes are realistic and well designed, then study validity is maximized and results enable researchers to deliver more valid inferences about the variables being studied.

Promoting the concept of validation and rigor in vignette development is found in the literature (Randhawa et al., 2015; McCrow et al., 2013; Paddam et al., 2010). Randhawa et al. (2015) stated that content validity of the vignette was “of paramount importance,” (p. 6). Use of focus groups, individual interviews, and expert panels were recommended to establish content validity (Randhawa et al., 2015; Paddam et al., 2010). Paddam et al. (2010) recommended that vignettes be rated by expert panel of the field being studied, then modified and re-assessed. While promoting content validity, few articles explain the details on how it should be carried out. The focus of this paper is vignette development and content validation using methods that are objective, guided by experts and transparent to facilitate replication in the future.

Purpose

The purpose of this paper is to describe a method for developing and establishing validity of seven vignettes that will be used to test a decision support tool for use by primary care providers in managing pain and minimizing risk for opioid misuse and opioid use disorder (OUD). These vignettes typify an encounter with patients who are in pain and with unknown risk of opioid misuse or opioid use disorder. The application of this three-phase vignette development process incorporates transparency and rigor in developing and quantifying content validity of these vignettes. By describing the process of development and validation of vignettes, we provide an exemplar method that others can use and evidence for the use of these vignettes in education and research.

Methods

The method for developing and establishing content validity of the vignettes included quantitative and qualitative approaches. Content experts in pain and OUD completed surveys and an interview so that the vignettes were evaluated using quantitative and qualitative data. This rich feedback supported revisions of the vignettes and additional measurement.

Three-Phase Development and Validation Framework

A three-phase approach for development and validation framework developed by the authors provides a method to test content validity of vignettes with objectivity and transparency in order for vignette-based studies to provide better data (Figure 1). Phase one consists of the initial development of the vignettes, along with the development of the content validity index (CVI) quantitative survey and the qualitative interview guide. Phase two solicited input from four content experts with extensive knowledge in the field of pain and OUD to provide feedback and ratings of content validity -- including clarity, importance, and relevance -- and to provide recommendations for improving the vignettes during qualitative interviews. The seven vignettes were revised based on this expert feedback. Phase three involved eliciting ratings of content validity of the revised vignettes from the same expert panel, followed by a comparison of ratings between phase two and phase three CVIs.

Figure 1.

Phases in Vignette Development and Validation

Phase One: Vignette Development

During phase one, the initial development of the vignettes, adaptation of the CVI survey, and construction of an interview guide occurred. The initial development of the vignettes followed a process consistent with Jackson et al. (2015) and was based on best practices and content expertise of the Principal Investigator (PI). Best practices resulted from consensus-derived core competencies for pain management within domains of multidimensional nature of pain, pain assessment, management (treatment) of pain, and application of knowledge in various populations and context (Fishman et al., 2013; Herr et al., 2015; Hoeger Bement et al., 2014). Combining these best practices with content expertise from a clinically competent nurse practitioner in the field of pain and OUD (B.S.) resulted in vignettes that captured various situations from actual clinical practice. The initial vignettes were reviewed by research team members (B.S., A.J.) and a sample vignette is shown in Table 1.

Table 1.

Original Vignettes (Phase Two) and Revised Vignettes (Phase Three)

Original Vignette #1 Phase Two	Revised Vignette #1 Phase Three
A 50-year-old man presents with low back and bilateral leg pain. He has had chronic back pain for many years, which has progressed during the last several months. He describes aching pain that is localized to his lumbosacral spine with some radiation into this buttocks and hips bilaterally, however, burning pain in his right leg down to his knee. Numeric pain intensity is 7 out of 10 at worst and 5 out of 10 at best. Recent yard work led to an acute exacerbation of his pain with 10 out of 10 intensity and he was bed-ridden for 2 days following, due to pain. He denies LE weakness and loss of bowel or bladder control. Pain is better with rest and codeine/acetaminophen received from a family member. Pain is worse with doing housework activities. Sleep is poor (4 hours/night, no socialization due to fear of pain, no more golfing or softball teams. His past medical history includes hypertension, hyperlipidemia, depression, and morbid obesity. Current medications are lisinopril, simvastatin, one baby aspirin, self-prescribed codeine/acetaminophen, two OTC ibuprofen (200 mg/tablet) and three tablets of extra-strength acetaminophen (500 mg/tablet) about 4 times/day with minimal relief of his pain. His habits include cigarettes two ppd, marijuana use daily, six beer per week. Occupation: Quit job as a carpenter earlier in the year due to pain.	Chief Complaint
	A 50-year-old Hispanic man known to you in primary care presents with low back and bilateral leg pain. Recent yard work led to an acute exacerbation of his leaving him bedridden with 10 out of 10 pain intensity.
	History of Presenting Illness
	He has had chronic back pain for many years, which has progressed during the last several months. Current medications provide minimal relief of his pain. He describes aching pain that is localized to his lumbosacral spine with some radiation into his buttocks and hips bilaterally, and burning pain in his right leg down to his knee. Numeric pain intensity is 7 out of 10 at worst and 5 out of 10 at best. He denies lower extremity weakness and loss of bowel or bladder control. Pain is better with rest and Tylenol #3 received from a friend. Pain is worse with doing housework activities. He is also complaining of poor sleep, receiving less than 4 hours a night.
	Past Medical/Surgical History
	Hypertension, hyperlipidemia, and morbid obesity. No surgical history.
	Smoking Status- Current
	Packs/day- 2
	Years- 25 years
	Alcohol Use- Yes
	Drinks/Week- 6 beers
	Drug Use- Yes
	Type- Marijuana (legal use)
	Frequency- Daily
	Last Use- Yesterday
	Medications
	Codeine/acetaminophen, self-prescribed
	Ibuprofen 200 mg, 2 tablets, four times a day
	Tylenol 500 mg, 3 tablets, four times a day
	Social History
	Married, spouse teaches, 4 children ages 6 to 12. Quit job as a carpenter earlier in the year due to pain. Socially isolated.

Phase Two: Validation and Revisions

Phase two involved four experts reading the vignettes and providing quantitative data on vignette validity using the CVI survey and completing semi-structured interviews. Qualitative interviews were conducted by one researcher (B.S.) inquiring about each of the vignettes. Input from the expert panel provided information to guide the research team in revisions of all seven vignettes.

Content Validity.

Content validity for surveys and measurement instruments is defined as the degree of appropriate item sampling for the construct measured, representation of items in the content domain, and the degree the items match the domain of interest when measuring a phenomenon (Bailey et al., 2010; Lynn, 1986; Polit & Beck, 2006). Content validity is used to measure items for surveys or instruments, but there is no content validity measure specifically for vignettes. Content validity is an important concept for vignette development (Randhawa et al., 2015; Lanza, 1988), and one study used an expert vignette review questionnaire, not identified as a measure of content validity, but similar in that it assessed plausibility, clarity, simplicity, accuracy to domains, and selection of diagnosis from a drop-down list (McCrow et al., 2013). The use of a CVI for vignette validation is a logical method that could provide appropriate data for establishing validity.

Content Validity Index (CVI).

CVI is the most common quantitative method used to assess content validity (Rodrigues et al., 2017). The expert participants scored each vignette by rating on three dimensions of content validity: clarity, relevance, and importance (Table 2). Questions asked of the participants were about the clarity of the vignette, the vignette’s relevance to clinical practice, and the importance of the vignette for learning. Three-choice rating options were used and were consistent with publications on content validity for instruments (McCrow et al., 2013; Bailey et al., 2010; Randhawa et al., 2015). A rating of 1 on each dimension means the vignette was judged as not clear, not relevant, or not important. A rating of 3 means the vignette was very clear, very relevant, or very important (see Table 2). The CVI is calculated for each dimension of the vignette’s content validity as the number of experts provide the highest rating (“3”) on the dimension divided by the total number of experts. An overall CVI for each vignette is calculated by averaging CVIs across the three dimensions. For example, if all the participants provided a rating of 3 across all dimensions, the vignette attained CVI=1, the highest possible value, which indicates 100% agreement among experts. A CVI between 0.80 to 1.0 is interpreted as acceptable, between 0.70–0.79 as needing revisions, and below 0.70 as unacceptable (Rodrigues et al., 2017; Lebet et al., 2018). This paper explored the application of CVI measurement to the content validity of vignettes using the same definition as is used in item development.

Table 2.

Content Validity Ratings

	Rating
Dimension	1	2	3
Clarity	Unclear	Somewhat clear	Very clear
Relevance	Not relevant	Somewhat relevant	Very relevant
Importance	Not important	Somewhat important	Very important

Qualitative Interview.

The research team developed the qualitative interview guide to gather information from the content expert panel. Questions were derived from the PI’s clinical experiences (B.S.) and from the perspective to improve the validity of vignettes. Questions asked were about missing information from the case vignette that could create barriers to understanding the vignette, unnecessary information included in the vignette, and whether the vignette was real enough to seem like a real patient encountered in clinic. This interview guide allowed the expert panel to expand on specific alterations within the vignettes and was used in conjunction with the CVI to inform revisions and improve validity.

Phase Three: Re-Validation

The revised vignettes were distributed to the same expert panel as in phase two. The experts completed the CVI survey for each revised vignette. CVIs were compared between phase two and phase three.

Participants

Five experts with extensive knowledge and clinical expertise in pain and OUD were identified from publications and national presentations on the topic. Following approval from the University Institutional Review Board, recruitment of participants began through email. Five experts were approached and four agreed to participate. It was determined that the sample of four experts was adequate based on previous studies using two to twenty interdisciplinary expert panels (Lynn, 1986; Polit & Beck, 2006; McCrow et al., 2013; Van Hulle Vincent et al., 2011). The inclusion criteria determined they were experts in the field with current practice in the area of pain and OUD, they owned a DEA licensure, they had published work in the area of pain and OUD, they spoke and read English, and they were willing to participate in two rounds of surveys and a phone interview. Exclusion criterion was the inability to both complete the study surveys or participate in an interview. All participants met criteria for participation in validation and re-validation in phases two and three.

Data Collection

The online demographic questions, validation survey, and semi-structured interview were used to collect data for this study. Demographic data included practice degree, sex, years in clinical practice, work status, years working with patients with pain and OUD, and information on current state and DEA licensure. Each expert read all seven vignettes and completed the CVI survey for each vignette. Both the demographic data and the CVI survey data were gathered using RedCAP^® (Research Electronic Data Capture) platform. RedCAP^® is managed by the Institute for Clinical and Translational Science, The University of Iowa, supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002537. Qualitative interviews of the expert panel were conducted for each of the vignettes by phone interviews of each expert lasting approximately 30 minutes. The interviews were audio-recorded and sent to a transcription service via a secured link, with transcriptions stored in a password protected computer on a secure shared drive in NVivo^®. Any hard copies were secured in a locked file cabinet in the locked office of the PI. The interview guide is available upon request.

Data Analysis

Demographic data were summarized using descriptive statistics. CVIs, previously described, were calculated in phases two and three and compared for all vignettes. The qualitative interviews were analyzed by two researchers (B.S., A.J.) using content analysis. Data analysis was inductive and informed by procedures described by Sandelowski (1995), Graneheim and Lundman (2004), and Elo and Kyngäs (2008). Interview transcripts were read and re-read and comments were made in the margins of the transcription on overall impressions, points of interest, and units of analysis were determined (Mayan, 2001). Coding sheets were developed using the notes from the margins (Fonteyn et al., 2008). The two researchers established data reliability and coder interrater reliability at weekly meetings by comparing codes, categories and emerging themes, and they discussed issues that arose during the coding process to achieve consensus on the themes. Categories were generated and grouped for higher order headings, clustering the categories so they belong to groups. This created the coding grid that was uploaded into NVivo^® so that the grid could be applied to all transcripts, organized and interpreted for a description of our phenomenon, that is, the expert recommendations for revisions of the vignettes. Revisions occurred only through consensus between the two researchers.

Results

Original vignette development began in spring 2018, and validation and revisions of the vignettes continued through spring 2019 to complete validation of the seven vignettes. The original vignette #1 is shown on Table 1. In accordance with the process outlined on Figure 1, content experts completed content validity surveys (phase two and three) and a qualitative interview following the first content validity survey (phase two). An example of an abbreviated vignette #1 can be used for comparisons between phase two and phase three in vignette development and revision (Table 1).

Phase Two

Four experts in the area of pain and OUD held current state licensure from four different states across the nation, plus federal DEA licensure. Among the experts were three nurse practitioners and one physician, three females and one male, two worked part-time in clinical practice and two worked full-time. Clinical experience ranged from 8 years to 27 years (mean=17.5), and number of years working with patients with pain and OUD ranged from 9 years to 18 years (mean=13.6).

Initial CVI Results

The initial CVI results are summarized in Table 3. In phase two, no expert assigned a rating of 1 to the vignettes, meaning unclear, not relevant, or not important. The mean CVI across all the original vignettes in phase two was 0.93 for relevance and 0.85 for clarity. The mean CVI of 0.68 for the importance dimension was unacceptable.

Table 3.

Content Validity Index Results in Phase Two and Phase Three

Phase two (original text)					Phase three (revised text)
	Clarity	Relevance	Importance	Overall CVI	Clarity	Relevance	Importance	Overall CVI
Vignette 1	1.00	1.00	0.25	0.75	1.00	1.00	1.00	1.00
Vignette 2	1.00	1.00	0.25	0.75	0.50	1.00	1.00	0.83
Vignette 3	0.75	1.00	1.00	0.92	1.00	1.00	0.75	0.92
Vignette 4	0.75	0.75	1.00	0.83	0.75	0.75	0.75	0.75
Vignette 5	0.67*	1.00	1.00	0.89	1.00	1.00	1.00	1.00
Vignette 6	1.00	0.75	0.75	0.83	0.75	1.00	1.00	0.92
Vignette 7	0.75	1.00	0.50	0.75	0.50	1.00	1.00	0.83
Average %	0.85	0.93	0.68	0.82	0.79	0.96	0.93	0.89

^*missing one rating in phase two.

Revision Recommendations by Expert Panel

Experts completing the qualitative interviews after the first CVI survey recommended format revisions and essential background information as a means to inform the healthcare provider and make the vignette valid and relatable. These recommendations were to format vignettes according to electronic health records; and to add content such as ethnicity/race, social context, medication lists, and laboratory values and radiology results. These findings informed revisions and enhanced the details of each vignette.

Structure and Format.

The experts recommended that the vignettes follow a similar format as electronic health record. They recommended replacing a single paragraph format with sections labeled: chief complaint, history of presenting illness, patient goals, past pain treatment, past medical/surgical history, medications, allergies, social history, physical examination, and diagnostic workup. One participant stated this format change would make it quick and easy to review. The participants were consistent in using terms for labeling section as “recognizable” and “similar” to what they encountered daily in their practice.

Ethnicity and Race.

The experts recommended that ethnicity be used in these vignettes and stereotypes and stigma in the wording be avoided. Ethnicities were added to the vignettes: Hispanic, African American, Caucasian, and Nepali. Although the vignettes cannot cover all ethnicities in America, three of the most common ethnicities were included in the vignettes, and one not as common. Furthermore, all the ethnicities were selected by the experts because they were represented in their practices.

Social Context.

To provide a thorough background to the individual vignettes, the experts recommended adding social context. They recommended including information about immigration status; involvement of family members; number of people residing in their homes including children; community participation; and occupation of the individual described in the vignette including shift work, and activity level. The experts stated this information would be consistent with important information about people in pain found in the electronic health record and created greater authenticity to the vignettes.

Medication List.

The experts suggested that all the vignettes contain a medication list that used generic medication names, dosage, and frequency. If opioids were listed, the experts recommended including how many tablets were used per day.

Laboratory and Imaging Results.

Experts wanted the laboratory values and imaging results included in the vignettes to better inform the reader and assist their decision making. Additional pertinent information to these results was also requested by the experts when applicable. The research team placed this information under the heading Diagnostics.

Revisions

All seven vignettes were revised based on this expert panel CVI ratings and recommendations. The revised vignettes were used for phase three.

Phase Three

In phase three, the CVI survey was used to evaluate the revised vignettes. An abbreviated revised vignette #1 is shown in Table 1. In phase three, again, no expert assigned a rating of 1 to the vignettes. Table 3 presents a comparison of CVIs from phase two to phase three showing that average percentage of experts providing the highest rating increased from 0.93 to 0.96 on relevance, and from 0.68 to 0.93 on importance. These two factors, relevance and importance, are critical to users of vignettes (Evans et al., 2015). After revision of the original vignettes, all four experts assigned the highest ratings for the relevance dimension to six vignettes, and importance dimension to five vignettes. Importance ratings for Vignettes 3 and 4 decreased from 1.00 to 0.75, due to one of the four experts decreasing their rating from 3 to 2, meaning somewhat important. The overall CVI improved for five vignettes (Vignettes 1, 2, 5, 6, and 7), stayed the same for Vignette 3 (0.92), and decreased for Vignette 4 (0.83 vs. 0.75).

Additionally, there was an average reduction in clarity after the revision (0.85 vs. 0.79). The three vignettes that accounted for the decrease in CVI for clarity following revision were Vignette 2 (1.00 vs. 0.50), Vignette 6 (1.00 vs. 0.75), and Vignette 7 (0.75 vs 0.50). The research team chose to explore this decline in clarity when importance and relevance improved following expert-guided revisions. Two areas were explored, changes in vignette word count and readability between the original vignettes and the revised vignettes.

The original vignettes had word counts ranging from 252 to 460 and the revised vignettes had word counts ranging from 445 to 626. The percentage increase in word count following the revisions ranged from 25% to 140%. Vignettes 2, 6, and 7 showed a decrease in clarity after revision and had a word count increase of 109%, 25%, and 140%, respectively.

The readability of the original and revised vignettes was evaluated and compared between phases two and three. The Flesch Reading Ease (FRE) index and the Flesch-Kincaid Grade Level (FKGL) index, found in most word processing programs, were computed for the seven vignettes (Figure 2). The FRE is calculated by taking the average length of sentences and the average number of syllables per word. Possible scores range from 0 (the text is very difficult to read) to 100 (the text is very easy to read). FKGL uses the same information as the FRE (i.e. sentence length and word length) but different weighting factors and presents the score as a United States education grade level for ease of interpretation (Zamanian & Heydari, 2012; Betschart et al., 2017).

Figure 2.

Flesch Reading Ease (FRE) index for Vignettes in Phase Two & Phase Three

The original seven vignettes scored an average FRE of 38.1, meaning the text is difficult to read but understood by college students, and had an average FKGL index of 11.7 indicating the reading level was corresponding to 12^th grade level. Following the vignette revisions, the average FRE was 44.1, a higher FRE than the original vignettes and indicated an improvement in readability, though it still indicated that the text was appropriate for college students. The example of an abbreviated revised vignette #1 showed the FRE increased from 38.3 to 49.8, indicating improvement in readability (Figure 2). The clarity dimension CVI for this vignette was 1.00 before and after revision, while importance improved from 0.25 to 1.00. The average FKGL index also showed improvement for the revised vignettes with an average score of 10.0 indicating a reading level corresponding to a 10^th grade level. The intended users of the vignettes are healthcare providers who all have college education so the FRE may have more meaning for this evaluation than the FKGL index. The increases in FRE of individual vignettes between the original and revised versions ranged from 1.8% to 55.5%. Overall, the readability ease improved for all vignettes and the average grade reading level decreased from 12^th grade to 10^th grade.

Discussion

This paper documents a carefully designed method using a three-phase approach to develop and validate vignettes for research. The descriptions of context and behavior in the vignette need to represent real-world phenomena in order for research findings and conclusions to be trusted and generalized (Evans et al., 2015). The three-phase process used to develop these vignettes was comparable to the approach by Jackson et al. (2015) for construction of research scenarios. An important distinction of vignette-based studies is that much of the validity is reliant on the vignette content being clear, relevant, and important in order to elicit responses based on the intended phenomena; in our study, the phenomena were pain and unknown risk for opioid misuse and OUD. This three-phase method facilitates establishing content validity and is a natural extension of the validation of the items or surveys about the vignette (Lynn, 1986; Polit & Beck, 2006). The method presented here (Figure 1) provided diagnostic information about the vignettes applying two rounds of expert panel judgment (Lynn, 1986). This method resulted in expert panel informed decisions for vignette revisions and established acceptable CVI of the final revised vignettes.

This method worked well and resulted in improved ratings for relevance and importance following revisions. Evans et al. (2015) stated that two domains that are most significant to users of vignettes are relevance and importance. The relevance ratings were high for the original vignettes and improved further in the revised vignettes. When vignettes depict a scenario appropriately, they are considered relevant and plausible; and when they do not, confusion, distress, or disinterest in study participation can result (Jenkins et al., 2010; Hughes & Huby, 2002). The importance ratings improved substantially between phase two and phase three (0.70 to 0.93), which is a very positive finding across the seven vignettes when using the three-phase method for testing content validity.

On the other hand, the average clarity rating decreased somewhat following revision and was the result of lower ratings in three out of seven vignettes, two of which became unacceptable (0.50). The content experts encouraged the revision of the vignettes from paragraph text to a structure familiar to clinicians that can be rapidly read to discern important data. Revising the format and adding additional content consistent with the electronic medical records meant the word count increased with more detailed vignettes, which resulted in lowering clarity for three vignettes. The research team explored further ways to assess clarity. The Flesch Reading Ease index and the Flesch-Kincaid Grade Level index showed that readability (measuring sentence and word length) actually improved in both indices in the revised vignettes (Zamanian & Heydari, 2012; Betschart et al., 2017).

In reviewing the outcome of our three-phase method for vignette development and validation, there are two alternative approaches to consider in order to improve clarity. One alternative is to present the vignettes for a mocked-up patient using an electronic medical record format from the beginning of development when the target population is familiar with using electronic medical records for information gathering. A second alternative is to follow the individual vignette readability indices (e.g., FRE and FKGL) through the phases as an additional measurement to guide revisions. The research team discussed the findings for all seven vignettes (i.e., improved importance and relevance, and reduced clarity) in the context of increased word count yet improved readability and decided no further revisions. The rationale for this decision was that the literature acknowledges relevance and importance as the most significant domains and these vignettes will be used to test a decision support tool applied by highly educated primary care health care providers where an electronic medical record format is typical.

CVI ratings are commonly used for item development and measurement, and they can serve a useful purpose in validating vignette content validity. Studies describe the importance of content validity for vignettes, use expert panel interviews to discuss vignette content, and consider the responses to assess face validity of newly develop vignettes (Randhawa et al., 2015; McCrow et al., 2013; Lanza, 1988; Paddam et al., 2010). However, these studies did not establish content validity with a quantitative index, such as the CVI. Using the methods in this paper, the CVI was quantified and served as fundamental and objective means for validating the content of vignettes and highlighting areas needing change.

There are no standards for vignette validation in research or education. Testing content validity of vignettes used for research should be seriously pursued as the very essence of research is quality data, and without quality data there cannot be quality data analytics. The method outlined can help improve quality data with additional strategies to assess clarity when needed. Vignettes used for healthcare education to measure knowledge of students or clinicians also require validation of those vignettes to ensure quality of the data collected. Standards for validation need to be established that are objective, logical and informed, and we propose this three-phase process as a method to do that.

Using this three-phase method for developing and establishing content validity of vignettes is the first step in determining the validity of the vignettes. The final step is to pilot test the vignettes to see how they perform. Pilot testing the validated vignettes is not within the scope of this paper, however, it needs to occur before they are used to assess healthcare providers’ decisions in developing treatment plans to manage pain while minimizing risk with or without using a decision support tool. For sensitive, complex, and high-risk scenarios, we may see increased reliance on vignettes for education and research in a contemporary world where pandemics (or other health circumstances) limit real-life engagement with patients. This rigorous and feasible method of vignette content validation may be cost-effective, low risk, and noninvasive for students, study participants, patients, and faculty.

Limitation of the methods we used for vignette development and validation is the amount of time spent validating seven vignettes. This three-phase approach for vignette validation demanded rigor and time. Content validity requires rigor in assessment, but is critical when using vignettes in research and interpreting research findings, and for evaluating student performance (Lynn, 1986). However, the rigor and provisions involved may surpass practical application when vignette development time is short and resources limited.

In conclusion, a clear method of development and validation of vignettes is important to evaluate the quality and appropriateness of vignettes for use in evaluating learning and decision-making. The three-phase approach presented in this paper includes initial vignette development; expert panel review, completion of content validity index survey and expert interview for revision recommendations; and revisions followed by re-evaluation of the content validity index. This three-phase approach is a viable method for developing and validating vignettes for use in health care education and research. Vignettes that are validated have greater plausibility, can portray complexity to guide knowledge attainment, and are useful tools for education and research measurement.