Educator's blueprint: A how‐to guide for collecting validity evidence in survey‐ based research

Abstract

Surveys are descriptive assessment tools. Like other assessment tools, the validity and reliability of the data obtained from surveys depend, in large part, on the rigor of the development process. Without validity evidence, data from surveys may lack meaning, leading to uncertainty as to how well the survey truly measures the intended constructs. In documenting the evidence for the validity of survey results and their intended use, it is incumbent on the survey creator to have a firm understanding of validity frameworks. Having an understanding of validity evidence and how each step in the survey development process can support the validity argument makes it easier for the researcher to develop, implement, and publish a high‐quality survey.

BACKGROUND

Surveys are a common tool used to evaluate educational initiatives and conduct data collection for research. Therefore, it is essential that clinician educators incorporate thoughtful design to ensure meaningful results. The first paper in this series on survey development and implementation focused on the initial steps of survey development: deciding whether a survey is the appropriate tool for assessing variables of interest, settling on the objectives of the survey, picking constructs to assess, and writing and formatting survey items. ¹

However, those steps are insufficient in isolation, and it is equally important to collect evidence to support the validity argument for the survey's results and how those results will be used. There are many steps that can be taken during the development process to establish and properly document validity evidence within a chosen framework. In this paper, we discuss validity frameworks and steps that can be taken in the development process to provide validity evidence.

VALIDITY FRAMEWORKS

In simple terms, validity typically refers to how well the data produced by an assessment tool reflect the intended construct. In other words, does the survey accurately measure what it intends to measure? It is notable that the instruments (in this case, the surveys) themselves are not said to be “valid,” but instead it is the inferences made based on survey results that are more or less valid for a specific purpose, in a specific context, for a specific population, and, often, at a specific time in history. ² It is also notable that there is no single piece of evidence that can certify a survey's results as “valid.” Instead, designers collect validity evidence for the survey results and their intended use. In this way, collecting validity evidence is akin to establishing a clinical diagnosis. There is no one clear test that cinches the diagnosis. Instead, through an accumulation of evidence, a case is built to establish the likelihood of a disease.

Over the years, a number of frameworks have been developed to provide structure for assembling a validity argument. These are primarily directed at the validity evidence of assessments and can be applied to survey instruments. The classical validity framework outlines three types of validity (content, criterion, and construct). However, modern interpretations of validity contend that there is only one type of validity—construct validity—and they therefore organize their frameworks under this construct validity banner. ³ , ⁴ , ⁵ These modern validity frameworks seek to gather evidence showing a connection between assessments and specific constructs, arguing that assessment results are only useful if they are theoretically and empirically linked to a construct. ⁵

Messick's unified validity framework is the most commonly used approach in the medical education literature and has been adopted and modified by the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education as part of their Standards for Educational and Psychological Testing. ⁶ , ⁷ The Standards define five sources of validity: content, response processes, internal structure, relations to other variables, and consequences of testing (see Table 1 for definitions and examples of how authors have documented and demonstrated validity using these five sources of validity evidence).

TABLE 1.

Messick's unified validity framework as articulated in the standards for educational and psychological testing

Category	Definition	Evidence for validity in the context of a survey	Example of validity evidence documentation
Content	The appropriateness of survey content in light of the construct the tool is intended to measure	Based on previously developed instruments Robust derivation of survey items stemming from well‐defined constructs of interest Focus groups Expert reviews of draft items	Padela et al. 19 describe a comprehensive literature review, searching multiple databases. They clearly describe the expertise of the consensus panel that arrived at the constructs of interest. An expert panel reviewed the draft survey “to assure that the clinical vignettes were specific and realistic, to reduce ambiguity within question stems.”
Response processes	The psychological processes or cognitive operations of survey takers and the “detailed nature of the performance … actually engaged in” while completing the survey 6	Following best practices for question item formulation Cognitive interviewing of respondents	Example of cognitive interviewing: “We performed cognitive, ‘think‐aloud’ interviews with two senior medical students who had recently matched in EM and two rising fourth‐year medical students pursuing non‐EM.” 20
Internal structure	The relationships among survey items or sections of a survey, including score consistency/reliability and subscale structure	Reliability calculations (Cronbach's alpha, inter‐rater reliability, factor analysis, generalizability theory) Data can come from pilot testing but should also be calculated after full data collection	Pickett et al. report Cronbach's alpha for their survey of EM resident training in psychobehavioral conditions 15
Relationship to other variables	The associations (positive or negative) between the survey scores and data on other variables	Associations between survey scores and external variables with theoretical associations Data from pilot testing	“As expected, confidence levels generally increased by PGY; however, this was not always the case among PGY‐4s whose confidence ratings were lower than the mean across several skill areas.” 1
Consequences of testing	The positive or negative, intended or unintended effects of survey use	Behavioral changes as a result of survey administration (mere‐measurements effects) 21 Iterative curricular improvements with post‐course surveys Psychological impact of surveying sensitive constructs	Rarely included as a component of initial published works. 21 May require follow‐up studies to assess.

Note: This table provides definitions for the five sources of validity in Messick's framework. Examples of how a survey designer can collect evidence for validity in a given domain are also provided.

Educators can also use the validity framework developed by Kane, ³ which is a stepwise, argument‐based approach to validity. It has the advantage of relying less on psychometric data and being applicable to both quantitative and qualitative assessments. It is less commonly used in survey research but depending on the role of a survey in an assessment process, it may provide a flexible approach to documenting validity evidence. In this framework, the validity of an assessment is examined methodically starting with a single response item through the final real‐world implementation. By collecting and adjudicating validity evidence at each step (scoring, generalization, extrapolation, and implication), assessment developers cohesively build an argument for or against the validity of an assessment's scores and intended use. ⁸

COLLECTING EVIDENCE FOR VALIDITY USING MESSICK'S UNIFIED VALIDITY FRAMEWORK

In the development and implementation of a survey, it is important that the survey designer uses methods that allow for the accumulation of validity evidence. Given its widespread use in medical education and its adoption by the measurement community as articulated in the Standards for Educational and Psychological Testing, this section will situate the utility of these methods in the context of Messick's unified validity framework. ⁷ The researcher, however, may use the data from these methods to report validity evidence for their survey results using the framework best suited to their assessment. Deciding on the most appropriate framework is a matter of balancing several factors: (1) which framework is most familiar in a given context, (2) the researcher's own skill and comfort, (3) the type of evidence that can be collected, (4) the complexity of the validity argument, and (5) the stakes of the inferences being made. ³

We have previously outlined several processes in the early stages of survey development that help build a case for content validity. ¹ First, a detailed literature search can help identify previously derived survey instruments assessing the same or similar construct(s) and will help the researcher better refine the phenomena they intend to measure. ⁹ However, just because a survey has been published does not mean validity evidence has been collected in a rigorous manner. ¹⁰ Taking stock of any validity evidence provided in previous publications is important. Researchers should not focus solely on the survey instrument in isolation but instead should consider its intended use in a potentially new context or new population. When applied in a new way, a previously published survey may or may not yield high‐quality results that can be used to make valid inferences. For example, a survey that works well for assessing burnout in medical students would need to be tested (and additional validity evidence collected) if that same tool were to be used in a new context (e.g., within a large health care system) or with a new population (e.g., residents). With that said, the efforts at identifying, using, or adapting previously derived surveys add strength to an argument for content validity. Where previously derived instruments are not available, focus groups, expert panels, or Delphi consensus approaches can be used to identify constructs of interest. ¹¹

After the clear identification of the objectives of the survey and constructs to be assessed, the next step in survey development is to write initial drafts of survey items. ¹¹ Following best practices for item writing, including question wording and response option formatting, helps increase the likelihood that a survey respondent will comprehend and respond to items as intended. ¹² The survey items should then be reviewed by content area experts to further collect content validity evidence. In this step, a group of content area experts is tasked with evaluating how well the survey items reflect the constructs, the clarity of the written items, the relevance of the items to the construct(s) being assessed, and the likely distribution of responses. ⁹ Integrating the feedback from the expert review allows for additional iterative changes to the survey, readying it for the next steps of development: cognitive interviewing and pilot testing.

Cognitive interviewing is a process that allows the survey designer to gather qualitative data on how a small group of respondents actually engage with the prospective survey. ¹³ This step adds important evidence about response processes as it provides a means to empirically study the way in which survey respondents mentally process individual survey items, respond using the given response categories, and work their way through the overall survey instrument. ⁸ As a qualitative technique, cognitive interviews rely on in‐depth interviews with a relatively small sample of volunteers whose characteristics resemble those of the ultimate survey sample.

Two common methods for conducting cognitive interviews are the think‐aloud and verbal probing techniques. In the think‐aloud technique, the interviewer encourages the respondent to verbalize all of their thoughts as they answer each individual survey item. The transcripts from these interviews can then be analyzed to assess whether or not the respondents are engaging with the survey items as intended. In the verbal probing technique, the interviewer asks directed questions of the respondent. These questions can vary in timing and form. They can occur concurrently to the participant responding to survey items, in a retrospective fashion after the respondent has completed a section of the survey, or during natural breaks in the process of filling out the survey (immediate retrospective probing). ⁹ The format of the questions in this process may include asking the respondent to paraphrase/restate a survey item, describe what they understand of the intent of an item, and defend why they answered in a particular way (among other questions). ¹³ Taken together, cognitive interviews are a valuable way to both verify problems that survey designers suspect might cause difficulties (e.g., problems understanding the meanings of certain words) and discover unexpected response process challenges (e.g., faulty interpretations that the survey designer did not foresee).

Prior to the full distribution of the survey, it is wise to pilot test the survey on a small group of potential respondents. There is no consensus on the size of the sample used in a pilot test. Instead, the sample size is usually based on the purpose of the study and the analyses that will be conducted using the pilot data. For example, if reliability or factor analyses are to be conducted, then a larger pilot sample may be needed (e.g., 10 respondents per survey item for factor analysis). Generally, the pilot sample should mirror the intended sample but include individuals outside the group ultimately to be surveyed. Despite one's best efforts, a survey designer never knows exactly how well their instrument will function until it is distributed to potential respondents and completed under typical conditions. This step also allows the researcher to trial the distribution methods and data collection mechanisms, ensuring they function as intended. Finally, pilot testing allows the researcher to test out their analytical approach and begin getting a sense for things like item distribution and variability, internal structure, and sometimes even relations to other variables (on a small scale).

Following pilot testing, or sometime after full distribution of the survey, the researcher should plan to collect data about the internal structure of the survey. For surveys that use scales (i.e., sets of items designed to measure a construct or constructs), factor analysis is often used to investigate the unidimensionality of a given construct, that is, the degree to which the items measure only a single construct. Although the details of factor analysis are beyond the scope of this paper, several useful resources exist. ¹⁴ Following factor analysis, the researcher should consider calculating the reliability of the survey scores, using internal consistency reliability calculations between items (i.e., Cronbach's alpha) or test–retest methods. Both factor analysis and reliability analysis help establish validity evidence based on internal structure.

Additionally, the researcher can examine the correlations between survey responses and other variables thought to be related. For example, a survey assessing residents as teachers might expect to find (based on theory and/or prior research) a positive correlation between confidence in bedside teaching skills and advanced levels of training (where senior residents will have had more training and practical experience serving in a supervisory role). By establishing that expected relationships do indeed exist, the researcher adds to the validity argument by showing evidence based on relationship to other variables.

The final form of validity evidence that a survey creator should consider is consequential validity. Consequential validity documents the effects of the survey, including effects of survey administration and the resulting inferences that might be made using those results. Much as we, as clinicians, worry about the downstream consequences of testing (e.g., will this cardiac CT lead to potentially unnecessary invasive testing?), we should also attend to the downstream consequences of assessments, including surveys, in medical education. Indeed, surveys and other assessments should be thought of as interventions with the potential to have intended and/or unintended, beneficial and/or negative effects on those being surveyed, and on educators, researchers, and the larger systems within which the surveys function. ¹⁵ For example, a postcourse survey of a novel curriculum that results in iterative improvements to the content, structure, or administration of that curriculum is an example of an intended, beneficial impact of a survey.

It has been shown that administering surveys that assess behavioral intention and socially desirable behaviors can, in some cases, make those behaviors more likely to occur. ¹⁶ This change in behavior that results from survey administration is referred to as the mere‐measurement effect. ¹⁷ Behavioral change that results from survey administration is, however, not always positive. In fact, high‐stakes survey instruments may lead to behavioral changes that threaten the validity of the survey data. For example, Appelbaum and colleagues ¹⁸ found significant differences in wellness scores on internal surveys versus Accreditation Council for Graduate Medical Education (ACGME) surveys, postulating in part that the differences could have been the result of coaching bias or social desirability bias in respondents taking the ACGME survey, since those scores are used to make high‐stakes accreditation decisions.

CONCLUSIONS

In the process of survey development, there are several steps that can be taken to support the validity argument for a survey and its intended use. Building this argument requires a firm understanding of validity frameworks and of how the development process intersects with these frameworks. Though several validity frameworks exist, because of its common use and acceptance, we have focused on Messick's validity framework here, showing how it can be used to guide the development process and support a given validity argument. Evidence based on content is collected through a detailed approach to survey purpose and construct selection. Response processes are examined through an evidence‐based approach to response item writing and formatting as well as cognitive interviewing and pilot testing. Internal structure is supported by factor and reliability analysis. Documenting the ways in which survey scores correlate with other known variables extends the validity argument by examining expected relationships. Finally, documenting the intended or unintended, beneficial or harmful effects of survey administration and use further supports the validity argument.

Having made a rigorous validity argument, collected the necessary data, and conducted the appropriate analyses, researchers are now ready to administer their survey. In our next paper, we will cover the challenges of survey administration, focusing on how to reach the intended audience and how to optimize a survey's response rate.

CONFLICT OF INTEREST

The authors declare no potential conflict of interest.

Hill J, Ogle K, Gottlieb M, Santen SA, Artino AR Jr. Educator's blueprint: A how‐to guide for collecting validity evidence in survey‐ based research. AEM Educ Train. 2022;6:e10835. doi: 10.1002/aet2.10835