Abstract
Purpose
Mapping or matching the items in a clinical outcome assessment (COA) to concepts that define a condition is a common method for evaluating a COA’s concept coverage. The purpose of this research was to address the lack of formal guidance for conducting this task by developing a framework for best practices in COA concept mapping and applying it to a case study.
Methods
To develop the framework, we examined the literature and created a draft set of best practices which was then reviewed by experienced researchers through focus groups before being finalized. To conduct the case study, we extracted data from a systematic review of knee osteoarthritis (KO) symptoms and impacts and used the framework to map relevant concepts to items in the SF-36v2® Health Survey (SF-36v2).
Results
The framework guides researchers in defining the purpose of and data sources for the mapping, establishing guiding principles and decision-making thresholds, and conducting the mapping exercise. The results of the case study demonstrate the usefulness of the framework in identifying 27/36 items (75%) in the SF-36v2 that addressed concepts that define KO.
Conclusion
This case study illustrates how the framework for best practices in COA concept mapping may be used, highlighting how establishing clear concept definitions and guiding principles and following a structured process throughout can help produce consistent, reliable, and reproducible results. The results from this rigorous approach can provide valuable evidence to support decisions about the appropriateness of a COA for the intended patient population.
Supplementary Information
The online version contains supplementary material available at 10.1007/s11136-024-03773-2.
Keywords: Concept mapping, Clinical outcome assessment, Qualitative, Patient-reported outcome
Plain language summary
In health-related quality of life research, mapping items in a clinical outcome assessment (COA) to concepts that define a health condition is one way to evaluate an instrument’s content validity (or, how well the instrument addresses the concepts it intends to measure). Without formal guidance on how to do this mapping, researchers can be inconsistent. This article describes the development of a framework for best practices in COA concept mapping. Informed by the literature and input from researchers with expertise in COA development and evaluation, the final framework guides researchers through the mapping process from start to finish, from helping to define the purpose of the task and identify the data sources, to establishing guiding principles and decision-making thresholds, conducting the mapping, and displaying the results. A case study—in which items from the SF-36v2® Health Survey were mapped to concepts from a systematic review of knee osteoarthritis symptoms and impacts—shows the framework in action, demonstrating how following the best practices can lead to consistent results that can support the evaluation of an instrument’s content validity.
Supplementary Information
The online version contains supplementary material available at 10.1007/s11136-024-03773-2.
Introduction
When evaluating the content validity of a clinical outcome assessment (COA), it is essential to examine how well the measure aligns with the concepts that reflect the patient experience of the condition in which the measure will be used. In health outcomes research, this assessment occurs after key concepts of interest—typically symptoms and impacts on daily life—have been identified and defined through literature review and/or concept elicitation interviews, and before evaluating a measure or measures through cognitive debriefing interviews. For this assessment, researchers map the concepts of interest to specific items in a measure. Despite how often this type of mapping occurs, to date, there has been no actionable guidance for determining when and how to do this task, nor even consensus on what to call it. Indeed, a December 2023 search in the PubMed database found that articles describing this task of evaluating the concept coverage of a COA used the terms “mapping” [1–7] and in some cases, specifically, “concept mapping,” [7] “matching,” [8] or “linking.” [8–11] This lack of consistent terminology is further complicated by the existence of an unrelated research method termed “concept mapping.” [12] Interestingly, this unrelated method is defined in the ISOQOL Dictionary of Quality of Life and Health Outcomes Measurement [13], but the exercise of concept mapping described here is not.
Among the published articles that describe mapping, matching, or linking concepts to a COA, the prevailing approach [1, 2, 5, 6, 8–10] is to use the guidance from Cieza et al. (2005, 2019) [14, 15] which outlines broad rules for linking content in health outcomes measures and/or interventions to the World Health Organization’s International Classification of Functioning, Disability and Health (ICF) codes. These rules include recommendations to clearly identify concepts addressed by each item in a measure as well as the purpose for which the measure will be used, consider the response options when relevant, and disregard recall periods. The rules also provide specific ways to notate and organize the mapping results [14]. While this is a helpful starting point and can provide a template for the mapping process, these rules are specific to working with the ICF codes and, thus, require adaptation or supplementation when evaluating COAs.
To address the lack of formal, COA-specific guidance and consensus around concept mapping as it is used to examine the concept coverage of one or more COAs, we developed a framework of best practices to standardize and document the process and applied it to a case study to demonstrate its use.
Best practices framework
Development
The framework development process was led by a working group composed of 3 health outcomes researchers (MO’C, LB, MLC) with direct experience with COA-specific concept mapping. The working group also solicited input throughout the development process from a group of 4 scientific advisors (KJ, MC, MK, LTW), each with 20 or more years of relevant experience in qualitative research and/or psychometrics.
To develop the framework, the working group conducted a targeted review of the published literature to identify the different terms used to describe COA-specific concept mapping, the different purpose(s) it serves, and the various methodological considerations researchers must consider when designing and executing the task. Published literature was collected from desk searching as well as database searches in PubMed, CINAHL, and SocINDEX. The literature yielded just 6 articles after title/abstract screening and full-text review (see Online Resource 1 for search strings and PRISMA diagram). To supplement the information from the published literature, the working group also compiled a list of methodological considerations derived from their experience mapping concepts from health-related quality of life (HRQoL) studies to items in COAs in more than a dozen past research studies.
The working group relied on the findings from the literature and internal reviews to develop a draft framework of best practices, which was shared with and reviewed by two focus groups of researchers (n = 4; n = 5) who had experience conducting mapping themselves or providing oversight to studies with a mapping component. The 1-hour focus groups were conducted virtually via videoconference and facilitated by two members of the working group (MO’C, LB; using the same set of pre-established questions for both groups) while a third member (MLC) kept detailed notes. Focus group participants included researchers with 4–38 years of experience working in COA and HRQoL research; participants were trained in qualitative research (n = 4), quantitative research/psychometrics (n = 4), and mixed methods (n = 2) (see Online Resource 1 for details). Each group included a mix of individuals with different research training (e.g., public health, anthropology, epidemiology, psychology). All but one participant had first-hand experience conducting a mapping-style exercise within COA research. The focus group participants provided input on the draft framework of best practices, examining and debating each aspect of the proposed purposes for COA concept mapping, and brainstormed a list of potential names for the task.
The working group reconvened to review the meeting notes and discuss updates to the framework. Overall, focus group members agreed with the draft framework but provided input that helped refine it to reflect a wider set of real-world experiences with concept mapping. For example, the groups encouraged greater detail on the individual steps within the mapping process to make it easier to follow regardless of a user’s background knowledge or experience with mapping. The groups also suggested clearly pointing out which aspects of the process could be flexible and how (e.g., the number of mappers), to aid users in different contexts. The results of the focus groups also led to the selection of a final name for this task (“COA concept mapping”).
Framework overview
We define COA concept mapping as a semi-structured methodology for connecting the concepts that define and/or describe the lived experience of a condition with the content in one or more COA measures. The framework for best practices in COA concept mapping asks researchers 4 key questions, the answers to which will help guide the mapping process (Fig. 1); these questions help in identifying the purpose and data source(s), and, in turn, the appropriate process to use and display for results.
Fig. 1.
Framework for best practices in COA concept mapping
What is the purpose of the exercise?
The purpose determines the goal(s) of the mapping, which is typically to either evaluate concept coverage of one or more pre-identified COA measures or to create a short form from an item bank or longer measure. In both cases, COA concept mapping can help to identify whether an existing measure needs modifications (or to be paired with other measures) to make it better align with the concepts that define the condition of interest.
What are the data sources?
Data sources can differ, ranging from literature review findings and/or results from concept elicitation interviews or focus groups, to data from online platforms, health records, or established lists of health domains, such as ICF codes. The list of concepts derived from the data should also be considered at this point: a clear definition for each concept is essential to ensure that each member of a study team has a shared understanding of the concepts that will be used in the mapping, and to allow for reproducibility. This step mirrors the first rule for linking to ICF codes from Cieza et al. (2005), which stresses that “before one links meaningful concepts to the ICF categories, one should have acquired good knowledge of the conceptual and taxonomical fundaments of the ICF, as well as of the chapters, domains, and categories of the detailed classification, including definitions.” [14].
What process should be followed?
While the specific steps taken to complete the mapping may differ depending on the purpose and data sources of a study, the overall process remains the same. The process starts with establishing guiding principles, including language and wording preferences (e.g., a focus on specific terms), and thresholds for decision making. Guiding principles act as criteria for how connections between concepts and COA items are made. Such principles may focus on particular facets of a concept, study-specific need (e.g., will this be used in a clinical trial that requires the measurement of a particular subset of concepts?), population need (e.g., is it important that items be written or constructed in a certain way to maximize comprehensibility?), or the overall purpose of the task, in addition to item language and/or wording choice. Decision-making thresholds contribute towards reaching consensus, a key aspect of the framework approach. Before executing the mapping, all study team members should be well-trained on its purpose, the definitions for the concepts of interest (derived from the data source), and the pre-established guiding principles and thresholds.
Execution involves mapping, i.e., matching or linking, the clearly defined concepts of interest to individual items or domains in a measure, which can be followed by mapping items or domains back onto concepts as a cross-check (or vice versa, mapping items or domains to concepts and then concepts to items or domains as a check). Items can be categorized as matching a concept, not matching a concept, or in need of further discussion; the third option may be useful when an item could be interpreted in more than one way.
Whenever possible, two or more study team members should conduct the initial mapping independently; those results should then be compared to identify areas of convergence and divergence and to discuss and resolve any differences. If resources do not allow for multiple team members to map independently, it is recommended that the initial mapping results be reviewed by another researcher familiar with the work to validate the mapping results. Full study team discussion should follow and can often be the last step in the process; however, depending on the study, there may be an opportunity for additional review from a study sponsor, outside expert, measure developer, or other well-informed resource. If another reviewer is added, the process of reviewing, discussing, and confirming the results should be repeated until the full team comes to consensus on the final results.
How should results be displayed?
Results should be displayed in a way that aligns with the purpose of the mapping, best suits the data, and effectively communicates the findings to the target audience. For example, tables, graphs, or figures may be used to show how concepts are addressed by specific items, or by different COA measures (see Online Resource 2 for examples).
Case Study: mapping concepts in knee osteoarthritis to the SF-36v2
Methods
To demonstrate the use of the framework with open-access materials (rather than confidential individual interview transcripts, which are often used in such endeavors), we mapped concepts from a systematic review of knee osteoarthritis (KO) symptoms and impacts by Wallis et al. (2019) [16] to items in the SF-36v2 Health Survey® (SF-36v2). The Wallis et al. (2019) review was selected as the case study condition based on the article’s well-described methods and ample patient descriptions of the experience of KO (which are compiled based on the results of multiple studies). The SF-36v2 is a widely used 36-item patient-reported outcome (PRO) measure that captures insights into patient health across 8 domains, including physical functioning, mental health, physical and emotional roles, social functioning, bodily pain, and vitality [17].
We extracted concepts from the systematic review to describe the patient experience of KO, namely, the various aspects of daily life affected by the condition. Definitions (based on details in the review) included an overall description and, where relevant, details on different facets of the concept. For example, the definition for the impact on physical health included challenges with general function and movement (e.g., walking), but also on activities of daily living (e.g., eating, bathing, dressing) and instrumental activities of daily living (e.g., cleaning, cooking, and driving). The concepts were prioritized according to the number of studies in which they were identified.
Following the framework, we defined the purpose of the mapping, described the data source, established guiding principles and decision-making thresholds, mapped items in the SF-36v2 to concepts in KO, and held two consensus meetings to confirm the results with the full study team before finalizing the results (Fig. 2).
Fig. 2.
Framework for best practices in COA concept mapping: mapping KO to the SF-36v2
Identifying the purpose and data source
The purpose of this exercise was to evaluate the concept coverage of the SF-36v2 in relation to KO. The key concepts that defined KO included impacts on physical, social, and emotional functioning, role participation, and pain.
Establishing guiding principles and thresholds for decision making
The study-specific principles included: considering the language of the item to ensure it aligned with or mirrored the definition of the concept and considering the domain of the SF-36v2 an item belonged. Consensus, to be determined following the independent mapping results, was defined as agreement among all three mappers whether each item was relevant (i.e., that the item addressed the concept). Items that were deemed relevant/not relevant by 2/3 mappers required discussion to come to consensus.
Mapping and consensus meetings
The study team reviewed the definitions thoroughly to ensure familiarity with each concept. Three team members (MO’C, LB, MLC) independently mapped the concepts to items in the SF-36v2. Using a shared Microsoft (MS) Excel template, the mappers categorized each item within each concept as “yes” (matches concept definition, is relevant), “no” (does not match the definition, is not relevant), and “requires discussion” (might match the definition, discussion needed to determine the relevance). The results of the independent mappings were compiled into a single MS Excel worksheet that clearly displayed where the three mappers converged and diverged. The same three members of the study team reviewed the compiled results and discussed and resolved discrepancies where possible. Next the full study team reviewed and discussed the updated mapping results, again resolving any disagreements to come to consensus on all items. The final results were compiled into a table.
Results
The independent mapping results showed consensus across all three mappers for 67% (24/36) of the items in the SF-36v2. All three mappers agreed that 22 items each clearly addressed at least one concept extracted from the literature and therefore did not require further discussion; all three agreed that 2 items required further discussion. The remaining 12 items were deemed relevant by one or two of the mappers, but not all three, and therefore required discussion. For example, for 2 items, all three mappers agreed on relevance to one concept, while one mapper considered a second concept also potentially relevant.
The first consensus meeting resulted in the 14 items in need of discussion being recategorized: 5 were classified as relevant, while the remaining 9 were classified as not relevant (Table 1). Next, the full study team met to review the results. All team members agreed with the decisions made by the three independent mappers. The final set of results showed that 27/36 items (75%) addressed the concepts extracted from the literature: 10 items were relevant to the physical impacts, 2 to the social impacts, 6 to the emotional impacts, 7 to role participation, and 2 to pain. The 6 general health items and 3 vitality items did not map to any specific concepts related to KO.
Table 1.
Final concept mapping results
Discussion/Conclusion
The results of COA concept mapping are valuable for comparing measures, identifying gaps in concept coverage (both within and among measures), and making decisions as to which COA(s) are most appropriate for a particular context of use. Each of these benefits is a key step in the process of evaluating the content validity of a COA and developing a COA measurement strategy. The results from COA concept mapping can help researchers determine whether a measure is worth further investigation through patient cognitive interviews by generating objective data on how thoroughly it addresses the key concepts and the balance of relevant versus nonrelevant items.
The case study above provides a useful example of how these results can help with decision making. In terms of concepts, the results show that the SF-36v2 addresses all of the domains of the key concepts related to KO; if key concepts were missing, a second measure might be needed to fill those gaps. In terms of items, the results show that 75% of the items in the SF-36v2 are relevant to KO; but when deciding whether this is an appropriate measure to pursue for a KO-specific study, it is important to balance the usefulness of the relevant items against the 9 general health and vitality items that were not related to key concepts (e.g., would those 9 items be burdensome for patients and researchers to complete or analyze? Do those items provide other useful information? And if irrelevant items were to be dropped, how would such a change impact the ability to score the measure?). The comprehensiveness of the SF-36v2’s content coverage and minimal waste (i.e., items that are not directly mapped to concepts) suggests that in the case of KO, the SF-36v2 is worth pursuing for additional content validation in this condition. It is also important to note that the SF-36v2 lacks attribution to a specific disease, so, while useful, a disease-specific instrument would likely demonstrate even stronger mapping results.
This case study demonstrates just one use of the framework for COA concept mapping. Because COA concept mapping can be performed for different purposes, and use a variety of data sources, the process can differ slightly each time. Thus, the framework has both structure and flexibility: the structure allows for standardization and continuity (supporting reproducibility), and the flexibility allows for its use across studies, regardless of purpose, data source, or even study resources/limitations. In the case study presented here, we applied the framework to map concepts identified in a systematic literature review to an existing COA instrument, the SF-36v2, but it can just as easily be applied to the development of a short form from a large item bank, such as those within the PROMIS®, Neuro-QOL™, and ASCQ-Me measurement systems. Similarly, this case study used concepts extracted from a literature review as its data source; while this is a common data source in health outcomes research, the framework can accommodate data derived from concept elicitation interviews or focus groups, as well as health records and online discussion boards, among others (and any combination thereof). Indeed, including results from concept elicitation interviews is an effective way to include the patient voice in this task. This study also used three team members to conduct the initial mapping independently. While this is ideal from a triangulation perspective, the framework allows for alternate approaches to help confirm mapping results. This structured but flexible approach mirrors the linking rules established by Cieza et al. (2005, 2019) for mapping to the ICF codes. [14, 15]
This framework is intended to serve as a roadmap for researchers engaging in COA concept mapping as part of content validity evaluation process and can play an important role in helping researchers refine a list of candidate COAs early in the measurement strategy development process. Indeed, investigators often turn to existing instruments when the time and resources required to develop a new, condition-specific COA may be greater than those already allocated to a clinical study or clinical trial. In these circumstances, the results from COA concept mapping can help researchers determine if existing COAs are sufficient for measuring a condition, or if a new measure needs to be created to effectively cover the key concepts that define that condition. The framework presented here helps to promote consistent, reliable, and reproducible results. While it is possible to evaluate content validity without conducting COA concept mapping, such an approach risks the selection of a COA (or multiple COAs) that ask about irrelevant concepts or are missing key concepts. Concept mapping also minimizes the risk of conducting burdensome cognitive debriefing interviews in which patients will be tasked with completing and discussing COAs that may not have been fully vetted as appropriate. Including mapping as an interim step (i.e., after concept elicitation interviews or a literature review and before cognitive debriefing interviews or psychometric analysis) can help to streamline the COA selection and evaluation process by only advancing measures that are well-suited to a condition or by identifying early on that modifications to a measure, or even a new measure altogether, may be needed. Together, the framework and case study presented here demonstrate how a systematic assessment of the comprehensiveness and relevance of a COA can be achieved by establishing clear concept definitions and guiding principles before mapping and following a structured process throughout.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Author contributions
All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by MO’C, LB, and MLC, with additional review from MC, KJ, LTW, and MK. The first draft of the manuscript was written by MO’C, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Funding
The authors are all employed by Quality Metric. No outside funding was received for this work.
Declarations
Ethics approval
As all participants in the process to develop the framework of best practices for COA concept mapping were acting in their capacity as experts, this work did not require ethics approval.
Consent for publication
Not applicable.
Competing interests
Although the SF-36v2 Health Survey® is owned and licensed by Quality Metric, the authors have no relevant financial or non-financial interests to disclose and thus, no competing interests.
Footnotes
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Anderson, J., Levy, D. S., Lumry, W., Koochaki, P., Lanar, S., & Henry Li, H. (2021). Letting the patients speak: An in-depth, qualitative research-based investigation of factors relevant to health-related quality of life in real-world patients with hereditary angioedema using subcutaneous C1 inhibitor replacement therapy. Allergy Asthma Clin Immunol, 17, 60. 10.1186/s13223-021-00550-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Andresen, E. M., Fried-Oken, M., Peters, B., & Patrick, D. L. (2016). Initial constructs for patient-centered outcome measures to evaluate brain-computer interfaces. Disability and Rehabilitation. Assistive Technology, 11, 548–557. 10.3109/17483107.2015.1027298 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Crudgington, H., Collingwood, A., Bray, L., Lyle, S., Martin, R., Gringras, P., et al. (2020). Mapping epilepsy-specific patient-reported outcome measures for children to a proposed core outcome set for childhood epilepsy. Epilepsy & Behavior, 112, 107372. 10.1016/j.yebeh.2020.107372 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.De Oliveira Faria, S., Hurwitz, G., Kim, J., Liberty, J., Orchard, K., Liu, G. (2021). Identifying Patient-Reported Outcome Measures (PROMs) for Routine Surveillance of Physical and Emotional Symptoms in Head and Neck Cancer Populations: A Systematic Review. JCM ;10:4162. 10.3390/jcm10184162 [DOI] [PMC free article] [PubMed]
- 5.Kuspinar, A., & Mayo, N. E. (2013). Do generic utility measures capture what is important to the quality of life of people with multiple sclerosis? Health and Quality of Life Outcomes, 11, 71. 10.1186/1477-7525-11-71 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Shayestehpour, S., Sharma, K., Mosor, E., Omara, M., Ritschl, V., Shayestehpour, S., et al. (2022). PATIENT-REPORTED OUTCOME MEASURES FOR PEDIATRIC DENTAL PATIENTS: A METHODOLOGICAL REVIEW AND MAPPING EXERCISE. Journal of Evidence-Based Dental Practice, 22, 101661. 10.1016/j.jebdp.2021.101661 [DOI] [PubMed] [Google Scholar]
- 7.Watson, C., Scippa, K., Barlev, A., Kresa-Reahl, K., & Cole, J. C. (2022). Results from patient interviews on fatigue in progressive multiple sclerosis and evaluation of fatigue patient-reported outcome (PRO) Instruments. Neurol Ther, 11, 725–739. 10.1007/s40120-022-00337-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Gladwell, P. W., Cramp, F., & Palmer, S. (2020). Matching the perceived benefits of Transcutaneous Electrical nerve stimulation (TENS) for chronic musculoskeletal pain against patient reported outcome measures using the International classification of Functioning, disability and health (ICF). Physiotherapy, 106, 128–135. 10.1016/j.physio.2019.01.017 [DOI] [PubMed] [Google Scholar]
- 9.Majewski-Schrage, T., Evans, T. A., & Snyder, K. R. (2019). Identifying Meaningful Patient outcomes after Lower Extremity Injury, Part 2: Linking outcomes to the International Classification of Functioning, disability and health. Journal of Athletic Training, 54, 869–880. 10.4085/1062-6050-233-18 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Mueller, M., Schuster, E., Strobl, R., & Grill, E. (2012). Identification of aspects of functioning, disability and health relevant to patients experiencing vertigo: A qualitative study using the international classification of functioning, disability and health. Health and Quality of Life Outcomes, 10, 75. 10.1186/1477-7525-10-75 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Bernardelli, R. S., Santos, B. C., Scharan, K. O., Corrêa, K. P., Silveira, M. I. B., & de Moser, A. D. (2021). Application of the refinements of ICF linking rules to the Visual Analogue Scale, Roland Morris questionnaire and SF-36. Cien Saude Colet, 26, 1137–1152. 10.1590/1413-81232021263.03502019 [DOI] [PubMed] [Google Scholar]
- 12.Trochim, W. M. K. (1989). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12, 1–16. 10.1016/0149-7189(89)90016-5 [Google Scholar]
- 13.Mayo, N. E. (Ed.). (2015). Dictionary of quality of life and health outcomes measurement. First edition. Milwaukee, WI: International Society for Quality of Life Research (ISOQOL).
- 14.Cieza, A., Geyh, S., Chatterji, S., Kostanjsek, N., Ustün, B., & Stucki, G. (2005). ICF linking rules: An update based on lessons learned. Journal of Rehabilitation Medicine, 37, 212–218. 10.1080/16501970510040263 [DOI] [PubMed] [Google Scholar]
- 15.Cieza, A., Fayed, N., Bickenbach, J., & Prodinger, B. (2019). Refinements of the ICF linking rules to strengthen their potential for establishing comparability of health information. Disability and Rehabilitation, 41, 574–583. 10.3109/09638288.2016.1145258 [DOI] [PubMed] [Google Scholar]
- 16.Wallis, J. A., Taylor, N. F., Bunzli, S., & Shields, N. (2019). Experience of living with knee osteoarthritis: A systematic review of qualitative studies. British Medical Journal Open, 9, e030060. 10.1136/bmjopen-2019-030060 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Mariush, M. E. (Ed.). (2011). User’s manual for the SF-36v2 Health Survey (3rd ed.). QualityMetric Incorporated.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.