CodeMergeR: A Shiny‐Based Application for Codelist Integration

ABSTRACT

Objective

To describe CodeMergeR, an open‐source R shiny application developed within the VAC4EU network for the standardization, cleaning, and integration of individual concept codelists into a master file for real‐world evidence (RWE) studies.

Materials and Methods

CodeMergeR was designed to process codelists exported by CodeMapper v1.0. The application includes two modules: (1) Conformance and Coherence, which checks file naming conventions, metadata consistency, and structural integrity; and (2) Cleaning and Standardization, which removes duplicates, corrects formatting issues where a valid match is available, flags and excludes uncorrectable codes (e.g., scientific notation, dashes). Functional and performance testing were conducted using real‐world library metadata and codelists from the VAC4EU library, including manually introduced errors to assess detection capabilities.

Results

The CodeMergeR identified a wide range of structural and semantic issues, achieving an overall error detection rate of 96.8% (61/63 errors). Depending on the issue type, detected errors were either automatically corrected (e.g., rounding, when a validated code was found), flagged and excluded from the master codelist if unresolved (e.g., scientific notation, dashes), or flagged for manual review (e.g., metadata mismatches, missing concepts). Processing of 50 clinical concept folders and 10 Algorithms completed in under 1 min during performance testing.

Discussion

CodeMergeR detected structural and formatting errors with an accuracy of 96.8%, supporting its use as a quality‐check step prior to deploying codelists in analytical pipelines. In contrast to existing, CodeMergeR fills a distinct quality assurance role in codelist generation and transparency.

Conclusion

CodeMergeR contributes to improved reproducibility, scalability, and transparency in codelist management in RWE studies.

Key Points

• Health data collected via electronic records (real‐world data, RWD) have shown to be a valuable source of information to generate real world evidence (RWE) on effectiveness and safety of medicines and vaccines.

• Very little guidance exists on codelist integration, even though they are essential for semantic harmonization and identification of clinical concepts in RWD. Their manual integration is error‐prone and time‐consuming, making this an important and under‐addressed opportunity to improve RWE quality.

• CodeMergeR is an open‐source R Shiny application developed to standardize, verify, and integrate individual concept codelists into a harmonized master file for analytical pipelines in RWE studies.

Plain Language Summary

Researchers who study the real‐world use and effects of medicines often use data from electronic health records. This type of data, called real‐world data (RWD), is generally analyzed in a federated manner (data stay local) using a common protocol, common analytics and a common data model. In this way RWD from different geographical regions and systems can be used even if diagnoses and treatments are recorded in different vocabularies. To make sure these can be retrieved, compared and combined, researchers use codelists, a set of files with a list of medical codes related to the clinical event of interest. However, concatenation of files can be difficult and time‐consuming due to differences in formats, naming styles over time and several people contributing to its creation. To help with this, we created CodeMergeR, a free, open‐source tool that helps researchers clean, check, and combine codelists automatically. It finds and fixes common problems, like missing files, different naming styles, or formatting errors. CodeMergeR fits easily into the tools built on the ConcePTION common data model by the Vaccine Monitoring Collaboration for Europe (VAC4EU) and has been shown to detect almost all issues (96.8%) quickly and accurately. CodeMergeR helps make studies using codelists more reliable, faster, and easier to repeat. In the future, it could also be used in other research networks that use multiple vocabularies.

1. Introduction

The growing availability of electronic health records (EHR), including medical records, administrative databases (e.g., billing and claims), and disease registries is transforming healthcare research. Collectively known as Real‐World Data (RWD), these data sources play an important role in generating real‐world evidence (RWE) [1, 2, 3]. To transform RWD into RWE, researchers must first define study variables, that is, specific clinical concepts with roles such as outcome, exposure, or co‐variate. These variables are typically retrieved using standardized codes for diagnosis, treatments, procedures, and laboratory results from vocabularies like the International Classification of Diseases (ICD), Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT), Read Codes, and International Classification of Primary Care (ICPC), among others [4, 5, 6, 7].

A codelist is a curated set of codes that represents a clinical concept across different vocabularies. For example, Guillan‐Barré Syndrome may be recorded in ICPC with N94.01, in SNOMED CT with 40 956 001, and in ICD‐10‐CM with G61.0 [8, 9, 10]. Codelists are essential for semantic harmonization in RWE, as they enable automated and transparent code retrieval across data sources [11]. However, creating and updating codelists remains an important challenge due to dynamic nature of vocabularies which require frequent updates.

Research networks have taken different approaches towards semantic harmonization. The Observational Health Data Sciences and Informatics (OHDSI) network, for example, requires translation of all original codes into a common vocabulary and uses ATLAS to compile concepts. The CodelistGenerator from DARWIN EU supports the systematic extraction and aggregation of codelists for vocabularies in the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM). VAC4EU and EU PE&PV research networks use the ConcePTION CDM, which eliminates the need for upfront vocabulary translation by the data holder [12]. Instead, it performs vocabulary mapping within the study scripts, for speed, transparency and flexibility.

Currently, researchers often create individual codelists stored in various formats such as spreadsheets or text files, without consistent naming conventions or standardized format structures. As highlighted by Matthewman et al. [11] the use of metadata and clear, consistent naming conventions is essential for ensuring codelist's transparency, minimizing errors, and promoting reuse in future studies. Within VAC4EU, a harmonized approach is implemented: codelists are first created by a coordinator using CodeMapper and then tagged by medical reviewers using specific labels to indicate the level of specificity for each code in relation to the study objective [13]. However, creating, tagging, and maintaining codelists with multiple reviewers requires substantial effort and oversight to ensure high‐quality codelists. The process of merging individual concept codelists into a single master study codelist is error‐prone, as it can lead to issues such as code duplication, formatting inconsistencies, and accidental omission of relevant concepts.

In this article, we introduce CodeMergeR, an open‐source R shiny application developed and validated within the VAC4EU network that provides an efficient and user‐friendly tool to support researchers in the standardization, cleaning, and integration of individual concept codelists into a harmonized study codelist for RWE studies.

2. Methods

2.1. File Workflow

CodeMergeR is currently designed to align with a prespecified folder and file structure used in VAC4EU. VAC4EU has already adopted many of the indicators explained by Matthewman et al. [11], such as metadata, clinical concept definition, review and publication. A dedicated taskforce composed by a coordinator (SM) and medical doctors is responsible for creating and reviewing collaborative codelists files obtained from CodeMapper v1.0, which maps clinical concepts across multiple vocabularies based on the Unified Medical Language System (UMLS) [13]. If available, local vocabularies (i.e., not necessarily standardized internationally) are provided by collaborators and manually mapped to the corresponding clinical concepts.

Usually, researchers need hundreds of individual codelists for specific clinical concepts relevant to their study. After creation of codelists in CodeMapper v1.0, these files were uploaded to Sharepoint within a nested folder structure. All folders followed standardized naming conventions. The folder name comprises four elements: first, a letter indicating the system related to the codelists (e.g., “E” for Endocrine, metabolic, or nutritional diseases or “I” for Infectious diseases); second, an abbreviation representing the concept; third, the type of variable for which the concept is made, such as “COV” for covariate or “AESI” for outcomes, and fourth, the full name of the codelist. These four elements are separated by underscores, resulting in three underscores in total within the folder name. The name of the file (codelist) containing the diagnostic codes is derived from the folder name, except for the full name of the concept. This combination is referred to as the variable_name. A worksheet with the same variable_name must be present. The file format extension *.xls or *.xlsx was chosen for uploading due to its user‐friendly interface, enabling the medical team to add columns, store comments and highlight specific cells for discussion and approval.

2.1.1. Mandatory Columns in Working Files

Each codelist file was required to have a set of mandatory columns which will be used to create a master file (Box 1). These columns are different for the files in the Algorithms/folder and the ones in the clinical concept folders (Box 2). Examples can be found in the File S1.

BOX 1. Mandatory fields or columns for the file in each clinical concept folder.

Fields a	Definition	Example
Coding system	Standardized vocabulary used to classify medical concepts, such as diseases, procedures, medications, and clinical events, through unique codes	ICD10
Code	Unique identifier, typically alphanumeric, assigned to represent a specific medical concept	E10
Code name	Label or description assigned to a specific set of medical codes representing a condition, procedure, or medication	Brittle diabetes (mellitus)
Concept	Unique, unified representation of a medical or health‐related concept that links equivalent terms, codes, and descriptions from different vocabularies and coding systems based on UMLS (i.e., concept unique identifier CUI as exported by CodeMapper)	C0011854
Concept name	UMLS based descriptive term that represents a unique clinical or health‐related concept and serves as a label for the concept	Diabetes mellitus, Insulin‐dependent
Tags	Level of specificity used when selecting codes to represent a medical concept	Narrow

Note: Letter case and spelling is important when running the app.

^a These fields are retrieved from the CodeMapper v1.0 tool.

BOX 2. Mandatory fields or columns for the file in Algorithms/folders.

Fields	Definition	Example
Codesheet_name a	Name of each concept codelists needed to create the rule‐based algorithm of interest	P_MATERNALTORCH_COV/I_CMV_COV
		I_SYPHILIS_COV
		I_HEPATITISB_COV
		I_HSV_COV
		I_TOXOPLASMOSIS_COV
		I_RUBELLA_COV
		I_HIV_COV
		I_ZIKA_COV

Note: Letter case and spelling is important when running the app.

^a This field is used to store names of concepts codelists used inside each Algorithms folder.

2.1.2. Library Metadata

The library metadata file consists of several worksheets that document information about codelists for various clinical concepts, such as concepts, procedures, and medicines. It serves as a central repository to store information about existing codelists, review status, and use of codelist across studies, as well as codelists to be created. The worksheet particularly relevant for CodeMergeR is called CDM_Events which focuses on Algorithms and clinical concepts, and contains key fields that facilitate the identification, creation and processing (i.e., quality‐checking and combination) of codelists (Box 3).

BOX 3. Mandatory fields or columns in the excel worksheet CDM_Events.

Fields a	Definition	Example
Event_abbreviation/variable name	Represents the variable_name derived from the folder and file naming convention	C_AMI_AESI_Acute mycardial Infarction
Covariates with no code	Identifies concepts for which codelists are not yet available (i.e., an Excel file is not present in the corresponding folder), those planned for future development, or those that have been archived and are no longer in use	Yes—When codelist is to be created. (empty)—When codelist is available.
Study name	Indicates the studies requiring specific Algorithms or CONCEPTSs, enabling filtering of the codelist library metadata based on the needs of a particular research project	COVID19_AESI
Type of events	Specifies the classification of the event	AESI

Abbreviations: AESI, Adverse Event of Special Interest; COV, Covariate; PrA—Codes used in the pregnancy algorithm developed within ConcePTION [14].

^a While these administrative columns are valuable for broader codelist management, they are not directly utilized by CodeMergeR.

2.2. CodeMergeR

2.2.1. Design & Architecture

We aimed to develop a tool that incorporates automated checks to detect invalid folder names and flag inconsistencies within independent codelist files. The design of CodeMergeR consists of four components: input files, a user‐friendly front‐end interface, a back‐end processing dataflow, and export files.

In Figure 1, the app workflow begins with the input and import phase, where users can upload multiple Excel files containing independent codelists and the library metadata, and specify the study name. The conformance and coherence module which is part of the back‐end processing ensures file compliance with predefined requirements, such as the number and type of files, file extensions, and naming conventions, while also verifying consistency across library metadata, algorithm folders, and clinical concepts. Next, the Cleaning and Standardization module identifies and removes duplicate entries, corrects formatting issues, and eliminates invalid characters from key fields such as code and vocabulary. This results in a curated, standardized codelist ready to be exported. Boxes 1, 3 (Section 2.1) summarize the required input structure (folder/file naming conventions, mandatory fields); Boxes 4, 6 summarize CodeMergeR's internal checks and behavior. The specific checks performed by CodeMergeR in both modules are described in Boxes 4 and 5. The handling of each detected issue (i.e., automatic correction, flagging with exclusion, or flagging for manual review) is specified in the System Behavior column. Furthermore, the export functionality enables users to download the master file (i.e., the concatenated study codelist) and the pregnancy algorithm (PrA) codelist in CSV format. The PrA codelist is exported separately because pregnancy codes are retrieved via exact match, unlike the master file, which relies on the hierarchical structure of the coding system. Therefore, the PrA codelist is maintained and reused independently of study‐specific codelists within CodeMapper [15]. Also, files reporting errors (when applicable) and two summary tables of included concepts, vocabularies and tags (tags_details.csv and summary.csv). All export files are saved locally within the “cleaned_codelist” folder within the app main directory, which is the local folder where the code of the app is saved.

FIGURE 1.

CodeMergeR. The process begins with the input of codelist files and the library metadata file. The front‐end interface allows users to specify the study name, upload files, and view validation results. The back‐end processing includes two modules: Module 1 performs conformance checks on folder and file structure, while Module 2 conducts standardization checks to clean and harmonize the data. The final output includes a cleaned master codelist and associated summary files, which are exported for use in RWE studies.

BOX 4. Module 1 conformance and coherence.

Check name	Description	System behavior	Displayed messages a , c
#1 Folder Names	Checks whether all folder names have 3 underscores	Valid folders are copied into a newly created clinical concepts folder named. Conceptsets/Algorithms within the app directory. Erroneous folders are excluded from further processing. List of erroneously named folders.	“All folders' names in clinical concepts are correctly formatted” “Concept folder name does not have at least three elements”
#2 Files with no codes	Checks for presence of Excel file (.xls or .xlsx) inside the concept folder for those folders that have been specified as “Covariates with no codes” in the library metadata file	Erroneous folders are removed from further processing. List of erroneous folders.	“This folder is marked as “Concept with no codes” but contains an Excel file, which is not expected. This folder will be removed” “All “Concepts with no codes” folders are correctly specified and contain no Excel files”
#3 Study name presence b	Checks if the name of the study in the app is present as a field name in the library metadata file	App stops	“The study name you entered is found as a column in library metadata” “The study name you entered is not found in the library metadata. Please check for typos or column naming”
#4 Excel File Count	Check if only one Excel file (.xls or .xlsx) is present inside the concepts folder	Erroneous folders are removed from further processing. List of erroneous folders.	“All concepts' folders contain exactly one Excel file” “The following folders contain zero Excel files. Create the clinical concept Excel file” “The following folders contain multiple Excel files. Keep only one Excel file in case of multiple files”
#5 Folder and Excel File Name Match	Check if the name of the concept folder and the Excel file inside match	Erroneous folders are removed from further processing. List of erroneous folders.	“The following folders have mismatched names between the folder and the Excel file. Match the name of the Concept folder and Excel file located inside the folder” “All folders and their Excel files are correctly named and match”
#6 Excel worksheet Name Match	Check if the worksheet inside the Excel file has the same name as the file itself	Erroneous folders are removed from further processing. List of erroneous folders.	“The following folders have mismatched names between the Excel file and the worksheet inside. Match the name of Excel file and at least one worksheet of inside this Excel file” “All Excel files contain a worksheet name that matches the file name”
#7 Mandatory Columns	Check if all mandatory columns are present within Excel files	Erroneous folders are removed from further processing. List of erroneous folders.	“All Excel files in Concepts folders contain the required columns” “Missing mandatory columns: [columns' names]. Review requirements and add the necessary column names”
#8 Concepts and Algorithms Match	Check if all concepts necessary to create the Algorithms exist in the Concepts folder	Erroneous folders are removed from further processing. List of erroneous folders.	“The following folders are present in Algorithms but missing from Concepts: [folder names]. Create the necessary Concepts folder for each Algorithm or tag them to the “Covariates with no codes” column” “All folder names are consistently present in both Algorithms and Concepts”

Note: Valid folders pass through, invalid folders are excluded from downstream processing (with folder‐level granularity—an error anywhere in a folder removes that whole folder), except Check #3, which is a blocking error that stops the app rather than excluding anything.

^a Same displayed messages but adapted to Algorithms/folder context.

^b Not applicable for either concepts or Algorithms/folder.

^c Success and failure message displayed in every check.

BOX 6. Internal Quality Checks Implemented in CodeMergeR.

Check type	Description	System behavior	Displayed messages a
Library metadata presence	Check that the library metadata file can be loaded correctly	App stops	“Error reading the library metadata. Please check the file format or permissions”
Concepts folder presence	Check for presence of at least one Concepts folder in the uploaded data	App skips all checks	“No folders are found in the directory to check”
Concepts folder—Library metadata comparison	Verifies if all required folders according to the library metadata file are present in the concept folder directory	Displays error (flagged for manual review)	“These Concepts folders are missing but present in the library metadata. Please create it and re‐upload before running this check”
Codelist metadata	Verifies the structure and content of the library metadata file.	Missing worksheet CDM_Events Displays error (flagged for manual review)	“Required worksheet not found in the library metadata. Review requirements and add the necessary worksheet”
Library metadata file structure	Checks for missing required fields in the library metadata file	Displays error (flagged for manual review)	“Required columns [‘Event_abbreviation/Variable Name’, ‘Covariates with no codes’] not found in the library metadata. Review requirements and add the necessary columns”
File format	Ensures that only valid file types are uploaded. For example, only. zip files	Displays error	“Only. zip files accepted in Upload Codelist Folder box”
Folder availability	Checks for required folders before performing naming convention checks	Displays error; List missing folders	“No folders are found in directory to check” “The Concepts folder is missing. Please ensure it exists before running the check”

^a Message displayed only on failure.

BOX 5. Cleaning and standardization.

Check name	Type	Description	System behavior	Displayed messages
#1 Available Study Concepts	Informational	Show all available concepts folders that have passed all checks of module 1 and are specified as necessary for the study in the “library metadata file”	All available folders are copied in a new “Available” folder in the back‐end processing. List of folders present.	“The following concepts have passed module 1: [folder names]”
#2 Missing Study Concepts	Informational	Show all concepts folders that are specified as necessary for the study in the library metadata but are not found in the Concepts folders	List of folders unavailable	“The following concepts are present in library metadata but missing in concept directory: [folder names]. Create these folders if necessary or delete them if not needed from library metadata. If they are planned to be created but haven't been created yet tag them in the ‘Covariates with no codes’ column” b
#3 Merge Codelists	—	Combine all separate codelist in a single master file without applying any cleaning action in the master file	File “Processed_codelist.csv” in the folder “Untouched codelist” in the back‐end processing. The “Untouched codelist” folder is automatically created when we start the app and is in the main app directory	“Merge Complete! Results saved in folder ‘Untouched Codelist’” “Merge Failed! Check logs” a
#4 Clean master file	Fix	Clean the “Processed_codelist” file in preparation for the master file stored in “Cleaned_codelist”	Remove rows with missing values in the fields of codes or coding systems. (automatic correction) Replace commas with dots where applicable Check for misspelling in tags; allowable values are “narrow,” “possible,” “exclude” and “ignore” (automatic correction) Verify rounding of codes and apply a cross‐check procedure to correct available codes. In all codes with a length of 15 or above, the last 2 digits (for possible rounding issue) and 3 digits (for rounding issue) are checked if equal to 0. For codes with rounding_issue set to TRUE a comparison with available codes is performed, and the erroneous codes are replaced with the validated codes, if present. (flagged for manual review) Create a summary of concepts and vocabularies and export as summary.csv and tags_summary.csv in the Cleaned_codelist folder Append local dictionaries to master file Ensure vocabularies contain only letters or numbers, except for SNOMEDUS_CT, free_text, ITA_procedures_coding_system where the underscore is allowed	“The following codes contain scientific notation: [codes and concept names]. Fix codes if necessary or otherwise they will be automatically excluded from the master codelist” “None—No codes with scientific notation were found” a
#5 Scientific Notation	Flagged, excluded if unresolved	Check rows with scientific notation (E+) in the code column	List of erroneous folders.
#6 Dash Detection	Flagged, excluded if unresolved	Check for records with (−) in the code column	List of erroneous codes and corresponding concepts.	“The following codes contain dashes: [codes and concept names]. Fix codes if necessary or otherwise they will be automatically excluded from the master codelist” “None—No codes with dashes were found” a
#7 Missing values	Informational		A table showing the number of records with empty values in columns code, coding system and tags. Empty table—if no codes are identified.
# 8 Rounding Issues	Flagged for manual review	Check all codes longer than 15 digits and end with “000”	A table showing the code, vocabulary and the name of the concepts with rounding issues. Empty table—if no codes are identified.	“The following codes have rounding issues: [codes and concept names]. Fix codes if necessary or otherwise they will introduce misclassification or might not fully capture the concept of interest” “None—No codes with rounding issues were found” a
# 9 Possible Rounding Issues	Flagged for manual review	Check all codes longer than 15 digits and end with “00”	A table showing the code, vocabulary and the name of the concepts with possible rounding issues. Empty table—if no codes are identified.	“The following codes have possible rounding issues: [codes and concept names]. Fix codes if necessary or otherwise they will introduce misclassification or miss to retrieve the concept of interest” “None—No codes with possible rounding issues were found” a

^a Success and failure message displayed.

^b Message displayed only when the check identifies an issue.

Internal validity checks are included through various steps of the app to ensure input files conform to expected standards. These checks assess the structural and formal quality of a codelist (e.g., file format, naming, duplicate detection); they do not assess whether a code is the clinically or epidemiologically correct choice for a given concept. For example, if the library metadata file contains all required fields. Details of each check are provided in Box 6.

2.2.2. User‐Interface and Usability Features

CodeMergeR starts with a pop‐up message when initializing the app, which describes a summary of the steps within the app. After, two tabs open, one for module Conformance and Coherence (Table 1) and one for module Cleaning and Standardization (Table 2). Table 1 displays information about the uploaded data folder, including information such as the file name, date of execution, and the results of each individual check in module conformance and coherence. A box explaining all rules and guidelines on folder structure and naming (conformance) and folder availability (coherence) is also present in the first tab. A clickable legend box is available for each tab where users can personalize which checks they would like to implement.

TABLE 1.

Summary of introduced vs. detected errors.

Error	Examples in test set	Checks (Figure)	Detected errors (n = 61/63%)
Incorrect folder names	B_COAGDISAESI_Coagulation disorders all, BAIHEMOLYTICANEMIA_AESI_Autoimmune hemolytic anemia	Module 1, check #1 (S4)	2/2 (100)
Missing or multiple files in several folders	C_ACI_AESI_Acute cardiovascular injury, C_AMI_AESI_Acute myocardial Infarction, C_ANGINA_AESI_Angina pectoris	Module 1, Check #2	3/3 (100)
Excel filename does not match folder concept name	Excel name not matching variable name (from folder name) B_ANAEMIA_COV_Anaemia	Module 1, check #5	1/1 (100)
Worksheet name does not match Excel file name	Worksheet name not matching Excel file name C_ACUTEHF_AESI_Acute heart failure	Module 1, check #6	1/1 (100)
Concepts and Algorithms mismatch	Algorithm references non‐existent concepts folders available (Table S3 in File S2)	Module 1 Check #7	21/21 (100)
Metadata mismatches in library metadata	Event listed in library metadata but not present in concept folder (Table S4 in File S2)	Module 2, Check 4	1/3 (33)
Missing required concept codelist for the study	Concept folders and Algorithms selected for the study but not present in the output codelist (Tables S3 and S5 in File S2)	Module 2, Check #2	26/26 (100)
Missing values (e.g., missing codes, tags)	Missing event code	Module 2, Check #7	4/4 (100)
Code formatting issue due to rounding	Codes longer than 15 digits ending in 3 zeros	Module 2, Check #8	2/2 (100)

Table 2 displays the summary of all included concepts and tags together with their corresponding vocabularies, and the results of each individual check in module cleaning and standardization. Each tab has an option to capture and save screenshots for documentation purposes. We have made available a short video in our repository for visualization (https://github.com/vjolahoxhaj/CodeMergeR).

2.2.3. Software

We developed CodeMergeR using R version 4.2.3 within the RStudio environment [16]. To improve usability, embedded JavaScript was used for dynamic UI components. A mandatory pop‐up ensures users read and acknowledge key information by clicking “OK” before use. The app integrates several R packages including shiny, DT, readxl, and tidyverse and was tested on Windows and macOS [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28]. Deployment was carried out via shinyapps.io, ensuring accessibility for end‐users without requiring local installation. The source code is maintained in a public repository (https://github.com/vjolahoxhaj/CodeMergeR) accompanied by user documentation to support reproducibility and adoption.

2.2.4. App Testing

We conducted functional and performance testing to ensure that CodeMergeR performed as intended using real‐world library metadata and codelists available in the VAC4EU library. We also tested edge cases, such as missing or incorrect input files, to confirm that the app handles errors appropriately.

For functional testing, we arbitrarily selected 10 clinical concepts to span multiple concept types in the library metadata and set them to “Yes”: B_ANAEMIA_COV (Anaemia), B_COAGDEF_COV (Coagulation, deficiencies), C_AMI_AESI (Myocardial infarction), C_HF_AESI (Heart failure), E_GOUT_COV (Gout), Im_ANAPHYLAX_COV (Anaphylaxis history), M_ARTRHEU_COV (Rheumatoid arthritis), N_CONVULSION_AESI (Generalized convulsion), N_EPILEPSY_COV (Epilepsy), and Onc_BREASTCANCER_COV (Breast cancer). This allowed us to verify that the app correctly processed selected clinical concepts.

For performance testing, we introduced a mix of correctly structured and manually flawed files in a copy of the VAC4EU library, to simulate common errors encountered in practice, such as incorrect folder naming, invalid code formats, missing files, and inconsistent metadata. An overview of the errors introduced is documented in File S2. The number of times an error was introduced depended on the type of input to which the corresponding validation rule applied and whether the input file had more than one error type introduced. Specifically, the input types included algorithm and concepts folders, whereas the error types included issues such as incorrect folder names, more than one or no excel file present in the folders, incorrect excel file name, etc. When the same validation rule applied to both algorithm and concepts, the corresponding error was introduced separately for each input type. We evaluated the correctness of the cleaning and integration processes by comparing the automated error outputs with manual review.

3. Results

3.1. Use Case

3.1.1. Installation Requirements and Environment

The app package was obtained from the corresponding GitHub repository and stored locally within the working environment. Detailed instructions for installation and execution are available in the repository. The preparation of the input files involved two components: (i) a. zip file containing the Concepts directory, and (ii) library metadata. The Concepts directory comprised an Algorithms subfolder, containing separate algorithm folders, along with distinct folders for the individual codelists. Within the library metadata, a new column was created to specify the study name. Concepts and Algorithms relevant to the study were marked as “yes” in this column. After input configuration, the torun. R script was executed and a web interface automatically opened in the default browser, guiding the user through the subsequent steps via an on‐screen pop‐up window.

3.1.2. App Testing: Functional Testing

Functional testing assessed whether CodeMergeR correctly processed real, unmodified codelists from the VAC4EU library. For functional testing, we deployed CodeMergeR using the tailored library metadata with 10 concepts as required for the study and the full list of independent codelists available in the VAC4EU library. The app correctly processed selected clinical concepts reporting issues in two checks from module Conformance and Coherence and four in module Standardization and Cleaning.

In Module 1, the first issue, check #4: Excel File Count, occurred in B_COAGDIS_AESI_Coagulation disorders all, where multiple Excel files were detected in the folder instead of the expected single file (Figure 2). The second issue, identified in Check #8: Not all necessary Concepts for Algorithm creation are available: C_ARRH_AESI and R_ASTHMAEXACERBATION_AES, was caused by the requirement of these two codelists for the creation of Algorithms present in the main directory (Figure 3).

FIGURE 2.

Visualization of module 1, check #4. The folder failed the check due to multiple Excel files inside, resulting in an “Invalid” status.

FIGURE 3.

Visualization of module 1, check #6. The folder failed the check due to a mismatch or formatting issue in the Excel file, resulting in an “Invalid” status.

In Module 2, check 21: Missing Study Concepts, involved B_ANAEMIA_COV being listed as missing: this concept was flagged and excluded from the master codelist until the discrepancy was resolved (Figure S1). Additionally, eleven structural issues were detected in Check #6: Dash Detection (Figure 4) and four issues in Check #7: Missing Values, all related to pregnancy clinical concepts labeled as PrA; as with dash‐containing codes, these were flagged and automatically excluded from the master codelist unless corrected by the user (Figure S2). Under Check #9: Possible Rounding, 60 codes were flagged with potential rounding issues, including E_GOUT_COV (n = 8), Im_ANAPHYLAX_COV (n = 12), N_CONVULSION_AESI (n = 1), Onc_BREASTCANCER_COV (n = 10), and pregnancy‐related clinical concepts (n = 29) (Figure S3); unlike dash and scientific‐notation errors, rounding issues were automatically corrected via a crosscheck against available codes whenever a validated code was found.

FIGURE 4.

Visualization of module 2, check #6. List of diagnostic codes containing range‐style values (dash) across different coding systems (ICD‐10, ICD‐10‐CM, ICD‐9‐CM) used in pregnancy‐related clinical concepts. These range‐style codes require special handling during validation and standardization to ensure accurate code representation.

3.1.3. Performance Testing

Performance testing assessed CodeMergeR's error‐detection accuracy and processing speed using a modified copy of the library containing manually introduced errors. For performance testing, we deployed CodeMergeR using the modified versions of library metadata and 50 concept codelists and 10 Algorithms from the VAC4EU library in which errors were introduced. This process ended within 1 min. The app was run locally on a MacBook Air 2024 equipped with an Apple M3 processor and 8 GB of memory, running macOS Tahoe 26.6.2. Table 1 we summarize the types and number of introduced errors correctly identified by CodeMergeR. In total, 63 distinct errors were introduced for testing, of which 61 were successfully detected, yielding an overall detection rate of 96.8%. Screenshot of the errors is available in File S3.

4. Discussion

The increasing use of RWD demands efficient and reproducible approaches for extracting clinical concepts through codelists. The development of CodeMergeR directly addresses the challenge of concatenating individual codelists through two integrated modules, offering a quality assurance solution for codelist concatenation.

4.1. Key Findings

CodeMergeR can reliably identify a wide range of formatting, structural, and semantic errors. For example, the code 12405,31000000103 Myocarditis due to disease caused by severe acute respiratory syndrome coronavirus 2 (disorder) in SNOMED vocabulary is rounded when used in Excel (i.e., the last 103 digits become 000). The findings in Module Conformance stress the importance of robust file structures and careful data formatting in the development and management of codelists. For example, the detection of multiple files within a single folder and the mismatch in file names illustrate how seemingly minor inconsistencies can disrupt automated workflows. Similarly, issues identified in Module cleaning and standardization such as missing entries in the concept directory, structural anomalies, and warnings related to potential rounding of codes, highlight the value of automated quality‐check tools. These types of errors would be difficult to detect manually in a timely and reliable manner, reinforcing the need for systematic quality checks during codelist preparation.

Organizing CDM such as ConcePTION (i.e., no semantic harmonization is required) generally require a broader set of quality checks than rule‐based CDMs like OMOP [12, 29]. Because the ConcePTION CDM allows multiple vocabularies, a summary of the all included vocabularies is necessary to detect missing or unregistered ones, handle range‐style values and reconcile conflicting tags in clinical concepts (i.e., the same code appears as high‐specificity and low‐specificity). In contrast, OMOP requires data holders to map the original codes to a common vocabulary (SNOMED CT) before analysis. Although errors in this mapping process are not always transparent, OMOP does not apply cross‐vocabulary reconciliation during analytical stage as ConcePTION CDM does [30].

The internal validation process, involving 63 manually introduced errors, confirmed an overall detection rate of 96.8%, with all major error categories such as naming convention issues, missing files, and erroneous codes successfully identified. This process is critical in the context of large‐scale RWE studies where even minor inconsistencies can lead to the exclusion of key clinical concepts or misclassification of outcomes.

4.2. Strength and Limitations

CodeMergeR processed individual concept codelist files including 50 clinical concepts folders and 10 Algorithms within 1 min, with consistent performance across operating systems. Its user‐friendly interface, including instructional alerts, clickable legends, and structured outputs, supports adoption by non‐technical users such as clinicians and researchers. The use of CodeMergeR within VAC4EU and EU PE&PV highlights its value for collaborative research environments, where codelists are iteratively reviewed, shared, and reused across studies. By centralizing and standardizing codelist processing, the application reduces duplication of effort, saves time, provides documentation on issues or errors, and enhances reproducibility.

The current version of CodeMergeR is tightly coupled to a particular folder structure. Vocabulary updates and version management are handled upstream by CodeMapper; CodeMergeR's role begins once updated codelists are available as input, so users must re‐run it whenever an updated combined codelist is needed. Additionally, the application does not currently version codelists across runs, only the execution date is recorded, and users must manually archive outputs to preserve a record of each run; future versions will aim to retain run‐level metadata on included/excluded concepts and corrected errors. Its open‐source license allows for adaptation to other international networks that need to deal with concept codelist concatenation. Integration with external code repositories or public‐facing platforms like OpenCodelists could further expand its utility and ensure alignment with evolving standards in the field [31]. The cleaning and standardization rules were developed and validated for the coding systems listed in Table S1 (File S4); users applying CodeMergeR to coding systems outside this list should manually verify that cleaning operations do not inadvertently alter valid values. The quality‐check process does not yet include comprehensive checks of the library metadata content, as reflected in a 33% detection rate for metadata‐related issues. When a metadata mismatch is detected, the affected concept is flagged and excluded from the master codelist until the user resolves the discrepancy and reruns the app. The concept is never silently dropped or erroneously included without warning. This app was initially created to check and harmonize individual codelists into a unified master file, but further enhancements should be added for issues such as misspelling of clinical concepts which might lead to missingness of necessary clinical concepts in the master codelist.

4.3. Comparison With Other Tools

Several tools exist to support the creation and management of codelists, each supporting different stages of the RWE workflow, summary can be found in Table S2 (File S4). DARWIN EU's CodelistGenerator is a data‐driven R package designed to work within the OMOP CDM [32, 33]. It connects directly to OMOP data instances and returns candidate codes using keywords/regex and options such as “include Descendants,” offering diagnostics to assess codes appearance in the data. Its primary function is to assist researchers in discovering and defining clinical concepts based on observed data. CodeMergeR complements CodelistGenerator by operating downstream in the workflow. Although CodelistGenerator helps identify relevant codes, CodeMergeR focuses on standardizing and quality‐checking researcher‐refined codelists, often compiled from multiple sources and vocabularies into a harmonized master file. Together, these tools could form a pipeline: discovery with CodelistGenerator, followed by quality assurance and integration with CodeMergeR.

OHDSI ATHENA serves as a vocabulary browser and downloader for OMOP‐standardized vocabularies, but lacks functionality for integrating or cleaning externally sourced or multi‐vocabulary codelists [34]. CodeMergeR could bridge this gap by transforming heterogeneous codelists into OMOP‐compatible formats, enabling smoother downstream use in tools like ATLAS [35]. OpenCodelists, developed by OpenSAFELY, provides a collaborative web interface for building, reviewing, versioning, and sharing codelists [31]. Although it supports transparency and reuse, it lacks automated harmonization features. CodeMergeR could serve as a pre‐processing step before uploading to OpenCodelists, ensuring that lists are clean and structurally sound for the platform.

4.4. Disclaimer

Given the lessons learned during the development of CodeMergeR, VAC4EU has improved CodeMapper leading to a version 2.0 (manuscript in preparation) which integrates all steps of codelist creation, tagging, and concatenation. Despite the overlap, CodeMergeR remains relevant as an independent quality‐assurance tool: its conformance, coherence, cleaning, and standardization checks are distinct from CodeMapper's scope. Its applicability to codelists from sources other than CodeMapper has not yet been demonstrated and remains a direction for future quality‐assurance work.

5. Conclusions

CodeMergeR is an open‐source R Shiny application developed to streamline the processing of codelists for RWE studies. It automates the standardization, cleaning, and integration of files containing clinical codes from multiple vocabularies that have been collaboratively and iteratively reviewed. By enforcing metadata standards and automating quality checks, the tool improves reproducibility, minimizes manual effort, reduces workload, and supports scalable, transparent codelist management. Future development will focus on expanding compatibility with alternative metadata formats, folder system structures, incorporating user feedback and integrating CodeMergeR with public tools and repositories to enhance interoperability and broader adoption, improving the codelist lifecycle.

Author Contributions

V.H. and C.L.A.N. started with the initial design; V.H. programmed the R scripts; V.H. and C.L.A.N. wrote the initial draft of the manuscript. V.H., C.L.A.N., J.R.‐A., S.M., and M.C.J.M.S. provided critical feedback and edited the manuscript. C.L.A.N. and M.C.J.M.S. provided supervision. This paper only reflects the personal views of the stated authors.

Funding

The authors have nothing to report.

Ethics Statement

The authors have nothing to report.

Conflicts of Interest

V.H., C.L.A.N., J.R.‐A., S.M., and M.C.J.M.S. are currently salaried employees at University Medical Center Utrecht, which receives institutional research funding from pharmaceutical companies and regulatory agencies and is administered by University Medical Center Utrecht.

Supporting information

File S1: Examples of the input files for CodeMergeR.

Table S1: Algorithms' Excel file structure.

Table S2: COIs' Excel file structure.

Table S3: Codelist library metadata file structure.

File S2: Description of the errors introduced to the codelist to test app performance.

Table S1: List of Algorithms' folders and manually introduced errors.

Table S2: List of available COIs folders and manually introduced errors.

Table S3: List of individual concepts necessary for the available Algorithms.

Table S4: Library metadata file and manually introduced errors.

Table S5: Library metadata file (the first 4 columns are to be found in the Codelist library metadata Excel file, the last column display whether this an error or not).

File S3: File containing the supplemental figures.

Figure S1: Visualization module 2 check #1.

Figure S2: Visualization module 2 check #7.

Figure S3: Visualization module 2 check #9.

Figure S4: Visualization of Incorrect folder names.

File S4: Summary tables with coding system characteristic and available tools.

Table S1: Description of coding systems.

Table S2: Comparison of CodeMergeR, CodelistGenerator, ATHENA, and OpenCodelists.

Data Availability Statement

The authors have nothing to report.

CodeMergeR availability: CodeMergeR is available as an open‐source R script in the following link https://github.com/vjolahoxhaj/CodeMergeR. The repository has a MIT License (NonCommercial).