LLM-augmented semantic embeddings enable Cross-Lingual mapping of medical procedure terms

Abstract

Cross-lingual information retrieval limits global exchange of data because of the high diversity in the methods to classify, document and encode medical procedures. Traditional keyword-based or single-language systems are not able to align data from surgical and interventional procedures, especially from non-English healthcare systems. This study aims to develop a pipeline for cross-lingual retrieval and integration of medical procedures data. MAP-CARE is a novel framework that leverages Large Language Models (LLMs) for translating and transforming medical procedures into a unified multilingual embedding space. Semantic embeddings are used to enhance retrieval accuracy and interoperability across languages and healthcare systems. MAP-CARE demonstrated high accuracy in the translation and mapping of clinical terms. Its cross-language translation performance proved robust, achieving up to Acc@5 = 0.90 in translating procedure classification codes across English, German, French, and Italian. The cross-classification mapping workflow also showed high accuracy in aligning two different national procedure classifications, with exact and near matches exceeding 53.8% at the most granular level. MAP-CARE offers a flexible, scalable, and robust solution for the multilingual and cross-system integration of medical procedural data. Its innovative use of large language models (LLMs) combined with semantic embeddings sets a new standard for the accessibility and utility of multilingual medical information. The framework is designed for easy extension from a terminology file in CSV format and is publicly available.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-025-34778-7.

Introduction

Harmonizing data from electronic health records (EHRs) remains a global challenge. Initiatives such as the Unified Medical Language System (UMLS)¹ and the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM)² aim to standardize and integrate heterogeneous health data vocabularies and sources. These frameworks are crucial for ensuring interoperability in multilingual environments, where variations in language morphology can lead to significant inconsistencies and misinterpretations in medical data. The current linguistic and structural diversity of medical terminologies calls for robust tools that can not only translate but also semantically align concepts across languages and classification systems. While platforms like UMLS and OHDSI’s Athena tool offer comprehensive vocabularies and mappings, they primarily focus on established standardized terminologies and may lack the flexibility to accommodate country-specific or health-system-specific classifications.

Significant challenges remain in aligning data from medical procedures and surgical procedures from non-English speaking regions. Classifications of medical procedures encode medical interventions and are essential for monitoring, billing, quality control, and research within the healthcare environments. Medical procedure codes are typically multiaxial by including abbreviations and free-text descriptors, which are often ambiguous, lack specificity, or include multiple possible interpretations, making semantic mapping to a standardized code difficult. Additionally, they often include complex inclusion and exclusion criteria and are context-sensitive, with modifiers such as material used or surgical approach further influencing their meaning. Retrieving a specific code relying only on keyword matches often fails to accommodate the complexity and diversity of medical terminologies and classifications, overlooking nuances and differences crucial for precision. For instance, a “gallbladder removal” query may not retrieve records labelled as “cholecystectomy,” the medical term for the same procedure. Similarly, a “knee replacement surgery” search might miss entries listed under more formal terms such as “total knee arthroplasty”. Given that English is widely regarded as the standard language for knowledge exchange, terminology standardization, and research, there is also a need for data accessibility through English for term searches to bridge linguistic and classificatory gaps.

Contextual and national procedural classifications often rely on specialized terminology, and their integration into international standards such as International Classification of Health Interventions (ICHI)³ is fraught with complexities due to their hierarchical structures and linguistic nuances. For example, procedural terminologies and coding systems widely used in the U.S., such as the Current Procedural Terminology (CPT) of American Medical Association⁴, the Healthcare Common Procedure Coding System (HCPCS)⁵, are not always directly translatable to systems used elsewhere. As a result, importing or aligning information from national systems such as Germany’s Operations and Procedures Catalogue (OPS)⁶, or France’s Classification Commune des Actes Medicaux (CCAM)⁷, Switzerland’s Operations and Procedures Catalogue (CHOP)⁸, to name a few, remains challenging in the majority of countries worldwide.

Several systems have been developed to facilitate the automated extraction and linking of clinical terms to standardized vocabularies. Over the past 20 years, rule-based mapping has been the most commonly used method, applied across various classifications and use cases. For example, MetaMap⁹ was designed to identify UMLS concepts within free-text clinical narratives. Similarly, systems like those by Liu et al.¹⁰, cTAKES¹¹, HeTOP¹², and FasTag¹³ employ rule-based methods to extract structured information from clinical text and map it to international classifications, ontologies, and terminologies such as International Classification of Diseases (ICD)¹⁴, RxNorm¹⁵, the Human Phenotype Ontology (HPO)¹⁶, and Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT)¹⁷. The design of SNOMED CT supports its role as a universal language for health care and reflects features typical of natural language, including synonymy, hierarchical structure, compositionality, and contextual adaptability^18,19. However, despite the effectiveness of these systems in identifying terms from free text, challenges remain in dealing with non-standardized or evolving procedural codes—particularly in non-English contexts.

Recent advances in natural language processing (NLP) have introduced powerful tools to process, interpret, and standardize language in complex domains such as healthcare. Among these, word embeddings have emerged as a particularly promising technique. By transforming words into numerical representations that capture contextual and semantic relationships, embeddings approximate human-like understanding of language^20–23. This capability enables computers to interpret medical texts with a nuance approaching that of human reasoning. Different methods, such as pretrained language models (PLMs) and key-word extractors were compared to populate defined procedural elements^24–27. However, their application in healthcare remains a work in progress, due to the unique linguistic demands and conceptual complexity of biomedical terminology as pointed out by Chiu & Baker²⁸. The rise of large language models (LLMs) has further advanced the field, providing AI systems capable of extracting insights, summarizing complex information, and supporting clinical decision-making. Nonetheless, effective deployment of LLMs in healthcare requires domain-specific adaptation to ensure both reliability and scalability²⁹. Despite growing interest in clinical NLP, there remains a notable scarcity of research on terminology embeddings in non-English languages, which poses significant challenges for accurate information retrieval from multilingual electronic health records (EHRs). Beyond rule-based systems and general-purpose word or sentence embeddings, several recent methods learn multilingual medical concept embeddings specifically for term normalisation. SAPBERT³⁰ and CODER³¹ pretrain BERT-based encoders on UMLS synonyms and, in the case of CODER, additional knowledge-graph relations, an achieve state-of-the-art performance on biomedical entity linking and cross-lingual term normalisation. Likewise, Wajsbürt et al.³² propose a concept-normalisation system for French that leverages multilingual contextual embeddings and UMLS-based supervision. These models, however, are trained and evaluated on large, established terminologies such as UMLS and SNOMED CT and are primarily targeted at linking free-text mentions to existing concept identifiers, rather than at representing and aligning national procedure classifications such as CHOP and OPS.

To address these longstanding challenges, this study introduces MAP-CARE (Multilingual Approach for Procedures in Clinical and Retrieval Embeddings)—a novel framework that redefines access to multilingual medical procedural data. MAP-CARE uniquely enables seamless integration of multilingual data and allows retrieval of medical information documented in non-English languages using English as the query language. By leveraging large language models (LLMs) and advanced multilingual embeddings, the framework encodes medical terminologies into a unified semantic space, capturing cross-lingual and cross-classification relationships with high precision. This approach moves beyond traditional rule-based mapping by offering a scalable, data-driven solution for semantic interoperability. The following sections outline the methods underpinning MAP-CARE and demonstrate how these conceptual innovations are operationalized into practical tools that significantly enhance cross-lingual alignment of surgical and diagnostic procedures. To our knowledge, this is the first comprehensive work on the multilingual representation of medical procedures within the full domain of surgical and diagnostic classifications.

Methods

Workflow overview

The MAP-CARE framework, depicted in Fig. 1, begins by processing a (non-English) medical classification, exemplified by the Swiss Classification of Operations and Procedures (CHOP). Each term from the classification undergoes a series of transformations. First, terms are sanitised (e.g. removing self-references, substitution of standard abbreviations. Second, the term is translated and contextually augmented using an LLM, Gemma 2³³. Third, the enhanced description is transformed into a semantic numeric representation. The enhanced term’s text is encoded into a 1024-dimensional numeric vector that captures its semantic meaning using the embedding model mxbai-embed-large-v1^34,35. Finally, the generated vectors from the complete classification are stored in ChromaDB³⁶, a database designed for efficient vector management and retrieval. The framework is designed for easy extension from a terminology file in CSV format and is publicly available^37,38.

Fig. 1.

MAP-CARE workflow for multilingual classification integration. In the MAP-CARE pipeline for integrating non-English medical classifications, input terms such as “Diagnostischer Ultraschall des Auges,” are sanitised and translated into English using the LLM Gemma 2 with contextual augmentation. The augmented terms, containing detailed descriptions, are then converted into a 1024-dimensional vector using the mxbai-embed-large-v1 model. This procedure yields a single shared multilingual embedding space: each vector represents a concatenation of the original-language label together with its English translation and description, allowing cross-language retrieval by embedding query terms in the same space. These vectors are stored in ChromaDB for semantic search and retrieval, enabling seamless access to multilingual procedural data.

Rationale for the evaluation design

We designed three complementary evaluations, each targeting a distinct aspect of MAP-CARE. Evaluation 1 assesses the quality of the curated English translations. Evaluation 2 examines CHOP–OPS alignment using SNOMED CT as a pragmatic external reference. Evaluation 3 evaluates direct semantic similarity between CHOP and OPS without relying on SNOMED CT. These components are intentionally separate and are not intended to be merged into a single performance metric; instead, they provide complementary perspectives on translation fidelity, semantic equivalence, and cross-system mapping behaviour.

Translation and preprocessing of CHOP and OPS codes

The Swiss Classification of Operations (CHOP) is a descendant of the ICD-9-CM developed by the Swiss Federal Statistics Office independently since 2008. CHOP is structured hierarchically in a tree-like architecture, organizing medical procedures across multiple levels of specificity. Anatomical regions are indicated at the first and second levels using two-digit codes, while procedural invasiveness and methods are detailed at the third and fourth levels with four-digit codes. The hierarchy culminates in highly specific terminal codes at the six-digit level, resulting in over 10,860 distinct entries. Although a hierarchical structure is present, it is inconsistently applied—particularly where child concepts do not inherit all explicitly defined features, and levels of specificity are not systematically represented. The classification is translated across three Swiss national languages: French, German, and Italian. Despite its comprehensive nature, the classification exhibits semantic inconsistencies and is limited to formats in PDF and CSV. CHOP version 2023 was used for this project.

The German Operationen- und Prozedurenschlüssel (OPS) is an adaptation of the International Classification of Procedures in Medicine (ICPM) specifically tailored for the German healthcare system. Maintained by the Federal Institute for Drugs and Medical Devices (BfArM), the OPS categorizes surgical operations and medical procedures in a hierarchical structure, similar to the CHOP system used in Switzerland, and consists of ca. 33’750 distinct terminal codes in the 2023 version⁶. The OPS codes are organized from broader categories to highly specific procedures, represented by a combination of digits and letters that detail procedural types and methods. The XML of the 2023 OPS was parsed into a CSV file for further analysis.

The sanitised CSV files for CHOP and OPS were translated from German/French/Italian into English using the Gemma 2 LLM accessed via the ollama interface³⁹. We did not perform any parameter fine-tuning of the model; instead, we used one-shot prompting tailored to the clinical translation task. Each prompt followed a consistent template consisting of (i) a system instruction that constrained the domain (clinical procedures), output style, and required format (English translation followed by a concise explanation separated by a vertical bar), (ii) one in-context example (“diagnostic ultrasound of the eye”), and (iii) the new CHOP or OPS term to be translated. The example term was selected manually from the source classification and was not optimised on any held-out evaluation subset. The complete prompt templates (“model files”) for German, French, and Italian are provided in Supplementary Table 1. The LLM (Gemma 2) occasionally produced verbose or conversational outputs (e.g., “In English, the entry translates to…”, “This entry refers to…”) rather than a clean label and description. To normalise these outputs, we implemented a simple rule-based post-processing pipeline. First, the raw LLM response was automatically split into (i) a short English label and (ii) a longer explanatory description using heuristic delimiters such as line breaks, “:”, “–”, or sentence boundaries. Second, we applied a deterministic cleaning function that removed a predefined list of boilerplate phrases and meta-text (e.g., “Translation”, “In English, the entry translates to”, mentions of the “Swiss classification of surgical interventions (CHOP)”) and corrected a few recurrent tokenisation errors (e.g., “Imcorporated” → “Incorporated”). This procedure does not alter the semantic content of the translation but removes LLM-specific noise and ensures consistent formatting. The resulting cleaned label and description constitute the English translations used in all subsequent evaluations. Beyond the sanitising pipeline, only limited manual spot-checking was performed. The most common remaining issues were occasional leftover meta-phrases, minor formatting inconsistencies, or inclusion of non-procedural information (e.g., billing-related text). Manual edits were restricted to removing such non-essential fragments; the medical procedure descriptions themselves were never deleted, modified, or enriched. Spot-checking was carried out by two authors who are experienced clinical coding specialists working in the hospital coding department and familiar with both CHOP and OPS (KT and OE). The quality of CHOP translations was validated by using the OHDSI Usagi mapping workflow tool⁴⁰. The English translation of the German version of CHOP was uploaded into the Usagi tool. First, the mapping of the translated terminal CHOP codes (10,860 in total) to corresponding vocabulary concepts (SNOMED CT) was done automatically. Subsequently, manual validation of the CHOP to SNOMED CT mapping was conducted for the representative subset of terms and the quality of translation was assessed by using OHDSI mapping equivalence framework.

Embedding generation

Following the translation and cleaning of the CHOP codes, embeddings were generated to encapsulate the semantic relationships and nuanced meanings embedded within the data. For this purpose, the mxbai-embed-large-v1 model was accessed via the ollama interface. This model was chosen for its capability to produce high-dimensional vector representations while maintaining a small memory footprint.

The preparation of data for embedding involved encoding each CHOP entry into a structured format. Each entry was formatted into a single line, with key fields concatenated and separated by a vertical bar (“|”). Included in these fields were the unique identifier for each CHOP code (zcode), the code’s title in its original language (German, French, or Italian), the translated title in English, and a translated description of the CHOP procedure. A similar format was used for OPS. Following the developer recommendations for the mxbai-embed-large-v1 model⁴¹, each query string was prefixed with the instruction “Represent this sentence for searching relevant passages:” to align with the model’s training setup for retrieval tasks. The resulting set of vectors, referred to as the “embedding space,” was produced from this detailed preparation. This space now serves as a comprehensive semantic landscape, enabling advanced searches and analyses based on the contextual relationships defined by the CHOP codes.

Expert validation of CHOP-to-OPS mappings

MAP-CARE was evaluated for its ability to map Swiss CHOP codes to German OPS codes across a sample of 494 terms that included a diverse range of common and rare surgical procedures across various medical specialties, methods, used material and anatomic locations. The mapping follows the strategy of directly query a term in one language within the shared multilingual embedding space and retrieve nearest neighbours whose original labels are written in another language, as opposed to comparing text directly (Fig. 2). The two expert evaluators (KT and OE) are physicians with ongoing roles as terminology manager and expert coders in our institution. Two manual expert evaluations were performed: Evaluation 1 (conducted by KT and OE, Supplementary Table 2) focused on the fidelity of German CHOP texts relative to their English translations, specifically whether the meaning of the original text was preserved. Evaluation 1 is performed exclusively on these cleaned translations, which represent the actual curated resource used by MAP-CARE, rather than on the raw LLM responses. In order to assess the interrater reliability, the resulting values of both raters were compared. Evaluation 2 (conducted by KT, Supplementary Table 3) verified whether the translated CHOP codes semantically matched their translated OPS counterparts, using the OHDSI Usagi tool for SNOMED CT–based guidance, then manually reviewing correctness and procedural specificity of each suggested mapping for similarity by medical coding experts (KT and OE). Because SNOMED CT does not fully cover national procedure classifications and Usagi’s automatic suggestions are imperfect, we use SNOMED-based alignment only as a pragmatic proxy for mapping quality. As SNOMED CT only partially aligns with CHOP and OPS, the evaluation assessed the semantic similarity of the resulting SNOMED concept pairs rather than the accuracy of the underlying CHOP–OPS mappings themselves.

Fig. 2.

Cross-system mapping workflow. For mapping terms between two classification systems (Classification A and Classification B) using MAP-CARE, terms from Classification A are represented as embeddings in a high-dimensional vector space. A similarity search is then performed against the embedding representations in Classification B, resulting in the identification of the mapped term from Classification B that is semantically closest to the input term from Classification A.

For both evaluations (1: German → English; 2: SNOMED CT of CHOP → SNOMED CT of OPS), each match was rated according to the OHDSI mapping equivalence framework:

• Equal: Original meaning is fully preserved.

• Equivalent: Slight inaccuracies, but essential meaning intact.

• Wider: Mapped term is a broader parent concept.

• Narrower: Mapped term is a more specific child concept.

• Inexact: Partial overlap, with some meaning lost.

• Unmatched: No suitable mapping was identified.

• Unreviewed: Cases not validated by both reviewers.

In parallel, Evaluation 3 (Supplementary Table 4) was conducted using an LLM-as-a-judge strategy using the following rubric:

• Exact Match: The retrieved term perfectly corresponds with the intended term in both meaning and context.

• Near Match: The retrieved term is closely related but exhibits slight variations in specificity, context, or comprehensiveness.

• Partial Match: The retrieved term shares certain semantic features with the target term but lacks essential details necessary for an accurate match.

• Mismatch: The retrieved term has no semantic relation to the target term.

Baseline retrieval models and standardized metrics

To contextualize MAP-CARE retrieval performance, we selected the terms evaluated as “equal” from Evaluation 3 as input for three CHOP to OPS retrieval tasks: DE2DE (German CHOP query vs. German OPS labels), EN2EN (English CHOP translation vs. English OPS translation), and EN + DESC2EN (English CHOP translation concatenated with its English description vs. English OPS translation). We compared basic lexical methods (naïve keyboard search, TF-IDF, and BM25) and three embeddings fine-tuned for medical tasks: SAPBERT_ENG (original SAPBERT based in PubMedBERT³⁰, SAPBERT_MULT (a multilingual UMLS/XLM-R variant⁴², and CODER³¹. We pre-processed the corpus for the lexical methods by lowercasing, removal of non-alphanumeric characters while retaining Latin-1 accents (e.g., äöüß), and whitespace collapsing; tokenization was performed by whitespace splitting after normalization. Results were ranked using cosine similarity. For each task and each method, we computed Acc@1 (top-1 accuracy; correct OPS code is ranked first) and Acc@5 (top-5 accuracy; correct OPS code appears anywhere among the first five retrieved).

Multilingual terminology mapping evaluation

The terminal codes across the CHOP classifications in German, French, and Italian were compiled. Each code, potentially linked to multiple entries due to variations in descriptions and contextual details across languages, was retained for further analysis.

The mapping process followed the strategy depicted in Fig. 2. Each entry from the CHOP classification was queried against its counterparts in the other two languages using ChromaDB. Specifically, for each entry, a query was executed to retrieve the single most relevant result (the top-1 result) from the databases of the other languages. To broaden the scope of the semantic analysis, the query was expanded to also retrieve the top-5 results for each entry. The accuracy of this mapping strategy was assessed by quantifying the frequency with which codes from one language’s CHOP classification matched correctly with the other language.

Results

Machine translation

The quality of the translations of CHOP codes was rigorously assessed using the similarity of terminal codes with mapped SNOMED CT terms as a proxy for semantic equivalence. Through the OHDSI Usagi mapping tool, a total of 10,860 CHOP terminal codes were linked to corresponding SNOMED CT concepts. This linkage was subjected to manual validation for a subset of codes, where each mapping was reviewed to assess similarity.

Cross-system matching

Expert evaluations

The MAP-CARE system’s functionality was tested through cross-system and cross-language matching tasks. First, the MAP-CARE system’s performance was evaluated across a dataset of 494 medical procedure mappings, manually analysing the efficacy and accuracy of semantic matching between the Swiss CHOP and German OPS coding systems (Fig. 3 and Supplementary Table 1). A set of comparative analyses was conducted to determine how consistently the two human expert evaluations (Evaluation 1 for translation and Evaluation 2 for semantic matching) aligned with each other and with a semantic matching using an LLM-as-a-judge approach (Evaluation 3).

Fig. 3.

Mapping evaluation results. (a) Translation evaluation conducted by two human evaluators, who categorized the CHOP to OPS translations into six categories: equal, equivalent, inexact, narrower, wider, and unmatched. The confusion matrix displays the distribution of agreed and disagreed labels across evaluators. (b) Distribution of translated CHOP to OPS mappings according to the OHDSI mapping equivalence framework, with percentages shown on the y-axis and absolute counts displayed above each bar. (c) Evaluation conducted using a large language model (LLM) acting as an expert, with mappings classified into four categories: Exact Match, Near Match, Partial Match, and Mismatch.

Evaluation 1 (translation, 2 raters)

The evaluation of translations (Fig. 3a) was conducted using six categories to capture varying degrees of alignment between the original text and the translated content: equal in all axes (anatomical target, procedure, method, organ-system specificity, approach) or equivalent (e.g. eponymous operations), inexact (deviation in the axes addressed, but from a clinician’s judgement identical), narrower or wider in at least one axis, and unmatched when no axis was aligned. Table 1 provides an example and rating for each category.

Table 1.

Translation evaluation categories with one CHOP example per category.

CHOP code	Name (German)	Name (translated to English)	Translation rating	Comment
Z00.66.42	Selektive Embolisation von Koronarien mit embolisierenden (ablösbaren) Ballons	Selective coronary embolization with embolizing (detachable) balloons	equal	equal in all axes
Z57.21	Vesikostomie	Urinary diversion to the skin	equivalent	eponymous
Z86.7 A.8E	Insellappenplastik, grossflächig, an anderer Lokalisation	Inlay flap surgery, extensive, at another location	inexact	Insellappen translates “island flap” or “island pedicle flap”
Z00.22	Intravaskuläre Bildgebung von intrathorakalen Gefässen	Intravenous imaging of intrathoracic vessels	narrower	“intravenous” narrower than “intravascular (parent concept)”
Z52.11.12	Feinnadelbiopsie am Pankreas, endosonographisch	Fine-needle aspiration biopsy of the pancreas, endoscopic	wider	“endoscopic (parent concept: all endoscopic interventions)” wider than “endosonographisch (endoscopy + sonography)”

The distribution of these categories in the confusion matrix is shown in Fig. 3. The rater percentage agreement was 75.9% and the interrater reliability (Cohen’s Kappa) between the 2 raters was 0.34 while considering the six labels, which corresponds to “fair” agreement under standard interpretations. In our setting, κ is penalised by two structural factors: (i) the OHDSI equivalence framework contains six ordered categories, many of which differ only by degree of specificity rather than by meaning, and (ii) most disagreements occurred between adjacent categories (e.g. equal vs. equivalent, equivalent vs. inexact), rather than between clinically incompatible interpretations. Raw agreement was 75.9%, and no cases showed contradictory assessments across raters. We therefore interpret κ in the context of this ordinal, fine-grained scale, where modest κ values are expected even when the substantive judgements are closely aligned.

The majority of translated codes were classified as equal, with both evaluators agreeing on this category in 339 cases (68.6%). This suggests that a substantial portion of the translations preserved the original clinical meaning without deviation. In 23 cases, one evaluator marked a code as equal while the other classified it as equivalent, indicating slight inaccuracies or minor wording differences that did not compromise the essential meaning. A total of 2 cases were mutually identified as equivalent (0.4%), reinforcing that a small subset of translations contained subtle inaccuracies yet still conveyed the core concept accurately. The inexact category accounted for 27 instances (5.5%) of agreement between evaluators. These cases reflect translations where some degree of semantic overlap existed, but certain details or nuances were lost. Notably, 18 cases initially labelled as “equal” by one evaluator were instead considered “inexact” by the other, suggesting that some translations perceived as precise by one reviewer were seen as incomplete or ambiguous by the other. The presence of codes classified as inexact, narrower, or wider suggests that certain clinical terms may pose particular challenges in translation, requiring improved guidance or clearer criteria for distinguishing between these nuanced categories.

Evaluation 2 (semantic matching)

Evaluation 2 assessed the semantic accuracy of mappings, evaluating whether CHOP and OPS codes maintained conceptual alignment despite the absence of direct linguistic equivalence (Fig. 3b). The evaluation was conducted using the OHDSI Usagi tool, followed by a manual review to ensure correctness and procedural specificity.

Across the mappings, the system achieved 57 of 494 SNOMED CT concept pairs (11.5%) were rated as equal (identical SNOMED CT concept identifier) confirming that the procedures described in CHOP and OPS were fundamentally the same in both meaning and medical intent and 20 of 494 (4%) as semantically equivalent (different SNOMED CT concept identifier) indicating minor semantic variation that did not impact clinical interpretation. These mappings typically involved synonyms, slight phrasing differences, or minor variations in procedural description without altering the core medical concept. A total of 221 cases (44.7%) were classified as inexact, meaning that the mapped procedures shared some commonalities but differed in key details, such as anatomical specificity or procedural approach. Further distinctions were observed in 132 cases (26.7%) categorized as wider, where the CHOP term encompassed a broader procedural scope than the OPS counterpart. Conversely, 64 cases (13%) were narrower, meaning that the CHOP term was more restrictive than the corresponding OPS term, limiting the procedural scope. Notably, no mappings were categorized as unmatched.

Evaluation 3 (LLM-based semantic matching)

While evaluating the matching with an LLM (Fig. 3c), the system achieved 127 exact matches, representing 25.7% of the evaluations, where procedures were correctly aligned with high fidelity in both procedural specificity and anatomical locations. For example, “coronary angioplasty with antibody-coated balloons” was precisely matched, linking CHOP code Z00.66.22 to OPS code 8-83b.b1. Near matches, more frequent with 144 instances (29.1%), were correct in alignment but lacked critical details; for instance, “removal of an intracranial implant” (CHOP Z01.39.50) was matched to “removal of a neuroprosthesis” (OPS 5–029.b). While the core procedures align, their specificities differ. Partial matches were the most common, observed in 176 cases (35.6%), and indicated correct categorization but missing specific procedural or anatomical details. A notable example includes the difference in invasiveness between “other craniotomy for evacuation of an epidural hematoma” (CHOP Z01.25.11) and “therapeutic percutaneous puncture of an epidural hematoma” (OPS 8-159.4); the former is an open surgery, while the latter is a minimally invasive procedure. Mismatches occurred in 47 instances (9.5%), showing fundamental errors, such as confusing “instillation of a uterine tube” (CHOP Z66.8) with “foetal implantation of a pacemaker” (OPS 5-755.8). A chi-square test of goodness-of-fit was conducted to assess how well the observed distribution of these categories fits an expected uniform distribution across the four outcomes. The chi-square statistic was calculated at 73.21 (), suggesting that the observed frequencies of match categories are not evenly distributed and thus indicating that the system performs differently across these categories.

The system’s performance varied significantly across different hierarchical levels (Fig. 4):

Fig. 4.

Mapping accuracy across hierarchical levels of CHOP and OPS codes. The accuracy outcomes are categorized into exact match, near match, partial match, and mismatch at four hierarchy levels.

• Level 3: Displayed an equal distribution of exact and near matches (41.67% each) but also presented a notable rate of mismatches (8.33%), pointing to difficulties at more abstract coding levels.

• Level 4: Showed less success in achieving exact matches (21.43%) and more frequent partial matches (35.71%), suggesting challenges in maintaining specificity amidst generalization.

• Level 5: achieved 50% exact and near matches, underscoring the system’s strength in detailed procedural matching.

• Level 6: Exhibited a balanced distribution across all categories, with exact matches at 26.36% and partial matches at 36.68%, though mismatches persisted at 9.51%.

Baseline comparison on the “Exact Match” subset from evaluation 3

We benchmarked lexical baselines (naïve keyword search, TF-IDF, BM25) and biomedical embedding encoders (two SAPBERT variants, CODER) against MAP-CARE’s embedding model (mxbai-embed-large-v1) (Table 2). We restricted this comparison to the N = 127 mapped codes labelled “Exact Match” by the LLM evaluation rubric (Supplementary Table 4), and evaluated retrieval in three scenarios (DE2DE, EN2EN, EN + DESC2EN) using Acc@1 and Acc@5. Lexical baselines performed weakly overall, with keyword search failing in all but the simplest settings and failing entirely when descriptions were added. In the German-only condition (DE2DE), BM25 remained competitive at rank 1 (Acc@1 = 0.583), while SAPBERT_MULT achieved the highest top-5 accuracy (0.819), indicating that multilingual biomedical encoders and keyword matching both retain utility for structured German procedure names. In contrast, in the English-only condition (EN2EN), MAP-CARE (mxbai-embed-large-v1) achieved ceiling performance (Acc@1/Acc@5 = 1.000), substantially outperforming all baselines. When English descriptions were appended (EN + DESC2EN), mxbai-embed-large-v1 remained the strongest method (Acc@1 = 0.740; Acc@5 = 0.961), whereas lexical and general biomedical encoders degraded more markedly. Overall, these results show that semantic embedding–based retrieval becomes dominant once translation and contextual information are introduced, while purely lexical methods are brittle to increased linguistic variability.

Table 2.

Retrieval benchmark on evaluation 3 “Exact Match” subset (n = 127). DE2DE: German CHOP query vs. German OPS labels; EN2EN: english CHOP translation vs. english OPS translation; EN + DESC2EN: english CHOP translation concatenated with its english description vs. english OPS translation. Acc@1: fraction of cases where top result was the correct code; Acc@5: fraction of cases where correct code was on the top five results; mxbai: mxbai-embed-large-v1.

Task	Method	Acc@5	Acc@1
DE2DE	SAPBERT_MULT	0.819	0.606
	mxbai	0.740	0.528
	BM25	0.724	0.583
	TFIDF	0.717	0.535
	CODER	0.709	0.543
	SAPBERT_ENG	0.630	0.394
	Keyword search	0.126	0.118
EN2EN	mxbai	1.000	1.000
	SAPBERT_ENG	0.906	0.677
	SAPBERT_MULT	0.874	0.740
	BM25	0.803	0.575
	TFIDF	0.803	0.575
	CODER	0.740	0.543
	Keyword search	0.102	0.094
EN + DESC2EN	mxbai	0.961	0.740
	SAPBERT_ENG	0.882	0.614
	SAPBERT_MULT	0.764	0.559
	TFIDF	0.748	0.457
	BM25	0.693	0.425
	CODER	0.567	0.362
	Keyword search	0.000	0.000

Cross-language matching

In order to test the cross-language capabilities of MAP-CARE, we developed an automated strategy to assess the mapping accuracy of CHOP codes within the MAP-CARE embedding space across German, French, and Italian versions. Each term is mapped between language pairs to verify if the original CHOP code corresponds to the mapped code in another language’s vector store (Fig. 2).

The evaluation of the mapping accuracy was conducted in two stages: top-1 match (Acc@1), and top-5 matches (Acc@5), visualized in Fig. 5. This measure allows for a broader assessment of the system’s capability to identify relevant terms, even if they are not the highest ranked, which is crucial for applications where multiple similar options may be clinically relevant. The performance at the top-1 result level demonstrates that the system achieves a moderate level of accuracy, with more than 59% accuracy across all language pairs.

Fig. 5.

Cross-language mapping performance for CHOP codes. The performance of MAP-CARE for mapping CHOP codes across German, French, and Italian was evaluated. The left panel shows Acc@1, with mapping success ranging from 0.59 (German to Italian) to 0.75 (French to Italian). The right panel shows improved accuracy when considering the top-5 results (Acc@5), exceeding 0.70 for all pairs and peaking at 0.90 (French to German).

When the evaluation criteria were expanded to include the top five matches, the system’s performance notably improved, achieving over 70% accuracy across all language pairs. Upon manual review of the mismatches, it became apparent that many were synonymous entries within the CHOP classification. This broader matching criterion enables the system to more effectively capture semantic relationships between terms, even when direct matches are not the highest ranked.

Discussion

The MAP-CARE workflow facilitates the integration and accessibility of multilingual medical procedural data through the application of large language models (LLMs) and vector embeddings. In contrast to established systems such as UMLS, which align numerous terminologies, MAP-CARE introduces a dynamic approach to (i) automatically align previously unmapped terminologies, thereby reducing the costs associated with manual mapping efforts, and (ii) assist users in efficiently navigating extensive medical classifications by ensuring that query results include all semantically relevant codes, eliminating the need for exhaustive exploration of the entire classification. This is particularly critical for elemental classifications, typically available in CSV or PDF formats, which are often searchable only by explicit terms and lack synonym recognition. A straightforward baseline for code retrieval is exact or approximate string matching over the original labels. While such methods succeed for simple, unambiguous terms, they frequently fail in the presence of synonyms, abbreviations, reordered modifiers or differences in granularity between classifications. In our coding department, coders therefore combine keyword search with manual navigation of hierarchical lists. MAP-CARE is designed to augment this workflow by ranking semantically related entries across languages and classifications, rather than to replace free-text concept-normalisation tools such as MetaMap⁴³ or cTAKES¹¹, which are optimised for English clinical narratives and rely on different underlying vocabularies. MAP-CARE’s capability to address linguistic complexities, coupled with the granularity and variability of non-English and less standardized terminologies, positions it as a valuable tool where conventional keyword-based search methods are inadequate. Although a brief expert quality-control step is required when onboarding a new terminology, this is a one-time curation step; once the translated resource is prepared, all subsequent embedding, retrieval, and cross-terminology alignment operations in MAP-CARE run fully automatically.

MAP-CARE was evaluated on three complementary tasks: translation fidelity (Evaluation 1), SNOMED-based cross-system alignment (Evaluation 2), and direct CHOP–OPS semantic similarity (Evaluation 3) and are not intended to be combined into a single performance metric. Rather, we interpret their results jointly, acknowledging discrepancies and using them to triangulate the strengths and weaknesses of MAP-CARE under different perspectives. MAP-CARE effectively circumvents linguistic barriers inherent in medical terminology by enabling precise mapping across four languages and two different classifications. The system demonstrates only 9.5% mismatches at the most detailed hierarchical levels, is therefore robust and particularly of interest since these final levels precisely describe a procedure, often including an exact anatomical region, approach, method and material. These results are promising because the system helps leverage the most granular and, therefore, the most work-intensive mapping level. The cross-language evaluation within CHOP (German/French/Italian) benefits from the fact that all language versions share the same underlying classification structure. As a result, the reported accuracy likely overestimates performance for mappings between truly independent terminologies. We therefore interpret these results as a best-case estimate of multilingual behaviour and complement them with the more challenging CHOP–OPS experiments, where no official mapping exists and structural differences between the systems are substantial. Such a mapping increases the potential for valuable knowledge exchange and collaboration between countries. This lack of mapping arises from fundamental structural differences between the two systems, even though both are available in German, and the manual mapping process is highly burdensome due to their complexity.

The level of aggregation can be achieved through explicit customisation and system training and by using cross-mapping to the refined hierarchical terminologies such as SNOMED CT¹⁸. However, using the OHDSI Usagi tool does not provide a sufficient quality of the SNOMED mappings when executed automatically which led to only 273 (55.3%) concept pairs in total rated as equal, equivalent, narrower or wider. The evaluation that uses SNOMED CT as a reference should be interpreted as a proxy rather than as a strict gold standard. SNOMED CT provides incomplete coverage of CHOP and OPS, and the limited accuracy of automatic OHDSI Usagi tool mappings (55.3% acceptable) constrains their use for formal validation. Nonetheless, this highlights the difficulty of aligning national procedure classifications and motivates the need for more flexible embedding-based approaches. For this reason, SNOMED-based evaluation is complemented with direct expert assessments and an LLM-based judging procedure.

Next steps in the project include creating embeddings of established international medical lexical database, such as SNOMED CT, ICD-10, OPS, ICHI and Classification of Operations and Procedures (OPCS-4), National Health System NHS⁴⁴. Through expanding the hierarchical nature of SNOMED, a knowledge graph representation enables more effective semantic reasoning, enhanced data interoperability, and improved clinical decision support by capturing complex relationships and contextual nuances among medical concepts⁴⁵. Recent concept-embedding approaches such as SAPBERT, CODER and related multilingual normalisation methods provide powerful representations for standardised vocabularies such as UMLS and SNOMED CT. We benchmarked MAP-CARE against both embedding-based baselines and simpler lexical retrieval methods, revealing systematic differences in how models respond to linguistic variability. All baselines were evaluated without task-specific fine-tuning and under identical preprocessing, ranking, and metric definitions, ensuring a fair comparison focused on retrieval behaviour rather than training advantage. Among embedding baselines, SAPBERT and CODER showed competitive performance in several scenarios, with the multilingual SAPBERT variant performing best in German-to-German retrieval, where rigid OPS/CHOP phrasing and compound-heavy morphology can still favour surface-level or multilingual matching. In contrast, MAP-CARE embeddings remained more robust once linguistic variability was increased through translation or the addition of contextual descriptions (EN + DESC2EN), indicating that the model benefits from richer procedural context rather than isolated short labels. The ceiling performance observed in the English-only Exact Match condition further suggests that translation combined with retrieval-oriented embeddings can fully resolve alignment when query and target descriptions are structurally equivalent, and that residual errors in other settings are driven primarily by linguistic noise rather than embedding capacity. These results should be interpreted in light of the evaluation design: the Exact Match subset was derived from mappings generated within the MAP-CARE workflow, which likely favours embedding representations closer to those used in MAP-CARE. Consequently, this benchmark is intended as a controlled diagnostic experiment that isolates retrieval behaviour under varying linguistic conditions, rather than as a comprehensive or unbiased estimate of overall CHOP–OPS mapping difficulty.

While evaluating MAP-CARE on cross-language tasks, it showed high efficacy in French and Italian. Contrastingly, German presented unique challenges that underscore the complexities of semantic interoperability. The observed lower accuracy rates in mappings from German to French and Italian (61% and 59%, respectively) suggest an underlying structural linguistic divergence, for example a typical characteristic to condense complex ideas into single words. These findings might reflect the intricate nature of German medical nomenclature and CHOP, which often lacks direct equivalents in the more Latinate vocabularies of French and Italian. This divergence not only impacts the system’s efficiency but also highlights the broader challenges of standardizing medical terminology across languages with disparate etymological roots. Enhancements to MAP-CARE’s performance were observed when the criteria were expanded to include the top five matches, where accuracy rates exceeded 70% across all language pairs. This improvement demonstrates the system’s capability to capture broader semantic relationships, emphasizing the potential of flexible matching criteria to accommodate linguistic variations.

However, challenges remain, particularly in integrating specific operational markers like “**” in OPS, which are critical in distinguishing between procedural nuances such as open versus laparoscopic approaches. Another challenge identified is when the nomenclature relies on mixed-term coding where multiple concepts are embedded within a single term, which is the case for CHOP. This lack of stringent compositionality which reduces semantic precision and impedes data interoperability across systems. For instance, procedural codes that fail to specify methods or surgical techniques obscure essential details, as evidenced by mismatches during CHOP-to-OPS mapping. Such ambiguity hinders cross-system integration and restricts the ability to conduct detailed procedural analyses. One possible solution to this issue can be segmenting procedures in granular, composable elements (e.g., device type, surgical approach, and anatomical site). Integrating this methodology into MAP-CARE would enhance its ability to represent and map medical procedures across classifications and languages semantically by moving beyond mixed-term coding and addressing marker code integration. An important limitation of the present work is the absence of evaluation on established concept normalisation benchmarks. Existing datasets largely focus on English free text and UMLS-style vocabularies and do not directly cover national procedure classifications in German, French and Italian. Extending MAP-CARE to such benchmarks and adapting benchmark suites to better represent non-English procedure codes, is an important direction for future work. Because no official CHOP–OPS mapping exists, our expert ratings should be viewed as a pragmatic operational gold standard rather than an absolute truth. The raters are experienced coding specialists, but their judgements inevitably involve some subjectivity, particularly when distinguishing between neighbouring equivalence categories. A formal, consensus-based reference mapping between CHOP and OPS does not yet exist and would itself constitute a substantial project. Tools such as MAP-CARE may eventually help accelerate the creation of such resources.

Finally, prospective applications of MAP-CARE include its use in feature engineering for machine learning. For example, semantic cluster assignments could be utilized as a new feature to implicitly model the invasiveness of procedures, enriching predictive analytics and downstream decision-making. Furthermore, MAP-CARE’s architecture allows for seamless extensions to facilitate natural language searches within the embedding space. Concept embedding can be used to harmonize heterogeneous datasets and mitigate local learning bias in federated learning⁴⁶. By converting queries into their embedded vector representations, the system can efficiently identify and retrieve the most relevant terms, broadening its utility in diverse clinical and research settings.

Conclusion

MAP-CARE represents a transformative approach in healthcare informatics, successfully leveraging the power of large language models and advanced embedding techniques to address the complexities of multilingual medical terminology. Its ability to accurately align and interpret medical procedures across languages supports the accessibility and utility of medical procedural information across linguistic and healthcare domains. Future work will aim to expand the application of MAP-CARE to include additional medical terminologies and refine its analytical capabilities to extract even more detailed insights from complex medical data sets, thereby increasing accuracy. Enhancing the system’s ability to integrate with various medical terminologies, frameworks, and electronic health record systems could further amplify its impact. This continued development will optimise the system’s architecture and scalability to support a broader range of clinical and administrative applications.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

Conceptualization: HGR, OE, KT, GB; Methodology: HGR, OE, KT; Formal analysis and investigation: HGR, OE, KT, BP; Writing - original draft preparation: HGR; Writing - review and editing: HGR, OE, KT, GB, CGB; Funding acquisition: OE, GB.

Funding

The research leading to these results received funding from Swiss Personalized Health Network SPHN under the Demonstrator Project INFRA: INFection RAdar.

Data availability

No patient data was used in this project. All materials related to MAP-CARE **,** including the code, prepared catalogues, and accompanying documentation are available in the Guillen H HugoGuillen/MAPCARE repository on zenodo and GitHub.

Declarations

Competing interests

The authors declare no competing interests.