Relation extraction using large language models: a case study on acupuncture point locations

Abstract

Objective

In acupuncture therapy, the accurate location of acupoints is essential for its effectiveness. The advanced language understanding capabilities of large language models (LLMs) like Generative Pre-trained Transformers (GPTs) and Llama present a significant opportunity for extracting relations related to acupoint locations from textual knowledge sources. This study aims to explore the performance of LLMs in extracting acupoint-related location relations and assess the impact of fine-tuning on GPT’s performance.

Materials and Methods

We utilized the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard) as our corpus, which consists of descriptions of 361 acupoints. Five types of relations (“direction_of”, “distance_of”, “part_of”, “near_acupoint”, and “located_near”) (n = 3174) between acupoints were annotated. Four models were compared: pre-trained GPT-3.5, fine-tuned GPT-3.5, pre-trained GPT-4, as well as pretrained Llama 3. Performance metrics included micro-average exact match precision, recall, and F1 scores.

Results

Our results demonstrate that fine-tuned GPT-3.5 consistently outperformed other models in F1 scores across all relation types. Overall, it achieved the highest micro-average F1 score of 0.92.

Discussion

The superior performance of the fine-tuned GPT-3.5 model, as shown by its F1 scores, underscores the importance of domain-specific fine-tuning in enhancing relation extraction capabilities for acupuncture-related tasks. In light of the findings from this study, it offers valuable insights into leveraging LLMs for developing clinical decision support and creating educational modules in acupuncture.

Conclusion

This study underscores the effectiveness of LLMs like GPT and Llama in extracting relations related to acupoint locations, with implications for accurately modeling acupuncture knowledge and promoting standard implementation in acupuncture training and practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in natural language processing.

Introduction

Acupuncture, originating from ancient Chinese medicine, has a history dating back thousands of years.^1,2 It was introduced to the United States in the 1970s, gaining gradual acceptance within the medical community for its therapeutic effects.^3,4 According to the World Health Organization 2019 report, acupuncture is the most widely used traditional and complementary medicine, practiced in 113 out of 120 countries.⁵

Acupuncture involves the insertion of thin needles into specific points on the body known as acupoints, which are believed to correspond to channels that conduct Qi, or vital energy.^6,7 Clinical studies have shown acupuncture to be effective in treating a variety of conditions, including chronic pain, migraines, and osteoarthritis.^8–13 It is also used to alleviate symptoms associated with chemotherapy, such as nausea and vomiting.¹⁴ The World Health Organization recognizes acupuncture as a viable treatment option for a range of disorders, including respiratory, digestive, and neurological conditions.^15–17

The rationale behind acupuncture lies in its emphasis on the precise location of acupoints.^18,19 These points are located along meridians, or energy pathways, where Qi flows.²⁰ In practice, the accurate location of acupoints is essential to ensure the proper flow of Qi, and acupuncturists need to understand the anatomical structures of the body to accurately locate acupuncture points and ensure the safety and effectiveness of acupuncture procedures.²¹ Moreover, acupuncture localization also adheres to the concept of “body cun” (同身寸), where acupuncture points are determined based on certain lengths on the patient’s body surface. Acupuncture therapy does not rely on fixed physical dimensions but adjusts according to the patient’s body characteristics. While some acupoints are easily identifiable, others may be more challenging to locate, especially for beginners.²² Acupuncturists undergo extensive training to master the palpation techniques and anatomical knowledge necessary for accurate point location. Several studies suggested that the accuracy and precision vary when locating acupuncture points using different methods.²³ Given the variance in treatment outcomes among these point location techniques, it is essential to leverage informatics technologies to structuralize and computerize the most critical point location knowledge for assisting acupuncture training and practice.

While there are numerous studies on relation extraction (RE) from biomedical literature, none have yet delved into RE from textual acupuncture knowledge. He et al used the CHEMPROT dataset from BioCreative VI and the drug-drug interaction (DDI) dataset to develop a specialized prompt tuning model for biomedical RE, showing its effectiveness in few-shot learning.²⁴ El-Allaly et al propose ADERel, an attentive joint model with a transformer-based weighted GCN for extracting adverse drug event (ADE) relations.²⁵ ADERel formulates the ADE RE task as an N-level sequence labeling, leveraging contextual and structural information.²⁵ The performance of traditional machine learning and deep learning models for RE can be limited by several factors. One major challenge is accurately capturing complex relationships between entities, especially in context-dependent or inter-sentence scenarios, leading to reduced performance in tasks requiring high precision and recall.²⁶ Additionally, model performance can suffer from inadequate or noisy training data, affecting generalization.^26,27 Moreover, the reliance on large pre-trained language models such as Bidirectional Encoder Representations from Transformers (BERT) or Long Short-Term Memory (LSTM), can also pose challenges, as these models may not be well-suited for all RE tasks and may require extensive fine-tuning and adaptation.²⁸

Large language models (LLMs) have revolutionized natural language processing (NLP) by leveraging deep learning techniques to understand and generate human-like text.^29,30 Among these models, Llama, which includes several iterations like Llama 2 and the latest Llama 3 has made substantial contributions to NLP by offering robust performance across various tasks. The Generative Pre-trained Transformer (GPT) stands out as another prominent example. Developed by OpenAI, GPT has gained widespread attention for its ability to perform a variety of language tasks, including text completion, translation, and summarization.^31,32 What sets GPT apart is its architecture, which is based on transformer neural networks.³³ This architecture allows GPT to effectively capture long-range dependencies in text, enabling it to generate coherent and contextually relevant responses.^34,35 Li et al evaluated multiple pre-trained and fine-tuned LLMs on their ability to extract adverse events (AEs) using notes from the Vaccine Adverse Event Reporting System (VAERS), which has shown LLMs’ capability in named entity recognition (NER) tasks.³⁶ However, few studies have explored the use of LLMs for extracting relations between entities, especially for a specific medical domain. To investigate the ability of LLMs in modeling acupuncture knowledge, we launched this study as an in-depth extension of our prior work in acupuncture knowledge extraction, which focused on extracting acupuncture point location entities based on the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard).³⁷

In this study, our objective is to leverage the power of LLMs for RE in the context of acupuncture points and human anatomy. We also aim to compare the performance between pre-trained and fine-tuned LLMs. This comparative analysis will provide insights into the effectiveness of using GPT and Llama for RE tasks in the domain of acupuncture, and highlight the potential of LLMs to advance modeling of acupuncture knowledge.

Methods

Figure 1 shows the workflow of our study, which aimed to extract relations between acupuncture points using various models. We utilized data from the WHO Standard as the data source. We manually annotated and studied 5 types of relations. Four models were studied: pre-trained GPT-3.5-turbo, fine-tuned GPT-3.5-turbo, pre-trained GPT-4, and pre-trained Llama 3.

Figure 1.

Overview of the framework.

Data source

We utilized WHO Standard as the data source, a key resource released in 2008 to standardize acupuncture point locations.^37,38 It elaborates on the locations of 361 acupoints across 14 systems and specifies the methodology to locate acupoints of the human body.^37,38 This standard aims to improve scientific communication and evidence-based practice in acupuncture, enhancing the accessibility and value of Eastern medicine in healthcare.^37,38

Annotation

In this study, we used the annotated entities from the research carried out by Li et al.³⁷ There are 5 types of entities associated with this task: Acupoint, Anatomy, Direction, Distance, and Subpart. To precisely locate acupoints, we investigated 5 relation types: direction_of indicates the Direction of one relative Acupoint or Anatomy to the Acupoint of interest, distance_of denotes the Distance to the Acupoint/Anatomy, part_of represents the Subpart of the Anatomy, near_acupoint refers to adjacency to the relative Acupoint, and located_near signifies proximity of the Anatomy to the Acupoint of interest (shown in Table 1). The total annotated relation pairs are 3174. All annotations were performed using CLAMP (version 1.6.6).³⁹

Table 1.

Types of relations annotated for SP8 (Diji) in WHO Standard.

Relation type	Definition	Example
direction_of	the Direction of one relative Acupoint or Anatomy to the Acupoint of interest
distance_of	the Distance to the Acupoint/Anatomy
part_of	the Subpart of the Anatomy
near_acupoint	adjacency to the relative Acupoint
located_near	proximity of the Anatomy to the Acupoint of interest

In our annotation examples, acupoints were described using detailed spatial relationships, such as SP8 (Diji): “On the tibial aspect of the leg, posterior to the medial border of the tibia, 3 B-cun inferior to SP9.” This narrative includes various key relationships, as illustrated in Table 1. For instance, the terms “inferior to” and “SP9” describe the acupoint’s position below SP9, highlighting the “direction_of” relationship. The distance between “3 B-cun” and “SP9” is expressed as a “distance_of” relation, indicating that SP8 is located 3 B-cun inferior to SP9. Furthermore, relationships such as “tibial aspect of” and “leg”, and “medial border of” and “tibia,” provide additional context by specifying which part of the body the acupoint is located on and its proximity to specific anatomical landmarks, representing “part_of” relationships. Additionally, the “near_acupoint” relation is demonstrated by the proximity of “SP8” and “SP9”, indicating that these acupoints are close to each other. Lastly, the “located_near” relation is exemplified by “SP8” being near both the “leg” and the “tibia”, highlighting its close proximity to these anatomical structures.

Two domain experts were invited to perform the annotation task. To evaluate the inter-annotator agreement, we employed Cohen’s kappa statistic for each relation type. The kappa scores provided a quantitative measure of consistency between the annotators’ evaluations. In cases where conflicts arose, the annotators engaged in discussions and negotiations to resolve discrepancies, ultimately reaching a consensus. This process ensured the construction of a reliable and consistent gold standard for our study.

Model and experiment setup

In this study, we explored the effectiveness of pre-trained GPT-3.5, fine-tuned GPT-3.5, pre-trained GPT-4, and Llama 3 models in extracting acupoint-based relations. GPTs, including models like GPT-3.5 and GPT-4, are Transformer-based models pre-trained on vast text corpora.⁴⁰ They advance in architecture and parameter size, generating coherent, contextually relevant text for diverse NLP tasks like NER, question answering, translation, and summarization.^40–42 Llama 3, developed by Meta, enhances natural language understanding and generation, improving upon Llama 2 in architecture, efficiency, and performance. It supports 8-70 billion parameters and processes sequences up to 8192 tokens, handling complex tasks effectively.^43,44

In the experiment setup, the dataset consisting of 361 acupoints was randomly divided into training and test sets in an 8:2 ratio. This division ensured that 80% (N = 288) of the acupoints were allocated to the training set for model training, while the remaining 20% (N = 73) were reserved for evaluating the model’s performance on unseen data in the test set.

We further analyzed the dataset to examine the distribution of each relation type within the training and test sets. This analysis provided insights into the balance of relation types and ensured that the model was trained and evaluated on a diverse set of relations, contributing to its robustness and generalization ability. The counts of each relation type in the training and test sets are summarized in Table 2 below.

Table 2.

Relation statistics in the training and test set.

Relation type	Training	Test	Total
direction_of	694	160	854
distance_of	238	66	304
part_of	467	107	574
near_acupoint	154	49	203
located_near	992	247	1239

GPT

In Table S1, we present the key parameters used for the GPT models in our study. For all GPT models, including the pre-trained GPT-3.5, fine-tuned GPT-3.5 and pre-trained GPT-4, we set the temperature parameter to 0.3 and the maximum number of tokens to 4096. In the case of fine-tuned GPT-3.5, the number of epochs (value = 2), batch size (value = 1), and learning rate multiplier (value = 1) were used during the fine-tuning process. These parameters were chosen to balance the model’s performance and computational efficiency, ensuring effective training and inference for RE tasks.

Llama

In this study, we utilized the Llama 3 model, specifically the Llama 3-8b instruct variant, to perform our experiments. This model configuration includes 8 billion parameters, making it a robust choice for our NLP tasks. We set the maximum sequence length to 8192 tokens, allowing the model to handle extensive and detailed input data effectively. The maximum batch size was set to 6, which optimizes the computational efficiency and ensures that the model processes data in manageable chunks without compromising performance. This setup facilitated our exploration of spatial RE within the context of traditional Chinese medicine (TCM) and acupuncture, leveraging the advanced capabilities of Llama 3 to achieve accurate and contextually relevant results.

Prompt

In this study, we used the pretrained GPT and Llama model for each relation type using specific prompts tailored to extract the corresponding relations. For instance, the prompt for the “direction_of” relation was formatted as follows:

"Textual information is below.\n——————————-\n{txt}\n——————————-\nEntity information is below(Format is:'T(id)\t(entity_type) (start_offset) (end_offset)'\t'(entity_text)'){entities}\nQuery:Now we have to conduct the relation extraction task. We want to extract 'direction_of' first. 'direction_of' refers to the direction of one relative 'Acupoint'or 'Anatomy' to the 'Acupoint'. The starting entity type of 'direction_of' is 'Direction' entity and the end entity of 'direction_of' is the relative 'Acupoint' or 'Anatomy' entity. Entities({entities}) have been annotated. And each entity is formatted as "'Entity sequence number''\t''Entity type'' ''starting character location'' ''end character location''\t''Entity'". To facilitate output extraction, please output the relation directly and format the relation as "'Relation sequence number(Starting from R1, R2,…)''\t''direction_of'' ''Arg1:Entity sequence number'' ''Arg2:Entity sequence number''\n'"from the following text. If no relations, please do not output anything. "

In Figure 2, we provided a training example for the “direction_of” RE using the acupoint BL12 as an illustrative case. This approach allowed us to fine-tune the GPT model for each relation type, enhancing its ability to extract specific relations accurately and efficiently from textual knowledge.

Figure 2.

One example of training set for “direction_of” relation of BL12 (Fengmen).

Fine-tuning and inference for the pre-trained GPT models were conducted on a server equipped with 8 Nvidia A100 GPUs, each with a memory capacity of 80 GB. In contrast, Llama was evaluated on a server with 5 Nvidia V100 GPUs, each providing a memory capacity of 32 GB.

Evaluation

In evaluating the performance of our model, we used our annotations as the gold standard. The micro-average approach calculates the total true positives, false positives, and false negatives between the gold standard and the predicted results across all classes and then computes precision, recall, and F1 score. The formulas for precision, recall, and F1 scores are shown below:

$$\mathrm{Precision}\mathrm{ }=\mathrm{ }\frac{\mathrm{True}\mathrm{ }\mathrm{positive}}{\mathrm{True}\mathrm{ }\mathrm{positive}+\mathrm{False}\mathrm{ }\mathrm{positive}},$$

$$\mathrm{Recall}\mathrm{ }=\mathrm{ }\frac{\mathrm{True}\mathrm{ }\mathrm{positive}}{\mathrm{True}\mathrm{ }\mathrm{positive}+\mathrm{False}\mathrm{ }\mathrm{negative}},$$

$$\mathrm{F}1\mathrm{ }=\mathrm{ }\frac{2\times \mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.$$

Results

Table 3 shows high inter-rater agreement for all relation types, with kappa statistics ranging from 0.87 to 1.00 and an overall value of 0.91.

Table 3.

Inter-rater agreement.

Relation type	Kappa statistics
direction_of	0.91
distance_of	1
part_of	0.87
near_acupoint	0.94
located_near	0.92
overall	0.91

The performance of different models is displayed in Figure 3. Figure 3A shows the precision results for different models across various relation types. The fine-tuned GPT 3.5 model achieved the highest precision of 0.96 for the “direction_of” relation, demonstrating its accuracy in identifying relative acupoint directions. It also performed well for the “located_near” relation with a precision of 0.95. In contrast, the Llama 3 model exhibited lower precision across all relation types. Overall, the fine-tuned GPT 3.5 model had the highest micro-average precision of 0.91.

Figure 3.

Performance on each relation type by different large language models. Note: Scores were averaged after 10 runs. (A) Precision. (B) Recall. (C) F1.

Figure 3B highlights the recall results, showing that the fine-tuned GPT 3.5 model achieved the highest recall for most relation types, particularly excelling in “direction_of” and “located_near” relations with recalls of 0.99 and 0.97, respectively. The pre-trained GPT-4 model had lower recall scores, suggesting difficulty in capturing acupoint-related details. The Llama 3 model had varied recall performance, higher for some relations like “direction_of” and “part_of” but lower for others. The fine-tuned GPT 3.5 model demonstrated the highest micro-average recall of 0.94.

Figure 3C presents the F1 scores, with the fine-tuned GPT 3.5 model achieving the highest scores across most relation types, especially in “direction_of” and “located_near” with F1 scores of 0.97 and 0.96, respectively. The pre-trained GPT-4 model had lower F1 scores, while the Llama 3 model showed varied performance. Overall, the fine-tuned GPT 3.5 model had the highest micro-average F1 score of 0.92, highlighting its effectiveness in RE for acupuncture.

Discussion

The proposed study provides a comprehensive analysis of RE tasks. Our study compared the performance of LLMs in RE tasks related to acupoint locations. We also examined the performance difference between pre-trained LLMs and fine-tuned GPT 3.5 model.

Our results indicate that the fine-tuned GPT-3.5 model outperformed pretrained GPT and Llama models across all evaluated relation types. Specifically, the fine-tuned GPT-3.5 model achieved the highest F1 scores, demonstrating its effectiveness in accurately extracting relations related to acupoints. Furthermore, we observed that pre-trained GPT models, while showing strong performance in general language tasks, did not perform as well as fine-tuned GPT models in RE for acupuncture point locations. This suggests that fine-tuning the GPT model on domain-specific data significantly improves its ability to extract complex body location relations related to acupoints.

Additionally, fine-tuning the GPT model on domain-specific data enables it to learn the nuances and complexities of RE in acupuncture, leading to improved performance compared to models that are not fine-tuned. Fine-tuning allows the model to adapt its weights and parameters to the specific characteristics of the acupuncture RE task, enhancing its ability to extract accurate and contextually relevant relations.

Our study contributes to the field of healthcare informatics, as well as traditional and complementary medicine, in several significant ways. Firstly, our information extraction approach facilitates the automatic discovery of acupuncture knowledge, enriching the existing body of knowledge and potentially uncovering novel insights into acupoint relationships and their therapeutic effects. Accurate acupoint locations are vital for achieving desired therapeutic effects, and our work contributes to a more accurate and reliable foundation for acupuncture practice.^23,45 Secondly, we provide a comparative analysis of the performance differences between LLMs, as well as between pre-trained LLMs and fine-tuned LLMs in the domain of information extraction. This comparative study not only sheds light on the effectiveness of using LLMs like GPT and Llama 3 for RE tasks but also provides insights into the benefits of fine-tuning models for specific domains, such as acupuncture. While it is well-known that fine-tuning generally improves performance, our study quantifies these improvements in the specific context of acupuncture-related information extraction, offering a more comprehensive understanding of how different LLMs perform on domain-specific tasks. This can inform future research and practical applications by highlighting the specific gains achievable through fine-tuning LLMs in specialized fields. Thirdly, our study significantly advances the field of spatial RE by leveraging and fine-tuning LLMs, an approach that has not been explored in the context of TCM before. Unlike conventional anatomical RE, which often relies on established Western medical concepts and structured anatomical hierarchies, TCM involves unique acupoints and meridians with distinct, complex relationships that do not directly align with Western anatomical frameworks, necessitating tailored NLP techniques for accurate understanding.^37,46,47 Our approach, using LLMs fine-tuned for TCM, offers a novel contribution by adapting state-of-the-art NLP techniques to this specialized domain. The application of LLMs to this domain not only demonstrates their versatility but also sets a new benchmark for NLP applications in alternative medicine.

From the clinical application perspective, our study offers important insights into the use of LLMs in clinical settings, particularly for developing acupuncture clinical decision support (CDS) tools. Integrating extracted acupoint location relations into these tools can help acupuncturists with precise acupoint selection and real-time recommendations tailored to patient-specific factors, enhancing treatment precision and effectiveness while providing personalized, context-aware guidance to improve clinical outcomes. Furthermore, the findings from our study can be utilized to develop educational modules and training programs for acupuncturists. These programs can include interactive tools and simulations that demonstrate the relationships between acupoints and anatomical structures. By incorporating these insights into training curricula, precise acupoint locations can improve the accuracy and effectiveness of acupuncture practice. The use of advanced technologies such as virtual reality and augmented reality in these educational tools can provide acupuncturists with immersive learning experiences, enabling them to master complex concepts and techniques more effectively.

One of the key strengths of our study is the utilization of the WHO Standard as our corpus. This widely accepted resource provides a formal and standardized conceptual framework for acupuncture point locations, ensuring the accuracy and reliability of our results. Moreover, our study highlights the potential of LLM and their fine-tuning for RE in traditional and complementary medicine domains, showcasing their adaptability and versatility.

Limitations of this study include the use of a specific dataset with limited size and format that may not fully represent the diversity of textual knowledge related to acupuncture, and the challenges of manual annotation prone to error and bias. Additionally, traditional cross-validation techniques to the fine-tuning of GPT models is not supported. Instead, we addressed this limitation by conducting multiple runs and averaging the scores. This approach aims to provide a more reliable estimate of model performance while acknowledging the inherent limitations of fine-tuning GPT models. Future work would focus on domain adaptation and acupuncture-related knowledge integration to improve model performance in textual knowledge.

Error analysis

We conducted an error analysis of the fine-tuned GPT-3.5 model’s performance in relation identification, as shown in Table 4. The analysis revealed varying error rates across different relation types, with notable findings in the “near_acupoint” relation. This relation type exhibited a high false positive rate of 32.5% and a relatively high false negative rate of 26.53%. These results suggest that the model struggles with accurately identifying acupoints that are close to each other, possibly due to the complexity of spatial relationships and the nuanced context required for such distinctions.

Table 4.

Error analysis for fine-tuned GPT-3.5.

	False positive (out of machine annotated entities)	False negative (out of human annotated entities)	Incorrect relation type (out of machine annotated entities)
direction_of	11/168, 6.55%	1/160, 0.63%	0/168, 0%
distance_of	13/75, 17.33%	1/66, 1.52%	0/75, 0%
part_of	16/119, 13.45%	1/107, 0.93%	0/119, 0%
near_acupoint	13/40, 32.5%	13/49, 26.53%	0/40, 0%
located_near	13/252, 5.16%	4/247,1.62%	1/252, 0.4%

Additionally, all classes showed high false positive rates compared to the other 2 error types—false negative and incorrect relation type. Despite these challenges, the model demonstrated a low error rate in assigning incorrect entity types, indicating a strong understanding of entity types in the context of acupoint descriptions. These findings highlight the model’s strengths and weaknesses in relation identification, pointing to areas for improvement, particularly in spatial relation understanding.

Based on the detailed error analysis of the fine-tuned GPT-3.5 model in RE for acupoint descriptions, several key observations and patterns were uncovered. The model occasionally fails to accurately identify adjacent acupoints, as seen in the description of ST38 (shown in Figure 4) where it misses the “near_acupoint” relationship between ST38 and ST35 or ST41. This indicates a difficulty in understanding complex spatial relationships in complex descriptions.

Figure 4.

Error analysis for ST38 (Tiaokou).

The model also struggles with identifying relationships that span across sentences, as seen in the description of ST31 (“St31 is located at the deepest point in the depression inferior to the apex of this triangle. Note 2: ST31 is located at the intersection of the line connecting the lateral end of the base of the patella with the anterior superior iliac spine, and the horizontal line of the inferior border of the pubic symphysis.”) where it incorrectly identifies a “direction_of” relationship between “inferior to” and “anterior superior iliac spine.” This error may stem from the model’s inability to effectively track and integrate information across multiple sentences. It may lack the contextual understanding which causes it to recognize that “inferior to” relates to “the anterior superior iliac spine.” Additionally, in the description of LR10 (“LR10: On the medial aspect of the thigh, 3 B-cun distal to St30, over the artery.”), the model incorrectly identifies a “part_of” relationship between “medial aspect of” and “thigh.” This error suggests that the model may struggle with accurately identifying anatomical relationships between acupoints and body parts, possibly due to a lack of profound understanding of anatomical structures and their spatial relationships.

While less frequent than false positives, false negatives also occur, as seen in the description of BL38 (“BL38: On the posterior aspect of the knee, just medial to the biceps femoris tendon, 1 B-cun proximal to the popliteal crease. Note: With the knee in slight flexion, BL38 is located medial to the biceps femoris tendon, 1 B-cun proximal to BL39.”) where the model fails to detect the “located_near” relationship between “BL38” or “BL39” and “knee.” This indicates a limitation in the model’s ability to capture subtle spatial relationships. The errors may be due to a lack of comprehensive knowledge about acupoints, preventing it from integrating acupoint information with anatomical knowledge effectively. Additionally, the model may not have been trained on a sufficiently large corpus of acupoint-related data, limiting its ability to learn complex relationships. Relations between anatomical structures and acupoints are inherently complex and may be challenging for the model to detect accurately.

These errors highlight limitations in the model’s ability to contextualize information over multiple sentences and to accurately interpret complex anatomical relationships. To address these issues, it may be beneficial to expand the model’s knowledge base on acupoints, expand its training data with a larger and more diverse corpus, and enhance its ability to comprehend context over multiple sentences. Improving the model’s contextual understanding and strengthening its knowledge of anatomical structures and their relationships could also help mitigate these errors and elevate its performance in RE for acupoint descriptions, upgrading its accuracy and reliability in this domain.

Conclusion

This study underscores the effectiveness of LLMs like GPT in extracting relations related to acupoint locations, highlighting their value in the acupuncture domain. By utilizing LLMs, we have demonstrated the potential for accurate modeling of acupoint location knowledge and further promoting precise locating of the acupoints during acupuncture practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in NLP.

Author contributions

Yiming Li and Xueqing Peng designed the study. Yiming Li developed the pipeline. Yiming Li and Jianfu Li built the model, and Yiming Li performed visualization. Cui Tao and Xu Zuo offered technical resource support. Donghong Pei contributed to data collection. Na Hong and Suyuan Peng participated in data annotation. Yiming Li drafted the manuscript. Hua Xu and Na Hong supervised the study, and Na Hong critically revised the manuscript.

Supplementary material

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflicts of interest

The authors have no competing interests to declare.

Ethics approval and consent to participate

Not applicable.

Code availability

The code underlying this article will be shared on reasonable request to the corresponding author.

Data availability

The data underlying this article will be shared on reasonable request to the corresponding author.