Ensemble transfer learning for classifying physical examinations in GP consultation: a multi-model approach to human-object and human-to-human activity recognition

Abstract

Objectives

This study aims to automatically classify physical examinations performed during general practitioner (GP) consultations using a deep learning fusion model. The model distinguishes between two interaction types: Human–Object Activities (HOA), such as blood pressure measurement, and Human–Human Activities (HHA), such as gland palpation.

Material and Method

A multi-component ensemble transfer learning framework was developed that integrates spatial and temporal feature analysis. The model comprises: (1) a CNN–LSTM module for spatial feature extraction and sequential modelling, (2) an ensemble of EfficientNet-B7, DenseNet-121, and Inception-v3 to capture diverse spatial representations, and (3) a fusion module that concatenates outputs from both streams, refined by an attention mechanism to prioritise salient features. Transfer learning was applied to fine-tune pre-trained networks on GP consultation video data. Model performance was evaluated using five-fold stratified video-level cross-validation, reporting mean ± SD for precision, recall, F1-score, specificity, Cohen’s κ, and PR-AUC.

Results

The fusion model achieved robust overall performance, with a precision of 92.1 ± 1.4%, recall of 89.9 ± 1.8%, F1-score of 90.9 ± 1.5%, specificity of 93.1 ± 1.3%, Cohen’s κ of 0.90 ± 0.02, and PR-AUC of 0.935 ± 0.02. It consistently outperformed ten state-of-the-art baselines, while ablation analysis showed F1-score improvements of 17% over CNN–LSTM and 16% over the ensemble model, confirming the benefit of combining spatial and temporal analysis.

Conclusion

The proposed fusion framework accurately recognises physical examinations in GP consultations and supports future telehealth and diagnostic research.

1. Introduction

General practitioners (GPs) are vital in primary care, with physical examinations central to diagnosis and clinical decision-making.¹ These activities range from basic tasks, such as blood pressure measurement, to more complex procedures, including gland palpation. However, the COVID-19 pandemic disrupted conventional face-to-face consultations, prompting a rapid shift toward telehealth to maintain care while observing social distancing.²

Telehealth has since become an integral part of healthcare delivery in countries such as Australia, the UK, and the US, offering convenience, accessibility, and reduced travel time.^3,4 Government support through initiatives like Australia’s Medicare Benefits Scheme (MBS)^5,6 and the UK’s updated Care Quality Commission (CQC) reflects efforts to integrate telehealth into healthcare systems permanently.⁷ The United States of America has also broadened its mental health programs by implementing telehealth services. Nevertheless, concerns persist around its limitations, particularly the inability to conduct physical examinations virtually, raising questions about the clinical completeness, which requires continuous evaluation and improvement.⁸

Studies measuring the effectiveness of telehealth have indicated that patients report high satisfaction with telehealth for administrative or low-risk clinical tasks and express concern over the absence of physical examinations.⁹ Therefore, it is crucial to understand what physical examinations can be feasibly conducted during in-person GP consultations in telehealth settings.

Despite the significance of physical examinations, there is a lack of detailed information regarding the specific examinations performed in primary care settings, particularly those involving Human-Object Activities (HOA) (such as using medical devices like blood pressure monitors) and Human-to-Human Activities (HHA) (such as using direct physical interaction for gland palpation).¹⁰ Prior studies report broad estimates of physical examination frequency across countries.^11–14 However, they rarely distinguish between different types of examinations or interactions.

Our study proposes an automated approach to identify and classify physical examination activities from GP consultation video recordings covering HOA and HHA interactions to address this. Manual coding is time-consuming and inconsistent; thus, we introduce a Fusion Model combining CNNs, LSTM, and ensemble learning to classify these activities automatically. By capturing spatial and temporal patterns, our method aims to provide robust, scalable insights into physical examination practices, supporting the transition from in-person to telehealth consultations.

2. Materials and methods

2.1. Dataset

The dataset comprises 169 GP consultation videos,¹⁵ with frames extracted for physical examination activities. See Appendix A, Section 2.1 for details.

2.2. Eligibility criteria

A total of 281 GP consultations from the HaRI dataset were analysed, and 169 videos met the eligibility criteria, which focused on consultations where a physical examination was performed. Further details on dataset privacy measures and ethical considerations are provided in Appendix A, Section 2.2.

2.3. Label generation and data coding

The text-based analysis in our previous study¹⁶ involved eight clinical students and a clinical practitioner who manually reviewed GP consultation transcripts to identify keywords related to physical examinations. The detected keywords were used to generate timestamps, facilitating the extraction of corresponding image frames for this experiment.

Following frame extraction, an interaction analysis was performed to classify the frames into Human-Object Activities (HOA) and Human-to-Human Activities (HHA), as detailed in Appendix A, Section 2.3. The classification framework is outlined in Table S1 for HOA categories and Table S2 for HHA categories.

2.4. Proposed methodology

2.4.1. Overview

We propose a Fusion Model to classify physical examination activities during GP consultations. These activities fall into two categories: Human-Object Activities (HOA), which involve the use of medical instruments (eg, blood pressure measurement), and Human-to-Human Activities (HHA), which involve direct interactions between a GP and a patient (eg, gland palpation). Figure 1 illustrates examples of both activity types.

Figure 1.

The figure represents examples of Human-Object Activities (HOA) and Human-to-Human Activities (HHA) observed during GP consultations. Screenshots are derived from the HaRI dataset, developed by Prof. Fiona Stevenson and her team at University College London. Reproduced with permission from the HaRI project under the dataset’s research use agreement. All video frames were de-identified to protect participant privacy.

Inspired by Hirooka et al.,¹⁷ the proposed framework consists of three interconnected layers that extract and analyse both spatial and temporal features. The first layer employs a CNN-LSTM model, which integrates Convolutional Neural Networks (CNNs) for spatial feature extraction (such as object shapes and body positions) and Long Short-Term Memory (LSTM) networks for temporal sequence modelling (such as the sequence of GP movements during an examination). The CNN component utilises transfer learning with a pre-trained ResNet-18 model (trained on the ImageNet dataset) to extract high-dimensional spatial features relevant to physical examinations.¹⁸

These extracted features are then passed to the LSTM, capturing sequential dependencies in HOA (eg, inflating a blood pressure cuff) and HHA (eg, a GP moving their hands during palpation). This sequential processing ensures the model can accurately interpret time-dependent variations in physical examination activities.

The second layer integrates multiple CNN architectures (EfficientNetB7, InceptionV3, and DenseNet121)¹⁷ to form an ensemble model for further enhancing classification performance. Each architecture specialises in extracting distinct spatial representations, contributing to a more comprehensive feature set. The ensemble approach improves accuracy by dynamically weighting the outputs of each CNN model, enabling the selection of the most relevant spatial and temporal patterns for robust classification. Additionally, adaptive average pooling standardises feature dimensions across different architectures, ensuring smooth integration of their outputs. The ensemble model mitigates the limitations of individual architectures by leveraging their complementary strengths, leading to improved generalisation and robustness in classification.

The final layer, the Fusion Model, integrates the outputs of the CNN-LSTM and Ensemble Model to form a comprehensive feature set. An attention mechanism¹⁹ prioritises the most salient features, ensuring the final classification decision is guided by the most informative spatial and temporal representations. This fusion process enables the framework to combine the sequential dependencies captured by CNN-LSTM with the diverse spatial representations extracted by the Ensemble Model, enhancing classification accuracy and robustness. Figure 2 illustrates the overall architecture, highlighting the interaction between these three layers.

Figure 2.

The figure represents an overview of the proposed Fusion Model. The model consists of three key components: (1) CNN-LSTM Model for extracting spatial features and capturing temporal dependencies; (2) Ensemble Model integrating multiple CNN architectures for enhanced spatial representation; and (3) Fusion Module, which combines outputs from both models using an attention mechanism to refine classification. This approach ensures robust recognition of Human-Object Activities (HOA) and Human-to-Human Activities (HHA) in GP consultations.

2.4.2. Layer 1: CNN-LSTM model

Transfer learning is a widely used approach in deep learning that leverages knowledge from pre-trained models to improve efficiency and performance on a new domain-specific task.²⁰ This study employed transfer learning for spatial feature extraction with ResNet-18, a pre-trained Convolutional Neural Network (CNN) on the ImageNet dataset. ResNet-18 extracts high-dimensional spatial representations from individual frames, such as object shapes, human body positioning, and the presence of medical instruments.²¹ This spatial feature extraction benefits Human-Object Activities (HOA), where static image analysis is crucial for recognising medical instruments like stethoscopes, blood pressure monitors, and otoscopes.

However, physical examinations involve sequential movements, and spatial features alone are insufficient to model the temporal dependencies in Human-to-Human Activities (HHA), where a GP’s interaction unfolds over multiple frames. To address this limitation, we integrated Long Short-Term Memory (LSTM), a Recurrent Neural Network (RNN) architecture, which models sequential patterns (temporal features) by learning dependencies between consecutive frames. The LSTM enables the model to track the progression of GP movements, such as hand placement during palpation or step patterns in gait assessments, distinguishing meaningful motion sequences from isolated gestures.

Further detailed justification of CNN-LSTM^22–27integration and additional configuration technical details provided in Appendix A, Sections 2.4. 2.1 and 2.4. 2.2, respectively.

2.4.3. Layer 2: Ensemble model

The Ensemble Model is a complementary module to the CNN-LSTM model, focusing on enhancing spatial feature extraction by leveraging multiple pre-trained CNN architectures (EfficientNetB7, InceptionV3, and DenseNet121).²⁸

While Layer 1 (CNN-LSTM) effectively captures spatial and temporal dependencies for HHA activities, its reliance on a single CNN architecture (ResNet-18) may limit its ability to generalise across different physical examination settings, particularly those involving Human-Object Activities (HOA). HOA, such as interactions with a GP inflating a blood pressure cuff or using a stethoscope, often depend on fine-grained spatial features, such as the presence, positioning, and orientation of medical instruments. Unlike HHA, these activities do not necessarily require sequential motion tracking but instead rely on precise static spatial representations.

The Ensemble Model integrates multiple CNN architectures to enhance spatial feature learning, each contributing unique strengths in spatial representation learning. Specifically, EfficientNetB7, InceptionV3, and DenseNet121 provide complementary feature extraction capabilities, improving the model’s ability to classify diverse physical examination activities. The key benefits of these architectures are:

• EfficientNetB7 enhances fine-grained detail extraction while maintaining computational efficiency.²⁹

• InceptionV3 captures multi-scale spatial features, accommodating object size, shape, and positioning variations.

• DenseNet121 improves gradient flow, ensuring effective feature propagation and reuse, which enhances hierarchical spatial representation.³⁰ Further details on model selection and prediction averaging are provided in Appendix A, Section 2.4.3.

2.4.4. Layer 3: Fusion model

The Fusion Model, the final stage of the proposed methodology, integrates the outputs of the CNN-LSTM model (Layer 1) and the Ensemble Model (Layer 2) to leverage their complementary strengths. While CNN-LSTM specialises in capturing sequential patterns in HHA (eg, tracking hand movements during palpation), the Ensemble Model enhances static spatial feature extraction for HOA (eg, detecting the positioning of a blood pressure cuff or stethoscope on the patient’s body). This ensures a robust classification framework that effectively recognises diverse physical examination activities.^31–33 Further implementation details are provided in Appendix A, Section 2.4.4.

3. Experimental set-up

A detailed experimental setup is provided in Appendix B, covering dataset preparation, class imbalance handling, model tuning, evaluation metrics, and an ablation study.^34–39 Pre-oversampling class distributions are summarised in Table S3, and the workflow of the proposed fusion approach is illustrated in Figure S1.

4. Results

4.1. Performance of baseline models

To evaluate the effectiveness of our Fusion model, we benchmarked its performance against 10 state-of-the-art image-based deep learning models. These baseline models were selected based on their established performance in spatial feature extraction, contextual feature representation, and long-range dependency modelling. The models were categorised as follows:

• CNN-based models: Inception-v3,⁴⁰ ResNet-152,⁴¹ EfficientNetV2,⁴² DenseNet-264,⁴³ and MobileNetV3.⁴⁴ These architectures are widely employed in image classification tasks due to their strong capacity for spatial feature extraction, enabling effective representation of visual patterns and object structures.

• Hybrid architectures- based models: Inception-ResNetv2,⁴⁵ ConvNeXt-XL, SE-ResNet-154.^46,47 These models integrate deep convolutional networks with enhanced feature fusion mechanisms, improving contextual awareness and robustness in complex classification tasks.

• Transformer-based models: ViT-L/16,⁴⁸ this model employs self-attention mechanisms to capture long-range dependencies and global contextual relationships, making it particularly well-suited for tasks requiring an extensive understanding of spatial relationships.

Each baseline model was evaluated on its ability to classify Human-Object Activities (HOA) and Human-to-Human Activities (HHA) within the HaRI dataset. The comparative assessment provides a benchmark reference against which the performance of the Fusion model can be measured, offering insights into the relative strengths and limitations of different deep learning architectures for physical examination recognition.

4.1.1. Fine-Tuning of baseline models

All baseline architectures were fine-tuned end-to-end on the HaRI dataset following a unified optimisation and augmentation pipeline to ensure comparability. Full preprocessing, parameter, and augmentation details are provided in Appendix C, Table S4.

4.1.2. Comparative performance of baseline and fusion models

The classification performance of the baseline models and the proposed Fusion model is summarised in Table 1, which reports the mean performance across five-fold validation for both Human-Object Activities (HOA) and Human-to-Human Activities (HHA). Among the baseline models, Inception-v3 has consistently higher performance than other architectures, achieving the highest F1-score for both HOA (89.0 ± 0.6%) and HHA (87.6 ± 0.7%). This was accompanied by strong PR-AUC values (0.912 ± 0.006 for HOA, 0.898 ± 0.007 for HHA) and high specificity and agreement scores. The proposed Fusion model further improved upon these results, demonstrating superior overall performance (details provided in Appendix C, Section 4.1.2).

Table 1.

The table presents performance comparison of baseline models and the proposed Fusion model for classifying Human-Object Activities (HOA) and Human-to-Human Activities (HHA). Metrics are reported as mean ± standard deviation across five video-level cross-validation folds

Model	Precision (HOA)	Recall (Sensitivity) (HOA)	F1 (HOA)	Specificity (HOA)	κ (HOA)	PR-AUC (HOA)	Precision (HHA)	Recall (Sensitivity) (HHA)	F1 (HHA)	Specificity (HHA)	κ (HHA)	PR-AUC (HHA)
Inception-v3	89.0 ± 0.8	88.9 ± 0.7	89.0 ± 0.6	90.4 ± 0.7	0.86 ± 0.01	0.912 ± 0.006	87.6 ± 0.7	87.5 ± 0.8	87.6 ± 0.7	89.1 ± 0.8	0.84 ± 0.02	0.898 ± 0.007
ResNet-152	73.2 ± 1.1	73.0 ± 1.0	73.1 ± 0.9	88.6 ± 1.2	0.78 ± 0.02	0.825 ± 0.010	82.3 ± 1.0	82.5 ± 1.1	82.4 ± 1.0	87.3 ± 1.1	0.80 ± 0.02	0.856 ± 0.011
EfficientNetV2	78.5 ± 0.9	78.3 ± 0.8	78.4 ± 0.7	89.2 ± 0.9	0.81 ± 0.02	0.861 ± 0.008	84.6 ± 0.9	84.5 ± 0.8	84.6 ± 0.8	88.8 ± 1.0	0.82 ± 0.02	0.874 ± 0.009
DenseNet-264	79.6 ± 1.0	79.7 ± 0.9	79.6 ± 0.8	88.5 ± 1.1	0.79 ± 0.02	0.847 ± 0.009	76.8 ± 1.1	76.7 ± 1.2	76.8 ± 1.0	86.6 ± 1.2	0.77 ± 0.02	0.835 ± 0.011
Inception-ResNet v2	79.8 ± 0.9	79.5 ± 0.8	79.6 ± 0.8	89.0 ± 1.1	0.80 ± 0.02	0.853 ± 0.009	79.1 ± 1.0	79.0 ± 0.9	79.0 ± 0.9	87.8 ± 1.1	0.79 ± 0.02	0.843 ± 0.010
MobileNetV3	78.5 ± 0.8	78.3 ± 0.7	78.4 ± 0.7	88.8 ± 0.9	0.80 ± 0.02	0.852 ± 0.008	79.8 ± 0.9	79.9 ± 0.8	79.8 ± 0.8	87.9 ± 0.9	0.79 ± 0.02	0.846 ± 0.008
Xception	82.1 ± 1.0	82.0 ± 0.9	82.1 ± 0.8	89.6 ± 1.0	0.83 ± 0.02	0.876 ± 0.008	75.6 ± 1.1	75.5 ± 1.2	75.5 ± 1.1	86.0 ± 1.2	0.76 ± 0.02	0.831 ± 0.010
ViT-L/16	79.1 ± 1.1	79.2 ± 1.0	79.1 ± 0.9	88.7 ± 1.2	0.80 ± 0.02	0.848 ± 0.010	69.8 ± 1.3	69.5 ± 1.1	69.6 ± 1.2	84.4 ± 1.4	0.71 ± 0.03	0.782 ± 0.012
ConvNeXt-XL	81.7 ± 0.9	81.8 ± 0.8	81.7 ± 0.8	89.5 ± 0.9	0.82 ± 0.02	0.869 ± 0.008	78.0 ± 1.0	77.9 ± 1.0	78.0 ± 0.9	86.8 ± 1.0	0.78 ± 0.02	0.839 ± 0.009
SE-ResNet-154	80.1 ± 0.8	80.0 ± 0.9	80.0 ± 0.8	89.1 ± 1.0	0.81 ± 0.02	0.864 ± 0.009	81.6 ± 0.9	81.5 ± 1.0	81.6 ± 0.9	88.1 ± 1.1	0.81 ± 0.02	0.858 ± 0.010
Fusion Model (Proposed)	92.7 ± 0.6	90.8 ± 0.7	91.7 ± 0.6	94.3 ± 0.6	0.91 ± 0.01	0.958 ± 0.005	91.6 ± 0.7	89.0 ± 0.8	90.3 ± 0.6	92.1 ± 0.7	0.90 ± 0.01	0.942 ± 0.006

4.1.3. Visual analysis of baseline model performance

To complement the quantitative results, Figure  3 presents a heatmap visualisation of baseline model performance across 22 physical examination activities. The heatmap presents the average 5-fold classification accuracy, with darker shades indicating higher accuracy and lighter shades indicating reduced performance. This visualisation enables a more granular comparison of model strengths and weaknesses across diverse tasks.

Figure 3.

The graph represents averaged 5-fold classification accuracy of baseline models across 22 physical examination activities. Darker cells indicate higher accuracy. The heatmap highlights variability in model performance across tasks, reflecting the relative strengths and limitations of each architecture.

Overall, the results indicate that while several models achieved high accuracy in specific categories, performance varied notably across tasks, underscoring the complexity of physical examination classification and motivating the development of more robust fusion-based approaches. Detailed task-wise performance metrics for each baseline model are presented in Appendix C (Section 4.1.3).

4.2. Performance of fusion model

The proposed Fusion model demonstrated superior performance compared to all baseline architectures, effectively addressing the limitations observed in individual networks. By integrating multiple convolutional backbones (Inception-v3, EfficientNet-B7, and DenseNet-121) with an LSTM-based sequential learning component (ResNet-18 + LSTM), the Fusion model jointly leverages spatial and temporal representations, yielding enhanced accuracy for physical examination recognition.

Figure 4(a and b) present the weighted averaged 5-fold confusion matrices corresponding to Human–Object Activities (HOA) and Human–Human Activities (HHA), respectively. The model achieved strong and consistent recognition across both categories, reflecting robust classification of instrument-assisted and clinician–patient interaction tasks.

Figure 4.

(a) The figure represents the weighted average 5-fold confusion matrix for Human-Object Activities (HOA) classification using the proposed Fusion Model. Darker cells indicate higher classification accuracy, highlighting strong performance across most object-based physical-examination categories. (b): The figure represents the weighted average 5-fold confusion matrix for Human-Human Activities (HHA) classification using the proposed Fusion Model. Darker cells indicate higher accuracy, demonstrating the model’s reliability in distinguishing diverse clinician–patient interaction patterns.

Overall, these results demonstrate the model’s ability to generalise across heterogeneous visual contexts encountered in general-practice consultations. Detailed per-class accuracies and confusion matrix breakdowns are provided in Appendix C, Section 4.2.

Table 2 shows the overall performance with a precision of 92.08%, a recall of 89.81%, and an F1-score of 90.93%, which highlights the model’s superiority and robustness.

Table 2.

The table below presents performance metrics (Precision, Recall, F1-score) of the proposed Fusion Model across different activity categories

Metric	Human-Object Activities (HOA)	Human-to-Human Activities (HHA)	Overall
Precision (%)	92.7 ± 1.3	91.5 ± 1.5	92.1 ± 1.4
Recall/Sensitivity (%)	90.8 ± 1.7	88.9 ± 1.9	89.9 ± 1.8
F1-Score (%)	91.7 ± 1.4	90.2 ± 1.6	90.9 ± 1.5
Specificity (%)	93.5 ± 1.2	92.8 ± 1.3	93.1 ± 1.3
Cohen’s κ	0.91 ± 0.02	0.89 ± 0.03	0.90 ± 0.02
PR-AUC	0.94 ± 0.01	0.93 ± 0.02	0.935 ± 0.02

As summarised in Table 2, the Fusion model achieved an overall F1-score of 90.9 ± 1.5%, precision of 92.1 ± 1.4%, and recall of 89.9 ± 1.8%, highlighting its balanced and consistent classification performance. Specificity was similarly high (93.1 ± 1.3%), and inter-rater reliability remained strong (Cohen’s κ = 0.90 ± 0.02), indicating the model’s agreement with manual ground truth labels. The area under the precision-recall curve (PR-AUC) reached 0.935 ± 0.02, further confirming its robustness across imbalanced activity categories.

By combining spatial encoding with temporal dynamics, the Fusion model effectively generalises across a diverse range of physical examinations, including those involving clinical instruments (eg, stethoscope, thermometer) and direct patient interaction. These results reinforce the model’s suitability for real-world general practice settings, where multimodal activity classification can support future telehealth adaptations and consultation support tools.

4.3. Ablation study results

The ablation study was conducted to evaluate the contribution of each component of the Fusion Model to its overall performance. Table 3 presents a comparative analysis of different configurations, where specific components (LSTM and Ensemble) were selectively removed to assess their impact on physical examination recognition.

Table 3.

The table below presents ablation study results, analysing the contributions of CNN-LSTM and Ensemble Model to overall performance

Configuration	Description	Precision (%)	Recall (%)	F1-Score (%)
Configuration 1: Full Fusion Model	CNN-LSTM + Ensemble	92.08	89.81	90.93
Configuration 2: Without LSTM	Ensemble Model Only	76.97	72.65	74.75
Configuration 3: Without Ensemble	CNN-LSTM Model Only	76.95	71.27	73.76

The results highlight that the Full Model achieved the highest performance, with an F1-Score of 90.93%, benefiting from both sequential modelling (LSTM) and diverse spatial feature extraction (Ensemble). When the LSTM component was removed, the model’s ability to capture sequential dependencies declined, leading to a significant drop in F1-Score for Human-to-Human Activities (HHA). Conversely, removing the Ensemble component reduced the model’s ability to capture fine-grained spatial details, impacting classification performance for Human-Object Activities (HOA). These findings reinforce the importance of combining spatial and temporal feature extraction to achieve robust and accurate classification of physical examination activities. For a detailed analysis of the individual contributions of each configuration, see Appendix C, Section 4.3.

5. Discussion

5.1. Main findings

The proposed Fusion Model outperformed ten state-of-the-art image-based architectures, achieving an overall precision of 92.1 ± 1.4%, recall of 89.9 ± 1.8%, and F1-score of 90.9 ± 1.5%, demonstrating robust and balanced performance across both Human-Object Activities (HOA) and Human-Human Activities (HHA). By integrating multiple convolutional backbones with an LSTM-based temporal learning component, the model effectively captured both spatial and sequential dynamics underlying clinician–patient interactions.

In addition to these core metrics, the model also achieved high specificity (93.1%), strong inter-rater agreement (κ = 0.90), and an elevated PR-AUC (0.935), reflecting its reliability in distinguishing true clinical actions from visually similar non-target behaviours. These complementary measures further confirm the model’s stability across imbalanced categories and its close alignment with human expert annotation.

Collectively, these findings demonstrate that the Fusion architecture generalises well across examinations of varying visual complexity and interaction types. The results underscore the importance of jointly modelling spatial context and temporal motion cues for accurate clinical activity recognition, particularly in real-world general-practice settings where object-based and interpersonal tasks coexist. Detailed per-category performance examples are presented in Appendix D, Section 5.1, for reference.

5.2. Strengths and limitations

The primary strength of this study lies in developing a robust Fusion model that accurately classifies physical examinations during GP consultations, encompassing both Human-Object Activities (HOA) and Human-to-Human Activities (HHA). By integrating the capabilities of multiple architectures and using an attention mechanism, the model demonstrates high-performance metrics, notably achieving a precision of 92.1 ± 1.4% and an F1-score of 90.9 ± 1.5%. The combination of CNN, LSTM, and ensemble learning enhances both spatial feature extraction and temporal modelling, making it well-suited for recognising complex examination activities. However, certain limitations must be acknowledged. The model was trained and evaluated on the HaRI dataset, which, while comprehensive in capturing a variety of GP consultations, is limited to 169 videos. This may impact its generalizability to more extensive and more diverse datasets. Additionally, the computational complexity of the Fusion Model is substantial, which could hinder real-time deployment in clinical settings. Future work should optimise computational efficiency and validate performance on larger, multi-institutional datasets to enhance scalability and applicability.

5.3. Comparative study analysis

Deep learning has significantly advanced human activity recognition (HAR), yet its application to physical examination recognition remains largely unexplored.^49–51 Our Fusion Model integrates CNN-LSTM and ensemble learning to classify these activities, bridging this research gap. See Appendix E for an extended discussion on related work and prior HAR-based models.

5.4. Implications for health informatics practice

Although this study primarily focuses on the technical development of an image-based deep learning framework, its findings have direct implications for health informatics and clinical practice. The proposed model automatically identifies and classifies physical examinations performed during in-person GP consultations, producing structured insights into when, how, and what types of examinations occur in routine care. These results provide a data-driven foundation for improving workflow design, clinical decision support, and telehealth system optimisation.

In the context of teleconsultation, the capacity to distinguish between Human–Object Activities (HOA) and Human–Human Activities (HHA) offers actionable evidence for adapting clinical workflows. HOA tasks involving instruments such as pulse oximeters or blood pressure monitors can inform the development of remote-monitoring tools and home-based examination kits. Conversely, HHA activities that depend on direct physical contact highlight the limitations of telehealth and the need for targeted training, patient guidance, or novel digital solutions to preserve examination quality.

Beyond telehealth, the framework contributes to consultation analytics by transforming unstructured video data into interpretable, quantifiable metrics. These outputs can support clinical documentation, quality-improvement initiatives, and the evaluation of clinician–patient interaction patterns, core objectives within health informatics practice. Ultimately, the proposed method bridges advanced AI research with real-world digital health applications, enabling more efficient, data-driven, and patient-centred care delivery.

5.5. Future work

Future research will enhance the proposed model by addressing its computational demands and optimising it for real-time application in telehealth scenarios. Expanding the model’s training and validation on larger, more diverse datasets across various clinical environments will be a priority, ensuring its applicability in real-world GP settings. Additionally, incorporating automated feedback loops could allow the model to adapt and improve continuously as more data from telehealth and in-person consultations becomes available.

Furthermore, integrating advanced interpretability techniques will be explored to offer more transparent predictions, aiding healthcare providers in understanding the model’s decision-making process. This transparency could facilitate trust in AI-based assessments, paving the way for seamless integration into clinical workflows and augmenting GP capabilities in telehealth and in-person consultations.

6. Conclusion

This study presents a comprehensive analysis of the Fusion Model for classifying physical examination activities encompassing both Human-Object Activities (HOA) and Human-to-Human Activities (HHA). The Fusion model, integrating CNN-LSTM and Ensemble components, demonstrated superior performance with a precision = 92.1 ± 1.4%, recall = 89.9 ± 1.8%, and F1-score = 90.9 ± 1.5. %It outperformed 10 state-of-the-art models, highlighting its capability to capture spatial and temporal dependencies effectively.

For HOA tasks, the Fusion model achieved a notable improvement of 14–19 percentage points in F1-Score compared to standalone CNN-LSTM and Ensemble models, showcasing its strength in accurately classifying interactions involving physical objects by incorporating temporal and fine-grained spatial features. Similarly, for HHA tasks, the Fusion model delivered approximately 14 percentage points higher precision and recall, underscoring its robustness in handling dynamic temporal interactions. These results establish the proposed Fusion Model as a reliable and accurate solution for automated classification of physical examination activities in GP consultations, contributing to advancements in AI-driven healthcare applications.

Ethics approvals

Ethics approval for the secondary analysis of the HaRI dataset was granted by the Macquarie University Human Research Ethics Committee for Medical Sciences (Ref: 52020558018892) and the NHS (REC Ref: 19/LO/0364; IRAS ID: 257924). The original HaRI project received NHS ethics approval (REC Ref: 16/LO/1029; IRAS ID: 197875).

Author contributions

• Conceptualization: AL, MW, HX.

• Methodology: MW, HX, AL.

• Data Curation: MW, KT.

• Formal Analysis: MW, HX.

• Writing-original draft: MW.

• Writing-review and editing: AL, HX, KT, M.

Supplementary Data

Supplementary material is available at Journal of the American Medical Informatics Association online.

Funding

MW was supported by the Centre for Health Informatics, Macquarie University (Pro Vice-Chancellor’s Research Excellence Scholarship). AYSL was supported by the NSW Health Early-Mid Career Fellowship and the NHMRC Centres of Research Excellence in Digital Health (APP1134919) and Connected Health (1170937).

Conflict of interest

The authors declare that they have no competing interests.

Data availability

This study uses the HaRI dataset of GP consultation videos and transcripts collected under ethics approvals held by the original custodians. Due to patient privacy and institutional data-use agreements, the underlying video data cannot be shared publicly. Aggregated results and model outputs supporting the findings are provided in the manuscript and Supplementary Material. Analysis code used to train and evaluate the models will be made available upon reasonable request to the corresponding author and subject to institutional approvals; a public repository link will be added upon acceptance, if permitted by data-governance constraints.