Automating the segmentation, date extraction, and classification of multi-report PDFs in outside medical records using optical character recognition and generative artificial intelligence

Abstract

Objectives

Patients referred for specialized care often arrive with outside medical records (OMRs) compiled into multi-report PDFs that include imaging, pathology, and clinical notes in unstructured formats. Reviewing these records is time consuming and mentally taxing, increasing the risk of delayed care, clinician frustration, and missed information affecting quality of care. This study aimed to automate the segmentation, classification, and date extraction of scanned OMRs, with a focus on records relevant to breast cancer care.

Materials and Methods

We used optical character recognition (OCR) to extract machine-readable text from 1303 scanned PDF documents from 116 distinct external institutions. Gemini 1.5, a large language model (LLM), was then used to segment multi-report files into individual documents, classify them into clinically meaningful categories such as mammograms and pathology reports, and extract study dates to build diagnostic timelines. Document categories were informed by clinical workflows in a breast cancer center.

Results

The system achieved an F1 score of 0.95 for segmentation, 0.96 for classification, and 0.90 for date extraction. In a pilot of 45 records reviewed by clinicians, only 2 classification errors and 1 date error were reported. Clinicians estimated that the tool reduced OMR review time by 40%, improved workflow efficiency, and increased satisfaction.

Discussion

Our findings demonstrate that combining OCR with LLMs can significantly enhance the processing of unstructured medical records, reducing manual burden and supporting timely clinical decision-making.

Conclusion

This study demonstrates the successful application of OCR and LLMs for organizing scanned OMRs within a specialty clinic. By automating a previously manual process, the approach supports scalable review of incoming outside records and has potential for adaptation to other clinical workflows. Future work will focus on evaluating the system across additional specialties and institutions.

Introduction

The complexities of modern health care necessitate precise and timely access to structured patient records to ensure effective diagnosis and treatment across various medical disciplines.^1,2 Many patients, particularly those referred for specialized care, arrive at hospital systems with outside medical records (OMRs) that are typically scanned documents, often compiled into multi-report PDFs lacking organization.³ These records encompass diverse document types, such as imaging studies, pathology reports, and clinical notes, aggregated into a single unstructured file.⁴ Without a standardized format, clinicians must manually navigate these documents, a labor-intensive process that disrupts workflows, delays decision-making, and increases the risk of overlooking critical patient data.^2,5

At tertiary or quaternary referral centers like Mayo Clinic, which receive high volumes of patient referrals, this challenge is particularly pronounced.⁶ These centers frequently obtain OMRs from a wide range of external institutions, often as scanned faxed documents with inconsistent formatting.⁷ The absence of a standardized structure complicates information retrieval, adding to the administrative burden and increasing the risk of delayed care. While efforts to integrate electronic health record (EHR) interoperability standards have improved aspects of data exchange, scanned documents remain a persistent challenge, requiring extensive manual review.⁸ Current software solutions are either limited in functionality or prohibitively expensive, failing to provide the level of detailed classification and data extraction needed for optimal clinical workflows.^9–11

Cancer care exemplifies these challenges.^12,13 Patients undergoing multidisciplinary treatment require timely access to structured clinical documentation to ensure accurate assessment and treatment planning.^2,14 However, their OMRs frequently contain disorganized imaging studies, pathology reports, and treatment histories, making it difficult for clinicians to construct a coherent diagnostic timeline.^15,16 Given that cancer treatment decisions rely on precise sequencing of diagnostic and therapeutic interventions, inefficiencies in document processing can directly impact clinical outcomes.^17,18

Optical character recognition (OCR) has been widely used in health-care settings to extract structured text from unstructured documents, demonstrating its effectiveness in digitizing scanned medical records.^4,19 Studies have shown that OCR-based approaches improve access to critical clinical data by converting handwritten or printed text into machine-readable formats, reducing the reliance on manual data entry.²⁰ Similarly, large language models (LLMs) have shown promise in automating document classification, segmenting multi-page PDFs into meaningful sections, and extracting key clinical details with high accuracy.^21,22 Recent advancements in natural language processing have demonstrated that LLMs can effectively categorize medical documents, summarize clinical information, and extract dates from medical records, thereby enhancing workflow efficiency.^23,24 These developments suggest that integrating OCR and LLMs could be a viable solution for structuring unorganized OMRs and improving information retrieval for clinicians.

To address these challenges, we implemented a structured pipeline combining OCR and LLMs to automate the segmentation, classification, and date extraction of scanned OMRs.^25,26 Our focus is on applying these tools within a clinical workflow to reduce the burden of manual review and better organize unstructured documentation. At high-volume referral centers like Mayo Clinic, where clinicians routinely handle complex, multi-report OMRs, this approach offers a practical solution to improve document triage, streamline review processes, and support timely clinical decision-making.

Figure 1 outlines the complete artificial intelligence (AI)-driven pipeline used to process OMRs, from patient referral to structured output generation.

Figure 1.

AI-driven processing pipeline for outside medical records (OMRs), including OCR preprocessing, LLM-based segmentation, classification, and date extraction, leading to structured clinical outputs. Abbreviations: LLM, large language model; OCR, optical character recognition.

Methods

Data preparation and OCR processing

The dataset used in this study comprised 1303 multi-report PDFs from 116 distinct external institutions, each containing an assortment of document types, including imaging reports, pathology results, clinical notes, and laboratory reports. Given the unstructured nature of these documents, anOCR system was employed to convert the scanned content into machine-readable text.

To ensure data integrity, OCR accuracy was manually verified on a randomly selected subset of documents. This verification process involved assessing the fidelity of extracted text against the original document, particularly for critical clinical terms, dates, and structured elements such as headers and tables. Any OCR errors were documented for further analysis.

Large language model selection and implementation

An LLM was leveraged to automate 3 key document processing tasks:

1. Segmentation: Identifying and delineating individual reports within multi-report PDFs.

2. Classification: Categorizing segmented documents into predefined clinical types.

3. Date extraction: Identifying and extracting key dates relevant to patient history and treatment timelines.

We selected Gemini 1.5, an off-the-shelf LLM, due to its demonstrated proficiency in document handling tasks and its ability to operate effectively without extensive fine-tuning. Unlike traditional machine learning models that require domain-specific training, Gemini 1.5 provided a solution without finetuning, reducing deployment complexity while maintaining high performance.

The text extracted via OCR was formatted into structured prompts designed to enhance segmentation, classification, and date recognition accuracy. The model was configured to recognize contextual indicators of document boundaries and was further refined using breast cancer clinic guidelines to ensure alignment with clinical documentation standards.

Workflow integration

The structured segmented reports, predicted document categories, and extracted dates were delivered to clinicians via a secure internal web application. This tool was accessed through a custom launcher embedded in our EHR system, enabling providers to review structured data alongside the original scanned documents without leaving their standard workflow. The interface presented each segmented report in a split-view format, allowing users to view the original scan and model-generated metadata. This configuration supported efficient triage of incoming outside records and was incorporated into the existing clinical intake process for new breast cancer patients, with no additional steps required by providers.

Evaluation metrics

The performance of the proposed system was assessed using both quantitative and qualitative metrics, including:

1. Segmentation: Evaluated using F1 score to measure boundary detection between distinct reports in a PDF.

2. Categorization: Evaluated using macro-F1 score to account for class imbalance across document types.

3. Date extraction: F1 score computed for correctly identifying clinically relevant dates within a ±3-day window.

Additionally, clinical feedback from breast cancer care providers was collected through structured interviews and surveys, focusing on workflow impact, time savings, and usability.

Survey methodology for workflow evaluation

To assess workflow impact and provider satisfaction, we conducted a survey targeting individuals who had used the tool at least once. A total of 570 responses were collected across physician, nursing, and allied health (nonphysician, non-nursing clinical) staff. The survey included a combination of Likert scale and open-ended questions to evaluate time savings, satisfaction, and ease of use. Participants were asked to estimate time saved per patient when reviewing OMRs and to rate their satisfaction with the new tool compared to prior workflows.

Results

Segmentation and classification performance

The proposed AI-driven solution achieved an F1 score of 0.95 for segmentation, correctly delineating individual reports within multi-document PDFs. The classification model achieved a macro-averaged F1 score of 0.96 across clinically relevant categories, accurately labeling reports as imaging studies, pathology results, clinical notes, and other key document types.

These results show the robustness of the model in processing diverse document types and aligning with real-world clinical workflows in breast cancer care.

Date extraction performance

The date extraction component demonstrated an accuracy of 90%, successfully identifying key clinical dates from pathology and imaging reports. This capability enhances longitudinal patient tracking, ensuring that clinicians have immediate access to critical historical data without manual review.

In addition to automated evaluation, a manual review of 45 records was performed to verify model outputs in a clinical context. Clinicians identified only 2 classification errors and 1 date extraction error across this set, confirming the system’s high accuracy in practical use.

Quantitative evaluation metrics

Table 1 provides F1 scores for classification across different document categories.

Table 1.

F1 scores by document category.

Category	Total reports	F1 score
Radiology and imaging	256	0.98
Procedure and operative notes	742	0.97
Non-procedural clinical notes	1225	0.93
Diagnostic tests	324	0.96
Other	378	0.96

Table 2 presents the F1 scores for the 3 main AI tasks in the study.

Table 2.

F1 scores for document tasks.

Task	F1 score
Document splitting	0.95
Document categorization	0.96
Date extraction (±3 days)	0.90

These results indicate that OCR and LLM integration can significantly enhance the accessibility and usability of outside medical records, providing structured insights with minimal human intervention.

Workflow impact and provider satisfaction

The introduction of the OCR- and LLM-based tool substantially improved workflow efficiency and clinician satisfaction. Based on pilot surveys, clinicians reported an average time savings of 11 min per patient, corresponding to an estimated 40% reduction in time spent reviewing OMRs.²⁷ This freed time could be reallocated to direct patient care and other clinical tasks.

To better characterize the impact across different clinical roles, we conducted a survey with 570 respondents, including physicians, nurses, and allied health staff (nonphysician, non-nursing clinical personnel). Key findings on time savings are summarized in Table 3.

Table 3.

Workflow survey results for OMR review (n=570).

Role	Average time saved (min patient−1)	Respondents (n)
Physician	7	120
Nursing	14	129
Allied health	11	123
Overall average	11	570

Abbreviation: OMR, outside medical record.

Baseline data collected prior to the pilot indicated an average of 29 min spent per patient and 1.5 h per day reviewing OMRs, with low satisfaction: 76% of users rated the prior process below neutral. Postimplementation satisfaction improved markedly, with the distribution of responses shown in Table 4.

Table 4.

User satisfaction (1=lowest, 5=highest) before and after tool implementation (n=570).

Satisfaction level	Pre-pilot (%)	Post-pilot (%)
1	30	6
2	46	10
3	17	20
4	7	33
5	0	31

These findings demonstrate that the tool not only reduces the manual burden of OMR review but also meaningfully improves clinician satisfaction with the process.

Discussion

The findings of this study highlight the effectiveness of OCR and LLMs in automating the segmentation, classification, and date extraction of multi-report PDFs from OMRs. The proposed AI-driven approach demonstrates high accuracy in structuring unstructured patient data, significantly improving workflow efficiency and reducing clinician burden.^28,29 These results align with prior research indicating that AI-based text extraction and classification methods enhance the usability of scanned medical records in EHR systems.

Comparison with prior work

Previous studies have explored various techniques for processing scanned documents in EHRs, primarily leveraging OCR and rule-based approaches. While these methods have demonstrated utility in extracting structured text, they have historically suffered from limitations in handling document segmentation and classification at scale. Recent advances in deep learning, particularly with LLMs, have expanded the scope of information extraction by incorporating contextual understanding and automated categorization. Our study builds upon this foundation by integrating OCR- with LLM-based classification, demonstrating that this hybrid approach yields substantial improvements in accuracy and efficiency.^28,29

Challenges in handling unstructured OMRs

One of the most persistent challenges in managing OMRs at quaternary referral centers like Mayo Clinic is the inconsistency in document formats. As shown in prior work, scanned documents received from external institutions vary significantly in layout, structure, and legibility. These variations complicate text extraction and necessitate manual review, which is both time-intensive and error-prone. Our findings reinforce the notion that without AI-driven automation, reliance on manual workflows increases the likelihood of delayed care and clinician fatigue. By addressing these inefficiencies, our system reduces the cognitive burden on providers and streamlines document processing.

Limitations and future directions

Despite the promising results, the limitations must be acknowledged. First, while our model demonstrated high accuracy in segmentation and classification, its performance may vary when applied to OMRs with extreme variations in formatting or poor scan quality. Future studies should explore methods for improving robustness, such as incorporating additional image preprocessing techniques to enhance OCR accuracy. Additionally, while our approach focused on breast cancer care, the principles underlying this work could be extended to other specialties where OMR management poses similar challenges.

Another avenue for future research involves expanding the capabilities of LLMs to perform automated summarization and information extraction beyond classification tasks. Prior studies have suggested that AI models can assist in clinical summarization, reducing the need for manual chart review. By integrating summarization features, future iterations of our system could further optimize clinician workflows and enhance patient care.

Our document corpus included records from 116 distinct external institutions, providing a diverse range of formatting and layouts. This diversity supports the plausibility of generalizability within breast surgical oncology workflows, with formal multisite evaluation planned for the future.

By automating segmentation, classification, and date extraction of outside medical records, this approach naturally supports scalability by reducing the manual effort required to process large volumes of multi-report PDFs. Although this study focused on breast cancer care, similar referral-based specialties that rely on imaging, biopsy, and histopathology reports, such as other surgical oncology services, could benefit from an adapted version of this workflow.

Conclusion

This study demonstrates the practical application of OCR and LLMs for structuring unorganized outside medical records within a specialty care workflow. By automating segmentation, classification, and date extraction, our system helped reduce clinician burden and improve review efficiency in a real-world setting. These findings add to the growing number of translational efforts showing how existing AI tools can support documentation workflows. Future work will evaluate adaptation to additional specialties and settings, as well as improve robustness for more variable document formats.

Author contributions

Shivam Damani (Conceptualization, Data curation, Formal analysis, Validation, Visualization, Writing—original draft, Writing—review & editing), Benjamin Hinton (Conceptualization, Data curation, Formal analysis, Methodology, Writing—review & editing), Tanner Hunt (Conceptualization, Data curation, Methodology, Software, Writing—review & editing), Nicholas Lawrence (Conceptualization, Data curation, Methodology, Writing—review & editing), Kurt Miller (Conceptualization, Data curation, Methodology, Writing—review & editing), Melinda Rice (Project administration, Supervision, Writing—review & editing), Kevin Peterson (Methodology, Supervision, Writing—review & editing), Sarah McLaughlin (Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Writing—review & editing), and Alexander Ryu (Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Supervision, Writing—review & editing)

Supplementary material

Supplementary material is available at JAMIA Open online.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflicts of interest

The authors declare no competing interests.

Data availability

The dataset analyzed in this study contains patient information and cannot be publicly shared due to institutional and legal restrictions. Researchers interested in accessing deidentified aggregated data or the processing pipeline may contact the corresponding author, subject to Mayo Clinic’s data sharing policies and applicable data use agreements.