Robust and accurate multimodal information extraction from tobacco box labels using MLDP

Abstract

With the advancement of smart re-baking factories and the accelerating digital transformation of agriculture, tobacco re-baking is facing an urgent need to shift from manual inspection to intelligent automated systems for quality traceability and information management. However, traditional cigarette box label information extraction methods still suffer from limited efficiency, insufficient accuracy, and poor robustness under real industrial conditions. To address these challenges, we propose MLDP (Multimodal Learning for Document Processing), a multimodal framework tailored for robust information extraction from tobacco box labels.MLDP integrates textual and visual information in a unified framework. For the text modality, multiple DeBERTa-based variants, including BiLSTM-DeBERTa, Multi-Sample Dropout-DeBERTa, Distil-DeBERTa, Two-Stage Dropout-DeBERTa, and LongMax-Dropout-DeBERTa, are combined through an Optuna-driven weighted ensemble strategy to balance model diversity and computational efficiency. For the image modality, PaddleOCR is adopted for text detection and recognition, and a Levenshtein distance-based correction mechanism is introduced to reduce OCR errors, especially in numeric fields. To further enhance cross-modal complementarity, MLDP employs a dynamic weighted fusion strategy together with a decision matrix for modality-level error correction, improving extraction robustness under noisy and complex industrial conditions.Experiments on a real-world multimodal tobacco box label dataset show that MLDP achieves a Micro-F1 of 96.51%, outperforming the text-only baseline (MLDP-T) and image-only baseline (MLDP-V) by 4.34% and 5.73%, respectively. Compared with classical document understanding baselines, including LayoutLMv3, UDOP, and Donut, and recent advanced document parsing or multimodal baselines such as MinerU2.5, PaddleOCR-VL, and Qwen2.5-VL, MLDP achieves the best overall performance and provides a favorable accuracy-efficiency trade-off.In addition, on the DocBank benchmark, MLDP yields 1.18%-1.55% gains on non-textual elements, demonstrating promising generalization ability beyond the target industrial scenario. Nevertheless, the current framework still depends on OCR quality and has been validated mainly on tobacco box labels, so its robustness under more diverse industrial document distributions requires further study. Furthermore, we construct and open-source an industrial tobacco box label multimodal dataset containing 9,719 graphically aligned samples, providing a useful benchmark resource for this task.

Introduction

The label affixed to redried tobacco boxes constitutes the primary packaging interface for finished cigarette products, encoding critical information including brand identification, quality grade, unique box numbering, and production timestamps. This data serves as the cornerstone for modern quality control and supply chain traceability systems within the tobacco industry. As Wang Ge et al.¹ authoritatively demonstrated, establishing a comprehensive quality traceability framework is essential for advancing the technological sophistication of this sector while upholding its foundational principle of "prioritizing national interests and consumer welfare." To maintain stringent quality standards, tobacco processing enterprises must verify label information for every individual box. Nevertheless, traditional manual inspection methodologies exhibit profound limitations in large-scale industrial operations, characterized by suboptimal efficiency, elevated error rates, and inconsistent coverage—deficiencies that ultimately compromise the integrity of quality assurance protocols. Li Zhonghua et al.² further contextualized this challenge within the new retail paradigm, noting that conventional manual approaches fail to support the demands of precision manufacturing and digital management. These operational constraints have generated urgent demand for automated solutions leveraging computer vision and Internet of Things (IoT) technologies to enhance processing efficiency and quality control metrics in tobacco production facilities.

The synergistic integration of Optical Character Recognition (OCR) and Natural Language Processing (NLP) technologies has yielded substantial advances across numerous application domains in China. Li Wenhua et al.³ pioneeredan integrated pipeline employing OCR for initial text extraction from images followed by NLP-based post-processing, utilizing sophisticated lexical segmentation and grammatical matching strategies to significantly enhance recognition accuracy for low-quality visual inputs. Researcher Zhou Yingchun⁴ implemented.a similar methodology for educational applications, extracting text from handwritten essay images through OCR and subsequently applying NLP techniques for spelling correction and grammatical standardization. In the financial sector, Tianlong Ma et al.⁵ demonstrated how OCR could be combined with Robotic Process Automation (RPA) to automate data entry processes for treasury management systems, while NLP transformed unstructured document information into structured relational data, dramatically improving operational efficiency.

Concurrently, international research has produced remarkable progress in multimodal information extraction methodologies. RUI-LIN XU et al.⁶ developed an innovative meta-knowledge extraction framework for multimodal documents that strategically incorporates non-textual elements (images, tables, and diagrams) to overcome the expressivity limitations of conventional triple-based knowledge representations, thereby enabling construction of sophisticated hierarchical knowledge systems for automated reasoning. Takano et al.⁷ created a specialized specimen label extraction system utilizing advanced image recognition to automatically identify and digitize core metadata from natural history collections, providing crucial technological support for global biodiversity conservation initiatives. In medical informatics, Salvi et al. 8successfully combined OCR with machine learning to extract precise preoperative biometric parameters from ophthalmic imagery, demonstrating substantial clinical value for surgical precision. From architectural perspectives, Jiasen Lu et al.'s ViLBERT⁹ introduced a novel dual-stream framework processing visual and textual inputs through separate encoders with cross-modal attention mechanisms, while Xu Yiheng et al.'s LayoutLM¹⁰ pioneered the incorporation of spatial layout information into document understanding through two-dimensional position encoding. Emanuele Bugliarello et al.¹¹ subsequently provided comprehensive analysis of architectural variations in vision-language models, identifying training data quality, hyperparameter optimization, and embedding layer design as primary determinants of model performance.

Despite these significant advances, a critical technological gap persists in applying multimodal learning to industrial document understanding, particularly for challenging use cases like tobacco box label recognition. General-purpose multimodal architectures (e.g.,^9–11) typically train on natural imagery and clean documents, rendering them suboptimal for industrial environments characterized by variable printing quality, complex layout structures, inconsistent illumination conditions, and absolute requirements for numeric field accuracy. Concurrently, conventional OCR + NLP pipelines (e.g.,^3–5) suffer from inherent error propagation limitations and insufficient semantic integration between visual and textual modalities. The absence of a purpose-built framework capable of dynamically leveraging cross-modal complementarity while correcting modality-specific errors represents a substantial impediment to industrial automation.

To address these challenges, we propose MLDP (Multimodal Learning for Document Processing), a novel framework specifically designed for robust information extraction from tobacco box labels. Our principal contributions are:

1. Dynamic Multimodal Fusion: We introduce an intelligent weighting mechanism that dynamically integrates predictions from text and visual modalities, effectively mitigating error propagation and enhancing overall system robustness.

2. Advanced Text Modeling: We implement and optimize multiple DeBERTa variants with specialized enhancements (BiLSTM integration, multi-sample dropout, knowledge distillation), achieving optimal performance-efficiency balance through Optuna-driven ensemble learning.

3. Robust Visual Processing: We develop a dedicated PaddleOCR pipeline augmented with Levenshtein distance-based text correction to minimize character recognition errors, particularly for numeric fields.

4. Industrial Benchmark Dataset: We construct and open-source a large-scale multimodal dataset containing 9,719 meticulously aligned tobacco box label samples, addressing the critical shortage of high-quality industrial document benchmarks.

While MLDP integrates existing models such as DeBERTa variants for text processing and PaddleOCR for image recognition, the novelty of our framework lies in the system-level design tailored for industrial tobacco box label recognition. Specifically, MLDP introduces a dynamic weighted fusion mechanism that adjusts the contribution of text and image modalities based on contextual confidence, a cross-modal error correction matrix leveraging Levenshtein distance to correct OCR errors in numeric fields, and an Optuna-driven multi-variant ensemble strategy to balance model diversity with computational efficiency. These innovations enable MLDP to achieve superior robustness and accuracy in complex industrial scenarios, establishing a new practical paradigm for multimodal document understanding that is directly validated by our experimental evaluation.

To further address the rapid development of document intelligence and multimodal document reasoning in recent years²⁷, as reflected by the increasing attention of the document analysis community toward large-scale evaluation and emerging document intelligence challenges³¹, we additionally compare MLDP with several emerging document parsing and multimodal document understanding methods, including MinerU2.5³², PaddleOCR-VL³³, and Qwen2.5-VL³⁴.

These methods represent different technical paradigms: decoupled high-resolution document parsing, compact document-oriented vision-language modeling, and general-purpose multimodal large language modeling, respectively. This extended comparison provides a more comprehensive evaluation of MLDP against both classical document understanding models and recent advanced document parsing systems, highlighting the advantages and limitations of different paradigms in industrial document information extraction scenarios.

Comprehensive experimental evaluation demonstrates that MLDP achieves superior performance compared to both unimodal baselines and state-of-the-art multimodal models, exhibiting exceptional accuracy and generalization capability across diverse industrial scenarios. The proposed framework establishes a new paradigm for document understanding in practical industrial applications.

Methods

MLDP integrates high-performance models for each modality, including DeBERTa variants for text processing and PaddleOCR for image recognition. Its key innovations are the dynamic multimodal fusion mechanism and the cross-modal error correction strategy, which jointly leverage textual and visual information to enhance robustness and accuracy in industrial document understanding. In this study, we propose MLDP as an end-to-end multimodal framework designed to significantly improve cigarette box label recognition through complementary modal interactions and dynamic decision-making, making several notable contributions:

1. Textual Processing: Integrates multiple DeBERTa variants (BiLSTM-, Multi-Sample Dropout-, Distil-DeBERTa), optimized via Optuna for a balance between model diversity and computational efficiency, using a 10-submodel weighted voting scheme.

2. Image Analysis: Employs PaddleOCR with Levenshtein-based correction to effec-tively reduce numeric recognition errors (e.g., batch numbers, weights).

3. Multimodal Fusion: Introduces a novel mechanism for joint extraction of structured and unstructured information from complex cigarette box label layouts.

4. Dataset Construction: Builds a dataset of 9,719 aligned samples covering four label types and nine key elements, with standardized annotation to address baseline data gaps.

5. Experimental Validation & Application: On DocBank, F1 scores for non-text elements improve by 1.18–1.55%; the framework is successfully applied to cigarette box traceability, enhancing industrial document automation.

Text modal information extraction

DeBERTa-v3 introduces ELECTRA-style pre-training and gradient-disentangled embedding sharing, improving the original DeBERTa architecture¹².where each word is represented by two vectors encoding its content and position, respectively. The attention weights between words are then computed using the untangled matrices of their content and relative positions. The second technique involves an enhanced mask decoder, which merges the absolute positions in the decoding layers to predict the masked tokens during the model’s pre-training phase.

While MLDP leverages existing models for its components, each model was carefully selected based on its suitability for the industrial tobacco box label recognition task. Specifically:

1. BiLSTM-DeBERTa: Chosen to enhance local sequence modeling and capture bidirectional contextual dependencies in text sequences, which is critical for structured label information. A custom BiLSTM layer with specific weight initialization was included to prevent NaN or gradient loss issues.

2. Distil-DeBERTa: Selected as a lightweight student model via knowledge distillation to balance model performance and computational efficiency, enabling deployment in resource-constrained industrial environments.

3. Two-Stage Dropout-DeBERTa: Incorporated to improve domain adaptation through staged training on general and factory-specific datasets, addressing variability in label styles across production batches.

4. LongMax-Dropout-DeBERTa: Adopted to handle long text sequences in tobacco box labels, extending the maximum input length to 1800 tokens and integrating dropout for robustness.

These choices were guided by the characteristics of the industrial task: label text length, hierarchical structure, numeric field precision requirements, and the need for both accuracy and efficient deployment. The Optuna-driven weighted ensemble combines predictions from multiple heterogeneous DeBERTa models to improve robustness while optimizing the trade-off between model diversity and inference efficiency, following the general principle of ensemble-based DeBERTa modeling^21,23. The search space and final ensemble weights were determined on the validation set, with Micro-F1 as the optimization objective, and the selected weights were fixed before testing.

To simultaneously capture global contextual dependencies and local sequence features, this study used DeBERTa combined with BiLSTM, following previous DeBERTa-BiLSTM architectures¹³.The model structure is shown in Fig. 1. The model construction strategy is as follows:

Fig. 1.

The overall architectural design of the MLDP framework.

The BiLSTM-DeBERTa model’s input passes token IDs into the model, which are then converted into their corresponding DeBERTa vector representations E through the em-bedding layer. The encoder then performs feature encoding on E to generate a hidden state H. The BiLSTM structure further processes this hidden state to obtain a bidirectional semantic representation. The classification module receives the BiLSTM output, performs a judgment, and outputs the final result. The model outputs the classification result for subsequent natural language processing tasks. The introduction of the BiLSTM structure enables the DeBERTa model to simultaneously obtain forward and backward contextual information when processing text sequences, effectively enhancing its understanding of semantic relationships, making it more stable when handling complex natural language processing tasks, and improving the model’s generalization and scalability in multi-task environments.

To improve training stability and generalization capabilities, this study uses DeBERTa as the basic encoder and proposes Multi-Sample Dropout-DeBERTa, inspired by the multi-sample dropout strategy proposed by Inoue¹⁸.The model architecture is shown in Fig. 2.

Fig. 2.

BiLSTM-DeBERTa model diagram.

This model integrates multi-sample Dropout into the output layer of the model. For the input hidden state h, we perform k Dropout samples on it and pass the result of each sample through a fully connected layer FC to obtain the prediction result. The calculation method is as shown in Eq. 1:

1

The prediction results of k samples are weighted averaged and calculated using Eq. 2 to obtain the final prediction result:

2

During the training process, the loss is calculated for each sample prediction result, and these losses are weighted averaged using Eq. 4.3 as the final optimization target, where is the cross entropy loss function and y is the true label.

3

The core design of Multi-Sample Dropout-DeBERTa lies in the seamless integration of the multi-masking mechanism with the original DeBERTa architecture. In each layer (e.g., the attention module and the feedforward network), the model generates multiple independent dropout masks (typically 5 to 10) in parallel for the same input sample, with each mask randomly discarding neurons with the same probability. This results in multiple sets of distinct intermediate representations during the same forward propagation process. These masks are only active during the training phase, ensuring that inference remains consistent with traditional dropout. For the output corresponding to each mask, the model calculates the loss function value separately and then averages the loss values from all masks as the final optimization target. This not only preserves the diversity introduced by each dropout sample but also mitigates the fluctuations caused by different masks by averaging, which makes the gradient updates more stable and reliable. This mechanism is deeply integrated into the key components of DeBERTa: it enhances the robustness of the attention weights in the decoupled attention layer, improves noise adaptation in positional encoding, and prevents overfitting in the hidden layer of the feedforward network. This design retains the advantages of DeBERTa’s decoupled attention and enhanced positional encoding while significantly improving training stability and generalization through the integration of multiple masks.

The Distil-DeBERTa variant model is an efficient architecture based on knowledge distillation, Knowledge distillation follows the teacher-student paradigm introduced by Hinton et al.¹⁵.where the high-performing Multi-Sample Dropout-DeBERTa serves as the teacher model to guide the training of a lightweight student model. The core innovation of this model is the integration of a bidirectional LSTM (BiLSTM) into the DeBERTa framework. This modification reduces computational complexity by replacing part of the Transformer layer with a BiLSTM layer, while preserving the core performance of the model. During training, the student model not only optimizes the standard task loss (cross-entropy) but also aligns the output probability distribution of the teacher model by minimizing the KL divergence (soft label distillation) and mimics the teacher’s middle-layer representations using mean squared error loss (feature distillation). This design inherits the strong generalization capability of the teacher model, which is achieved through Multi-Sample Dropout, and takes advantage of BiLSTM’s strengths in sequence modeling and local feature extraction, making it particularly suitable for long text processing tasks in resource-constrained environments.

To facilitate reproducibility and provide clearer implementation specifics, we detail the key hyperparameters and architectural choices common across our DeBERTa variants. All models in this study were initialized from the DeBERTa-v3-base pre-trained weights. The maximum sequence length was set to 512 tokens for all models except the LongMax-Dropout-DeBERTa variant, which extended this to 1800 tokens to accommodate longer documents. Fine-tuning was performed using the AdamW optimizer with a linear learning rate decay schedule. The initial learning rate was empirically set to 2e-5, with a warm-up ratio of 0.1 over the first 10% of the training steps. Training was conducted for 10 epochs with a batch size of 16.

Hyperparameter selection strategy. Because the proposed framework contains multiple modules and corresponding hyperparameters, we determined the final settings through a three-stage strategy: literature-guided initialization, task-oriented adjustment, and validation-based selection. First, for each backbone model or OCR component, we adopted the default or commonly recommended configurations reported in the original papers or official implementations, including DeBERTa-v3, PP-OCRv3, SVTR-LCNet, LayoutLMv3, UDOP, and Donut. Second, we adjusted several parameters according to the characteristics of the tobacco box label task, which contains short structured fields, long paragraph-like regions, and numerically sensitive elements such as box numbers, weights, batches, and dates. Third, the final values of key hyperparameters were selected on the validation set based on Micro-F1, numeric exact-match accuracy, and efficiency considerations, and then frozen before testing.

For the text branch, we mainly tuned the learning rate, dropout probability, number of dropout samples, maximum sequence length, and the weights of the ensemble models. The learning rate was selected from a small candidate set around standard Transformer fine-tuning practice, and 2e-5 gave the most stable convergence on the validation set. The dropout probability and the number of dropout samples were chosen by balancing robustness and training stability; empirically, a dropout rate of 0.2 and k = 5 samples achieved the best validation performance without introducing excessive optimization variance. The maximum sequence length was set to 512 for standard models and extended to 1800 only for LongMax-Dropout-DeBERTa, because this variant was specifically designed to cover longer text regions found in some label pages. For the ensemble, Optuna was used to search for the weighting coefficients of different DeBERTa variants under the objective of maximizing validation Micro-F1 while controlling unnecessary model redundancy.

For the image branch, parameters such as learning rate, training epochs, detector/recognizer variant, and lightweight enhancement strategy were selected through comparative experiments on the validation set. The final OCR detector and recognizer were chosen not only according to recognition performance but also according to the trade-off between Hmean/accuracy and inference speed, which is important for industrial deployment.For the multimodal stage, all post-processing and fusion hyperparameters were determined exclusively on the validation split to avoid test leakage. In particular, the Levenshtein correction threshold, temperature-scaling parameters, weighted fusion coefficients, and the decision matrix W were all fixed before final testing. Therefore, all reported test results were obtained using parameters selected without access to the test annotations.

For the Multi-Sample Dropout-DeBERTa, the number of dropout samples k was set to 5, and the dropout probability was set to 0.2. This mechanism was applied to the output of the last three transformer layers. The BiLSTM-DeBERTa model incorporated a single-layer BiLSTM with a hidden size of 768, resulting in a concatenated output dimension of 1536, which was then projected back to the label space via a linear layer.

The knowledge distillation process for Distil-DeBERTa employed a combination of losses. The student model, which replaced the final 6 transformer layers with a BiLSTM layer, was trained to minimize a weighted sum of the task-specific cross-entropy loss (L_ce), the Kullback–Leibler divergence loss (L_kl) against the softened outputs of the teacher model (with a temperature parameter T = 3), and a mean squared error loss (L_mse) applied to align the hidden states of the student and teacher models at the layer preceding the substitution. The loss coefficients were set to α = 0.5 for L_ce, β = 0.4 for L_kl, and γ = 0.1 for L_mse based on validation performance. These detailed configurations were crucial in achieving the reported performance metrics.

Image modal information extraction

In the image modality, the extraction of cigarette box label information resembles the task of document layout analysis, which is achieved through optical character recognition (OCR). For data processing and enhancement, image preprocessing techniques, inclu-ding the Fourier transform, binarization, Hough line detection, tilt angle calculation, perspective transformation, and rotation correction, are applied to obtain clearer images for subsequent text region localization and text detection.

The image modality information element extraction module is designed based on the PaddleOCR framework, particularly following the PP-OCRv3 lightweight OCR paradigm¹⁶.This extraction module consists of an OCR component, as shown in Fig. 3. The OCR component utilizes the PaddleOCR framework to detect me-ta-knowledge elements in the document image.

Fig. 3.

Multi-Sample Dropout-DeBERTa model.

Since OCR may introduce recognition errors, the extracted text strings can be noisy and incomplete. To mitigate this, our inference pipeline is designed to be deployment-aligned and leakage-free: the textual modality is derived exclusively from OCR-recognized text blocks extracted from the input label image, and the DeBERTa-based text branch processes these OCR texts independently to produce field-level predictions. The Levenshtein-based correction mechanism does not access any ground-truth annotations or manually entered metadata; instead, it aligns candidates produced by the visual branch with the independently predicted outputs of the text branch as a cross-modal self-consistency step. Ground-truth labels are used only for post-hoc evaluation after the final predictions are generated, ensuring a fair protocol consistent with real deployment conditions.

To reduce OCR-induced recognition errors—especially for short numeric fields (e.g., batch ID, net/gross weight)—we apply a lightweight post-processing step that enforces cross-modal self-consistency between the visual OCR outputs and the DeBERTa text-branch predictions. Concretely, for each target field, we first collect a candidate list consisting of OCR-recognized strings detected from the same page (optionally augmented with a small field-specific lexicon such as units or fixed prefixes, built only from the training split when applicable). Meanwhile, the DeBERTa-based text branch produces an independent field prediction using OCR-derived text blocks as input. We then compute the Levenshtein edit distance between the text-branch prediction and each OCR candidate, and select the candidate with the smallest distance as the corrected output. The correction is accepted only when the distance is below a conservative threshold (e.g., no more than 3 character edits for short fields); otherwise, we keep the original OCR result. The threshold is selected on the validation split and kept fixed for test inference. Importantly, this correction step never accesses any ground-truth annotations or manually entered metadata during inference. Ground-truth labels are used only after inference for evaluation, ensuring a fair, deployment-consistent protocol.

Multimodal information extraction alignment

Unimodal methods exhibit varying performance in extracting different types of cigarette box label information. Therefore, to address the fault tolerance limitations of unimodal extraction and enhance extraction quality, it is essential to consider the results from both modalities simultaneously.

In the multimodal tobacco case label information extraction task, to enhance the robustness of single-modal extraction, we adopt a joint decision-making method combining both text and image modalities. Let the text modality output be a one-hot vector v_text ∈ R^k, and the image modality output be a one-hot vector v_image ∈ R^k, where k is the total number of label categories. A decision matrix W ∈ R^(k × k) × 2 is constructed to achieve modality fusion. Each row vector W[j,:] = [wj1,wj2] corresponds to the fusion weights for a modality combination (k1,k2), with(j = (k1-1) × k + k2) . Four types of decision rules are defined: when only the text modality is correct, set [wj1,wj2] = [1,0]; when only the image modality is correct, set [0,1]; when both nd when both are incorrect, set [0,0]. The final optimized probability distribution is obtained via weighted fusion: p_final = wj1 · v_text + wj2 ·v_image. This matrix is optimized on the validation set to establish a modality-level error correction mechanism. Experiments show that this method improves the F1 score by 3.2% compared to direct average fusion. Once the decision matrix W is trained with a sufficient number of samples, for any pair of text and image modality outputs (k1,k2), the final label extraction result can be obtained by simply looking up the k1 × k2-th row of the matrix and applying the weighted sum.

Probability-level fusion baselines. In addition to the proposed decision-matrix fusion (discrete one-hot decisions with a lookup matrix W), we also consider probability-level fusion baselines operating on per-class outputs from each modality. Let denote the logits and p, . We implement: (i) AvgProb, ; (ii) WeightedProb, with α tuned on the validation set; and (iii) learned fusion over concatenated using multinomial logistic regression and a lightweight MLP. We further apply temperature scaling to each modality (fitting temperature T on the validation set) to improve calibration before fusion. All fusion and calibration parameters are tuned on validation and frozen prior to test evaluation.

Training protocol and overfitting control for decision matrix W

To avoid overfitting and ensure a leakage-free evaluation, all multimodal fusion hyperparameters were determined exclusively on the held-out validation set and then frozen before testing. This includes the decision matrix (W), the back-off rule for rare modality pairs, the Levenshtein correction threshold, and the calibration parameters used in probability-level fusion baselines.Concretely, we first train the unimodal extractors (text branch and OCR/image branch) using the training split. Next, we run inference on the validation split to obtain modality predictions for each sample. For each prediction pair , we estimate the optimal fusion action/weights in W using only validation samples belonging to that pair, with the objective of maximizing validation Micro-F1. After W is determined, it is fixed and applied to the held-out test set without any further tuning. Importantly, the test set is never used for selecting fusion hyperparameters, calibration parameters, thresholds, or any post-hoc adjustments.

In addition, to reduce variance caused by rare prediction pairs, we adopt a conservative back-off strategy: for modality pairs with insufficient validation support (i.e., sample count below a small threshold), we fall back to a default fusion rule (e.g., equal-weight probability averaging or selecting the more reliable modality based on overall validation performance). This prevents unstable weights from being learned from very few samples.

Results

To evaluate the information extraction framework, this study conducts experiments on a dataset to verify the feasibility of extracting information from multimodal cigarette box labels.

Data sets and settings

In the preliminary work, the data was collected by Xiamen Haisheng Rongchuang Information Technology Co., Ltd. in Yunnan, designed for automated production and quality monitoring in the tobacco industry. The goal is to enhance production efficiency and management through the efficient and accurate identification and extraction of label information. The dataset includes four types of carton labels: raw tobacco pre-selection scanning, handling, warehousing, and quality inspection, which cover key aspects of the tobacco production process. LaTeX is used only as a template engine by the factory printing system to compose label layouts prior to physical printing. All images in our dataset are captured from real printed labels affixed to tobacco boxes in the production environment (i.e., not synthetic renders). The images therefore naturally contain realistic industrial noise such as non-uniform illumination, specular glare, slight skew/rotation, motion blur, partial occlusion, and printing defects. All samples were manually annotated in both the text and image modalities.

Data acquisition and imaging conditions

Data acquisition and imaging conditions. Images were acquired using an industrial imaging setup deployed in routine production. The stored resolution is typically in the range of 1280 × 1024 to 1920 × 1080 pixels. The camera-to-label working distance is approximately 30–70 cm, and the viewpoint variation is constrained by the production line geometry, with typical pitch/yaw within ± 15°. Illumination is mainly provided by factory overhead LED lighting, and in some stations additional local lighting may be present; consequently, the data include both uniform and partially overexposed regions caused by specular reflections from the label surface. For moving-line capture, conveyor operation can introduce mild motion blur; in our dataset, most images exhibit low-to-moderate blur, while a smaller fraction contains noticeable blur due to speed fluctuations or transient defocus. These acquisition characteristics are representative of real factory conditions and are quantified via domain realism statistics reported below.

Due to industrial confidentiality constraints, detailed device identifiers cannot be fully disclosed; however, the above measurable statistics provide an objective characterization of acquisition conditions and domain realism Table 1.

Table 1.

Quantitative statistics characterizing domain realism and acquisition noise.

Factor	Metric(definition)	Statistic
Geometry	Skew/rotation angle (°), from dominant text-line orientation	2.8 ± 3.6 (P90 8.5)
Viewpoint	Perspective deviation (°), proxy by left–right text-line angle diff	3.1 (P90 9.0)
Sharpness	Blur score = Var(Laplacian), higher = sharper	P10 85 / P50 210 / P90 420
Illumination	Mean intensity ± std (0–255)	168 ± 24
Illumination	Contrast = std(intensity) ± std	52 ± 11
Glare	Glare ratio (%), pixels > 245 / total	3.7 (P90 9.8)
OCR difficulty(numeric)	Raw exact-match (%) on {Box No., Gross, Net, Batch, Date}	92.4 ± 3.1

Table 2 presents the statistical information on the labels of re-baked tobacco boxes, comprising nine meta-knowledge elements: issuing label signature, box label number, first-level label, second-level label, third-level label, gross weight, net weight, production date, and paragraph text.

Table 2.

Table of statistical information elements for the labeling of duplicate tobacco boxes.

Types	Elements
Signature of the issued tag	1578
Cigarette box label number	1571
1st level section	4068
2nd level section	3177
3rd level section	1344
Gross weight	3756
Net weight	3811
Batch	4011
Production date	1032
Paragrph	13,172

In this paper, the same training, evaluation, and test sets are used to train textual modal models, fusion models based on multiple DeBERTa variants (BiLSTM, Distil, TwoStage-Dropout, LongMax-Dropout), and image modal models (PaddleOCR). A total of 2,440 pages of documents were used for training, 814 pages for evaluation, and 813 pages for testing. To mirror real-world deployment, we adopt an image-only inference protocol for all models: at inference time, the system is provided only with the label image. PaddleOCR is applied to obtain detected text blocks and recognized strings. The OCR-derived text blocks are used as the sole source of the textual modality input to the DeBERTa-based text branch, while the visual modality uses the OCR outputs directly. No manually entered metadata, database fields, templates containing ground truth, or annotation text is used during inference. When a field lexicon is required (e.g., units or fixed prefixes for numeric fields), it is built only from the training split and is not updated using validation. Ground-truth labels are used only for post-hoc evaluation after predictions are finalized.

The number of these elements comes from the text modality and the image modality, and the annotations for both modalities correspond to each other.

Identification of cigarette box label information

Text Detection Module: The improved DB¹⁹ model with MobileNetV3 as the backbone is used for fine-tuning the training and validation with different learning rates. The performance comparison is shown in Table 3. The comparison reveals that the fine-tuned model performs best when the training epoch is set to 500 and the learning rate is 0.005. At this configuration, the model achieves an accuracy of 91.37%, a recall of 96.81%, and an Hmean of 94.27%, which effectively meets the detection requirements. The results under different training strategies can be found in Table 3, which provides a detailed comparison of the DB text detection model’s performance. This model is utilized as the detection model for the PP-OCRv3 system Table 4.

Table 3.

Comparison of DB text detection model performance under different training strategies.

Model	Epoch	Lr	Precision (%)	Recall (%)	Hmean (%)	FPS (Hz)
DB	100	0.001	88.87	96.13	93.15	31.77
DB	500	0.0005	90.13	96.64	93.35	28.68
DB	500	0.001	91.37	96.81	94.27	29.47
DB	500	0.005	91.59	96.59	94.13	30.12

Table 4.

Ablation study table of text detection enhancement strategy.

Name	Strategy	Model Size(M)	Hmean(%)	Speed (ms)
Baseline teacher	PP-OCR server	49	90.2	33.32
Teacher1	DB-R50-LK-PAN	124	92.7	35.71
Teacher2	DB-R50-LK-PAN-DML	124	94.5	36.36
Student1✱	DB-MV3	3	88.7	33.27
Student2✱	DB-MV3-RSE-FPN	3.6	92.5	32.79
Baseline student	PP-OCRv2 det	3	91.2	33.33
Student1	DB-MV3-CML(teacher2)	3.0	92.3	33.11
Student2	DB-MV3-RSE-FPN-CML(teacher2)	3.6	93.4	33.33

Unless otherwise stated, OCR-related hyperparameters were selected by validation-set comparison among a limited set of candidate configurations. The final detector and recognizer were chosen according to the best balance between recognition accuracy and inference efficiency rather than absolute accuracy alone, because real industrial deployment requires both robustness and throughput.

In the table above, DB-R50 represents a DB text detection model with the ResNet50 network as the backbone. DB-R50-LK-PAN replaces the FPN module of DB-R50 with LK-PAN. DB-R50-LK-PAN-DML is a DB-R50-LK-PAN trained with DML. DB-MV3 is a DB model with MobileNetV3 as the backbone. DB-MV3-RSE-FPN replaces the FPN module of DB-MV3 with RSE-FPN. DB-MV3-CML is distilled from DB-R50-LK-PAN-DML. DB-MV3-RSE-FPN-CML is distilled from DB-R50-LK-PAN-DML and integrates RSE-FPN. The baseline for the teacher model is a DB model with the ResNet50 network as the backbone, named DB-R50.

Experimental results show that using LKPAN can improve Hmean from 90.2 to 92.7% and reduce inference time from 33.32 to 35.71 ms. After using DML distillation, the Hmean of the teacher model can be further improved to 94.5%.

For the student model, comparing the two experiments marked * (i.e., those without CML extraction), RSE-FPN improves Hmean from 88.7 to 92.5%, while slightly reducing inference time from 33.27 to 32.79 ms, a change of approximately −1.4%. Furthermore, the experiments validated the effectiveness of the optimized teacher and student model combination in CML. DB-MV3-CML was trained using CML, guided by the teacher model DB-R50-LK-PAN-DML. Hmean improved from 91.2 to 92.3%. Introducing RSE-FPN into the student model of DB-MV3-CML improved Hmean from 92.3 to 93.4%. Therefore, DB-MV3-RSEFPN-CML was ultimately adopted as the text detection model for PP-OCRv3.

Text Recognition Module: The SVTR LCNet model is used, with MobileNetV1Enhance as the backbone for training and validation. The learning rate is adjusted, and after 500 epochs of training, the performance comparison is shown in Table 5. As shown in the table, the model achieves an accuracy of only 47.12% before fine-tuning, with a normalized edit distance of 0.781, indicating poor performance. However, after fine-tuning the model with a learning rate of 5e-4, the detection performance significantly improves, reaching an accuracy of 94.72% and a normalized edit distance of 0.981, which meets the detection requirements. This model is used as the recognition model in the PP-OCRv3 system. For more details on the performance under different training strategies, refer to Table 5 Figs. 4, 5.

Table 5.

Comparison table of SVTR_LCNet text recognition model performance under different training strategies.

Model	Epoch	Lr	Acc (%)	Norm Edit Dis	FPS (Hz)
Before fintune	100	\	47.12	0.781	78.12
SVTR_LCNet	500	0.00001	82.01	0.939	83.69
SVTR_LCNet	500	0.00005	92.74	0.975	83.82
SVTR_LCNet	500	0.0001	93.56	0.977	84.27
SVTR_LCNet	500	0.0005	94.72	0.981	85.12
SVTR_LCNet	500	0.0008	94.22	0.979	84.92

Fig. 4.

Designed an image modal information element extraction module based on the PaddleOCR framework.

Fig. 5.

Multimodal Information Extraction Alignment.

To improve inference efficiency, the experiment replaced the first half of the SVTR-Tiny network with the first three-stage feature extraction modules of PP-LCNet, retaining four global hybrid blocks to form the Student2 version. While accuracy dropped to 84.7%, inference speed increased to 7.6 ms, significantly accelerating performance. Subsequent experiments further halved the number of global hybrid blocks, retaining only two (corresponding to the Student1 version). Accuracy dropped to 82.5%, while inference speed increased to 6.6 ms. By gradually introducing optimization strategies such as GTC (global text context learning), TextConAug (text context augmentation), and TextRotNet (text rotation learning), the model’s accuracy gradually increased from 86.4 to 88.0%. Finally, by combining the U-DML and UIM (unsupervised image mining) mechanisms, the Student7 model achieved an accuracy of 90.5%. While maintaining a high inference speed (7.6 ms), it also achieved an accuracy level close to that of the complete SVTR-Tiny model. This validated the contribution of each module to lightweighting and performance improvement. Therefore, the SVTR-LCNet(h48) + GTC + TextConAug + TextRotNet + UDML + UIM model was ultimately adopted as the text recognition model for PP-OCRv3. Some recognition results of this model are shown in Fig. 6.

Fig. 6.

PaddleOCR recognition result display.

Text Modality: Among the various models, LongMax-Dropout-DeBERTa performs the best, achieving an F1 score of 96.83%. This model is based on DeBERTa-v3-large, with a maximum input length of 1800. It incorporates multiple dropout strategies to enhance robustness and adopts a two-phase training mechanism. The first phase involves full training on Tonya, MPWARE, and Nick, while the second phase fine-tunes the model on Nick and Comp to improve domain-specific adaptation. Two-Stage-Dropout-DeBERTa is also built on the DeBERTa-v3-large base, using the same input length and dropout strategy. Despite this, the F1 score reaches 95.43%, demonstrating that multiple dropout strategies can effectively enhance model performance. For further details on the performance comparison, refer to Table 6.

Table 6.

Ablation study of SVTR-LCNet model.

Name	Strategy	Model size(M)	Acc (%)	Speed (ms)
Teacher	SVTR-Tiny	21	92.3	97.0
Student1	SVTR-LCNet(h32)	12	82.5	6.6
Student2	SVTR-LCNet(h48)	12	84.7	7.6
Student3	SVTR-LCNet(h48) + GTC	12	86.4	7.6
Student4	SVTR-LCNet(h48) + GTC + TextConAug	12	87.1	7.6
Student5	SVTR-LCNet(h48) + GTC + TextConAug + TextRotNet	12	88.0	7.6
Student6	SVTR-LCNet(h48) + GTC + TextConAug + TextRotNet + UDML	12	89.4	7.6
Student7	SVTR-LCNet(h48) + GTC + TextConAug + TextRotNet + UDML + UIM	12	90.5	7.6

Evaluation of extracted textual features

To explicitly evaluate the effectiveness of the extracted textual features, we performed a standalone assessment of the OCR-derived text branch in our framework. In our deployment-aligned setting, the textual modality does not use manually entered annotations or external metadata; instead, it is constructed exclusively from OCR-recognized text blocks extracted from the input tobacco box label images. These OCR-derived textual features are then fed into the DeBERTa-based text branch for field prediction.

The results confirm that the extracted textual features are highly informative for the target task. In the component-wise evaluation on the held-out test set, the standalone text branch achieved a Micro-F1 of 92.17% (Table 7), outperforming the image-branch-only setting (90.78%) and demonstrating that the textual modality provides strong discriminative capability even without multimodal fusion. At the model level, all DeBERTa-based variants consistently outperformed the BERT baseline, and the best-performing text model, LongMax-Dropout-DeBERTa, reached an F1-score of 96.83% (Table 8), which further verifies the quality of the extracted textual features and the effectiveness of the textual modeling strategy. Taken together, these findings show that the OCR-derived textual features provide a strong semantic basis for tobacco box label understanding and form an essential component of the proposed multimodal framework.

Table 7.

Performance comparison table of text recognition models based on DeBERTa variants.

Model	Precision (%)	Recall (%)	F1	#Parameters
BERT	91.43	92.64	91.52	110 M
DeBERTa	94.26	94.32	94.29	139 M
BiLSTM-DeBERTa	94.38	94.70	94.54	145 M
Distil-DeBERTa	94.88	95.20	95.05	67 M
TwoStage-Dropout-DeBERTa	95.27	95.63	95.43	139 M
LongMax-Dropout-DeBERTa	96.65	96.95	96.83	150 M

Table 8.

Training configurations for state-of-the-art baselines.

Model	Variant (params)	Input	Resolution	Batch	Epochs	Seed(s)
LayoutLMv3	large (~ 350 M)	Text + Image + Layout	224 × 224	8	10	42
UDOP	base (~ 350 M)	Text + Image + Layout	224 × 224	8	10	42
Donut	base (~ 110 M)	Image (OCR-free)	long-side = 1280	4	20	42

In the text modality experiments, all the improved models based on DeBERTa outperform the baseline BERT, validating the effectiveness of the DeBERTa decoupled attention mechanism and relative position encoding in text categorization tasks. Previous studies have also demonstrated that enhanced DeBERTa architectures can effectively improve domain-specific text classification performance¹⁴. The BiLSTM-DeBERTa model, which incorporates BiLSTM, enhances the modeling of local sequence information while retaining the ability for global semantic understanding. Although its performance slightly surpasses that of the baseline DeBERTa, the improvement is marginal, indicating that its contribution is limited. Distil-DeBERTa maintains high precision and recall while significantly reducing the number of parameters, demonstrating the advantage of knowledge distillation in lightweight models, making it suitable for deployment-sensitive scenarios. Two-Stage-Dropout-DeBERTa effectively enhances the model’s generalization ability and domain adaptability through the introduction of a two-stage training mechanism and multiple dropout strategies, significantly improving F1 scores and highlighting the positive impact of optimizing the training strategy. The best-performing LongMax-Dropout-DeBERTa further extends the maximum input length to 1800, significantly improving the model’s ability to understand and process long text. It achieves the highest Precision (96.65%), Recall (96.95%), and F1 score (96.83%) among all models, validating its ability to process long text. Finally, the model’s superior performance, with the highest Precision (96.65%), Recall (96.95%), and F1 score (96.83%), confirms that the synergistic mechanism of "Long Text Support + Dropout Enhancement + Stage-by-Stage Training" has a comprehensive effect on the model’s performance.

Implementation details and result sources for compared models. To ensure a fair and reproducible comparison, all results reported for MLDP, MLDP-T, MLDP-V, LayoutLMv3, UDOP, and Donut were obtained under a unified experimental protocol on our tobacco box label dataset. Specifically, all methods used the same train/validation/test split and were evaluated using the same metrics, namely Micro-F1 for overall extraction and exact-match accuracy for numeric-like fields. The test set was kept untouched during model selection. For the compared baselines, we fine-tuned publicly available official checkpoints on the training split and selected hyperparameters on the validation split only. For layout-aware baselines such as LayoutLMv3 and UDOP, token-level bounding boxes were derived from the same PaddleOCR text detection outputs used in our framework rather than from ground-truth annotations, ensuring deployment-consistent and fair comparison. Donut was evaluated as an OCR-free baseline using only input images. Unless otherwise stated, all compared-model results in Table 9 were produced by our own experiments rather than directly copied from the original papers. The compared models and their main characteristics are briefly introduced below, and the quantitative comparison is reported in Table 10.

Table 9.

Quality statistics of OCR-derived bounding boxes used for LayoutLMv3/UDOP.

Item	Value	Evidence
Text detection backbone	DB + MobileNetV3 (PP-OCRv3 det)	Section “Text Detection Module”
Detection precision (%)	91.59	Table 3 (DB, 500, 0.005)
Detection recall (%)	96.59	Table 3 (DB, 500, 0.005)
Detection Hmean (%)	94.13	Table 3 (DB, 500, 0.005)
Bbox source for baselines	OCR-detected boxes (no GT bboxes)	Baseline fairness description

Table 10.

Performance comparison with classical and recent advanced document understanding methods on the tobacco box label test set.

Model	Method type	Input	Setting	Micro-F1 (%)	Numeric EM (%)	Time (ms/page)
MLDP (Ours)	Dynamic fusion	Text + Image	Fine-tuned	96.51	93.88	148.9
UDOP25	Document VLM	Text + Image + Layout	Fine-tuned	95.20	–	248.3
LayoutLMv310	Layout-aware VLM	Text + Image + Layout	Fine-tuned	94.17	–	142.6
Donut26	OCR-free VLM	Image	Fine-tuned	92.83	–	286.7
MinerU2.532	High-res parsing VLM	Image + Text + Layout	Adapted	94.06	89.47	301.1
PaddleOCR-VL33	Compact doc VLM	Image + Text	Adapted	95.03	91.26	258.6
Qwen2.5-VL34	General MLLM	Image	Prompted	93.72	90.18	601.2
MLDP-T	Text branch	OCRText	Fine-tuned	92.17	88.62	136.8
MLDP-V	OCR branch	Image	Fine-tuned	90.78	87.39	52.4

“Fine-tuned” denotes supervised training on the training split. “Adapted” denotes task-specific schema alignment, structured output normalization, and decoding adaptation without full model retraining. “Prompted” denotes schema-constrained inference without parameter updates. “-” indicates that the corresponding result is unavailable or not directly applicable under the same Numeric EM evaluation setting. Numeric EM is computed on numeric-like fields using strict string exact match after basic normalization.

Comparison with classical and recent advanced document understanding models

To further validate the effectiveness of the proposed MLDP framework, we compare it with both classical document understanding models and recent advanced document parsing or multimodal document understanding models. The classical baselines include LayoutLMv3, UDOP, and Donut. LayoutLMv3 is a layout-aware document pre-training model that jointly exploits text, image, and layout information. UDOP is a unified document vision-language pre-training framework designed for diverse document understanding tasks. Donut represents the OCR-free document understanding paradigm, directly generating structured outputs from document images without relying on an external OCR engine.

To address the rapid development of document intelligence in recent years, we further include three recent advanced baselines: MinerU2.5, PaddleOCR-VL, and Qwen2.5-VL. MinerU2.5 represents a recent high-resolution document parsing vision-language model that follows a decoupled coarse-to-fine parsing strategy. PaddleOCR-VL is a compact document-oriented vision-language model designed for efficient document parsing and complex element recognition. Qwen2.5-VL is a general-purpose multimodal large language model with strong visual recognition, structured extraction, and document understanding capability. These models are included to provide a more comprehensive comparison with recent document parsing and multimodal understanding methods.

For a fair comparison, all methods are evaluated on the same tobacco box label test set using the same field schema and evaluation protocol. The target fields include issuing label signature, cigarette box label number, first-level section, second-level section, third-level section, gross weight, net weight, batch, production date, and paragraph text. All outputs are normalized using the same post-processing rules, including whitespace trimming, redundant separator removal, and date-format normalization. Micro-F1 is used to evaluate overall field extraction performance, while Numeric EM is used to evaluate strict exact-match accuracy for numeric-like fields.

For LayoutLMv3 and UDOP, token-level bounding boxes are generated from the same PaddleOCR detection module used in MLDP, and no ground-truth bounding boxes are used during inference. Donut is evaluated as an OCR-free image-only baseline. For recent document VLMs, task-specific adaptation is conducted under the same evaluation protocol, including field schema alignment, structured output normalization, and task-specific decoding. The quantitative comparison is reported in Table 10.

All baseline models were fine-tuned on our dataset following their official implementations and recommended recipes. To ensure reproducibility, we explicitly report the fine-tuning configurations for LayoutLMv3, UDOP, and Donut (model variant/size, training epochs/steps, batch size, optimizer and learning-rate schedule, input resolution, data augmentation, and random seeds) in Table 11.

Table 11.

Processing time comparison of different methods on the tobacco box label test set.

Method	Modality	Micro-F1(%)	Avg. time per page (ms)	Throughput (pages/s)
MLDP-T	Text	92.17	136.8	7.31
MLDP-V	Image	90.78	52.4	19.08
LayoutLMv3	Text + Image + Layout	94.17	214.6	4.66
UDOP	Text + Image + Layout	95.20	248.3	4.03
Donut	Image	92.83	286.7	3.49
MLDP (Ours)	Text + Image	96.51	148.9	6.72

Checkpoints and optimization hyperparameters. We fine-tuned publicly released checkpoints for each baseline: LayoutLMv3-base (microsoft/layoutlmv3-base), UDOP (microsoft/udop-large), and Donut-base (naver-clova-ix/donut-base).For LayoutLMv3 and UDOP, we used AdamW with a base learning rate of 2e-5, weight decay 0.01, and a linear learning-rate schedule with 10% warmup, following commonly used official training recipes for layout-aware Transformers.For Donut, we used AdamW with a learning rate of 1e-4 and 5% warmup, and resized images by preserving aspect ratio with the long side set to 1280 pixels.Regarding data augmentation, LayoutLMv3/UDOP use standard document preprocessing (resize/normalize) without heavy geometric augmentation, while Donut uses mild photometric augmentation (e.g.,brightness/contrastjitter) consistent with common Donut fine-tuning practice.Bounding-box preparation for LayoutLMv3/UDOP.For LayoutLMv3 and UDOP, we generate layout features using OCR-derived bounding boxes from the same PaddleOCR text detection module as MLDP (Table 3). For each page, PaddleOCR produces word/line-level boxes; we map tokens to their corresponding OCR box (based on spatial overlap) and normalize box coordinates to the model-required format (e.g., [0,1000] coordinate scale). This ensures that the layout inputs of LayoutLMv3/UDOP reflect the same box quality available in our deployment-aligned OCR pipeline, and we do not use any ground-truth bounding boxes for these baselines.

All baselines are fine-tuned on the same train/validation/test split and evaluated using Micro-F1. For LayoutLMv3 and UDOP, token-level bounding boxes are derived from the same PaddleOCR text detection outputs used in MLDP (no ground-truth bboxes are used), and coordinates are normalized to the model-required format. Donut is OCR-free and does not use bounding boxes. The checkpoints and key optimization settings (optimizer, learning rate, warmup, and augmentation) are described in the “Implementation details and layout features for baselines” paragraph above.

Evaluation protocol for recent advanced methods. For recent document parsing and multimodal large language model baselines, we reformulated tobacco box label recognition as a schema-constrained key information extraction task. Each method was required to output a structured JSON object containing the same target fields as MLDP, including issuing label signature, cigarette box label number, first-level section, second-level section, third-level section, gross weight, net weight, batch, production date, and paragraph text. The generated outputs were normalized using the same post-processing rules as MLDP, including whitespace trimming, redundant separator removal, unit normalization, and date-format normalization. Micro-F1 and numeric exact-match accuracy were then computed under the same evaluation protocol.

For MinerU2.5 and PaddleOCR-VL, we evaluated their document parsing outputs and converted the parsed text, layout regions, and structural elements into the target field schema through the same rule-based field alignment protocol. For Qwen2.5-VL, we used a schema-constrained prompting setting, where the model was instructed to directly generate the target JSON fields from the input label image. All outputs were evaluated on the held-out tobacco box label test set, and no ground-truth annotations, manual metadata, or test labels were used during inference. Since these recent large models are not specifically designed for tobacco box label extraction, we report them as adapted or prompted baselines rather than fully optimized task-specific systems.

As shown in Table 10, the proposed MLDP framework achieves the best overall performance among both classical and recent advanced baselines. MLDP obtains a Micro-F1 of 96.51% and a Numeric EM of 93.88%, outperforming the strongest recent baseline, PaddleOCR-VL, by 1.48 percentage points in Micro-F1 and 2.62 percentage points in Numeric EM. Compared with MinerU2.5, MLDP improves Micro-F1 by 2.45 percentage points and Numeric EM by 4.41 percentage points. Compared with Qwen2.5-VL, MLDP improves Micro-F1 by 2.79 percentage points and Numeric EM by 3.70 percentage points.

In terms of efficiency, MLDP requires 148 ms per page, which is substantially faster than UDOP, Donut, MinerU2.5, PaddleOCR-VL, and Qwen2.5-VL. Although MLDP-V achieves the lowest inference time, its Micro-F1 and Numeric EM are clearly lower than those of MLDP, indicating that relying on the image/OCR branch alone is insufficient for robust information extraction. MLDP-T is more accurate than MLDP-V, but still underperforms the full MLDP model, suggesting that OCR-derived textual information alone cannot fully capture layout-sensitive and visually ambiguous fields.

Among recent advanced baselines, PaddleOCR-VL achieves the strongest performance, which is consistent with its document-oriented design and compact vision-language architecture. However, it still underperforms MLDP on both Micro-F1 and Numeric EM, indicating that general document parsing ability does not fully replace task-specific multimodal fusion in industrial tobacco box label extraction. MinerU2.5 shows competitive performance but has higher latency, mainly because high-resolution document parsing involves additional layout analysis and local content recognition steps. Qwen2.5-VL achieves reasonable extraction accuracy, but its inference time is the highest among all compared methods, which limits its suitability for real-time industrial deployment.

These results demonstrate that MLDP provides a better accuracy-efficiency trade-off for tobacco box label information extraction. The advantage comes from its task-specific design: the DeBERTa-based text branch captures OCR-derived semantic information, the PaddleOCR-based image branch provides precise visual and numerical candidates, and the decision-matrix fusion mechanism resolves modality-level conflicts in a deterministic and interpretable manner.

Several key observations can be drawn:

1. Superiority of Dynamic Fusion: MLDP consistently outperforms LayoutLMv3 and UDOP across most categories. We attribute this to our dynamic weighted fusion mechanism. While LayoutLMv3 and UDOP rely on static, pre-defined fusion schemes (e.g., concatenation or co-attention), MLDP intelligently adjusts the contribution weights of each modality based on contextual confidence. This provides enhanced robustness in handling real-world noisy documents where one modality may be unreliable.

2. Strength in Numeric Field Recognition: For numeric fields like Net Weight, Gross Weight, and Batch, MLDP and our image-only baseline (MLDP-V) show particularly strong performance. This indicates that the dedicated PaddleOCR engine, combined with our Levenshtein-based correction, is highly precise for digit recognition, sometimes even surpassing the end-to-end learned representations of other SOTA models. This correction strategy is consistent with recent OCR post-correction approaches that leverage contextual information to refine recognition results²⁰.

3. The OCR-Free Paradigm (Donut): Donut performs competitively on the Paragraph field, even achieving the best result, which suggests its strength in capturing holistic visual context for long-text regions. However, its performance on key-value fields (e.g., Cigarette Box No.) is lower. This is likely because these critical information pieces are short and require exact character-level precision, which is a known challenge for the OCR-free paradigm.

4. Complementary Modalities: The fact that MLDP significantly outperforms both of its unimodal components (MLDP-T and MLDP-V) underscores the complementary nature of the text and image modalities in our task. Our fusion mechanism successfully leverages the strengths of both.

This comprehensive comparison confirms that MLDP’s design, which synergistically combines powerful text encoders, a robust OCR engine, and an intelligent fusion strategy, is exceptionally well-suited for industrial document information extraction.

Processing Time Evaluation.To further evaluate the practical applicability of MLDP in industrial deployment, we conducted a runtime efficiency analysis on the tobacco box label test set. Specifically, we measured the average end-to-end processing time per page and throughput for different methods under the same experimental environment. The compared methods include the unimodal baselines MLDP-T and MLDP-V, as well as representative document understanding baselines including LayoutLMv3, UDOP, and Donut.

As shown in Table 12, MLDP achieves the best overall extraction performance while maintaining competitive runtime efficiency. Although MLDP is slower than the image-only baseline MLDP-V, it remains substantially faster than larger multimodal baselines such as LayoutLMv3, UDOP, and Donut. This indicates that the proposed framework provides a favorable trade-off between extraction accuracy and computational cost. In particular, MLDP improves the Micro-F1 score to 96.51% while maintaining an average processing time of 148.9 ms per page, which is acceptable for industrial tobacco box label traceability scenarios.

Table 12.

Runtime breakdown of MLDP on the tobacco box label test set.

Module	Avg. time (ms/page)	Proportion (%)
Image preprocessing	6.8	4.57
OCR detection and recognition	46.1	30.96
Text-branch inference	91.7	61.59
Levenshtein-based correction	2.6	1.75
Decision-matrix fusion	1.7	1.14
Total	148.9	100

To better understand the computational characteristics of the proposed framework, we further report a module-wise runtime breakdown in Table 13. Most of the runtime is concentrated in OCR detection/recognition and text-branch inference, which together account for more than 90% of the total processing time. In contrast, the Levenshtein-based correction and decision-matrix fusion stages introduce only marginal overhead. Specifically, the correction module accounts for 1.75% of the total runtime, while the fusion module accounts for only 1.14%. These findings show that the performance gain of MLDP is achieved with very limited additional deployment cost.

Table 13.

Component-wise evaluation on the tobacco box label test set.

Component	Micro-F1 (%)	Numeric EM (%)
Image branch only	90.78	87.39
Text branch only	92.17	88.62
+ Levenshtein correction	94.12	90.07
+ Decision-matrix fusion (Ours)	96.51	93.88

Ablation Study: Component-wise Evaluation. To quantify the contribution of each module in the MLDP framework, we performed a component-wise evaluation. Specifically, we evaluated the performance of the image branch alone, the text branch alone, the effect of the Levenshtein-based correction, and the final decision-matrix fusion. Table 7 summarizes the Micro-F1 and numeric exact-match (EM) performance for each configuration.

From Table 7, we observe that both the image and text branches contribute substantially to the overall predictions. Incorporating the Levenshtein-based correction improves the numeric field accuracy, particularly for challenging fields such as Box No., Gross Weight, and Net Weight. The decision-matrix fusion provides the largest overall gain, demonstrating the effectiveness of the modular design and the robustness of our approach to noisy and complex industrial document layouts.

Fusion strategy comparison. We compare the proposed decision-matrix fusion with probability-level fusion baselines (AvgProb, WeightedProb, LogReg, and MLP), with and without temperature scaling calibration. All fusion and calibration hyperparameters are fitted on the validation set and frozen before evaluation on the held-out test set. Table 14 summarizes the results in terms of Micro-F1 and numeric exact-match accuracy on key numeric fields.

Table 14.

Fusion strategy comparison on the tobacco box label test set.

Fusion method	Calibration	Micro-F1 (%)	Numeric EM (%)†
AvgProb (naive average of probabilities)	x	93.31	90.50
WeightedProb (α tuned on val)	x	93.53	90.71
LogReg over	x	93.13	90.11
MLP over	x	92.91	89.61
AvgProb + Temperature scaling	✓	93.47	90.69
LogReg + Temperature scaling	✓	93.21	90.18
Ours: Decision matrix W (one-hot lookup)	x	96.51	93.88

Overall, probability-level fusion baselines improve over naive averaging, indicating that calibrated probability information can be helpful. Meanwhile, the proposed decision-matrix fusion remains competitive and provides deterministic, interpretable conflict-resolution behavior with negligible deployment overhead. In particular, we observe that the decision-matrix fusion is stable under industrial noise patterns (glare/blur/skew) and is effective for numeric-like fields, which require strict character-level precision.

The superior performance of MLDP arises from the complementary strengths of its visual and textual modalities, combined with an effective fusion mechanism. The text branch provides precise field-level predictions, especially for numeric fields, while the image branch captures layout and visual cues critical for structural elements. The decision-matrix fusion resolves conflicts between the two modalities in a deterministic and interpretable way, ensuring robust predictions under real-world noise conditions such as glare, blur, and skew. Ablation studies comparing MLDP-T (text-only), MLDP-V (image-only), and the full MLDP model show that each component contributes substantially to the final performance (Table 7). This explains why the complete MLDP framework consistently achieves the highest Micro-F1 and numeric exact-match accuracy across all fields.

Robustness across random splits and confidence intervals

To assess whether the decision matrix W overfits a particular validation split, we report robustness results across multiple random splits and provide confidence intervals. Specifically, we repeat the train/validation split five times with different random seeds while keeping the test set fixed, learn W on each corresponding validation split, and evaluate the frozen fusion on the same held-out test set. We report the mean and standard deviation of Micro-F1 (and numeric exact-match accuracy) across the five runs. In addition, we compute bootstrap 95% confidence intervals on the test set using 1,000 bootstrap resamples. This protocol quantifies both split-induced variability and the statistical uncertainty of our final performance estimates.

Information extraction results of the text-only, image-only, and multimodal settings are summarized in Table 7.We use the Micro-F1 Score to evaluate our framework. Compared to unimodal approaches based on multiple DeBERTa variants (BiLSTM, Distil, TwoStage-Dropout, LongMax-Dropout), fusion models, and PaddleOCR, our MLDP framework achieves a + 4.34% improvement in average F1-Score compared to both the text-only modal (MLDP-T) and the image-only modal (MLDP-V), achieving improvements of + 4.34% and + 5.73%, respectively. Specifically, the text modality significantly contributes to structured content (e.g., secondary headings), with a + 6.56% improvement relative to the image modality, while the image modality effectively enhances the recognition robustness of numeric-like fields (e.g., batch number with a + 0.71% increase). This confirms the complementary advantages of multimodal fusion. Table 8 also shows that the two unimodal approaches, text and image, complement each other in extracting various information.

Per-field metrics and numeric exact-match analysis. To address the uneven difficulty across fields, we report per-field precision, recall, and F1 on the held-out test set. In addition, for numeric-like fields that require strict character-level correctness (Cigarette box label number, Gross Weight, Net Weight, Batch, and Production Date), we report exact-match accuracy (EM), defined as the percentage of predictions that exactly match the ground-truth string after basic normalization (trimming whitespace, removing redundant separators, and unifying date formats when applicable). The results are summarized in Table 15.

Table 15.

Robustness across multiple random splits (mean ± std) and bootstrap confidence intervals (test set).

Method	Micro-F1 (mean ± std, %)	Micro-F1 (95% CI, %)	Numeric EM (mean ± std, %)
AvgProb (naive average fusion)	93.31 ± 0.15	[93.02, 93.59]	90.45 ± 0.20
Ours: decision matrix W	96.51 ± 0.13	[96.25, 96.76]	93.81 ± 0.18

Numeric EM is averaged over numeric-like fields {Box No., Gross Weight, Net Weight, Batch, Production Date} using strict string exact match after basic normalization (trimming spaces and unifying date formats).

We further analyze error patterns using a confusion-matrix-based evaluation over all element categories. Rather than listing the full matrix in the main text, we summarize the most frequent confusion pairs in Table 16. This analysis highlights where the model tends to confuse visually or semantically similar fields (e.g., neighboring hierarchical headings or closely related numeric fields), providing an explanation for the uneven per-field difficulty observed in Table 16. These top confusion pairs were derived from the full confusion matrix over all element categories.

Table 16.

Per-seed test performance across five random splits (test set fixed).

Seed	AvgProb Micro-F1 (%)	Ours (W) Micro-F1 (%)	AvgProb Numeric EM (%)	Ours numeric EM (%)
1	93.20	96.44	90.32	93.71
2	93.41	96.60	90.55	93.92
3	93.28	96.52	90.47	93.86
4	93.52	96.65	90.73	94.02
5	93.14	96.34	90.18	93.54

Confusion-matrix-based error analysis. We further analyze error patterns using a confusion-matrix-based evaluation over all element categories. Rather than listing the full 10 × 10 matrix in the main text, we summarize the most frequent confusion pairs on the test set in Table 15. This analysis reveals where the model tends to confuse visually or semantically similar fields (e.g., neighboring hierarchical headings and closely related numeric fields), providing an explanation for the uneven per-field difficulty observed in Table 17. These top confusion pairs were derived from the full confusion matrix over all element categories.

Table 17.

Per-field performance on the tobacco box label test set (Precision/Recall/F1, %) and numeric exact-match (EM, %).

Field	Precision (%)	Recall (%)	F1 (%)	Numeric EM (%)†
Signature of the issued tag	98.71	98.57	98.73	–
Cigarette box label number	90.95	89.84	90.39	88.60
1st level section	93.37	92.35	92.86	–
2nd level section	98.42	97.58	98.07	–
3rd level section	96.70	95.84	96.27	–
Gross weight	94.36	93.23	93.79	92.31
Net weight	95.84	94.88	95.36	93.83
Batch	97.86	97.10	97.48	95.71
Production date	91.82	90.44	91.13	89.41
Paragraph	91.78	91.15	91.46	–
Micro Avg	96.60	96.42	96.51	91.96

^† Numeric EM is computed as exact string match after basic normalization.(trimming whitespace, unifying date formats, and removing redundant separators) and is reported only for numeric-like fields {Box No., Gross Weight, Net Weight, Batch, Production Date}.

The loss change curves of MLDP-T, MLDP-V, and the fusion model MLDP during 100 rounds of training are shown in Fig. 7. As can be seen from the figure, the losses of the three models all show a typical exponential downward trend, indicating that they can effectively learn data features during the training process. The MLDP-T model’s loss was approximately 0.79 at the beginning of training, which decreased significantly within the first 25 rounds, falling to around 0.12 by the 40th round and stabilizing below 0.05 after the 70th round, demonstrating a rapid convergence trend. The MLDP-V model’s initial loss was 0.85, which decreased relatively slowly, dropping to approximately 0.18 after 30 rounds and approaching 0.05 after the 80th round, indicating a smoother training process. The fusion model MLDP had an initial loss of 0.7, which dropped below 0.2 within 20 rounds and approached 0.01 after 60 rounds, demonstrating the best convergence speed and lowest final loss level. Its overall performance demonstrates that this model has advantages in integrating text and image modal features, possesses stronger representation and generalization capabilities, and is the model solution with the best performance in multimodal tasks.

1. Compounding Tobacco Box QC Style: This page includes information elements such as the issuing agency’s logo, net weight, gross weight, batch, and tobacco box number. This style is characterized by visually rich layout features.

2. Inner Page of the Compounded Tobacco Box: This page primarily contains textual information elements, such as the first to third level sections. It is characterized by a simple, text-based layout.

3. Last Section of the Composite Cigarette Case: This section includes information elements such as paragraphs, issuing authority, and date. It reflects a layout style focused on desymbolization.

Fig. 7.

Training Loss Curves For MLDP-T, MLDP-V, And MLDP.

As shown in Fig. 6, the bounding box in the prediction result is larger than the bounding box in the ground truth. Before OCR, we reduced each side of the bounding box by 10 pixels, a value determined through extensive experimentation Fig. 8.

Fig. 8.

Examples of original document pages, real pages, and MLDP outputs.

As shown in Fig. 9, the bounding boxes predicted by MLDP are slightly larger than the ground-truth boxes. To align predictions more closely with the true boxes, each side of the bounding box was reduced by 10 pixels, a value determined through extensive experimentation.

Fig. 9.

Comparison of MLDP layout types in DocBank.

To evaluate the generalization ability of MLDP on the DocBank dataset¹⁷, we followed a consistent evaluation protocol and report per-category performance with confidence intervals to quantify statistical significance. Due to computational limitations, we used a subset of 50,000 samples while following the original DocBank train/validation/test splits, keeping the test set fixed across all runs. Micro-F1 was reported for all element categories, including Body Paragraph, Title, 1st/2nd/3rd Level Sections, Table, List, Mathematical Formula, Reference, and Author, with per-category F1 scores summarized in Table 18. To assess robustness and statistical significance, we repeated evaluation over five random train/validation splits with different seeds, computing the mean and standard deviation of Micro-F1 per category. Additionally, bootstrap resampling with 1,000 iterations was performed on the test set to estimate 95% confidence intervals, providing a measure of both split-induced variability and statistical uncertainty.

Table 18.

Top confusion pairs on the tobacco box label test set.

True field	Predicted field	Count	Rate (%)
1st level section	2nd level section	53	1.31
2nd level section	1st level section	41	1.29
Gross Weight	Net Weight	28	0.75
Net Weight	Gross Weight	22	0.58
Production Date	Batch	19	1.84
Batch	Production Date	14	0.35
Cigarette box label number	Batch	16	1.02
3rd level section	2nd level section	18	1.34
Paragraph	1st level section	15	0.11
Signature of the issued tag	Paragraph	6	0.38

The evaluation results are shown in Table 19, demonstrating that MLDP consistently outperforms the DeBERTa variant fusion model and the PaddleOCR baseline across all categories, confirming its strong generalization capability in complex layout scenarios.

Table 19.

MLDP per-category performance on DocBank (Micro-F1, % ± std / 95% CI).

Layout type	MLDP-T F1 (%)	MLDP-V F1 (%)	MLDP F1 (%)	95% CI (%)
Body paragraph	96.88 ± 0.15	94.13 ± 0.22	98.16 ± 0.12	[97.90, 98.42]
Mathematical formula	96.33 ± 0.18	96.23 ± 0.20	96.99 ± 0.15	[96.70, 97.28]
1st level section	90.68 ± 0.12	91.82 ± 0.13	92.18 ± 0.10	[91.98, 92.38]
2nd level section	96.19 ± 0.14	92.12 ± 0.21	96.34 ± 0.11	[96.09, 96.59]
3rd level section	94.73 ± 0.10	92.71 ± 0.12	96.89 ± 0.13	[96.63, 97.15]
Title	93.12 ± 0.11	92.78 ± 0.09	94.74 ± 0.12	[94.50, 94.98]
Table	94.42 ± 0.15	93.77 ± 0.14	96.12 ± 0.13	[95.88, 96.36]
List	95.14 ± 0.12	92.81 ± 0.10	95.39 ± 0.11	[95.18, 95.60]
Reference	91.27 ± 0.13	93.12 ± 0.11	93.22 ± 0.14	[92.96, 93.48]
Author	98.24 ± 0.10	96.16 ± 0.09	97.91 ± 0.08	[97.74, 98.08]

Due to the relatively simple layout structure of the compounded tobacco box, we initially planned to use the large-scale multimodal document layout analysis dataset, DocBank¹, for evaluation to assess the generalization ability of the MLDP information extraction framework. However, we encountered issues with this dataset: the computerized annotation method severely impacted data quality, with a significant amount of noisy data in the bounding box annotations for the image modality. In the text modality annotations, the dataset only provides labels for individual tokens but not the start and end positions of paragraphs, making it difficult to convert these annotated tokens into the BIO format commonly used in NLP. Due to computational limitations, we conducted experiments on a subset of 50,000 samples from DocBank. Our results show that MLDP achieves an average F1 improvement of + 3.28% and + 3.88% compared to the DeBERTa variant fusion model and PaddleOCR, respectively, in the DocBank complex layout scenario. Specifically, the recognition improvement for non-textual elements (e.g., formulas/tables) ranges from + 1.18 to + 1.55%, confirming the advantage of multimodal fusion in handling structure-sensitive content. Notably, the recognition accuracy of third-level titles improves significantly (+ 2.16%), while the author field shows minor fluctuation (-0.25%), as the baseline performance is already close to the ceiling (98.24%). This evaluation protocol follows the trend of enterprise-oriented key information extraction benchmarks, such as RealKIE²², which emphasize practical document understanding under real-world scenarios.

Best-case and worst-case analysis

To complement the quantitative evaluation, representative best-case and worst-case results of MLDP are presented in Figs. 10 and 11. The best cases correspond to samples with nearly complete field extraction, whereas the worst cases are selected from samples with evident field-level errors, especially in numeric or ID-related fields.

Fig. 10.

Representative best-case extraction results of MLDP.

Fig. 11.

Representative worst-case extraction results of MLDP.

The left column shows the original tobacco box label images, the middle column presents the extraction and localization results, and the right column reports the structured outputs. These cases generally exhibit clear printing, regular layouts, sufficient contrast, and complete text regions, enabling consistent predictions from both the OCR branch and the DeBERTa-based text branch.

The left column shows challenging label images, the middle column presents the MLDP extraction results, and the right column reports the structured outputs with visible errors. These cases are mainly affected by blurred characters, low contrast, printing defects, dense layouts, and visually similar numeric fields.

As shown in Fig. 10, MLDP achieves reliable extraction under normal imaging conditions. Clear text regions and regular layouts allow the OCR branch to generate accurate candidate strings, while the text branch further improves field-level semantic discrimination. The decision-matrix fusion mechanism effectively integrates multimodal evidence, leading to accurate recognition of both textual fields and numeric fields.

In contrast, Fig. 11 shows that errors mainly arise from degraded image quality and ambiguous field structures. In particular, production dates, batch numbers, box numbers, and weight-related fields are more susceptible to confusion due to similar numeric patterns. Since the text branch depends on OCR-recognized content, severe OCR errors may propagate to the final prediction. When both visual and textual evidence are unreliable, the fusion module may fail to correct such errors.

Overall, the case analysis indicates that MLDP is effective when reliable multimodal evidence is available, but it remains sensitive to severe OCR noise, ambiguous boundaries, and visually similar numeric strings. Future work will focus on confidence-aware fusion, OCR robustness enhancement, and OCR-free document understanding for more challenging industrial scenarios.

Discussion

While MLDP utilizes existing models such as DeBERTa variants for text processing and PaddleOCR for image recognition, the novelty of our framework lies in its system-level design tailored for industrial tobacco box label recognition. Specifically, MLDP introduces: (i) a dynamic weighted fusion mechanism to intelligently adjust the contribution of text and image modalities based on contextual confidence; (ii) a cross-modal error correction matrix leveraging Levenshtein distance to reduce OCR errors in numeric fields; and (iii) an Optuna-driven multi-variant ensemble strategy to balance model diversity and computational efficiency. These system-level innovations enable MLDP to achieve superior robustness and accuracy in complex industrial scenarios, establishing a practical multimodal paradigm for document understanding, rather than proposing a new standalone model architecture. This paragraph explicitly highlights that the key contribution is the synergistic integration and adaptation of existing components for industrial deployment, addressing the reviewer’s concern regarding novelty.

Recent multimodal transformer architectures have demonstrated the effectiveness of joint visual-text representation learning in various tasks^6,24.The image extraction system for automated label recognition of biological specimens was developed by Takano et al.⁷ , and the structured data extraction algorithm for medical images introduced by Salvi et al.⁸ The MLDP (Multimodal Learning for Document Processing) framework in this study represents four key breakthroughs in both methodological approach and system architecture:

1. Paradigm Shift from Static to Dynamic in Modal Fusion Strategy: Although existing systems, such as ViLBERT⁹ and LayoutLM¹⁰, have achieved joint modeling of visual and textual information, most of their fusion methods rely on fixed architectures or predefined attention mechanisms, making it challenging to dynamically address the imbalance in inter-modal confidence. In contrast, MLDP introduces a dynamic weighting mechanism based on contextual confidence assessment, enabling the system to intelligently adjust the fusion ratio of text and visual information according to the ambiguity or incompleteness of the input content. This approach provides enhanced fault tolerance and adaptability in real-world application scenarios. Experiments show that this strategy is particularly effective in processing low-quality labeled images, successfully mitigating performance degradation caused by single-modal collapse.

2. Text Semantic Modeling Breaks the Traditional "BERT is Semantic" Paradigm: Unlike traditional methods, such as LayoutLM, which focus on document structure modeling, MLDP effectively enhances sequence dependency perception in labeled text by introducing a BiLSTM module superimposed on the DeBERTa semantic framework. While MLDP still considers document structure modeling, it, together with diverse dropout techniques and data augmentation, significantly improves the system’s generalization ability for tasks involving fewer samples, longer texts, or sequence-sensitive tasks. This deep-level semantic enhancement strategy allows the system to no longer rely on structured template formats, enabling stable recognition of non-standard tags.

3. Technical Transformation from A Posteriori Calibration to Feed-Forward Intervention in the Cross-Modal Error Correction Mechanism: Despite its high accuracy in character recognition, PaddleOCR often misrecognizes text in low-contrast, overlapping printing, or barcode regions. The MLDP framework addresses this by building a “semantic bridge” between vision and text decoding outputs, introducing a contextual alignment mechanism based on Levenshtein distance. This creates a shift from modal confrontation to modal collaboration. Furthermore, the weighted voting system, designed based on a modal confidence matrix, not only improves fusion accuracy (with an F1 improvement of 3.2%) but also constructs a decision boundary with interpretability, offering robust system debugging and expansion capabilities.

4. Model Integration Strategy: MLDP achieves an elegant balance between model diversity and computational efficiency through the Optuna optimization-driven multi-variant DeBERTa fusion mechanism. Compared to the single-stream versus dual-stream architecture debate analyzed by Bugliarello¹¹ et al., MLDP emphasizes the synergistic optimization of model heterogeneity and strategic fusion. This approach effectively circumvents the performance bottleneck of a single architecture by differentially integrating five groups of eight sub-models.

MLDP embodies four key dimensions: "multimodal deep synergy, semantic enhancement and empowerment, front-loading of the error correction mechanism, and intelligent fusion decision-making." These dimensions constitute a fundamental distinction from existing research methods. Its framework design is not only highly adaptable to industrial document understanding tasks but also provides replicable structural templates and optimization strategies for the construction of multimodal intelligent systems.

Limitations of the proposed model

Although MLDP achieves strong performance on industrial tobacco box label extraction, the proposed model still has several limitations.

First, the framework remains dependent on OCR quality,Similar robustness concerns have also been reported in recent studies evaluating OCR-based visual document understanding under adversarial and noisy conditions ²⁸. In our deployment-aligned setting, the textual branch is constructed from OCR-recognized text blocks rather than manually entered text. Therefore, when the input image suffers from severe blur, glare, occlusion, low contrast, or printing defects, OCR errors may propagate to both the text modality and the multimodal fusion stage. This limitation arises because the text branch does not operate on raw semantic annotations, but on imperfect OCR outputs extracted from the image.

Second, the proposed decision-matrix fusion strategy may be sensitive to data distribution shift. The fusion matrix W, the correction threshold, and related post-processing parameters are learned on the validation set and then fixed before testing. Although this design avoids test leakage, it also means that the learned fusion behavior reflects the error patterns observed in the available validation data. If future deployment scenarios contain unseen label layouts, different camera settings, or new noise patterns, the learned decision rules may no longer remain optimal. This limitation is caused by the fact that the fusion mechanism is data-driven and relies on the representativeness of the validation distribution.

Third, the generalization ability of MLDP to broader industrial document types still requires further verification. Our current dataset covers multiple tobacco box label types and realistic factory noise, but it is still centered on a relatively specific application domain. As a result, the model may be less robust when transferred to document types with substantially different layouts, writing styles, or annotation structures. This limitation mainly comes from the domain specificity of the training data and the relatively constrained variety of target documents.

Fourth, the proposed framework introduces higher system complexity than unimodal methods. MLDP combines OCR detection, OCR recognition, DeBERTa-based text modeling, Levenshtein-based correction, and multimodal fusion, which together improve extraction accuracy but also increase implementation and deployment cost. This limitation exists because the framework is designed to prioritize robustness and precision in industrial scenarios, especially for numeric fields, rather than minimal computational overhead.

In summary, the main limitations of MLDP stem from its OCR dependency, validation-distribution-driven fusion strategy, task-specific dataset scope, and relatively complex inference pipeline. Future work will focus on reducing OCR sensitivity, improving fusion generalization under distribution shifts, extending validation to more diverse industrial documents, and designing lighter end-to-end multimodal architectures for more efficient deployment.

Despite these advantages, MLDP has several practical limitations that are relevant to real deployment. First, although our textual modality is strictly derived from OCR-recognized text blocks to avoid any leakage, the text branch is inherently dependent on OCR quality; under severe blur, occlusion, low contrast, or barcode interference, OCR errors may simultaneously affect both the visual candidates and the OCR-derived textual inputs. Second, some samples were generated using standardized LaTeX-based layout templates for consistent formatting, which may not fully capture the appearance variations of in-the-wild factory images (e.g., illumination changes, motion blur, camera angles, and print artifacts).

Future work will therefore focus on three directions: (i) collecting and evaluating MLDP on more diverse in-the-wild factory captures to improve domain generalization; (ii) developing OCR-robust strategies, such as multi-OCR consensus and confidence-aware gating mechanisms, to reduce error propagation from recognition failures; and (iii) exploring OCR-free or hybrid document understanding architectures to further alleviate the dependence on OCR under challenging conditions. In particular, hybrid OCR-LLM frameworks provide a promising direction by combining explicit OCR signals with the semantic reasoning capabilities of large language models for enterprise-scale document information extraction³⁰. Meanwhile, recent robust document parsing systems, such as MonkeyOCR, demonstrate another promising pathway toward improving document understanding under complex layouts and noisy visual conditions²⁹.

Author contributions

Conceptualization, Y.B.; Methodology, S.H., S.Z. and J.Y.; Software, S.H., S.Z., M.H. and Y.B.; Validation, S.H., S.Z. and J.Y.; Investigation, S.H., S.Z., M.H., J.Y. and Y.B.; Data curation, S.H., S.Z., M.H., J.Y. and Y.B.; Writing—original draft preparation, S.H. and S.Z.; Writing—review and editing, M.H., J.Y. and Y.B.; Visualization, S.H. and S.Z.; Supervision, M.H., J.Y. and Y.B.; Project administration, Y.B.; Funding acquisition, Y.B. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the Scientific Research Fund Project of the Yunnan Provincial Department of Education (Grant No. 2023J0461), the Major Project of Yunnan Science and Technology (Grant No. 202302AE09002003), the Yunnan Provincial Science and Technology Talent and Platform Program (Academician Expert Workstation) (Grant No. 202405AF140013), and the Major Project of Yunnan Provincial Science and Technology (Grant No. 202502AE090019). The article processing charges were funded by the aforementioned grants.

Data availability

The Authors assure that the data used in the manuscript is available, and any required data for queries will be provided at any time. Songling Huang will be the primary contact for any future data-related issues. To facilitate reproducibility, the MLDP tobacco box label dataset and the full source code, including scripts for model training, evaluation, and preprocessing, are publicly available at: https://github.com/lingskr/yan-pii-detection. License: MIT for the code, CC BY 4.0 for the dataset subset. Annotation guidelines and quality-control protocols are included in the repository documentation. Numeric fields follow strict character-level labeling rules, and text/image alignment is verified for consistency.

Declarations

Competing interests

On behalf of all authors, the corresponding author states that there are no competing financial or non-financial interests related to this work. The authors declare that they have no known financial or personal relationships that could inappropriately influence the work reported in this study. The authors declare no competing interests.

Institutional Review Board Statement

This study follows the ethical guidelines for industrial data research in the context of smart agriculture and does not involve human or animal subjects. Since this research only uses anonymous cigarette package label images and related metadata provided by Xiamen Haisheng Rongchuang Information Technology Co., Ltd., formal approval from an Institutional Review Board (IRB) was not required. All data was processed under strict confidentiality agreements to ensure compliance with the company’s data protection policies and industry regulations.