Deployable real-time spinal endoscopic instance segmentation with lightweight multi-scale attention mechanism

Abstract

Real-time instance segmentation in spinal endoscopy is vital for identifying and protecting key anatomy, but is hampered by narrow views, specular highlights, smoke/bleeding artifacts, fuzzy boundaries, and large-scale variation. Deployment further demands accuracy, speed, and stability under small-batch (often batch-size-one) settings. We present EndoSeg-RT, a deployable real-time framework co-designed across backbone, neck, and head with lightweight multi-scale attention. The backbone, C2f-Pro, combines RepViT-style re-parameterized convolutions with efficient multi-scale attention: multi-branch training for rich features that collapses to a single low-latency path at inference. The neck strengthens cross-scale consistency and boundary quality via Scale-Sequence Feature Fusion and Triple Feature Encoding, enhancing high-resolution anatomy. The head uses a Lightweight Multi-task Shared Head with shared convolutions and GroupNorm to cut redundancy and stabilize batch-size-one training/inference. We also release a clinically reviewed PELD dataset of 610 images from 61 patients with instance-level labels for adipose tissue, bone, ligamentum flavum, and nerve. Extensive experiments show EndoSeg-RT delivers competitive or superior accuracy with markedly lower complexity—only 1.8M parameters and 8.8 GFLOPs—outperforming heavier instance segmentation baselines. The model generalizes well to a public dental instance segmentation benchmark, demonstrating robustness beyond spinal endoscopy. Code and dataset are publicly available at: https://github.com/hhwmortal/PELD-Instance-segmentation.

Introduction

Spinal endoscopy has become an essential modality in minimally invasive spine surgery, offering reduced tissue disruption, faster postoperative recovery, and improved access to delicate anatomical regions^1–3. During these procedures, surgeons operate within a constrained visual environment and must continuously identify and preserve critical anatomical structures-including the ligamentum flavum, neural elements, osseous landmarks, and surrounding adipose tissue-to avoid iatrogenic injury and ensure procedural safety⁴. Consequently, deployable real-time and accurate delineation of anatomical structures directly from endoscopic video streams represents a key unmet clinical need for intraoperative guidance and decision support.

Instance segmentation is particularly well suited to this setting, as it provides pixel-level localization while explicitly distinguishing individual anatomical entities, enabling actionable visual overlays for navigation, dissection planning, and risk awareness during surgery^5–11. However, spinal endoscopic imaging constitutes an exceptionally challenging visual domain. Images are characterized by a narrow field of view, non-uniform illumination, frequent specular highlights, smoke and bleeding artifacts, ambiguous tissue boundaries, strong inter-class similarity, and pronounced scale variation across structures¹². Achieving robust, low-latency instance segmentation under these conditions-while remaining deployable on resource-constrained surgical hardware-remains a major barrier to clinical translation¹³.

Recent advances in deep learning have driven substantial progress in medical image segmentation^14,15, evolving from semantic labeling to instance-level understanding^16–18. Two-stage frameworks such as Mask R-CNN⁵ often provide strong accuracy, but their computational complexity and inference latency limit their suitability for real-time intraoperative use¹⁹. Single-stage methods, including YOLACT⁶, SOLOv2⁷, and recent YOLO-based variants^11,20, offer improved speed and deployability but frequently struggle with boundary precision and sensitivity to small, clinically critical structures in visually complex endoscopic scenes. Transformer-based and hybrid architectures enhance global context modeling^21,22, yet their memory and computational demands complicate stable deployment on edge devices such as endoscopy workstations. Despite numerous lightweight design strategies, there remains no end-to-end solution that simultaneously delivers fine-grained anatomical accuracy, consistent multi-scale fusion, and reliable real-time performance tailored to spinal endoscopy.

These challenges give rise to three persistent limitations in existing approaches. First, the persistent accuracy–efficiency trade-off under surgical constraints means that models with high accuracy are often too computationally heavy, while lightweight networks tend to degrade on fine boundaries and small structures such as thin ligaments or nerve margins. Second, insufficient multi-scale and fine-grained feature fusion leads to semantic–spatial misalignment, resulting in mask leakage or missed instances under glare, noise, and high tissue similarity. Third, limited robustness under small-batch training and domain variability—including batch-size sensitivity, class imbalance, and variations in anatomy and imaging conditions—reduces generalization and deployability.

To address these challenges, we propose EndoSeg-RT, a deployable real-time spinal endoscopic instance segmentation framework with lightweight multi-scale attention. EndoSeg-RT adopts a deployment-driven, top-down design that co-optimizes the backbone, neck, and head to balance anatomical fidelity with computational efficiency. The backbone alleviates the accuracy-efficiency bottleneck, the neck strengthens multi-scale and boundary-aware fusion, and the head improves stability under small-batch and real-time inference conditions typical of intraoperative deployment.

At the backbone level, EndoSeg-RT introduces C2f-Pro, which integrates Vision-Transformer-inspired re-parameterized blocks and efficient multi-scale attention into a lightweight convolutional framework. Multi-branch pathways enrich representation during training, while structural re-parameterization collapses them into a single low-latency path at inference, preserving complementary local and global context without the overhead of full transformer models. Building on this compact representation, the neck incorporates Scale-Sequence Feature Fusion (SSFF) and Triple Feature Encoding (TFE) to enhance cross-scale coherence and sharpen anatomical boundaries, particularly for small or visually ambiguous structures. A shared, GroupNorm-based segmentation head further ensures stable performance under batch-size-one inference, which is common in real-time surgical settings.

In summary, this work makes three key contributions.

• i.we introduce EndoSeg-RT, a lightweight and deployable framework that achieves a SOTA accuracy-efficiency trade-offfor real-time spinal endoscopic instance segmentation, requiring only 1.8 M parameters and 8.8 GFLOPs while operating at 189.3 FPS.
• ii.we present a set of deployment-oriented architectural components-including C2f-Pro, SSFF, TFE, and a lightweight multi-scale segmentation head-that jointly improve boundary fidelity, sensitivity to small anatomical structures, and robustness under clinical constraints.
• iii.we curate a clinically reviewed PELD spinal endoscopy dataset comprising 610 images from 61 patients with pixel-level instance annotations for key anatomical structures, enabling standardized evaluation and future benchmarking.

Results

Dataset definition

To evaluate EndoSeg-RT under realistic intraoperative conditions, we built a spinal endoscopic instance-segmentation dataset from percutaneous endoscopic lumbar discectomy (PELD) cases recorded on an IPS710A endoscopic system at Xiangyang Central Hospital. Digital videos from 61 patients were exported and parsed into frames. A standardized sampling-and-filtering pipeline was then applied: every video was sampled at 25 frames per second, followed by removal of frames with motion blur, severe overexposure, dominant instruments, or insufficient anatomical content. After screening, we retained 10 images per patient, yielding a total of 610 images at 1080 × 720 resolution (approximately 10 frames per case). The dataset can be download on https://drive.google.com/file/d/1Ha-UWBhVUrsgJWLx5Eo4igPxtwlViqcy/view.

Annotation protocol

Curated frames were annotated in LabelMe under the guidance of spine surgeons. Four clinically relevant structures were labeled with pixel-wise, instance-level masks: adipose tissue, bone, ligamentum flavum, and nerve (see Fig. 1). To control selection bias and enforce consistency, all data were manually annotated by two trained raters (R1, R2; PELD experience > 10 years) using a standardized protocol and nomenclature reviewed by a senior spine surgeon. An initial calibration phase with feedback ensured consistency. For each case, R1 and R2 produced the primary instance masks and class labels; disagreements were flagged and resolved in weekly adjudication meetings led by the expert (PELD experience > 15 years), with final masks approved by the expert once consensus was reached. We measured inter-rater agreement on a 10% double-annotated subset: mask-level agreement using Dice and boundary F₁ score, and instance matching via IoU≥0.5. Mean Dice across classes is reported in the paper; per-class agreement and example adjudication cases are provided in the supplement. Colors are used only for visualization (purple: adipose; red: bone; yellow: ligamentum flavum; green: nerve). The training pipeline consumes masks exclusively without relying on color cues.

Fig. 1. Representative samples from the PELD instance-segmentation dataset.

Top row: raw endoscopic frames captured at 1080 × 720 resolution. Bottom row: corresponding pixel-wise, instance-level annotations for four surgical structures-adipose tissue (purple), bone (red), ligamentum flavum (yellow), and nerve (green). Examples illustrate typical variations in viewpoint, illumination, bleeding, and specular highlights encountered during percutaneous endoscopic lumbar discectomy.

Procedural coverage and visual challenges

These labeled images span the typical PELD workflow and its visual challenges (Fig. 2). Early frames commonly show the establishment of the working canal with the ligamentum flavum covering deeper layers; subsequent steps include partial flavectomy that exposes subcutaneous fat and adjacent neural tissue; further clearance reveals bony landmarks and an intact nerve; and the procedure concludes with adequate decompression of the compressed nerve root. Consequently, the dataset naturally contains narrow fields of view, specular highlights, irrigation/blood contamination, and frequent instrument occlusions-conditions that stress-test small-structure segmentation.

Fig. 2. Percutaneous endoscopic lumbar discectomy (PELD) procedural progression.

A Establishment of the working channel with ligamentum flavum obscuring deeper structures. B Partial flavectomy exposing subcutaneous fat and adjacent neural tissue. C Further clearance revealing bony landmarks and an intact nerve. D Adequate decompression of the compressed nerve root. Typical endoscopic artifacts include narrow field of view, specular highlights, irrigation/blood contamination, and frequent instrument occlusions.

Per-class pixel distribution

To provide a more comprehensive view of the PELD dataset, we report the percentage of image area occupied by each class per image (Fig. 3). Figure 3 presents box-and-whisker plots overlaid with individual samples. This complements Table 1’s instance counts and reveals clear class imbalance: Ligament occupies the largest average share with wide variability (median ≈ 14%); Nerve (the primary target) shows moderate occupancy with a long upper tail (median ≈ 8%); Bone and Adipose have lower median coverage (median ≈ 5%). We include these distributions in the dataset card and release per-image statistics to facilitate stratified sampling, class-balanced batching, and cost-sensitive training.

Fig. 3. Per class pixel occupancy per image (%).

Box and whisker plots with jittered points summarize the distribution of mask area as a percentage of image area for Bone, Ligament, Adipose, and Nerve.

Table 1.

Dataset split and per-class instance counts

# Class	Train	Val	Test	Total
Adiposity	353	40	42	435
Bone	224	28	25	277
Ligament	433	45	32	510
Nerve	365	47	43	455
# Images	490	60	60	610

# Class and # Images denote the number of classes and the number of images in each split, respectively.

The split is stratified to balance class presence across subsets.

Percentage of labeled pixels per class

Complementing, Fig. 4 provides the dataset-level percentage of labeled pixels per class, showing dominance by Ligament (39.0%), followed by Nerve (27.7%), Adipose (17.6%), and Bone (15.7%). Together, these distributions support class-aware sampling, loss reweighting, evaluation stratified by tissue type, and the design of balanced training batches. We release per-image pixel totals and class proportions in the dataset card to facilitate reproducible and scale-aware experimentation.

Fig. 4.

Percentage of labeled pixels per class across PELD dataset.

Instance size distribution across the PELD dataset

To characterize object scale, we report the instance size distribution using the normalized scale $\sqrt{areainpixels}$ (Fig. 5). This transformation reduces skew and aligns with common detection heuristics. The distribution is right-skewed, with most instances concentrated in the ~90–220 px range and relatively few very small or very large objects. We also mark conventional scale regimes-small (<32 px), medium (<96 px), and larger instances beyond this threshold-to facilitate scale-aware analyses. These statistics are included in the dataset card and released as per-instance metadata to support scale-aware sampling, multi-scale training, and anchor/stride selection.

Fig. 5. Instance size distribution (instance scale = $\sqrt{areainpixels}$.

Bars show counts of annotated instances binned by scale.

Implementation details

Experimental settings

All experiments were conducted on a single NVIDIA GeForce RTX 3060 GPU with 12 GB memory. Our software stack consisted of Python 3.10 and PyTorch 2.10, with CUDA 12.1 for GPU acceleration. Input images were resized to 640 × 640 pixels. We trained each model for 300 epochs using a batch size of 8. Stochastic Gradient Descent (SGD) was adopted as the optimizer with momentum set to 0.9 and an initial learning rate of 1.0 × 10⁻³. Unless otherwise specified, all other hyperparameters followed the defaults of the corresponding implementations.

Dataset split and statistics

For fair and reproducible evaluation across methods, we applied a stratified split that preserves the relative frequency of all four classes. The 610 images are divided into train/val/test with an 8:1:1 ratio (490/60/60). Because a single frame may not include every category, per-class instance counts are lower than the number of images. The final statistics are summarized in Table 1.

Evaluation metrics

Following^14,15, we evaluate models using Dice, IoU, parameter count, FLOPs, and FPS. Dice and IoU quantify segmentation accuracy. The number of parameters reflects model size. FLOPs denote the computational cost for a single forward pass at a specified input resolution. FPS is the inverse of the average per-image inference time and, unless noted otherwise, excludes data loading and post-processing. All FPS measurements are obtained on the same hardware as Section “Implementation details” with a batch size of 1. Parameter counts and FLOPs are reported for the final trained models at an input resolution of 640 × 640 to ensure fair comparison.

Training details

We train with SGD (Nesterov) and a 300-epoch cosine LR schedule. The LR is linearly warmed from 0 to 1.0 × 10⁻³ in the first 5 epochs, then cosine-decays to 1.0 × 10⁻⁶ by epoch 300 (no restarts). We use momentum 0.9, weight decay 5 × 10⁻⁵, and exclude BatchNorm and bias from decay. The same setup is used for SE and EMA, with global batch size 16 and gradient clipping at 1.0. Full configs (optimizer, warmup, cosine params, seeds) are provided in configs/train.yaml.

Image-level results

To comprehensively evaluate the proposed method for instance segmentation on spinal endoscopic images. i.e., PELD dataset, we illustrated in Section “Dataset definition”. We perform comparative experiments on our PELD dataset against representative baselines: Mask R-CNN⁵, SOLOv2⁷, YOLACT⁶, and Swin Mask R-CNN⁸. These models encompass two-stage and one-stage detectors, as well as Transformer-based architectures, offering robust and diverse references. We report Dice, IoU, Parameters, FLOPs, and FPS to jointly assess segmentation quality and model efficiency.

As shown in Table 2, our method attains the best-or-second-best performance across multiple metrics. Traditional two-stage models (e.g., Mask R-CNN⁵) maintain reasonable accuracy but are hindered by heavy parameters and slow inference. One-stage methods (e.g., SOLOv2⁷ and YOLACT⁶) offer higher throughput, yet their segmentation quality degrades in complex scenes. Swin Mask R-CNN achieves competitive accuracy but at substantial computational cost, limiting deployment on resource-constrained devices. In contrast, our approach achieves superior Dice and strong IoU while using considerably fewer parameters and FLOPs, resulting in faster inference than most two-stage counterparts and improving practicality for real-world use.

Table 2.

Quantitative performance and efficiency comparison of instance segmentation methods on the PELD test set

Method	Accuracy				Efficiency
	Dice ↑	IoU ↑	Precision ↑	A.R. ↑	Params (M) ↓	FLOPs (G) ↓	FPS ↑
Mask R-CNN5	67.7	51.1	74.8	61.9	43.9	259.0	13.6
YOLACT6	63.6	46.5	68.5	59.2	34.7	61.3	46.2
SOLOv27	71.3	55.8	75.8	67.3	46.2	157.2	11.3
Swin Mask R-CNN8	71.6	56.0	75.2	68.4	47.4	264.0	9.3
Mask2Former9	71.4	55.6	78.4	65.6	44.4	262.1	17.0
ConvNeXt10	74.0	58.8	77.8	70.6	48.7	268.2	18.8
YOLOv8-seg11	72.7	57.0	75.4	70.3	32.2	120.0	170.5
YOLOv11-seg20	76.0	61.0	78.9	73.3	1.8	8.8	189.38
EndoSeg-RT	74.4	63.0	78.9	73.3	1.8	8.8	189.3

Larger (smaller) values indicate better performance for accuracy (efficiency) metrics. Bold and underlined values denote the best and second-best results per column, respectively.

Qualitative comparisons

Qualitative comparisons on representative spinal endoscopic images further corroborate these findings (Fig. 6): our method achieves a favorable balance between segmentation accuracy and efficiency, underscoring its suitability for spinal endoscopic instance segmentation.

Fig. 6.

Qualitative comparison on the PELD instance-segmentation dataset.

Patient-wise results

We evaluate the model using patient-wise 5-fold cross-validation, where each fold holds out a disjoint set of patients for testing and uses the remaining patients for training/validation. As summarized in Table 3, the model exhibits stable generalization across unseen patients. Fold-to-fold variation is modest, indicating that the method is not overly sensitive to which patients are held out.

Table 3.

Patient-wise 5-fold cross-validation (test splits are disjoint by patient)

Fold	Test patients (IDs)	#Image	#Instance	Dice (%)	IoU (%)	Precision (%)	Recall (%)
1	[1, 2, 3, 4, 5]	160	419	65.4	48.6	70.7	60.9
2	[6, 7, 8, 9, 10]	137	372	66.7	50.1	72.1	62.1
3	[11, 12, 13, 14]	180	448	64.8	47.9	68.2	61.7
4	[15, 16, 17, 18]	76	212	72.2	56.5	84.8	62.9
5	[19, 20, 21, 22]	59	184	66.2	49.4	64.6	67.8
Mean	–	–	–	67.1	50.5	72.1	63.1
Std	–	–	–	3.0	3.5	7.7	2.5

Per-class analyses in Table 4 further confirm robust segmentation quality. Precision/Recall pairs show balanced behavior for the clinically important Nerve class, while Ligament attains high Dice with strong precision (79.4%) and recall (74.4%). Bone also achieves competitive Dice with tight variance, suggesting consistent delineation across patients.

Table 4.

Per-class segmentation performance across 5 folds

# Class	Metrics	Fold1	Fold2	Fold3	Fold4	Fold5	Mean ± std
Adiposity	Dice (%)	57.9	49.3	37.4	48.2	36.7	45.9 ± 8.9
	IoU (%)	40.7	32.7	23.0	31.8	22.5	30.1 ± 7.6
	Pre. (%)	64.2	50.3	50.3	73.6	36.7	55.4 ± 15.4
	Recall (%)	52.7	48.0	29.8	35.6	30.5	40.5 ± 9.5
Bone	Dice (%)	61.1	67.6	64.3	73.5	79.3	69.2 ± 7.2
	IoU (%)	43.9	51.4	50.0	58.3	66.5	54.0 ± 7.3
	Pre. (%)	66.4	73.3	69.0	87.7	68.0	72.9 ± 8.7
	Recall (%)	56.5	61.9	60.2	63.3	95.2	67.6 ± 15.6
Ligament	Dice (%)	72.9	71.9	76.9	78.7	84.7	77.0 ± 4.7
	IoU (%)	57.4	56.1	62.1	64.5	71.3	62.3 ± 5.7
	Pre. (%)	80.9	76.9	70.6	91.6	71.5	79.4 ± 7.6
	Recall (%)	67.8	67.2	83.7	69.1	84.2	74.4 ± 7.1
Nerve	Dice (%)	69.7	77.9	77.8	84.7	81.9	78.4 ± 5.6
	IoU (%)	53.5	63.8	64.3	73.5	68.8	64.7 ± 7.4
	Pre. (%)	71.4	78.5	77.7	86.3	80.4	78.0 ± 5.8
	Recall (%)	68.1	70.1	78.9	84.1	80.3	76.3 ± 6.8

Overall, the patient-wise protocol demonstrates that the proposed model maintains reliable performance on patients unseen during training, with particularly strong and stable results for Ligament and Nerve. These outcomes support the method’s suitability for deployment scenarios where inference occurs on new patients without fine-tuning.

Confusion matrix analysis

We provide a per-class breakdown using a row-normalized confusion matrix (rows sum to 1, so the diagonal entries correspond to class recalls) in Fig. 7. Nerve performs best with a recall of 0.85, followed by Ligament at 0.73 and Bone at 0.71, while Adiposity is the most challenging at 0.52. The dominant error across classes is confusion with Background: Adiposity is often predicted as Background (0.44), Bone shows 0.19 to Background, and Ligament 0.14, indicating boundary and low-contrast issues. These results suggest the model is strong on Nerve and reasonable on Ligament and Bone but struggles to separate Adiposity from Background and could benefit from boundary-aware refinement or class reweighting.

Fig. 7.

Normalized confusion matrix for instance segmentation task on PELD dataset.

Qualitative analysis and failure cases

To provide a balanced assessment, we include a dedicated figure that pairs ground-truth masks (top row) with model outputs (bottom row) across five challenging scenarios (Fig. 8). The examples highlight typical failure modes that stress-test the segmentation pipeline: (a) blurred/indistinct boundaries that lead to boundary leakage; (b) intraoperative bleeding that alters local color/texture statistics; (c) specular highlights/overexposure that suppress informative appearance cues; (d) limited field of view near the periphery causing partial structures and context loss; and (e) instrument occlusion that hides target tissues. These cases reveal that most residual errors arise from weak edges, severe illumination changes, and missing context due to occlusions or cropping. They motivate boundary-aware refinement and robustness-oriented training (e.g., glare/occlusion augmentations) as potential remedies.

Fig. 8. Representative failure cases.

Top: ground truth masks (GT). Bottom: model outputs. Typical challenges include: a blurred boundaries; b intraoperative bleeding; c specular highlights; d limited field of view; and e instrument occlusion.

Ablation study

The ablation evaluates the effectiveness and lightweight design of the proposed modules for instance segmentation on PELD dataset by progressively inserting C2f Pro, the improved Neck, and LMSH into a YOLOv8 baseline has been summarized in Table 5.

Table 5.

Ablation study of EndoSeg-RT on the PELD dataset

Backbone	C2f Pro	Neck	LMSH	Accuracy		Efficiency
				Dice↑	IoU↑	Params (M)↓	FLOPs (G)↓	FPS↑
✓	–	–	–	72.7	57.0	3.2	12.1	170.8
✓	✓	–	–	72.8	57.2	2.5	10.3	189.0
✓	–	✓	–	71.0	54.7	3.2	12.4	166.3
✓	–	–	✓	68.7	52.3	2.5	10.0	195.3
✓	✓	✓	–	71.7	56.0	2.5	10.7	180.1
✓	✓	–	✓	72.7	57.0	1.9	8.4	191.1
✓	–	✓	✓	74.7	59.6	2.5	10.6	187.2
✓	✓	✓	✓	75.5	60.7	1.8	8.8	191.7

C2f Pro, Neck, and LMSH denote the three modules. Metrics are reported on the validation set. Notes that ↑ (↓) indicates that larger (smaller) values are better.

As reported in Table 5, C2f Pro-with structural re-parameterization and efficient multi-scale attention-reduces parameters from 3.2M to 2.5M and slightly improves performance, yielding a better accuracy-efficiency trade-off. The improved Neck, with almost no parameter change, strengthens cross-scale feature interaction and boosts results to 74.7/59.6. LMSH uses shared convolutions and lightweight normalization to stabilize batch-1 inference at negligible cost; used alone it remains competitive, and in combination it preserves or improves accuracy without added complexity. Combining all three modules gives the best overall outcome-75.5 Dice and 60.7 IoU-while keeping the model compact and fast (≈191.7 FPS), offering a strong balance of accuracy and deployability.

SE vs. EMA in C2f pro

To isolate the effect of the attention choice inside C2f Pro, we compare Squeeze-and-Excitation (SE) with Efficient Multi-scale Attention (EMA) on the PELD validation set, as reported in Table 6). EMA delivers higher Dice/IoU and recall with fewer parameters and lower latency, at a small precision trade off. The modest precision decrease (78.9 → 76.9) aligns with EMA’s stronger recovery of thin and ambiguous structures, yielding better boundary quality. These results support our choice of EMA for EndoSeg RT’s real-time deployment goals.

Table 6.

Ablation of attention choice in C2f Pro on the PELD validation set

Attention mechanisms	Dice↑	IoU↑	Pre.↑	Recall↑	Par. (M)↓	FLOPs (G)↓	FPS ↑	TRT FP16 (ms) ↓
SE	68.4	52.0	78.9	60.2	3.2	10.2	180.2	6.0
EMA	72.9	57.6	76.9	69.3	2.5	10.3	189.0	5.3
Δ (EMA − SE)	+4.5	+5.6	−2.0	+9.1	−0.7	+0.1	+8.8	−0.7

Dice, IoU, Pre. (Precision), Recall, Par. (parameters), FLOPs, FPS (only PyTorch FP32 model), and TRT FP16 (TensorRT FP16, ms) are measured end-to-end on an RTX 3060.

Efficiency analysis

Figure 9 compare model size, computation, and speed across representative baselines. EndoSeg-RT attains 1.8 M parameters, 8.8 G FLOPs, and 189.3 FPS, clearly occupying the low-FLOPs/high-FPS region of the trade-off plot. Relative to recent strong detectors (e.g., Mask2Former⁹ and ConvNeXt-based method¹⁰ at 44-49 M parameters and 262-268 G FLOPs), EndoSeg-RT reduces parameters by over an order of magnitude and cuts computation by roughly 30- while maintaining competitive accuracy.

Fig. 9. Efficiency comparison across instance-segmentation models on PELD dataset.

Top: bar charts of model size (Parameters, M), computation (FLOPs, G), and speed (FPS). Bottom: FLOPs-FPS trade-off scatter plot. EndoSeg-RT lies in the optimal region (low FLOPs, high FPS), achieving 1.8 M parameters, 8.8 G FLOPs, and 189.3 FPS.

The ablation trend in Table 5 and Fig. 10 highlights the sources of these gains. C2f-Pro and LMSH are the primary contributors to compactness, reducing both parameters and FLOPs without sacrificing speed; their combination yields the tightest efficiency envelope. The improved Neck mainly enhances discriminative capacity: although it adds FLOPs relative to C2f-Pro/LMSH alone, it synergizes with them to sustain high FPS. The final configuration balances all three modules, reaching 1.8 M/8.8 G with near-peak throughput, delivering a favorable accuracy-efficiency trade-off suitable for real-time, resource-constrained clinical deployment.

Fig. 10. Comparison of model complexity in the ablation study.

The bar chart reports Parameters (M) for each configuration, highlighting the reduction achieved by the proposed modules.

Analytical latency and deployment profiling

To evaluate deployment latency with strict control, we performed a stage-wise profiling experiment (Table 7). The end-to-end pipeline was partitioned into input/output (T_io), pre-processing ($T_{pre}$), model inference ($T_{{infer}_{mean}}$); mean/peak), and post-processing ($T_{{infer}_{pred}}$). Post-processing comprised Non-Maximum Suppression (NMS), mask upsampling, and thresholding. From Table 7 we can observed that Box-NMS is negligible on GPU (~0.16–0.17 ms), whereas Mask-NMS adds ~1.7–1.8 ms; Δ_mask−box quantifies this overhead. On CPU, inference and post-processing dominate, resulting in markedly higher end-to-end latency. Relative errors remain within single digits for both NMS variants, supporting the reliability of the analytical predictor.

Table 7.

End-to-end latency breakdown (ms) at 640 × 640, batch = 1

Device	RTX 3060		Intel Core i9-13900K
Precision	FP32	FP16	FP32
Tio	1.24	1.24	1.24
$T_{pre}$	1.50	0.18	0.22
$T_{{infer}_{mean}}$	11.60	12.36	56.8
$T_{{infer}_{pred}}$	12.80	13.11	51.92
$T_{{nms}_{box}}$	0.17	0.16	0.12
$T_{{nms}_{mask}}$	1.80	1.72	10.26
Δmask−box	1.63	1.56	10.14
$T_{{mask}_{up}}$	1.03	1.11	0.46
Tend2end,meas (box/mask)	15.54/17.17	15.05/16.45	58.84/68.98
Tend2end,pred (box/mask)	16.74/18.37	15.80/17.36	53.96/64.10
Err box (%)	7.72	4.98	8.29
Err mask (%)	6.99	5.53	7.07

Values are medians over 200 iterations after 50 warm-up runs; $T_{\mathrm{end}2\mathrm{end}}=T_{\mathrm{io}}+T_{\mathrm{pre}}+T_{\mathrm{infer}}+T_{\mathrm{post}}$, with T_post including Non-Maximum Suppression (NMS), mask upsampling, and thresholding; Err (%) = ∣pred − meas∣/meas × 100.

And Table 8 summarizes the latency-accuracy trade-off on RTX 3060 for FP32 vs FP16 and for Box-NMS vs Mask-NMS at IoU = 0.5. Precision (Prec.) indicates numeric format; FP16 generally lowers runtime with limited effect on accuracy. Dice and IoU measure segmentation quality. T_nms is the NMS cost, and T_end2end is total pipeline latency (I/O, pre-processing, inference, and post-processing, including NMS, upsampling, and thresholding). These metrics jointly quantify throughput versus accuracy under different precision and NMS settings.

Table 8.

Latency-accuracy on RTX 3060 across FP32 and FP16 precisions with NMS variants

Device	Prec.	NMS type	IoU thr.	Dice (%)	IoU (%)	Tnms	Tend2end
RTX 3060	FP32	Box-NMS	0.5	74.4	43.0	0.17	15.54
RTX 3060	FP32	Mask-NMS	0.5	–	–	1.80	17.17
RTX 3060	FP16	Box-NMS	0.5	71.6	41.7	0.16	15.05
RTX 3060	FP16	Mask-NMS	0.5	–	–	1.72	16.61

Prec., IoU thr., T_nms and T_end2end represent numeric precision, matching threshold, NMS latency, and end-to-end latency, respectively.

To better visualize the performance, Fig. 11 compares analytical end2end predictions with direct measurements across devices (RTX 3060, Intel Core i9-13900K), precisions, and input sizes. Points cluster near the identity line, indicating that the additive model T_end2end,pred closely matches observed latency. The inset for RTX 3060 shows only millisecond-level deviations across FP32/FP16 and NMS variants, supporting the model’s utility for deployment-time what-if analysis.

Fig. 11.

Comparison of analytical latency predictions against measurements for batch = 1 across input sizes and precisions on RTX 3060 and Intel Core i9-13900K.

Further analysis

Attention mechanism selection

To disentangle recognition quality from computational cost, we report accuracy and efficiency separately. Table 9 summarizes accuracy (Dice, IoU), Precision, and Average Recall (A.R.) across SE²³, AFGCAttention²⁴, CPCA²⁵, SimAM²⁶, and EMA²⁷. EMA²⁷ attains the best Dice (72.9) and IoU (57.6) with competitive Precision and A.R.; SE yields the highest Precision (78.9), while SimAM ²⁶ achieves the highest A.R. (71.6). CPCA ²⁵ provides balanced performance with strong Precision and recall-oriented accuracy.

Table 9.

Effect of different attention mechanisms in the EndoSeg-RT model on the PELD dataset

Attention mechanisms	Accuracy				Efficiency
	Dice↑	IoU↑	Precision↑	A.R.↑	Params (M)↓	FLOPs (G)↓	FPS↑
SE23	68.4	52.0	78.9	60.2	3.2	10.2	180.2
AFGCAttention24	66.2	49.5	71.3	61.9	3.2	12.0	183.8
CPCA25	69.5	53.4	74.3	65.3	3.3	12.4	182.3
SimAM26	64.0	47.5	67.8	71.6	3.2	12.0	187.1
EMA27	72.9	57.6	76.9	69.3	2.5	10.3	189.0

Metrics are reported on the validation set. Notes that ↑ (↓) indicates that larger (smaller) values are better. Bold and underlined mark the best and second-best results per column, respectively.

Complementarily, Table 9 and Fig. 12a show that EMA has the smallest parameter and FLOP footprint and the highest FPS among peers (2.5M params, 10.3 GFLOPs, 189.0 FPS), making it the most practical choice for real-time endoscopic deployment.

Fig. 12. Comparison of EndoSeg-RT module variants on the PELD dataset.

Bars indicate mAP50, mAP50 -95, Precision, and Recall (percent), as shown in the legend; higher values are better for all four metrics. All results are reported on the PELD validation set. a Attention mechanisms: SE, AFGCAttention, CPCA, SimAM, and EMA. b C2f variants: C2f-Star, C2f-CG, C2f-Faster, and C2f-Pro. c Segmentation heads: LADH, LQE, Efficient Head, and LMSH.

Backbone variants

To evaluate the effectiveness of the proposed C2f Pro design, we compare it with representative C2f variants-C2f-CG, C2f-Faster, and C2f-Star-under the same EndoSeg-RT framework on the PELD dataset (Table 10). Using the updated metrics, C2f Pro consistently achieves the best accuracy, reaching 71.3% Dice and 55.7% IoU, surpassing the second-best configuration. It also attains the highest Precision (76.9%) and strong Average Recall (66.4%), indicating improvements in both segmentation reliability and coverage.

Table 10.

Comparative performance and efficiency of different C2f modules in EndoSeg-RT on the PELD dataset

C2f Module	Accuracy				Efficiency
	Dice↑	IoU↑	Precision↑	A.R.↑	Params (M)↓	FLOPs (G)↓	FPS↑
C2f-CG28	63.5	46.4	65.9	61.3	2.3	10.9	184.5
C2f-Faster29	63.4	46.3	64.4	62.5	2.5	10.2	197.5
C2f-Star38	64.6	47.8	68.2	61.3	2.7	10.7	182.8
C2f Pro	71.3	55.7	76.9	66.4	2.5	10.3	189.0

Metrics are reported on the validation set. ↑ (↓) indicates that larger (smaller) values are better. Bold and underlined denote the best and second-best results per column, respectively.

From the efficiency perspective, C2f Pro maintains a favorable accuracy-efficiency trade-off. Although C2f-CG²⁸ and C2f-Faster²⁹ have slightly lower parameter counts or FLOPs, these savings come with noticeable performance drops. In contrast, C2f Pro preserves competitive efficiency with only 2.5M parameters and 10.3G FLOPs, while sustaining real-time inference at 189 FPS. These results demonstrate that the C2f Pro module effectively balances representational capacity and computational efficiency, making it well suited for real-time instance segmentation in resource-constrained scenarios.

Segmentation head comparison

To assess the effectiveness of the proposed LMSH segmentation head, we compare it with representative designs-LADH³⁰, LQE³¹, and EfficientHead³²-under the same EndoSeg-RT framework on the PELD dataset (Table 11). Using the updated metrics, LQE attains the highest Dice/IoU (70.3%/54.3%) with strong recall (68.5%), while LADH achieves the highest precision (75.4%). LMSH offers a more balanced accuracy profile, maintaining competitive precision (74.5%) and recall (63.7%) with solid Dice/IoU (68.7%/52.3%), indicating robust segmentation quality across diverse cases.

Table 11.

Performance and efficiency comparison of different segmentation heads on the PELD dataset

segmentation Head	Accuracy				Efficiency
	Dice↑	IoU↑	Precision (%)↑	Recall ↑	Params (M)↓	FLOPs (G)↓	FPS↑
LADH30	66.7	50.2	75.4	59.9	2.5	9.9	196.6
LQE31	70.3	54.3	72.1	68.5	3.2	12.0	165.3
EfficientHead32	65.1	48.6	76.5	56.5	4.0	11.9	172.9
LMSH	68.7	52.3	74.5	63.7	2.5	10.0	195.3

Metrics are reported on the validation set. ↑ (↓) indicates that larger (smaller) values are better. Bold and underlined mark the best and second-best results per column, respectively.

From the efficiency perspective, LMSH also demonstrates a favorable balance between computational cost and runtime performance. With only 2.5M parameters and 10.0G FLOPs, LMSH matches the most lightweight configuration (LADH) and achieves near-optimal inference speed of 195.3 FPS. In contrast, LQE and EfficientHead introduce additional computational overhead without proportional efficiency benefits. These results indicate that LMSH enhances feature aggregation for instance segmentation while preserving real-time efficiency, making it well suited for practical deployment scenarios.

As illustrated in Fig. 12, the proposed EndoSeg-RT consistently outperforms alternative designs across different architectural components, including attention mechanisms, C2f variants, and segmentation heads. These results demonstrate that each proposed component contributes positively to the overall performance, and their combination yields a complementary and cumulative improvement.

Generalization ability

To evaluate the generalization capability of the proposed method under distribution shift, we conduct experiments on the Teeth Segmentation dataset^33,34, which contains 598 dental radiographs with 15,021 annotated instances spanning 32 tooth categories (Fig. 13). Since this dataset is collected independently and follows a distribution distinct from the training data, it provides a rigorous out-of-domain test for assessing the generalization capability of the proposed method.

Fig. 13. Examples from the Teeth Segmentation dataset.

Each panel shows a dental X-ray and its corresponding instance-level annotations (ground truth).

As reported in Table 12, EndoSeg-RT delivers strong accuracy under distribution shift, achieving 91.1% Dice and 83.5% IoU with competitive Precision (91.3%) and A.R. (90.9%). Meanwhile, it maintains the lowest computational cost (1.8M parameters, 8.8 GFLOPs) and the highest inference speed (187.4 FPS) among all compared methods. Notably, despite using an order of magnitude fewer parameters and FLOPs than transformer-based approaches such as Mask2Former and ConvNeXt, EndoSeg-RT attains comparable Dice/IoU and markedly superior efficiency, demonstrating strong robustness to domain shift and suitability for real-time deployment.

Table 12.

Generalization performance on the Teeth Segmentation dataset under distribution shift

Methods	Accuracy				Efficiency
	Dice↑	IoU↑	Precision↑	A.R.↑	Params (M)↓	FLOPs (G)↓	FPS↑
Mask R-CNN5	85.7	75.2	89.4	82.2	43.9	259.0	11.7
YOLACT6	85.1	74.3	88.2	86.8	34.7	61.6	42.3
SOLOv27	87.2	77.3	87.5	86.9	46.2	15.2	12.5
Swin Mask R-CNN8	88.3	79.1	90.1	87.7	47.4	264.0	7.6
Mask2Former9	87.9	78.4	90.5	86.4	44.4	262.1	5.4
ConvNeXt10	87.7	78.0	90.3	83.8	48.7	268.2	9.3
YOLOv8-seg11	89.9	81.8	89.9	89.8	3.2	12.0	177.7
YOLOv11-seg20	91.3	83.8	91.9	90.9	2.8	10.2	170.2
EndoSeg-RT	91.1	83.5	91.3	90.9	1.8	8.8	187.4

Accuracy and efficiency metrics are reported on the test set. ↑ (↓) indicates that larger (smaller) values are better. Bold and underlined denote the best and second-best results per column, respectively.

Figure 14 (qualitative) contrasts predictions under challenging cases (specular highlights, occlusion by instruments, and motion blur). Methods emphasizing global context (Mask2Former⁹), Swin Mask R-CNN⁸) preserve coarse tooth extents but may erode boundaries near gingival margins. One-stage decoders (YOLACT⁶, SOLOv2⁷, YOLOv8/11-seg^11,20) are fast but show occasional under-segmentation at tight contacts. EndoSeg-RT preserves enamel-gingiva boundaries and interdental separations while sustaining real-time throughput, reflected by its simultaneous lead in AP₅₀ and A.R. with state-of-the-art efficiency.

Fig. 14.

Qualitative comparison on the Teeth Segmentation dataset under distribution shift.

Discussion

This work presents EndoSeg-RT, a lightweight multi-scale attention framework designed for deployable real-time instance segmentation in spinal endoscopy, where intraoperative scenes are characterized by narrow fields of view, specular glare, fluid interference, ambiguous tissue boundaries, and pronounced scale variation. EndoSeg-RT adopts a holistic, deployment-driven co-design across all network stages. Specifically, it introduces i) a C2f-Pro backbone that integrates RepViT-style re-parameterized convolution blocks with efficient multi-scale attention to achieve strong feature representation at low computational cost; ii) a dual-neck architecture combining Scale-Sequence Feature Fusion (SSFF) and Triple Feature Encoding (TFE) to enhance cross-scale consistency and boundary delineation; and iii) a Lightweight Multi-task Shared Head (LMSH) employing shared convolutions and GroupNorm to reduce parameter redundancy and ensure stable batch-size-one inference during real-time deployment.

In addition, we curate a clinician-reviewed PELD spinal endoscopy dataset comprising 610 images from 61 patients, with pixel-level instance annotations for adipose tissue, bone, ligamentum flavum, and neural structures, enabling standardized benchmarking in this domain. Extensive experiments demonstrate that EndoSeg-RT effectively balances segmentation accuracy and computational efficiency, achieving 74.4 mAP₅₀ and 43.0 mAP₅₀₋₉₅ on the PELD dataset with only 1.8 M parameters and 8.8 GFLOPs, while operating at ~189 FPS under batch-size-one inference at 640 × 640 resolution. Ablation studies further confirm that each proposed component contributes positively to segmentation performance and model compactness. Collectively, these results highlight EndoSeg-RT as a practical and robust solution for real-time spinal endoscopic guidance, with strong potential for clinical translation.

While EndoSeg-RT demonstrates competitive accuracy and notable efficiency on spinal endoscopic instance segmentation, several limitations remain:

i) Single-center data and modest scale. The PELD dataset is collected from a single clinical center with 61 patients and 610 images. Although clinically reviewed, its size and source may limit diversity in anatomy, device vendors, illumination conditions, and surgical styles. Multi-center, larger-scale cohorts are needed to improve generalizability.

ii) Image-only, frame-level evaluation. All experiments are performed on static frames at a fixed resolution (640 × 640). We do not exploit temporal coherence or video-based priors, which can stabilize predictions under rapid endoscope motion, occlusions, and smoke. Extending EndoSeg-RT to temporal models and evaluating end-to-end video performance are important future steps.

iii) Challenging artifacts and edge cases. Failure cases persist under severe specular highlights, heavy bleeding/irrigation, bone dust/smoke, and tool-induced occlusions, where thin boundaries (e.g., nerve margins, ligament edges) may be fragmented or leak. Artifact-aware augmentation, boundary-centric supervision, and uncertainty modeling could further mitigate these errors.

These limitations suggest several directions for future work: multi-center video datasets with richer artifact distribution; temporal and uncertainty-aware modeling; artifact-robust training schemes.

Methods

EndoSeg-RT architecture overview

To improve both efficiency and practicality for spinal endoscopic instance segmentation, we propose EndoSeg-RT, a lightweight attention-aware model composed of a compact backbone, a dual-branch neck, and a shared-convolution head (Fig. 15). The pipeline is optimized for batch-1 inference, boundary fidelity, and small-structure sensitivity at surgical resolutions (640-640 by default). This study was approved by the Institutional Review Board (IRB) of Xiangyang Central Hospital, China, under protocol 2025-199.

Fig. 15. Overall architecture of EndoSeg-RT.

A The complete network pipeline, consisting of a C2f-Pro-based backbone, a dual-branch feature fusion neck, and a lightweight shared-convolution segmentation head. B The proposed efficient multi-scale attention mechanism. C The C2f-Pro block used for efficient feature extraction and aggregation. GN denotes Group Normalization.

Backbone: Lightweight Multi-Scale Feature Extractor (LMFE) with C2f Pro. We build on a streamlined CSPDarknet53³⁵ (light variant) as the main feature extractor to produce multi-scale features while balancing semantics and details. Each stage adopts the proposed C2f-Pro module, which integrates re-parameterized convolutions in a ViT-style block (RVB) together with a selective Efficient Multi-scale Attention (EMA) module at the deepest stage to enhance long-range context and robustness to specular highlights. Given an RGB image $\mathbf{I}\in {\mathbb{R}}^{3\times H\times W}$, the backbone yields a pyramid

$$(P_3,,P_4,,P_5)=Backbone\left(\mathbf{I}\right),$$ 1

where $P_{\ell }\in {\mathbb{R}}^{C_{\ell }\times H_{\ell }\times W_{\ell }}$ (ℓ ∈ 3, 4, 5), and P₃, P₄, and P₅ are the feature maps at strides s ∈ 8, 16, 32.

Given the backbone pyramid P_ℓ LMFE aligns channels to C_f while preserving spatial resolution:

$$F_{\ell }=\varphi \left(P_{\ell }\right)\in {\mathbb{R}}^{C_f\times H_{\ell }\times W_{\ell }},\ell \in 3,4,5,$$ 2

where the per-scale mapping is

$$F_{\ell }=\mathrm{ReLU}!\left(\mathrm{GN}(\mathrm{Conv}1\times 1(P\ell ;\mathrm{out}=C_f\left)\right)\right).$$ 3

If a neck path applies stride-2 downsampling, insert an optional depthwise 3 × 3 blur before that step.

Neck: dual-branch neck for fusion

To fuse complementary information across scales, we newly designed a dual-module neck with two coordinated blocks. Scale-Sequence Feature Fusion (SSFF) block cross-scale dependencies by treating {P₃, P₄, P₅} as an ordered sequence and learning compact 3D kernels with non-negative aggregation. Triple Feature Encoding (TFE) reinforces fine details via large/middle/small receptive-field branches centered on the shallow features. These produce two aligned feature tensors for the head:

$$\begin{array}{c}{\mathbf{F}}_{\mathrm{SSFF}}=\mathrm{S}\mathrm{S}\mathrm{F}\mathrm{F}(F_3,F_4,F_5),\hfill \\ {\mathbf{F}}_{\mathrm{TFE}}=\mathrm{T}\mathrm{F}\mathrm{E}(F_3,F_4,F_5).\hfill \end{array}$$ 4

where we define their shapes on the stride-8 grid as ${\mathbf{F}}_{SSFF}\in {\mathbb{R}}^{C_f\times \frac{H}{8}\times \frac{W}{8}}$, and ${\mathbf{F}}_{TFE}\in {\mathbb{R}}^{C_f\times \frac{H}{8}\times \frac{W}{8}}$. The C_f is the fused channel width after alignment. In practice, SSFF treats {P₃, P₄, P₅} as an ordered scale sequence and performs a single bi-directional fusion pass, while TFE refines high-resolution details by upsampling deep features to the stride-8 grid and applying channel-spatial gating on P₃. Both tensors are channel-aligned and resolution-matched for the downstream head.

Head: lightweight shared-convolution segmentation head (LMSH)

LMSH employs two shared 3 × 3 convolutions with GroupNorm to stabilize batch-1 training while minimizing parameters. It supports classification and optional geometric prediction, and generates instance masks either directly or via a prototype-coefficient scheme:

$$\mathbf{p},\mathbf{b},\mathbf{M}=\mathrm{LMSH}({\mathbf{F}}_{\mathrm{SSFF}},{\mathbf{F}}_{\mathrm{TFE}},P_3,P_4,P_5),$$ 5

where p are class probabilities, b are optional box/center parameters, and M are instance masks.

C2f pro module

Spinal endoscopic images exhibit smoothly transitioning textures and thin, low-contrast boundaries between adjacent tissues, which makes purely local aggregation prone to boundary discontinuities or regional adhesion. The original C2f block-built on local convolutions and fixed shortcut interactions-therefore struggles to capture cross-channel and cross-scale dependencies in this setting. To address these limitations under real-time constraints, we redesign C2f into C2f-Pro by integrating re-parameterized ViT convolutional cells (RVB³⁶) and replacing SE²³ with Efficient Multi-scale Attention (EMA³⁷). This enhances training-time capacity and global awareness, while structural re-parameterization collapses the multi-branch design into a single path at inference, preserving low latency.

Given an input feature $X\in {\mathbb{R}}^{C\times H\times W}$ (e.g., X ∈ {P₃, P₄, P₅} after optional channel alignment), C2f-Pro first compresses channels and forms a shortcut/transform split:

$$X_0={\mathrm{Conv}}_{1\times 1}\left(X\right),[X_s,X_t]=\mathrm{Split}\left(X_0\right),$$ 6

where X₀ is the channel-reduced tensor after the initial 1 × 1 convolution; X_s is the shortcut branch; and X_t is the transform branch input.

We unify the notation to make explicit that EMA precedes RVB inside each RVB–EMA unit. Let the transform-branch input after the 1 × 1 split be Y₀ = X_t. For i ∈ {1, 2, 3}, the i-th unit applies EMA to Y_i−1 to obtain the EMA-weighted feature ${\tilde{Y}}_{i-1}$, which is then mixed by RVB to produce Y_i:

$$Y_i=\mathrm{R}\mathrm{V}\mathrm{B}\left({\tilde{Y}}_{i-1}\right),$$ 7

Given $Y_{i-1}\in {\mathbb{R}}^{C\times H\times W}$, EMA computes channel descriptors via global average pooling (rows/cols indexed by u, v):

$$z_c=\frac{1}{HW}\sum _{u=1}^H\sum _{v=1}^W{Y}_{i-1}(c,u,v),z\in {\mathbb{R}}^C.$$ 8

A lightweight mixing function $a=g\left(z\right)\in {\mathbb{R}}^C$ produces channel weights, and an optional spatial gate $S=\sigma \left(h\left(Y_{i-1}\right)\right)\in {\mathbb{R}}^{1\times H\times W}$ is obtained by a depthwise k × k conv h( ⋅ ) followed by a sigmoid σ. EMA reweights the input feature to yield the intermediate tensor ${\tilde{Y}}_{i-1}$:

$${\tilde{Y}}_{i-1}(c,u,v)=a_{c}S(u,v)Y_{i-1}(c,u,v).$$ 9

where makes explicit that in Eq. (9) is the intermediate, EMA weighted tensor that directly precedes RVB inside each RVB EMA block, and that Y_i in Eq. (7) is the post RVB output used for the next recursion.

Dual-branch neck for fusion

Spinal endoscopy requires a neck that fuses multi-scale features while preserving thin, low-contrast boundaries under strict real-time constraints. Building on the aligned outputs of Backbone, we structure the neck as two tightly coupled stages: first, Sequence-Style Scale Fusion (SSFF) exchanges information across scales in a single pass; then, Target-Focused Enhancement (TFE) leverages the fused semantics to refine the highest-resolution features where small structures reside. This top-down flow-from global fusion to local emphasis-keeps the design coherent and latency-aware.

Sequence-style scale fusion (SSFF)

With inputs standardized by Backbone, SSFF treats the pyramid as a short sequence indexed by scale and performs a single bi-directional pass to communicate both top-down and bottom-up information.

Let $F_{\ell }\in {\mathbb{R}}^{C_f\times H_{\ell }\times W_{\ell }}$ for ℓ ∈ {3, 4, 5} denote features at strides 8, 16, 32, ordered as F = [F₃, F₄, F₅]. Channel width C_f is already aligned by LMFE. We summarize each scale into a compact token and infer fusion coefficients via a lightweight mixer:

$$\begin{array}{c}{t}_{\ell }=\left[\mathrm{GAP}\left(F_{\ell }\right);\mathrm{SMP}\left(F_{\ell }\right)\right]\in {\mathrm{ℝ}}^d,\hfill \\ T=[t_3,t_4,t_5],\hfill \end{array}$$ 10

where GAP is global average pooling, SMP is 2 × 2 strided mean pooling followed by a linear projection, and [ ⋅ ; ⋅ ] denotes concatenation. A causal/anti-causal mixer $\mathcal{M}$ yields per-scale channel weights and gates:

$$[{\alpha }_3,{\alpha }_4,{\alpha }_5],[g_3^{\uparrow },g_4^{\uparrow }],[g_4^{\downarrow },g_5^{\downarrow }]=\mathcal{M}\left(T\right).$$ 11

The fused features are computed by a bi-directional update with lightweight resampling:

$$\begin{array}{ccc}{\widehat{F}}_{\ell }\hfill & =\hfill & {\mathrm{Conv}}_{1\times 1}\left({\alpha }_{\ell }\odot F_{\ell }\right)+{\underbrace{\mathrm{Up}\left({\mathrm{Conv}}_{1\times 1}\left(F_{\ell +1}\right)\right)\odot g_{\ell }^{\uparrow }}}_{\mathrm{t}\mathrm{o}\mathrm{p}-\mathrm{d}\mathrm{o}\mathrm{w}\mathrm{n}}\hfill \\ \hfill & \hfill & +{\underbrace{\mathrm{Down}\left({\mathrm{Conv}}_{1\times 1}\left(F_{\ell -1}\right)\right)\odot g_{\ell }^{\downarrow }}}_{\mathrm{bottom}-\mathrm{up}},\hfill \end{array}$$ 12

where Up/Down use nearest-neighbor resampling aligned to (H_ℓ, W_ℓ); an optional 3 × 3 blur precedes Down to curb aliasing. For boundary protection at the finest scale, an edge-aware gate may be applied after the upsampled path at ℓ = 3. The outputs ${\widehat{F}}_3,{\widehat{F}}_4,{\widehat{F}}_5$ provide globally informed yet resolution-faithful features for the subsequent refinement.

Target-focused enhancement (TFE)

Given the globally fused features from SSFF, TFE concentrates compute where small anatomical structures are best represented: the highest resolution (stride-8). It transfers deep context back to ${\widehat{F}}_3$ using low-cost channel and spatial emphasis.

We first upsample deeper features to the stride-8 grid and project to form a semantic prior:

$$S={\mathrm{Conv}}_{1\times 1}\left(\left[\mathrm{Up}\right({\widehat{F}}_4),\mathrm{Up}({\widehat{F}}_5\left)\right]\right)\in {\mathrm{ℝ}}^{C_f\times H_3\times W_3}.$$ 13

Channel and spatial gates are then computed as

$$\begin{array}{c}\mathbf{w}=\sigma \left(W_2\delta \left(W_1\mathrm{GAP}\left(S\right)\right)\right)\in {\left(0,1\right)}^{C_f},\hfill \\ M_s=\sigma \left({\mathrm{DW}}_{3\times 3}\left(S\right)+{\mathrm{DW}}_{5\times 5}\left(S\right)\right)\in {\left(0,1\right)}^{H_3\times W_3},\hfill \end{array}$$ 14

where δ and σ denote ReLU and Sigmoid, and DW are depthwise convolutions.

The gates modulate ${\widehat{F}}_3$ in a residual form,

$${\tilde{F}}_3=(\mathbf{w}\odot {\widehat{F}}_3)\odot M_s+{\widehat{F}}_3,$$ 15

achieving batch-1 stable enhancement that benefits boundary fidelity. To align feature edges with image cues during training, we add a zero-cost-at-inference gradient-consistency loss:

$${\mathcal{L}}_{\mathrm{gc}}={\lambda }_{\mathrm{gc}}\parallel {\nabla }_{\mathrm{Sobel}}\left({\tilde{F}}_3^{\mathrm{edge}}\right)-{\nabla }_{\mathrm{Sobel}}\left(I\right){\parallel }_1,$$ 16

where ${\tilde{F}}_3^{edge}$ is a 1 × 1 projection used only for supervision.

The neck outputs $\{{\tilde{F}}_3,{\widehat{F}}_4,{\widehat{F}}_5\}$, preserving SSFF’s ordering so downstream heads can attach consistently. In multi-scale heads, channels are aligned to C_h by shared 1 × 1 projections and depthwise separable 3 × 3 blocks per scale.

Lightweight shared-convolution segmentation head

The Lightweight Shared Convolution Segmentation Head (LMSH) transforms fused neck features into dense masks with minimal latency. In particular, LMSH primarily consumes the refined high-resolution feature ${\tilde{F}}_3$, while optionally incorporating cues from ${\widehat{F}}_4$ and ${\widehat{F}}_5$. A shared kernel bank applies identical weights across scales to reduce parameters and encourage cross-scale consistency. Predictions are generated at stride-4 or stride-8 (depending on memory constraints), and the logits are then upsampled to the input resolution. Let $\{{\tilde{F}}_3,{\widehat{F}}_4,{\widehat{F}}_5\}$ be the outputs of the neck with channel size C_f. We first align channels to a head width C_h using shared 1 × 1 projections:

$$Z_{\ell }={\mathrm{Conv}}_{1\times 1}^{\mathrm{shared}}\left(F_{\ell }\right),\ell \in \{3,4,5\},$$ 17

where the weights of ${\mathrm{Conv}}_{1\times 1}^{\mathrm{shared}}$ are tied across scales.

Each projected feature is refined by B shared depthwise-separable blocks with residual connections:

$$\begin{array}{c}{\mathrm{\Phi }}_{\ell }^{(b+1)}={\mathrm{\Phi }}_{\ell }^{\left(b\right)}+{\mathrm{PW}}_{1\times 1}\left(\delta \left({\mathrm{DW}}_{3\times 3}\left({\mathrm{\Phi }}_{\ell }^{\left(b\right)}\right)\right)\right),\hfill \\ {\mathrm{\Phi }}_{\ell }^{\left(0\right)}=Z_{\ell },b=0,\dots ,B-1,\hfill \end{array}$$ 18

where DW and PW denote depthwise and pointwise convolutions, and δ is ReLU. All block parameters are shared across ℓ.

To obtain a strong stride-8 representation while keeping compute low, we aggregate features to the H₃ × W₃ grid:

$$\begin{array}{c}{\mathrm{U}}_3={\mathrm{\Phi }}_3^{\left(B\right)},\hfill \\ {\mathrm{U}}_4=\mathrm{Up}\left({\mathrm{Conv}}_{1\times 1}\left({\mathrm{\Phi }}_4^{\left(B\right)}\right)\right),\hfill \\ {\mathrm{U}}_5=\mathrm{Up}\left(\mathrm{Up}\left({\mathrm{Conv}}_{1\times 1}\left({\mathrm{\Phi }}_5^{\left(B\right)}\right)\right)\right),\hfill \\ \hfill \end{array}$$ 19

and fuse them with a lightweight attention gate guided by U₃:

$$\begin{array}{c}A=\sigma \left({\mathrm{Conv}}_{1\times 1}\left(\right[U_3,U_4,U_5\left]\right)\right),\hfill \\ G=A\odot U_3+(1-A)\odot {\mathrm{Conv}}_{1\times 1}\left(\right[U_4,U_5\left]\right),\hfill \end{array}$$ 20

yielding the head feature $G\in {\mathbb{R}}^{C_h\times H_3\times W_3}$. This keeps U₃ dominant while still injecting deep context.

LMSH produces logits at stride-4 or stride-1 via a minimal decoder:

$$\begin{array}{c}{L}_8={\mathrm{Conv}}_{1\times 1}\left(G\right)\in {\mathrm{ℝ}}^{C_{\mathrm{c}\mathrm{l}\mathrm{s}}\times H_3\times W_3},\hfill \\ L_4={\mathrm{Conv}}_{3\times 3}^{\mathrm{DW}-\mathrm{PW}}\left(\mathrm{U}\mathrm{p}\left(L_8\right)\right),\hfill \end{array}$$ 21

where C_cls is the number of classes. If full-resolution masks are required, we apply one or two additional nearest-neighbor upsamplings with a final 1 × 1 convolution.

To further enhance thin structures, an auxiliary edge stream supervises high-frequency content without adding runtime cost at inference:

$$\begin{array}{c}E=\mathrm{C}\mathrm{o}\mathrm{n}{\mathrm{v}}_{1\times 1}\left({\nabla }_{\mathrm{S}\mathrm{o}\mathrm{b}\mathrm{e}\mathrm{l}}\left(G\right)\right),\hfill \\ {\mathcal{L}}_{\mathrm{e}\mathrm{d}\mathrm{g}\mathrm{e}}={\lambda }_{\mathrm{e}\mathrm{d}\mathrm{g}\mathrm{e}}\cdot \mathrm{BCEWithLogits}(E,\mathrm{Edge}(Y\left)\right),\hfill \end{array}$$ 22

where Edge(Y) is a binary edge map derived from the ground truth mask Y.

Ethics approval

This study received approval from the Institutional Review Board (IRB) of Xiangyang Central Hospital, China, under protocol number 2025–199.

Author contributions

Qi Lai: Conceptualization, Methodology, Formal analysis, Writing- Original draft; Qiang Cai: Data curation, Resources; Junyan Li: Software, Validation, Data curation; Chiman Vong: Conceptualization, Funding acquisition; Lei Jin: Data Analysis, Interpretation; Lei Wang: Reviewing, Investigation. Tao Yan: Reviewing, Funding acquisition; Xiaokun Liang: Reviewing, Supervision.

Data availability

This work did not generate any new physical materials. The trained model weights and configuration files (for training and inference) have been published. The dataset can be downloaded from: https://drive.google.com/file/d/1HaUWBhVUrsgJWLx5Eo4igPxtwlViqcy/view. Any additional artifacts can be provided by the first author upon reasonable request.

Code availability

Code has publicly available at: https://github.com/hhwmortal/PELD-Instance-segmentation.

Competing interests

The authors declare no competing interests.

Consent to participate

This study used archived endoscopic surgical video data from all patients who underwent orthopedic endoscopic procedures at Xiangyang Central Hospital, China, between Sep 2018 and Dec 2025. The study involved no risk to participants and did not affect patient care. The requirement for informed consent was waived by the Institutional Ethics Committee.