Fine-grained 3D rice phenotyping via multi-scale NeRF and multimodal segmentation

Abstract

Fine-grained 3D phenotypic analysis of rice plays a vital role in rice breeding and yield estimation. However, a comprehensive rice data acquisition and segmentation pipeline is still lacking. While Neural Radiance Fields (NeRF) have shown impressive results in crop-level 3D reconstruction, their high sensitivity to data volume and camera viewpoints often leads to reconstruction failures for rice. In addition, the large-scale rice point clouds, coupled with heavy occlusion and visual similarity among grains, pose significant challenges for fine-grained trait extraction. To address the challenge of reconstructing rice point clouds under low-quality data conditions, we propose a novel method named Multi-Scale NeRF(MSNeRF). This method incorporates a structure-detail collaborative reconstruction mechanism and a dynamic initialization density scheduling strategy. Furthermore, we introduce a multimodal and multitask rice dataset (MMR) as a benchmark resource for future research. For rice point cloud segmentation, we develop Vision Rice Knowledge Graph Network(VRKGNet), which comprises an image segmentation module, a projection module, and a point cloud segmentation module enhanced with a Transformer to enlarge the receptive field. VRKGNet performs standalone point cloud segmentation and integrates image segmentation results from multiple viewpoints as prior knowledge to enhance semantic and instance-level segmentation. Extensive experiments demonstrate that MSNeRF achieves high-fidelity point cloud reconstruction with as few as 10 viewpoints. VRKGNet achieves superior rice plant segmentation with a semantic segmentation mIoU of 88.79% and an instance segmentation AP₂₅ of 84.55%, outperforming mainstream algorithms.

1. Introduction

Rice 3D phenotype refers to the morphological and structural characteristics observed during growth and development [1,2], such as the number and size of grains on the panicle, leaf length and width, and stem attributes including height and diameter. These phenotypic traits reflect the plant's health status and correlate closely with yield and quality, playing a crucial role in rice breeding, agricultural production, and scientific research. In particular, precise quantification of mature grains on the panicle is essential for accurate yield estimation. However, compared with crops such as potato [3], tomato [4], and maize [5], phenotypic analysis of rice remains more challenging. This is primarily due to the small size and dense distribution of its grains, which are frequently occluded by leaves, leading to severe structural overlap and indistinct organ boundaries, thus posing significant challenges to traditional analytical methods.

Conventional crop phenotyping typically relies on manual annotation or contact-based measurements, which are labor-intensive, inefficient, and may damage the samples. With the advancement of computer vision technologies, 2D image analysis methods based on deep learning have been extensively applied to crop organ detection. For example, Zou et al. [6] and Deng et al. [7] identified rice panicle grains by flattening them and performing planar image detection. However, this approach requires destructive handling of the plants and fails to preserve their natural structure. To overcome the limitations of 2D image-based methods, researchers have introduced 3D modeling techniques such as laser scanning [8,9] and MVS(multi-view stereo) [1,10] reconstruction to acquire plant point cloud data. These approaches have been successfully applied to crops like potatoes [11] and sorghum [12], which feature distinctly separated organs and well-defined structures, making them suitable for 3D modeling. However, the high cost of 3D scanning equipment hinders its widespread adoption in large-scale agricultural applications. Tang et al. [1,10,13], and Sun et al. [14] have employed deep learning-based methods such as MVSNet [15] to generate point clouds for more precise modeling of plant structures. Nevertheless, the reconstructed point clouds still contain substantial noise [16], negatively impacting subsequent recognition and analysis tasks. NeRF [17] is an emerging 3D reconstruction technique that enables high-quality volumetric rendering by simulating light propagation through 3D space to estimate scene density and color fields. In recent years, NeRF and its extended variants have been widely utilized in vegetation modeling [[18], [19], [20], [21]], offering a realistic reconstruction of plant morphology while preserving structural continuity. Furthermore, with the advancement of 3D Gaussian [22], Gaussian point-based reconstruction methods have demonstrated superior efficiency and accuracy compared to traditional NeRF and have shown promising results in agricultural applications [[23], [24], [25]]. Despite these advancements, several key challenges remain in the 3D modeling of rice: (1) Stringent image acquisition requirements:Rice plants’ intricate structure and severe self-occlusion pose significant difficulties. Inadequate multi-view coverage or limited image quantity often leads to visual artifacts such as ghosting and severely degrading reconstruction quality. (2) Subtle panicle features:Rice grains exhibit compact size, high spatial density, and lack well-defined surface contours. These characteristics interfere with depth estimation, resulting in noisy points that compromise overall reconstruction accuracy. (3) Sparse point clouds of 3D Gaussian:Although 3D Gaussian modeling techniques offer high efficiency, the resulting point clouds are often too sparse to capture the fine-grained details of rice grains accurately. Additionally, the root cause of the ghosting phenomenon lies in the geometric inconsistencies across multi-view observations. High-resolution images contain abundant details but are more susceptible to inconsistency due to inaccuracies in camera pose estimation. In contrast, low-resolution images maintain structural consistency but lack the necessary detail for high-fidelity reconstruction. The fixed density scaling factor commonly employed in NeRF further exacerbates this issue:an overly high density leads to excessive attenuation of rays near scene boundaries, resulting in undersampling of central regions; conversely, overly low density weakens geometric representation, causing blurred textures and indistinct boundaries.

Although current techniques can acquire high-resolution 3D point clouds, the visual similarity in color and texture among rice grains, leaves, and stems significantly hampers the effectiveness of traditional organ segmentation methods based on spatial coordinates and color features. While such methods are effective for crops with well-defined structures and separated organs, their performance deteriorates markedly when applied to rice due to its compact morphology and overlapping components. Point cloud representations typically contain over 100,000 points to preserve the contour structure of rice panicles faithfully. This imposes substantial demands on the receptive field of convolution-based instance segmentation algorithms, thereby limiting their effectiveness in rice scenarios. In comparison, image segmentation achieves higher precision in distinguishing color and texture patterns, yet it cannot inherently accurately model occlusion relationships. Point cloud segmentation approaches possess inherent strengths in handling spatial occlusion; however, the limited availability of semantic information constrains their ability to perform fine-grained organ-level identification.

To address the aforementioned challenges, this paper explores the application of NeRF in the 3D reconstruction of rice and introduces a multimodal, multitask rice segmentation dataset named MMR (Multimodal Multitask Rice), which is specifically designed for indoor breeding scenarios and comprises the following components: (1) Multi-view images captured around rice plants; (2) Corresponding 3D point cloud data; (3) Intrinsic and extrinsic parameters of the camera. To efficiently acquire high-quality phenotypic rice data, we develop an automated data collection system that integrates a robotic arm with an Intel RealSense D435i depth camera, enabling comprehensive 360° imaging of rice plants. The system is equipped with uniformly distributed LED lighting to ensure consistent illumination and employs a pure black background to enhance the visibility of rice contours and fine structural details.

For 3D reconstruction, we propose a novel multi-scale neural radiance field method based on NeRFacto [26], referred to as MSNeRF, which incorporates two key mechanisms: (1) A structure-detail collaborative reconstruction mechanism:During early training, the model leverages both original and downsampled images to guide learning of global scene structure; in later stages, it focuses on high-resolution images to capture fine-grained local details and improve reconstruction fidelity; (2) A dynamic initial density scheduling strategy:We introduce a cosine-annealed initialization approach to adaptively control the average scene density throughout training, which enhances stability and training efficiency while mitigating reconstruction artifacts. With these mechanisms, MSNeRF achieves integrated modeling of both global structure and local details while preserving geometric consistency, substantially improving the 3D reconstruction quality of intricate plant structures such as rice.

Furthermore, to enable more precise and robust organ-level segmentation of rice, we present a 3D segmentation framework named VRKGNet, which integrates image and point cloud modalities and consists of the following three modules: (1) An image segmentation module that performs semantic segmentation on multi-view images; (2) An image-to-point-cloud projection module that maps the segmentation results onto the corresponding 3D point cloud to spatially align semantic information; (3) A point cloud segmentation module that conducts organ-level segmentation based on the fused 3D data. By effectively combining the semantic richness of 2D images with the spatial geometry of 3D point clouds, VRKGNet achieves superior accuracy and robustness in segmenting complex plant structures, providing a reliable foundation for high-precision phenotypic analysis of rice.

The main contributions of this paper are summarized as follows:

• •We propose MSNeRF, a novel approach for fine-grained 3D reconstruction of rice from low-quality images, specifically designed for breeding scenarios. The method integrates multi-scale image features and introduces a dynamic neural field density modulation mechanism to enhance reconstruction accuracy. Notably, MSNeRF can produce high-quality reconstructions even only with 10 input images.
• •We propose an integral VRKGNet framework that leverages point clouds and images for 3D instance and semantic segmentation, significantly enhancing performance and robustness for complex plant point cloud segmentation tasks.
• •RKM, a rice knowledge modeling pipeline, significantly mitigates resource burdens in building rice datasets. RKM rapidly generates multimodal data, including point clouds and images, ensuring high quality and consistency and enabling comprehensive rice phenotype analysis.
• •We introduce MMR, the comprehensive, multimodal, multitask rice dataset with high-quality labeling, enabling more detailed and accurate research in rice phenotyping.

2. Related work

2.1. Vegetation point cloud construction methods

Creating point cloud datasets is crucial for advancing phenotype studies in plant organ datasets. Various methods currently exist for constructing plant point cloud data.

Schunck et al. [27] generate point cloud data of maize and tomatoes using laser triangulation scanners. Yang et al. [28] utilize a 3D laser scanner for maize point cloud reconstruction, while Yan et al. [29] use a handheld binocular imaging scanner for cotton point cloud reconstruction. Zarei et al. [12] integrate scanners and cameras to provide sorghum point cloud and image data. These methods often yield point clouds that are sparse, noisy, and lack color information. To address these challenges, some methods employ multi-view stereo matching methods [15,30] to generate denser point clouds with color information. Luo et al. [31] use cameras to capture panoramic images of vegetation and employ Metashape software to create a temporal point cloud dataset of potatoes. Nevertheless, these methods are time-consuming for reconstruction and sensitive to lighting variations. Recently, Hu et al. [32] have utilized cameras to collect data and reconstruct point clouds of watermelons, grapes, potatoes, and oranges using the NeRF.

NeRF employs multi-layer perceptrons to streamline 3D scene learning and representation, significantly reducing reconstruction time. It has garnered attention in computer vision for its advanced visual quality and view synthesis capability. NeRF enhances reconstruction quality, accelerates training and inference speeds, and enables the reconstruction of both large and dynamic scenes. While it has been utilized for 3D crop scale measurements [32,33], its application in rice phenotypic research remains unexplored. NeRF leverages multi-view images to gather 3D data and then fits these perspectives using an implicit 3D model within a neural network. This integrates deep learning strengths with traditional 3D vision techniques, enabling dense vegetation point cloud reconstruction. Consequently, it presents substantial potential for densely packed objects like rice plants.

2.2. 3D point cloud segmentation methods

Segmentation of plant organs is crucial for phenotypic research, particularly with image instance segmentation techniques. López et al. [34] use Mask R-CNN to segment chilli peppers and their pedicels. Liu et al. [35] develop a specialized Mask R-CNN for accurate cucumber localization in greenhouses. To address occlusion challenges, Kim et al. [36] utilize a U-Net-based reconstruction network to segment cucumber morphology, enhancing the training dataset with synthetic data to improve occlusion handling. Despite these advances, this method still has limitations with occluded objects.

Researchers adopt point cloud instance segmentation as an alternative to traditional image segmentation for detailed phenotypic information on plant organs. Point cloud technology addresses occlusion challenges and enables precise 3D measurements of physical quantities like length and thickness. For instance, Li et al. [37] segment maize organs using PointNet architecture, while Luo et al. [31] develop the Eff-3DPSeg framework for effective plant organ segmentation and feature extraction in soybeans. Wang et al. [38] utilize PartNet methods for organ-level instance segmentation of plant point clouds. While point cloud segmentation offers improved spatial localization accuracy, it lacks the rich color and category information images provide.

Researchers have recently explored integrating image and point cloud data to overcome single-source limitations. The PlantPart project by Shi et al. [39] utilizes a multi-view approach that initially uses 2D images to segment isolated plant point clouds, followed by point cloud classification. However, further validation is needed to determine its applicability to rice plant organ segmentation. Effective merging image and point cloud data remains an unresolved research question in plant organ segmentation.

3. Materials and methods

3.1. MMR data collection

3.1.1. Vision Rice Knowledge Graph, VRKG

Given the complexity and diversity of rice data and annotations, we introduce the Vision Rice Knowledge Graph (VRKG) to better integrate multimodal data and corresponding annotations and standardize them. Knowledge graphs are instrumental for mapping relationships among various knowledge types. The VRKG comprises three main components:multi-view images, point clouds, and camera parameters. These components are detailed below and illustrated in Fig. 1.

Fig. 1.

The VRKG dataset defines a Vision Rice Knowledge Graph (VRKG), which encompasses three types of knowledge:multi-view images along with their annotations, instance and semantic segmentation labels for point clouds, and the projection matrices between images and point clouds.

Multi-View Images: To obtain comprehensive view information for an object, it is necessary to capture images from multiple perspectives. By combining these multi-view images, 3D structural information can be reconstructed. Furthermore, the images are annotated at the instance level, identifying each grain, leaf, and stem.

Point Clouds: After reconstructing the object's point cloud, it is stored and annotated. The annotation includes both semantic and instance segmentation. Semantic annotation distinguishes organ regions by treating all panicle grains as one whole. Instance annotation separates individual instances within each organ class, such as identifying each panicle grain as a separate instance. This enables accurate panicle grain counting through instance segmentation.

Camera Parameters: In NeRF, the camera parameters of each image are crucial as they capture the connection between the image and the actual object. The parameters saved include internal parameters (focal length, principal point, distortion) and external parameters (position, orientation) for each viewpoint. Ensuring accurate alignment and integration of the multi-view images significantly improves the precision and quality of the 3D reconstruction.

We create a comprehensive rice knowledge repository encompassing multimodal data and various annotations by integrating the aforementioned dataset into the knowledge graph. As shown in Table S1(in the support material), the knowledge graph constructed in this paper provides detailed annotation of rice across multiple modalities. This not only supports a variety of research tasks related to rice but also meets the needs for data fusion.

3.1.2. Rice knowledge modeling (RKM) pipeline

For rapid, low-cost, organized data collection, we construct the Rice Knowledge Modeling (RKM) Pipeline, illustrated in Fig. 2 a. RKM encompasses building a dataset in two stages:1) Setting up relevant acquisition equipment to construct the point cloud. 2) Denoising and annotating the point cloud and surrounding images.

Fig. 2.

(a):The pipeline of dataset building. (b):(1) The task-specific model acquisition setup includes the following components:1-4:Supplemental lighting. 6-8:Black background clothes. 9:Tray. 10:D435i camera. 11:Robotic arm. 12:Diffuser cloth. (2):The framework for a phenotyping system based on NeRF comprises four stages:(1) Data collection:Collecting images surrounding the rice plant. (2) Pose estimation:Determining camera internal and external parameters. (3) Model training:Training the NeRF using images and estimated poses. (4) Point cloud generation:Producing point clouds from the trained NeRF model.

Model Acquisition Equipment We set up a recording system to efficiently collect real-world rice models, as illustrated in Fig. 2 b. Detailed hardware components can be found in Table S2 of the supplementary material. The system comprises a robotic arm and a D435i camera that captures images at 1280x720 resolution. The robotic arm is designed for flexibility, enabling it to adjust to different heights and angles under program control and perform a 360-degree comprehensive scan. This flexibility allows for capturing complete information about the rice from multiple perspectives, covering both morphological and structural details. The system also features supplementary lighting and multiple backdrop cloths to ensure consistent image quality under various experimental conditions.

We use this system to perform a 360-degree scan of the rice plant and obtain multi-angle image data. For each plant, we obtain approximately 120 images surrounding it. As shown in Fig. 2 b, we then use the NeRF to reconstruct the 3D scene of the rice. We first use the COLMAP software [40,41] to process and analyze the collected multi-view image data. COLMAP can efficiently reconstruct the scene and extract the camera's internal and external parameters. The entire scanning and 3D reconstruction process can be completed within 5 min. This system is more convenient and cost-effective for acquiring point cloud data than traditional 3D hardware scanning methods.

Dataset Production After scene reconstruction, we utilize the median method to process the data. This approach determines depth by calculating the midpoint of each pixel's ray, effectively extracting the rice plant point cloud information and generating 3D point cloud data. We then employ a Gaussian distribution method for denoising, yielding cleaner, refined point cloud data. Subsequently, precise instance annotations are performed using the CloudCompare tool. To further enhance data quality, we reapply the Gaussian distribution method to denoise each individually annotated instance's point cloud. This process effectively removes errors arising from inaccurate annotations. We then downsample the point clouds, controlling the number per rice plant to approximately 100,000. This sampling adequately represents the detailed characteristics of the rice plants.

Subsequently, we write code to unify different instance labels, annotating various rice plant organs as “grain”, “leaf”, and “stem”, completing semantic segmentation annotation. Utilizing the camera's intrinsic and external parameters, we removed occluded points at specific perspectives. We precisely projected annotated point cloud information onto corresponding images. Optimizing projection results via morphological closing filled gaps and smoothed edges, preliminarily completing rough image annotation. Finally, we employed the X-AnyLabeling tool for enhanced annotation precision and detailed refinements. This meticulous processing ensured high visual quality and annotation standards in our data.

3.2. MSNeRF

While maintaining the expressive capabilities of NeRF, MSNeRF (as illustrated in Fig. 3) significantly enhances the reconstruction success rate by incorporating two key components:a structure–detail collaborative learning mechanism and a dynamic scheduling strategy for density initialization.

Fig. 3.

The architecture of MSNeRF comprises two main modules: (1) Structure-Aware Multi-Scale Supervision:Jointly leverages original and downsampled images in early training to learn global structure, and focuses on high-resolution details in later stages for finer reconstruction. (2) Dynamic Density Initialization Via CosineAnnealing:Adopts a phase-wise cosine scheduling strategy to dynamically adjust initial scene density, enhancing training stability and reconstruction accuracy.(HR means High-resolution, LR means Low-resolution).

3.2.1. NeRF rendering and volume integration

In NeRF's volumetric rendering process, a camera ray is parameterized as:

$$r(t)=\mathbf{o}+t\mathbf{d}$$ (1)

where $\mathbf{o}\in {\mathbb{R}}^3$ denotes the ray origin, $\mathbf{d}\in {\mathbb{R}}^3$ is the normalized direction vector, and t ∈ [t_n, t_f] is the ray depth. The model samples N points along the ray. For the i-th sample, it predicts the color c_i and volumetric density σ_i, and defines the spacing between adjacent samples as:

$${\delta }_i=t_{i+1}-t_i$$ (2)

The accumulated transmittance, opacity, and corresponding weight are computed as:

$$T_i=\exp \left(-\sum _{j=1}^{i-1}{\delta }_j{\sigma }_j\right),{\alpha }_i=1-\exp (-{\delta }_i{\sigma }_i),w_i={\alpha }_i{T}_i$$ (3)

The final pixel color is obtained via weighted summation:

$$C_r=\sum _{i=1}^N{w}_i{c}_i$$ (4)

The model is optimized by minimizing the mean squared error between predicted and ground-truth colors:

$$L_{\text{mse}}=\frac{1}{N}\sum _{r=1}^N{‖C_r-C_r^{gt}‖}^2$$ (5)

3.2.2. Structure-Aware Multi-Scale Supervision

Early-stage NeRF training often struggles to learn global scene structure, hindering effective fine-detail recovery. To address this, we introduce a multi-scale supervision strategy that leverages both the high-resolution image I_gt and its downsampled counterpart I_low to guide network learning. The corresponding loss function is defined as:

$$L_{\text{rgb}}=\left\{\begin{array}{ll}‖C_r-I_{gt}‖+\lambda ‖C_r-I_{\mathit{low}}‖,& \text{if count}\le Ratio\\ {‖C_r-I_{gt}‖}^2,& \text{otherwise}\end{array}\right)$$ (6)

where λ is the weighting coefficient for low-resolution supervision. Ratio denotes a range (5000–8000); within this interval, the probability of crop-structure reconstruction failure is significantly reduced. we set this value to 5000 in all experiments. Detailed parameter settings can be found in Table S3 of the support material. This strategy encourages the model to first capture global structure, then gradually refine local details, enabling joint optimization of structural consistency and fine appearance.

3.2.3. Dynamic Density Initialization

Since NeRF is sensitive to the initialization of volumetric density, a fixed value can lead to instability or suboptimal convergence. We propose a cosine-based scheduling approach that dynamically modulates the average initial density across training stages. The density is computed as:

$${\rho }_{\text{avg}}(t)={\rho }_{\min }+\frac{1}{2}({\rho }_{\max }-{\rho }_{\min })\left(1-\cos \left(\pi \cdot \frac{t-t_s}{t_e-t_s}\right)\right)$$ (7)

where t is the current training iteration, t_s, t_e are the start and end iterations of a scheduling stage, ρ_min, ρ_max denote the density bounds for that stage. The stage configuration is summarized in the Supplementary Materials Table S4. This scheduling mechanism ensures progressive adaptation to different learning phases, improving both training stability and convergence efficiency.

3.3. VRKGNet

This section introduces VRKGNet, a comprehensive processing framework for 3D point cloud semantic and instance segmentation. As shown in Fig. 4, VRKGNet comprises three sub-modules:an image segmentation module for instance segmentation of images; a mapping images to point clouds module mapping 2D images segmentation to the point clouds; and a point cloud segmentation module leveraging 2D segmentation results as prior knowledge for semantic and instance segmentation on the 3D point cloud.

Fig. 4.

The architecture of VRKGNet comprises three main modules: (1) Image Segmentation Module:Segmenting images from multiple perspectives. (2) Mapping Images to Point Clouds module:Integrating and projecting image segmentation results onto point clouds as prior knowledge. (3) Point Cloud Segmentation Module:Segmenting point clouds with prior knowledge. These modules enable efficient conversion and analysis from images to point clouds, ensuring thorough data integration and interpretation.

3.3.1. Problem formulation

For each rice plant, we collect both RGB images of its surrounding environment and corresponding point clouds as input data. Specifically, for each plant, the input images are $I={\{I_i\}}_{i=1}^M$, where I_i is an RGB image from the i_th perspective, and M is the total number of perspectives. The point clouds are $P={\{P_n\}}_{n=1}^N$, where P_n is the n_th point, and N is the total number of points. VRKGNet receives the input to obtain semantic segmentation results Ω^sem and instance segmentation results Ω^inst.

3.3.2. Image segmentation module

To obtain 2D prediction results for rice plant images, we first pass the multi-view images $I={\{I_i\}}_{i=1}^M$ through the Image Segmentation Module. This module uses the yolov8 [42], divided into Backbone, Neck, and Head.

Backbone: The Backbone uses a stacked CSP [43] structure with ELAN [44] concept to maintain a lightweight design while extracting richer gradient flow information.

Neck: The neck uses a PAN [45] architecture to enhance the Backbone output. The PAN not only propagates semantic information bottom-up and transfers it top-down.

Head: The Head module adopts the decoupled head structure, separating classification and detection tasks. We convert the instance segmentation results to obtain the semantic segmentation result F_i for I_i.

3.3.3. Mapping images to point clouds module

Fig. S1(in the support material) shows the Mapping Images to Point Clouds module projecting ${\{F_i\}}_{i=1}^M$ onto the 3D point cloud to aid 3D point cloud segmentation accuracy by providing predictive class information for each point. Nevertheless, projecting ${\{F_i\}}_{i=1}^M$ onto the point cloud may result in occluded objects sharing labels with foreground points, especially in densely packed objects like rice plants. To address this, we employ an occluded point removal method γ [46] to extract the foreground point cloud $\hat{P}$ γ requires P, the location of camera C_i and search radius R. We use the i_th camera external parameters E_i to determine the camera location.

$$\begin{array}{ll}{C}_i& =-E_i(R)\times E_i(T),\\ \hat{P}& =\gamma (P,C_i,R)\end{array}$$ (8)

We associate $\hat{P}$ with I_i using projection matrix T_i, projecting $\hat{P}$ onto I_i. The detailed projection process is as follows.

• 1.Utilize $\hat{P}$ coordinates $(\hat{x},\hat{y},\hat{z})$ and projection matrix T_i to determine image coordinates (u, v).

$${[{\hat{x}}^{\prime }{\hat{y}}^{\prime }{\hat{z}}^{\prime }]}^{\text{T}}=T_i\cdot {[\hat{x}\hat{y}\hat{z}]}^{\text{T}},u=\frac{{\hat{x}}^{\prime }}{{\hat{z}}^{\prime }},v=\frac{{\hat{y}}^{\prime }}{{\hat{z}}^{\prime }}$$ (9)

• 2.Convert floating point coordinates (u, v) to integer pixel indices, ensuring indices are within image dimensions w × h.

$$\begin{array}{ll}u& =\min (\max (\text{int}(u),0),w-1)\\ v& =\min (\max (\text{int}(v),0),h-1)\end{array}$$ (10)

• 3.Retrieve specific category annotations from segmentation results based on indices. Categories are:0 for “grain,” 1 for “leaf,” 2 for “stem,” and 3 for “unknown.” The “unknown” category includes background elements and incorrectly classified points. The prior knowledge of P is set to 3 by default.

$$F\widehat{P_n}=F_i[u,v]$$ (11)

Subsequently, we concatenate P coordinates (x, y, z) with prior knowledge $F\widehat{P_j}$ from sampled perspectives j = (1, 2, …, o) to form a new point cloud feature Q, where o is the number of images we uniformly sample from the surround view.

$$Q=\text{Concatenate}(P,{\{F\widehat{P_j}\}}_{j=1}^o)$$ (12)

3.3.4. PointCloud segmentation module

After obtaining the new point cloud Q, we design the PointCloud Segmentation Module that takes Q as input and performs both 3D instance and semantic segmentation simultaneously. Specifically, the process begins with feature extraction from the Q using a 3D-UNet, yielding F_vox. Subsequently, two separate branch models handle the 3D instance and semantic segmentation tasks. For 3D instance segmentation, we primarily construct it based on the SPFormer [47], which uses superpoints to reduce the number of points in the cloud. However, when calculating superpoints for rice plants using the method described in Ref. [48], the number of superpoints matches the number of points in the cloud. Thus, we eliminate the superpoints input from the SPFormer model and skip the process of converting features from voxel to point level to better suit dense environments and improve computational efficiency. Then, the voxel features F_vox are passed through three decoder layers, each consisting of cross-attention and self-attention modules. Compared to SPFormer, we introduce an explicit interaction mechanism between F_vox and query features during the decoding process. This enables the voxel-level point cloud features to be dynamically updated according to the evolving queries across multiple decoding stages, thereby enhancing the feature representation capability. After decoding, the updated query features are fed into the instance prediction head to produce the instance segmentation output, denoted as Ω^inst. For 3D semantic segmentation, we employ a Mamba module [49] to model long-range dependencies among point cloud features, effectively capturing global contextual information. The output features from the Mamba module are then fed into a MLP and a semantic prediction head to generate the final semantic segmentation output Ω^sem.

3.3.5. Loss module

During loss function computation, we employ cross-entropy loss for instance classification L_cls, binary cross-entropy loss and Dice loss for mask classification L_bce and L_dice, and mean squared error loss for score prediction L_score. Considering all these loss functions, the instance segmentation loss L_inst is formulated as:

$$L_{\text{inst}}={\lambda }_1{L}_{\text{cls}}+{\lambda }_2{L}_{\text{bce}}+{\lambda }_3{L}_{\text{dice}}+{\lambda }_4{L}_{\text{score}}$$ (13)

Specifically, λ₁, λ₂, λ₃, and λ₄ are set to 0.5, 1, 1, and 0.5, respectively.

The semantic segmentation loss L_seg is cross-entropy. The semantic segmentation loss is formulated as L_sem = λ₅L_seg, where λ₅ is set to 5.0. Two separate branch models allow the two segmentation tasks to effectively share the underlying network-extracted features, enhancing overall performance and efficiency.

4. Results

4.1. Dataset

To systematically evaluate the performance of the Structure-Aware Multi-Scale Loss and Dynamic Density Initialization strategies under real-world motion blur conditions, we developed the Blur Rice Dataset, which is specifically designed for rice 3D reconstruction. This dataset was acquired using crop phenotyping devices and comprises nine distinct rice scenes, each containing approximately 106 images. Following the standard 9:1 train/validation split in the NeRFstudio [26] framework, the validation set is constructed by uniformly sampling images from each scene, ensuring viewpoint independence between training and validation sets.

To emulate scenarios with restricted viewpoints, we further sampled 8, 10, 12, 14, 16, 18, and 20 frames at equal intervals from each full-view scene in the Blur Rice Dataset, forming the Sparse Blur Rice Dataset. This subset is employed to assess the robustness of reconstruction algorithms under limited viewpoint conditions and to evaluate point cloud reconstruction performance in sparse-view settings. The test set is derived from the Blur Rice Dataset with no overlap with the training set. For test images, background regions were manually removed, retaining only the foreground rice regions of interest (ROI), which are used for PSNR(Peak Signal-to-Noise Ratio) and SSIM(Structural Similarity Index) metric calculations.

For point cloud reconstruction quality assessment, we established high-quality pseudo-ground-truth point clouds. Specifically, NeRFacto [26] was trained for 90,000 iterations to generate point clouds, which were subsequently processed in CloudCompare for denoising, outlier removal, and downsampling (see Table S5 for parameters). The resulting point clouds were then manually reviewed and refined by multiple experts. This procedure provides a robust and reliable evaluation benchmark in the absence of real ground truth, facilitating quantitative metric analysis.

Additionally, we validated the performance of VRKGNet and its individual modules on the MMR dataset, which is organized by plant instance and split into training and validation sets at a 9:1 ratio. The validation set was constructed using an analogous interval sampling strategy.

4.2. Experimental setup

For the reconstruction task, NeRFacto implemented within the NeRFStudio [26] framework serves as our baseline, upon which MSNeRF is developed. Comparative methods include Instant-NGP, SplatFacto, and NeRFacto, all evaluated using the official implementations provided by NeRFStudio. FreeNeRF is also assessed using its NeRFStudio implementation, with model configuration details listed in Table S5.

For point cloud instance segmentation, SPFormer [50] is adopted as the baseline, and VRKGNet is implemented for comparison. Additional comparative models include SoftGroup [51], SoftGroup++ [52], and SPFormer [50].

In the point cloud semantic segmentation task, Pointnet++ [53] and PAConv [54] are evaluated using the MMDetection3D [55] framework, while MSC [56] and CAC [57] are implemented using the Pointcept [58] framework.

All other models are evaluated using either official releases or faithfully reproduced implementations within their respective frameworks to ensure fairness and reproducibility. Detailed configurations are provided in Table S5.

4.3. Evaluation metrics

4.3.1. Image reconstruction metrics

For image reconstruction and evaluation, the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) are commonly employed. These metrics quantify the differences between reconstructed and reference images in terms of pixel-wise error and structural similarity.

PSNR Let I denote the reference image, $\hat{I}$ the reconstructed image, with dimensions H × W, and L_max the maximum possible pixel value. The Mean Squared Error (MSE) is defined as

$$\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{HW}\sum _{i=1}^H\sum _{j=1}^W{\left(I(i,j)-\hat{I}(i,j)\right)}^2.$$ (14)

The Peak Signal-to-Noise Ratio is calculated as

$$\mathrm{P}\mathrm{S}\mathrm{N}\mathrm{R}=10{\log }_{10}\left(\frac{L_{\max }^2}{\mathrm{M}\mathrm{S}\mathrm{E}}\right)\text{dB}.$$ (15)

When images are normalized to [0, 1], L_max is set to 1. Lower MSE and higher PSNR indicate superior reconstruction quality.

SSIM SSIM evaluates image similarity based on luminance, contrast, and structural components. For I and $\hat{I}$, let ${\mu }_I,{\mu }_{\hat{I}},{\sigma }_I^2,{\sigma }_{\hat{I}}^2,{\sigma }_{I\hat{I}}$ denote the local mean, variance, and covariance within a local window, respectively. SSIM is defined as

$$\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}(I,\hat{I})=\frac{(2{\mu }_I{\mu }_{\hat{I}}+C_1)(2{\sigma }_{I\hat{I}}+C_2)}{({\mu }_I^2+{\mu }_{\hat{I}}^2+C_1)({\sigma }_I^2+{\sigma }_{\hat{I}}^2+C_2)}.$$ (16)

Here, $C_1={(k_1{L}_{\max })}^2$ and $C_2={(k_2{L}_{\max })}^2$ are stabilization constants to prevent division by zero, typically set as k₁ = 0.01 and k₂ = 0.03.

4.3.2. Point cloud reconstruction metrics

Chamfer Distance (CD) Chamfer Distance quantifies the average minimum distance between two point cloud sets, reflecting the similarity between reconstructed and ground-truth point clouds:

$$CD(P,Q)=\frac{1}{|P|}\sum _{p\in P}{\min }_{q\in Q}\Vert p-q{\Vert }_2+\frac{1}{|Q|}\sum _{q\in Q}{\min }_{p\in P}\Vert q-p{\Vert }_2$$ (17)

where P and Q denote the point cloud sets.

Normal Consistency (NC) Normal Consistency measures the alignment of normal vectors between corresponding points:

$$NC=\frac{1}{|P|}\sum _{p\in P}|{\mathbf{n}}_p\cdot {\mathbf{n}}_{q^{\ast }}|$$ (18)

where n_p is the normal vector at point p, ${\mathbf{n}}_{q^{\ast }}$ is the normal vector at the nearest neighbor q∗ in Q.

4.3.3. Point cloud semantic segmentation metrics

Mean Intersection over Union (mIoU) and overall accuracy (Acc) serve as the primary evaluation metrics. Given K semantic classes, let TP_k, FP_k, FN_k, and TN_k denote the true positive, false positive, false negative, and true negative counts for class k, respectively. The confusion matrix entry n_ij represents the number of points with ground-truth class i and predicted class j. The Intersection over Union (IoU) for class k is defined as

$${\mathrm{I}\mathrm{o}\mathrm{U}}_k=\frac{{\mathrm{T}\mathrm{P}}_k}{{\mathrm{T}\mathrm{P}}_k+{\mathrm{F}\mathrm{P}}_k+{\mathrm{F}\mathrm{N}}_k}=\frac{{\mathrm{n}}_{kk}}{{\sum }_{j=1}^K{\mathrm{n}}_{kj}+{\sum }_{i=1}^K{\mathrm{n}}_{ik}-{\mathrm{n}}_{kk}}.$$ (19)

Mean Intersection over Union is computed as

$$\mathrm{m}\mathrm{I}\mathrm{o}\mathrm{U}=\frac{1}{K}\sum _{k=1}^K{\mathrm{I}\mathrm{o}\mathrm{U}}_k.$$ (20)

4.3.4. Point cloud instance segmentation metrics

Average Precision (AP), based on IoU matching between predicted and ground-truth instances, is employed as the primary metric. For a given IoU threshold t, AP@t is computed from the precision-recall curve; the final AP score is averaged over the set of thresholds {0.50, 0.55, …, 0.95}:

$$\mathrm{A}\mathrm{P}=\frac{1}{10}\sum _{t=0.50}^{0.95}\mathrm{A}\mathrm{P}@t.$$ (21)

Additionally, single-threshold AP values are reported as follows:

$${\mathrm{A}\mathrm{P}}_{50}=\mathrm{A}\mathrm{P}@0.50,{\mathrm{A}\mathrm{P}}_{25}=\mathrm{A}\mathrm{P}@0.25.$$ (22)

4.4. Main results of MSNeRF

We systematically compare MSNeRF with Instant-NGP, SplatFacto, and NeRFacto on the Blur Rice Dataset, using evaluation metrics including image quality (PSNR, SSIM) and geometric consistency (CD, NC).

Quantitative Results Comparison As summarized in Table 1, the proposed MSNeRF consistently achieves superior performance over mainstream baselines across multiple key metrics. In terms of image quality, MSNeRF attains an average PSNR of 33.89 and an SSIM of 0.99, both markedly higher than those of competing methods. For geometric reconstruction, MSNeRF yields an average CD of 8.05 and NC of 0.87, representing the best performance among all methods and exhibiting stability across diverse scenes. In contrast, Instant-NGP and SplatFacto show inferior geometric quality, with mean CD values of 626.15 and 1446.35, and NC values of 0.58 and 0.52, alongside considerable variability across scenes. While SplatFacto achieves higher PSNR in certain viewpoints, its visual and geometric metrics are notably inconsistent, with CD two orders of magnitude higher than MSNeRF and lower normal consistency. NeRFacto performs slightly worse than MSNeRF in geometric metrics (CD = 10.66, NC = 0.84), further underscoring the superiority and robustness of MSNeRF in geometric reconstruction.

Table 1.

Reconstruction results on the validation set of the Blur Rice Dataset are presented, including image quality (PSNR/SSIM) and point cloud geometric consistency (CD; NC). Arrows indicate the preferred direction for each metric.

	Mean	demo1	demo2	demo3	demo4	demo5	demo6	demo7	demo8	demo9
PSNR→
Instant-NGP	31.64	33.68	32.33	29.78	32.02	31.53	31.87	30.97	31.32	31.26
Splatfacto	32.34	35.45	31.53	31.01	30.67	33.16	33.50	33.90	31.66	30.21
NeRFacto	30.41	32.56	31.04	29.24	30.59	29.83	30.36	29.98	29.98	30.11
MSNeRF	33.89	36.47	34.15	32.77	33.61	33.22	34.55	32.77	33.56	33.93
SSIM→
Instant-NGP	0.98	0.99	0.99	0.98	0.98	0.98	0.98	0.98	0.98	0.98
Splatfacto	0.99	0.99	0.99	0.99	0.98	0.99	0.99	0.99	0.98	0.98
NeRFacto	0.98	0.99	0.98	0.98	0.98	0.98	0.98	0.98	0.98	0.98
MSNeRF	0.99	0.99	0.99	0.99	0.98	0.99	0.99	0.98	0.99	0.99
CD↔
Instant-NGP	626.15	306.38	1677.27	415.58	428.25	241.63	211.82	270.51	1832.99	250.88
Splatfacto	1446.35	681.47	2240.14	1172.31	1042.34	1575.48	1437.83	2145.56	1186.56	1535.48
NeRFacto	10.66	9.25	12.46	9.59	8.86	12.56	11.26	9.38	12.80	9.82
MSNeRF	8.05	7.02	9.20	6.95	6.85	10.86	8.59	6.86	8.82	7.33
NC→
Instant-NGP	0.58	0.59	0.50	0.54	0.52	0.62	0.66	0.66	0.45	0.65
Splatfacto	0.52	0.52	0.64	0.40	0.56	0.36	0.53	0.56	0.53	0.57
NeRFacto	0.84	0.87	0.81	0.84	0.86	0.82	0.81	0.85	0.81	0.85
MSNeRF	0.87	0.88	0.85	0.87	0.88	0.86	0.86	0.88	0.86	0.88

Visual Results Analysis Fig. 5 presents a visual comparison of rendering and point cloud reconstruction results across methods. Instant-NGP and SplatFacto exhibit pronounced deficiencies in geometric boundaries and details, with substantial noise in point cloud reconstruction and blurred, distorted rendering outputs. NeRFacto demonstrates improvements in rendering quality and geometric reconstruction, yet still suffers from blurred edges and loss of fine details. In contrast, MSNeRF delivers the best overall performance, with sharp geometric details, high edge consistency, and significantly reduced point cloud noise. In summary, MSNeRF surpasses existing methods in both accuracy and efficiency. With comparable computational resources to NeRFacto, MSNeRF achieves higher-quality geometric reconstruction.

Fig. 5.

Visual comparison between MSNeRF and mainstream 3D reconstruction methods in terms of rendered image quality and point cloud generation.

4.4.1. Ablation study

MSNeRF incorporates strategies such as Structure-Aware Multi-Scale Loss and Dynamic Density Initialization. To assess the independent contributions and synergistic effects of these modules, we perform ablation studies on the NeRFacto baseline, employing unified training parameters and a fixed training schedule of 30,000 iterations.

As shown in Table 2, in the absence of any additional modules, the model achieves a CD of 10.66 and an NC of 0.835. Introducing only Dynamic Density Initialization reduces the CD to 8.76 and increases NC to 0.866, indicating a substantial improvement in reconstruction accuracy. Incorporating only the Structure-Aware Multi-Scale Loss further decreases CD to 8.24 and raises NC to 0.869, demonstrating that this loss function effectively captures structural information and enhances model performance. When both modules are utilized concurrently, the model attains optimal performance, with CD at 8.05 and NC at 0.870, demonstrating the complementary nature and effectiveness of both modules.

Table 2.

The effect of various module combinations on model performance is evaluated based on the NeRFacto baseline.

Dynamic Density Initialization	Structure-Aware Multi-Scale Loss	CD↔	NC→
		10.66	0.835
✓		8.76	0.866
	✓	8.24	0.869
✓	✓	8.05	0.870

4.4.2. Performance analysis of MSNeRF under sparse viewpoints

Table S6 presents a comparative analysis of MSNeRF against mainstream methods (SplatFacto, Instant-NGP, NeRFacto) based on PSNR, SSIM, CD, and NC metrics across different numbers of sparse viewpoints (ranging from 8 to 20). Detailed scene-wise results can be found in the Supplementary Table S7. MSNeRF consistently outperforms competing methods across all viewpoint counts, with particularly notable advantages under extremely sparse conditions. We provide point clouds exported by each model for the 10-view and 20-view configurations reported in Table S6 (see Supplementary Data S1).

Specifically, as the number of viewpoints increases, MSNeRF's PSNR and SSIM steadily improve, rising from 26.98 to 0.97 with 8 viewpoints to 31.26 and 0.98 with 20 viewpoints, substantially exceeding those of Instant-NGP and NeRFacto. MSNeRF also exhibits superior geometric reconstruction performance. For instance, with only 8 viewpoints, MSNeRF achieves a Chamfer Distance of 358.90 and a Normal Consistency of 0.65, whereas Instant-NGP reports a CD of 1833.33 and NC of 0.50. As the number of viewpoints increases to 20, MSNeRF's CD further decreases to 11.80 and NC increases to 0.83, outperforming other methods by a significant margin.

Fig. S2 presents visualizations of point cloud generation under the 20-viewpoint setting. Despite achieving a high PSNR score in demo1, SplatFacto fails to produce useable point clouds. NeRFacto yields better results than Instant-NGP and SplatFacto, yet its reconstructions are often affected by outlier points and ineffective point clouds in certain scenarios. In contrast, MSNeRF generates point clouds of markedly higher quality under the 20-viewpoint condition and maintains structural integrity even with as few as 10 viewpoints.

These findings indicate that MSNeRF exhibits a considerably lower reconstruction failure rate in practical applications. Its performance becomes increasingly reliable with more input viewpoints, underscoring its promising potential for real-world agricultural 3D reconstruction tasks. In summary, MSNeRF provides a robust and generalizable solution for 3D reconstruction under sparse sampling conditions, achieving superior reconstruction quality and geometric consistency.

4.5. Main results of VRKGNet

The effectiveness of our multimodal fusion approach for both semantic and instance segmentation is validated through comparisons with other mainstream models. The input to VRKNet is the coordinates of the point cloud and prior knowledge, i.e., $(x,y,z,F\hat{P})$, and the others is the coordinates and colors of the point cloud, i.e., (x, y, z, r, g, b).

Results of semantic segmentation. We compare our method VRKGNet with mainstream semantic segmentation methods on the MMR dataset. As shown in Table 3, our method achieves the best results in IoU Per Categor, mean IoU (mIoU), and mean accuracy (mAcc), surpassing the previous best model CAC [57] by 7.84 points in IoU (88.79% vs. 80.95%). The leaves and stems of rice plants have similar colors and grow compactly, making them difficult to segment. Our model outperforms CAC by over 16.27 points in “stem” (94.45% vs. 78.18%) and 6.67 points in “leaf” (80.40% vs. 73.73%). These experimental results demonstrate the effectiveness of our multimodal approach.

Table 3.

Comparison of mainstream semantic segmentation methods on the MMR dataset. “∗” indicates the method uses prior knowledge generated by the Mapping Images to Point Clouds module. “VRKGNet w/o PK” refers to VRKGNet without prior knowledge.

Method	IoU Per Category (%)			mIoU	mAcc
	grain	leaf	stem
PointNet++ [53]	89.27	57.57	64.41	70.18	88.07
PointNet++∗	93.19	78.64	84.96	85.60	93.99
PAConv [54]	88.22	54.97	59.40	66.63	86.61
PAConv∗	94.67	83.26	88.43	88.79	95.37
MSC [56]	86.72	56.87	55.68	66.43	76.33
MSC∗	91.01	68.49	77.76	79.09	91.01
CAC [57]	90.94	73.73	78.18	80.95	93.03
CAC∗	89.77	74.25	83.46	82.49	93.41
VRKGNet w/o PK	83.81	73.07	67.16	74.28	88.30
VRKGNet (ours)	91.52	80.40	94.45	88.79	94.13

Results of instance segmentation. As shown in Table 4, we set new benchmarks. We evaluate instance segmentation results with average precision (AP) metrics across thresholds (AP, AP₅₀, AP₂₅). We observe that VRKGNet achieves AP scores of 44.74%, 13.54 points higher than the previous best model Spformer (31.20%). Under AP₅₀ and AP₂₅, our model achieves scores of 63.97% and 84.55%.

Table 4.

Comparison of mainstream instance segmentation methods on the MMR dataset.

Model	AP				AP50				AP25
	mean	grain	leaf	stem	mean	grain	leaf	stem	mean	grain	leaf	stem
softgroup	22.00	0.30	50.60	17.50	29.90	0.30	63.60	25.70	49.60	0.30	81.80	66.70
softgroup∗	32.70	0.30	59.80	38.10	43.10	0.30	62.40	66.70	52.40	0.30	81.80	75.00
softgroup++	26.10	0.30	43.70	34.30	40.20	0.30	55.80	64.60	53.40	0.30	84.90	75.00
softgroup++∗	33.10	0.00	59.00	40.30	45.70	0.00	70.40	66.70	52.30	0.10	81.80	75.00
Spformer	31.20	19.10	41.30	33.40	56.30	45.50	63.60	59.90	73.70	77.80	69.10	74.30
Spformer∗	37.20	32.60	52.20	26.80	64.20	66.80	63.60	62.20	80.00	86.80	71.20	81.80
VRKGNet w/o PK	34.09	25.77	46.18	30.30	54.41	49.19	54.55	59.50	75.75	74.74	67.90	84.60
VRKGNet(ours)	44.74	39.23	46.04	48.95	63.97	67.15	51.94	72.82	84.55	83.36	82.28	88.00

Notably, VRKGNet outperforms mainstream methods comprehensively even without incorporating any prior knowledge, highlighting its robustness and generalization ability in challenging segmentation scenarios.

4.5.1. Ablation study

PointCloud Segmentation Module Performance To comprehensively evaluate the 3D segmentation method in VRKGNet, we conducte experiments without the Mapping Images to Point Clouds module. Specifically, we replace the image segmentation results with the color of the point cloud as prior knowledge in 3D segmentation, keeping the rest of the model unchanged. As shown in Table 3, Table 4, the experimental results underscore the exceptional capabilities of our 3D segmentation method. Notably, even without the image segmentation prior, our method outperforms mainstream methods.

In semantic segmentation, our module significantly outperforms other existing methods. Our approach offers advantages including simplicity, lightweight architecture, and ease of deployment from our straightforward segmentation network design. In instance segmentation, our method achieves significantly higher results than mainstream methods.

Mapping Images to Point Clouds module Performance To evaluate the effectiveness of the Mapping Images to Point Clouds module, we integrate its prior knowledge into mainstream semantic and instance segmentation methods for a detailed quantitative analysis. Table 3 shows significant performance improvements across mainstream semantic segmentation methods with prior knowledge. All methods exhibit substantial gains across all evaluation metrics, confirming that the Mapping Images to Point Clouds module substantially enhances recognition and segmentation of different point cloud components, improving overall segmentation accuracy. Table 4 demonstrates the positive impact of the prior knowledge from the Mapping Images to Point Clouds module on mainstream instance segmentation method performance. Since instance segmentation requires distinguishing instances and understanding their semantics, introducing the Mapping Images to Point Clouds module's prior knowledge significantly improves instance segmentation performance.

Mamba Performance As shown in Table S8, we systematically investigate three configurations on the Spformer baseline: no additional module, addition of an MLP layer, and replacement of the MLP layer with the Mamba module, assessing their impact on point cloud instance and semantic segmentation tasks. The results indicate that joint training with a semantic segmentation task effectively enhances instance segmentation accuracy; for example, introducing an MLP layer increases AP from 37.50% to 40.85%. This improvement can be attributed to the semantic segmentation task's ability to correct projection errors. However, the MLP layer is limited in modeling spatial relationships among point clouds, resulting in suboptimal semantic segmentation performance (mIoU:75.16%).

Replacing the MLP layer with the Mamba module yields substantial improvements in both instance and semantic segmentation tasks, with AP rising to 44.74% and mIoU reaching 88.79%. The Mamba module effectively captures spatial relationships within the point cloud, enabling more expressive feature learning and improved generalization(see Fig. S3). Overall, the Mamba module overcomes the spatial modeling limitations of MLPs and delivers stronger performance in point cloud scene understanding tasks.

4.6. Visualization results

As shown in Fig. 6, we present a qualitative comparison between VRKGNet and mainstream methods in 3D semantic segmentation. Our method most accurately aligns with the ground truth labels. Similarly, we visualize instance segmentation results in Fig. 7. Compared to other models, the instance segmentation results of our method are closer to the ground truth, especially for panicle grains. In instance segmentation, distinguishing panicle grains is challenging. We observe that Softgroup and Softgroup++ models cannot distinguish individual grains at all; Spformer can discern individual grains, but the boundaries are quite blurred. None of the compared models match our model's segmentation performance.

Fig. 6.

Qualitatively compare the semantic segmentation results of mainstream methods.

Fig. 7.

Qualitative comparison of instance segmentation results of mainstream methods.

5. Discussion

5.1. Mitigating geometric inconsistencies in real-world rice sparse-view 3D reconstruction

This paper presents MSNeRF, a novel model built upon NeRFacto that addresses the challenges of 3D reconstruction of real-world rice under low-quality image data. The model introduces two key components:a structure-detail collaborative reconstruction mechanism and a dynamic mean initial density scheduling strategy. While existing NeRF-based approaches typically rely on Colmap for camera pose estimation, they are prone to errors under sparse imagery or complex scenes, which can degrade reconstruction quality. MSNeRF is designed to improve robustness under low-quality image conditions while minimizing the reliance on expert-level data acquisition. Our research aligns with recent works such as FreeNeRF [59] and RegNeRF [60], targeting high-quality neural reconstruction from sparse inputs. FreeNeRF highlights that neural fields tend to overfit the training views during early training stages, limiting generalization to novel views. It addresses this via a frequency regularization strategy. Similarly, MSNeRF adopts a progressive reconstruction approach that begins with low-frequency structural learning and incrementally incorporates high-frequency details. In addition, MSNeRF introduces a dynamic mean initial density scheduling mechanism to mitigate early-stage depth overconfidence, a common issue in existing methods. This design enhances the model's ability to cope with geometric inaccuracies inherent in real-world data. Unlike FreeNeRF and RegNeRF, which primarily evaluate their methods on synthetic datasets, MSNeRF demonstrates superior robustness and practical applicability on real rice datasets. A significant observation is that FreeNeRF fails to reconstruct rice structures on the MMR real-world dataset despite its strong performance on synthetic benchmarks. This finding suggests that methods optimized for synthetic data may experience substantial performance drops when confronted with complex multi-view inconsistencies in real scenarios. In contrast, MSNeRF maintains stable performance on such data, underscoring its design's effectiveness and real-world utility.

5.2. A multimodal fusion framework for 3D instance segmentation of rice

This paper introduces VRKGNet, a novel 3D instance segmentation framework that leverages the complementary strengths of images and point clouds tailored for complex rice phenotyping scenarios. Unlike traditional single-modality approaches, VRKGNet integrates semantic cues from images with geometric information from point clouds to achieve superior segmentation performance. This is particularly beneficial for high-density and structurally intricate crops like rice, where VRKGNet demonstrates enhanced robustness and accuracy in recognizing key components such as grain, leaf, and stem. Rice point clouds pose significant challenges in 3D plant phenotyping due to their structural complexity and point density. CNN-based instance segmentation methods, such as SoftGroup and SoftGroup++, are constrained by limited receptive fields, which hinders their ability to model global structural relationships and impairs their discrimination of fine-grained components like grain. To address these limitations, VRKGNet employs a Transformer-based architecture to expand the receptive field of point cloud features and incorporates image-guided feature enhancement to enrich semantic representations. This design overcomes existing bottlenecks in both semantic and instance segmentation tasks. A key observation is that semantic and instance segmentation are inherently interdependent and can benefit from joint modeling. Whereas prior approaches often treat them as separate tasks, VRKGNet demonstrates that unified modeling improves grain-level accuracy and enhances generalization in complex environments. Moreover, although yolov8 excels in image-based segmentation, it struggles with occluded regions in rice panicles. In contrast, point cloud models are robust to occlusion but may confuse visually similar structures such as grain and leaf. This contrast highlights the necessity of fusing image and point cloud modalities for accurate and robust segmentation. Finally, the NeRF-based 3D modeling pipeline integrated into VRKGNet offers a promising direction for end-to-end modeling in agricultural phenotyping, bridging the gap from image data to structural understanding.

5.3. Limitation and future prospects

MSNeRF has demonstrated strong capabilities in rice panicle phenotyping, enabling precise reconstruction of complex 3D panicle structures. Nevertheless, its reliance on high-precision 3D modeling under controlled indoor conditions imposes certain limitations. Specifically, data acquisition must be performed under stable environments–typically in the absence of wind or under light breeze–which significantly restricts its operational flexibility and ease of deployment in real-world paddy field scenarios. For MSNeRF, we conducted an initial sensitivity analysis and carried out systematic experiments under controlled conditions, including uneven lighting, exposure variations, and a relatively high rotation speed (see Table S9 and Fig. S4). The results demonstrate that MSNeRF achieves greater adaptability and robustness compared to the baseline NeRFacto. Given MSNeRF's robustness in sparse-view reconstruction, in the future, we can utilize multiple cameras with synchronized exposure to capture high-quality images from diverse viewpoints simultaneously. This approach aims to mitigate the degradation of image quality caused by natural wind, thereby enhancing the method's applicability and reliability in outdoor agricultural environments.

6. Conclusions

This paper presents the MMR dataset, a multimodal resource for rice that integrates image and point cloud data to support fine-grained segmentation in agricultural applications. To efficiently collect high-quality multimodal data, we developed a rapid data acquisition system and constructed a dataset knowledge graph, termed VRKG. To address challenges posed by low-quality data, we propose MSNeRF, a method that substantially enhances reconstruction quality and success rates. Notably, MSNeRF can reconstruct rice scenes using as few as 10 images. Building on VRKG, we introduce the VRKGNet framework to enable effective fusion and analysis of multimodal data. Experimental results show that VRKGNet surpasses existing methods in accuracy and generalization across complex segmentation tasks. These findings highlight the potential of multimodal data fusion for advancing precision agriculture.

Future work will focus on expanding the MMR dataset and VRKG knowledge graph by incorporating additional crop types and modeling more diverse and complex environmental conditions. We also plan to improve the robustness and accuracy of the VRKGNet framework to support a broader range of agricultural scenarios. Additionally, we aim to extend MSNeRF to outdoor environments, enabling high-quality 3D reconstruction in open-field farming where data quality and lighting conditions are more variable. These developments are expected to contribute to the advancement of intelligent agriculture, fostering greater sustainability and efficiency in global food production.

Author contributions

F.Q., L.F. designed the study, performed the experiment, and wrote the manuscript. F.Q. developed the algorithm and analyzed the data. L.F, W.Z., W.S., J.W. developed the dataset for the algorithms. H.H. and Y.S. supervised experiments at all stages and revised the manuscript.

Funding

This research was supported by Anhui Provincial-Level Scientific and Technological Innovation Tackling Key Problems Plan Project (202523n10050008), Anhui Provincial Key Research and Development Project (2023n06020028) and Institute of Hefei Artificial Intelligence Breeding Accelerator Co. Ltd. (NB2024005-02).

Data availability

The source code for the proposed methods, MSNeRF and VRKGNet, is publicly available on GitHub. The released repositories contain testing scripts and test cases used to reproduce the main results presented in this paper:

• •MSNeRF:https://github.com/qfwysw/MSNeRF.git
• •VRKGNet:https://github.com/qfwysw/VRKGNet.git

The datasets used in the experiments are available from the corresponding author upon reasonable request. For access or further inquiries, please contact the corresponding author.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following are the Supplementary data to this article:

figs1.

figs2.