Machine Learning Approaches to Improve Three Basic Plant Phenotyping Tasks Using Three-Dimensional Point Clouds^,

Machine learning tools can improve plant phenotyping (e.g. lamina identification, stem skeletonization) using 3D point clouds.

Abstract

Developing automated methods to efficiently process large volumes of point cloud data remains a challenge for three-dimensional (3D) plant phenotyping applications. Here, we describe the development of machine learning methods to tackle three primary challenges in plant phenotyping: lamina/stem classification, lamina counting, and stem skeletonization. For classification, we assessed and validated the accuracy of our methods on a dataset of 54 3D shoot architectures, representing multiple growth conditions and developmental time points for two Solanaceous species, tomato (Solanum lycopersicum cv 75 m82D) and Nicotiana benthamiana. Using deep learning, we classified lamina versus stems with 97.8% accuracy. Critically, we also demonstrated the robustness of our method to growth conditions and species that have not been trained on, which is important in practical applications but is often untested. For lamina counting, we developed an enhanced region-growing algorithm to reduce oversegmentation; this method achieved 86.6% accuracy, outperforming prior methods developed for this problem. Finally, for stem skeletonization, we developed an enhanced tip detection technique, which ran an order of magnitude faster and generated more precise skeleton architectures than prior methods. Overall, our improvements enable higher throughput and accurate extraction of phenotypic properties from 3D point cloud data.

Emerging technologies for high-throughput plant architecture capture has increased the demand for automated phenotyping methods (Furbank and Tester, 2011; Fiorani and Schurr, 2013). Extracting plant features is a first step toward quantifying biomass and yield (Mathan et al., 2016), assessing flowering time and drought tolerance (Minervini et al., 2016), mapping genotypes to phenotypes (Balduzzi et al., 2017; Setter, 2012), and studying morphological properties of plant architectures (Conn et al., 2017a; Conn et al., 2017b; Bucksch et al., 2017; Li et al., 2018; Prusinkiewicz and Lindenmayer, 1996). Common phenotyping features of interest include the number, size, and shape of leaves (Wilf et al., 2016; Huang et al., 2018); plant height and growth rates (Madec et al., 2017); and branch lengths, diameters, and angles (Bucksch et al., 2017); among others.

There are many techniques used for noninvasive plant phenotyping, ranging from camera-based imaging (Heckwolf et al., 2015; Nguyen et al., 2015) to three-dimensional (3D) laser scanning (Paulus et al., 2013; Paulus et al., 2014), with advantages and disadvantages to each in terms of cost, accuracy, and throughput. For example, two-dimensional (2D) imaging methods remain a cheap, high-throughput solution (Perez-Sanz et al., 2017; Ubbens and Stavness, 2017; Gao et al., 2018; Zhu et al., 2018) and can be used in conjunction with structure from motion techniques (Szeliski, 2010) to generate point clouds (Dey et al., 2012; Santos et al., 2015; Gélard et al., 2017b) or mesh (Moraes Frasson and Krajewski, 2010; Paproki et al., 2012) representations of plants, the latter providing additional areal and volumetric information. This approach, however, faces several issues, including camera calibration, image registration, object occlusion, and inconsistencies in scaling and illumination. Images that contain an extra depth component (called RGB-D) have also been used in the context of plant phenotyping that helps overcome some of these challenges (Chene et al., 2012; Xia et al., 2015). Further, some platform-specific solutions have been proposed (Nguyen et al., 2016; Gibbs et al., 2018), although general-purpose solutions remain elusive.

On the other hand, light detection and ranging (LiDAR) and 3D laser scanning devices are considerably more expensive but generate point cloud representations of plant geometry with up to micron-level precision. When compared with most 2D methods, 3D can provide a more comprehensive and detailed view of the entire architecture. Prior works have used 3D scanning to study, for example, growth and development (Li et al., 2013) and light interception of leaves (Gaëtan et al., 2012), and these technologies are now being deployed in the field (Sun et al., 2018; Andújar et al., 2013). However, these techniques also face challenges, such as nonuniform sampling of points, outliers and missing data (Li et al., 2013), scalability, and automation (Chaudhury et al., 2018). Thus, developing robust computational methods to study plant shapes from 3D point cloud data has become increasingly important.

Here, we study three basic computational problems that are essential for downstream high-throughput phenotyping: classification of point cloud data into sets of stem and lamina points, lamina counting, and stem skeletonization (Fig. 1). We test our methods on a dataset of 54 3D laser-scanned shoot architectures from two dicot species (tomato [Solanum lycopersicum], Nicotiana benthamiana), each grown across three developmental time points, and five growth conditions (ambient, high-heat, shade, drought, high-light). Prior approaches to point cloud classification in context of phenotyping included both supervised (Paulus et al., 2013) and unsupervised (Wahabzada et al., 2015) machine learning methods, and hand-crafted methods specialized toward the geometry of particular species (Dey et al., 2012; Gélard et al., 2017b). For lamina counting, we extended a region growing algorithm to improve the robustness of lamina segmentation to natural variation, including ripples, wrinkles, and hairs. For stem skeletonization, our method improves tip and root detection to generate more precise skeletons, and we improve run-time by an order of magnitude. Finally, we combine these three pipelines and group lamina, petioles, and petiolules into individual tomato leaves. Our algorithms are open-source and freely available for the community.

Figure 1.

Overview. A summary of the three problems tackled: classification of lamina versus stem points, lamina counting/segmentation, and stem skeletonization.

RATIONALE

Dataset of 3D Plant Shoot Architectures

We used 3D laser scanning to generate point cloud representations of 54 full shoot architectures. We used the FaroArm EDGE model 14000, which is a noninvasive laser scanner and provides micron-level resolution. We scanned plants from two species: 36 tomato plants (Solanum lycopersicum cv m82D) and 18 Nicotiana benthamiana plants. Tomato plants were grown in five conditions (control/ambient light, high-heat, high-light, shade, and drought), and Nicotiana benthamiana plants were grown in three conditions (control, high-heat, and shade). There were two replicates per species-conditions pair. These conditions were selected because they represent a range of realistic environments regularly faced by many plants, and for which there is phenotypic plasticity that challenges accurate phenotyping. Each plant was scanned on developmental days 5, 12, and 20. Each of the 54 architectures was manually classified into sets of stem and lamina points. Full experimental details were previously described (Conn et al., 2017a; Conn et al., 2017b).

This dataset encapsulates a diverse set of architectures, with variation in many phenotypic features, including plant size (biomass), number and structure of lamina, and stem lengths and angles. For example, the plant volume, measured by the convex hull of the plant’s cloud points, of day 20 tomato plants varied across conditions by up to 50-fold: from 13.78 cm³ in high-heat to 668.68 cm³ in high-light (Conn et al., 2017b). The number of lamina for Nicotiana benthamiana plants varied from 4 to 12 from the young to old plants (Conn et al., 2017a). Over all 36 tomato plants, the range in number of cloud points was from 22,411 to 958,991, with an average of 233,917 cloud points per scanned architecture. Similarly, over all 18 Nicotiana benthamiana plants, the range was from 3,709 to 806,269 cloud points. For tomato, on average 84.7% of the cloud points of an architecture belonged to lamina, and 15.3% for stems. For Nicotiana benthamiana, on average 90.1% of the cloud points belonged to lamina, and 9.9% to stems. Thus, this dataset represents a diverse set of phenotypes to test the generality of our methods across two Solanaceous species.

Classification Framework and 3D Shape Descriptors

As input we are given a point cloud of points. Each point represents a 3-dimensional location on the surface of the shoot architecture. Each point is also associated with a normal vector, , indicating the orientation of each point in 3D space. Normal vectors were provided by the 3D scanner, and thus did not need to be determined computationally (Rusu and Cousins, 2011). We associate each point with a local neighborhood of points , defined by a sphere of radius . This neighborhood consists of all such that .

Our first goal is to learn a classifier that maps each point to one of two labels: lamina (such as leaflets and cotyledon blades) or stem tissue (including the main stem, petioles, and petiolules). Each point is represented by a feature vector describing the local geometry around the point. In principle, a good descriptor should capture general properties of the two classes (lamina or stems) and the variability within a class, without overspecifying shapes to a particular species or environmental growth condition.

Several geometric feature descriptors have been proposed, both within the plant phenotyping community (Paulus et al., 2013; Wahabzada et al., 2015) and in the computer vision community (Arbeiter et al., 2012; Guo et al., 2016) more broadly. Below, we summarize the descriptors compared here (Table 1).

Table 1. List of 3D shape feature descriptors.

The summary of features used with our dataset is shown. The size column represents the dimensionality of the feature vector per point in the point cloud.

Descriptor	Size	Reference
PFH	125	Rusu et al., 2008
FPFH	33	Rusu et al., 2009
Radius-based surface descriptor	289	Marton et al., 2010
SHOT	352	Tombari et al., 2010
Spin images	153	Johnson and Hebert, 1999
PC	2	Rusu and Cousins, 2011

Principal curvature

Intuitively, the principal curvature of a point is the speed with which a surface around “moves away” from a tangent plane defined at (Vemuri et al., 1986; Rusu and Cousins, 2011). Thus, the speed of moving away on a flat surface (e.g. a lamina) should be relatively small, and the speed of moving away on a surface of a cylinder (stem) should be relatively fast, at least along some axis.

To estimate the principal curvature of a point , first, every normal vector of is projected to a tangent plane formed by and point :

(1)

where is the 3x3 identity matrix.

Next we calculate the 3 × 3 covariance matrix of the projections:

(2)

where is the number of points in the neighborhood of and is the mean of all .

We then calculate principal curvatures from the nonzero eigenvalues of . These eigenvalues represent how much variation there is along each direction. Assuming , then for point , the feature , corresponds to the maximum and minimum curvatures, respectively.

Radius-based surface descriptor

This method was originally developed to estimate the radius of an object to determine if the object can be grasped by a robotic arm (Marton et al., 2010).

Given two points and their normal vectors, we can compute the radius of an assumed circle the two points lie on. The radius between and is approximated by first computing the angle between the normals and and second, computing the distance between and , which is equal to . Solving for gives us the radius between and .

The final radius-based surface descriptor feature can come in two versions. In the first version, will contain two values, the minimum and maximum radii computed between and each point . In the second version, will contain histograms of counts of the angles and distances between and each point in . The second version performed best, and we report these results below.

Point feature histograms

Rusu et al. (2008) derived a method that generalizes the mean curvature around a point that is more robust to noise. For each point , the method computes four features between all pairs of points . These features provide higher descriptive power by capturing all possible normal interactions between the points in the local surface.

For each pair of points , we first compute three vectors corresponding to the Darboux frame (Rusu et al., 2008):

We then use the Darboux frame to calculate four features for the pair: .

The final feature for point is based off the histograms of these four features.

Fast point feature histograms

Point feature histograms (PFH) features are computationally expensive to compute because the four features are calculated between all pairs of points in , which takes time , where is the size of the neighborhood. Fast PFH (FPFH) reduces this complexity to ) by only considering pairs that include (Rusu et al., 2009). FPFH also adds a distance-based weighing step that reduces the contribution of a feature based on how far the corresponding point is from . The two steps are:

1. Compute features between and , where and call this operation .

2. Scale each resulting histogram via distance-based weighting.

Here, equals the Euclidean distance between and . The result is a feature vector that is based off the histograms for each feature.

Signature of histogram of orientations

This descriptor captures local topology around a point by computing a histogram of the orientations of points in . The first stage of the descriptor is to establish a local reference frame, , around point . The goal of this frame is to make the feature locally invariant to geometric transformations (Tombari et al., 2010). The frame is computed using the eigenvalue decomposition of the weighted covariance matrix:

(3)

where . The directions of the reference frame are aligned to the directions where the majority of the points in the local neighborhood lie.

Next, we divide the neighborhood into multiple subvolumes of a sphere. The subvolumes are divided along its three spherical axes, and a local histogram is constructed for every subvolume. Each histogram contains bins of normalized point counts of points that lie within the angle between and every normal vector for each point in the subvolume. The final feature is a concatenation of all subvolume histograms, with a quadrilinear interpolation applied to remove abrupt changes in histogram boundaries.

Spin image

Unlike signature of histogram of orientations (SHOT), where the neighborhood is aligned with a local reference frame, the neighborhood in the spin image descriptor is aligned with a local reference axis (Johnson and Hebert, 1999). The local reference axis is established by using the normal vector of .

Intuitively, imagine a cylinder with a cylindrical axis defined by . The cylinder is subdivided into bins along its height and radius. Dividing along the height produces subvolumes of smaller cylinders. Dividing along the radius produces subvolumes of rings. The final feature consists of counts of points in each subvolume (after interpolation and distance-based weighting are applied, as above).

Testing and validation framework

The ground-truth classification was created by manually labeling each point in each architecture as either a lamina or stem point using the PolyWorks software tool. Because our data does not have a color component, there was some ambiguity in this labeling; for example, at locations near a stem where a lamina emerges, it was difficult, even for humans, to precisely delineate where the lamina begins and where the stem ends. We discuss this further in “Results.”

We tested performance using 6-fold cross-validation. We chose 6-fold because our labeled tomato dataset consists of 36 plants, thus each fold contains exactly 6 plants. We report the mean accuracy, and sd, over all folds. No preprocessing steps were applied to the point cloud data before computation of features and classification. This was by design to test robustness to noisy and imperfect data that may be expected in real-time scanning applications.

There are many classifiers, and performing cross-validation using all feature × classifier combinations is laborious. Thus, we first used cross-validation with a deep learning classifier (LeCun et al., 2015) to determine the best performing feature (FPFH; “Results”; Fig. 2), and we then tried many different classifiers using this feature. Specifically, we compared four classifiers: deep learning, linear support vector machines (SVM), K-nearest neighbors (KNN), and random forest. Prior work has used FPFH features for plant classification in conjunction with SVMs (Paulus et al., 2013) and k-means (Wahabzada et al., 2015). We also tested our how well classification accuracy generalized to environments that were not trained on (“Results”; Fig. 3).

Figure 2.

Impact of different classifiers and descriptor features on lamina versus stem classification accuracy. A, Results of 6-fold cross-validation for each descriptor using an underlying deep learning classifier. Error bars indicate sd across folds. PFH and FPFH features perform best. B, Results of 6-fold cross-validation for different classifiers using FPFH descriptors. The three bars for each descriptor/classifier show accuracy by class (stem, lamina, and overall for both classes). RSD, radius-based surface descriptor; SI, spin image; RF, random forest; n = 36 tomato plants.

Figure 3.

Generalization of classification accuracy across different environments and species. A to D, Results of training a classifier on tomato control plants, and testing on tomato plants for four other conditions (shade, drought, high-heat, and high-light). Over all descriptors, FPFH generalizes the best across conditions. E, Results of training a classifier on tomato data from all conditions and testing it on Nicotiana benthamiana data, using FPFH descriptors. Error bars indicate sd over replicates. Performance on the N. benthamiana data are shown separately per condition and time point. RSD, radius-based surface descriptor; SI, spin image; n = 54 tomato and Nicotiana benthamiana plants.

For each classifier, we performed a grid search to find the best model parameters, evaluated using 6-fold cross-validation. This search included different values of the neighborhood radius, . For linear SVM, we tested values for , where . For deep learning, we varied the number of training steps and the architecture of the neural network. The steps tested were 2000, 20000, and 200000. We tried six different architectures with as few as one hidden layer, as well as deeper networks with up to 10 hidden layers with various number of nodes in each layer (Goodfellow et al., 2016). The best architecture contained an input layer, three hidden layers with 33, 66, and 33 nodes, respectively, followed by the output layer. For KNN, we tested K values of 7, 77, 277, and 777. Finally, for the random forest classifier, we used 5, 25, 50, or 100 trees.

Lamina Segmentation Algorithms

The result of the classification yields two sets of points: lamina points and stem points. The lamina points by themselves are “free flowing” islands of points in space. The next challenge is to segment or cluster these lamina points into subsets, where each subset represents an individual lamina. Segmented lamina allow one to calculate traits, such as flowering time and drought tolerance (Minervini et al., 2016); different stresses can also affect lamina size (McCree and Davis, 1974), which can determine photosynthetic response (Valencia et al., 2016).

Lamina segmentation is a difficult problem because lamina can have many different shapes, poses, and orientations, including variation in wrinkles and ripples (Xia et al., 2015; McCormick et al., 2016). Further, the data itself can be noisy due to occlusion of certain parts of the lamina during scanning, issues in registration multiple scans, identification of small newly emerging lamina (Li et al., 2013), and close proximity and perhaps touching of two different lamina (McCormick et al., 2016). Although some methods have been proposed that solve these problems for specific types of lamina structure (e.g. sunflower lamina [Gélard et al., 2017b] or heuristics to deal with some types of touching lamina [McCormick et al., 2016]), here we seek a more general solution that is agnostic to any such prior.

Below, we describe two prior methods for segmentation, followed by our new proposed method. All methods use an iterative, region-growing approach to cluster lamina points into subsets, but each method uses a different criterion to determine whether to admit a candidate point into a cluster.

Euclidean clustering

The first method clusters lamina points into subsets based on Euclidean distances (Rusu, 2010). A point is picked randomly to start a cluster, and then that cluster is grown by admitting points that are less than a threshold distance away from at least one point from the cluster. This procedure is repeated until all lamina points have a cluster assignment. Ideally, the distance threshold should be larger than the average distance between two points on the same lamina, yet smaller than the distance between points on two distinct lamina. We set the threshold to 0.3 after performing a computational grid search to optimize lamina counting performance. In practice, the threshold is data-set sensitive, both with respect to the scanning hardware/resolution used and to variation in lamina density and structure of the species analyzed. It may be possible to learn the threshold from the dataset, for example, by manually classifying lamina for a few plants of interest, plotting distances between lamina points, and selecting a threshold that well separates different lamina. In addition, we filtered out clusters with less than 300 cloud points, as these were typically noise.

Segmentation using smoothness curvature

The second method clusters lamina points into subsets using normal vectors and estimated curvature values (called smoothness; Rabbani et al., 2006). The objective is to cluster sets of points that have a smooth surface. The method starts by computing and sorting the estimated curvature for all points. A cluster is initialized with the point with the smallest curvature (i.e. the surrounding region around has a flat surface). This cluster is grown by considering ’s neighbors; for each neighbor , if the angle between and is less than a threshold, is admitted to the cluster. To generate new seed points to grow the cluster, the curvature of is checked to see if it is less than another threshold; if so, is also added to a queue of seed points. If not, remains in the cluster with , but its neighbors are no longer candidates to further expand the cluster. Once all points from the seed queue are exhausted, a new cluster is started with next smallest curvature value (that has not already been assigned to a cluster) and the process repeats. The parameters for the normal calculation and curvature threshold values were selected using grid search on the entire dataset.

Our method

The third method, which we developed, clusters lamina points into subsets using distances, normal vectors, curvature, and FPFH features (Algorithm 1). The curvature around point is typically calculated as: , where, and where the eigenvalues are calculated using PCA on the neighborhood of . The drawback of this approach is that this measure sometimes produces sharp changes at the edges and in areas that are flat but have lower density. This can lead to oversegmentation.

Instead, we propose an anisotropic measure of smoothness: . This measure provides better clustering of points located on edges and in poorly scanned areas (lower point density or holes). Because this method does not consider the second eigenvalue, it produces a smoother curvature in flat areas, and the abrupt changes are more likely to happen where two lamina are in contact.

Our method also uses an additional check to determine whether a point should be admitted into a cluster based on the similarity of FPFH features. This approach is inspired by a similar method used for creating supervoxels for point clouds (Papon et al., 2013). Intuitively, using FPFH features will help capture the essence of the surface, despite possible hairs, wrinkles, or tears on the lamina.

The complexity of FPFH allows this technique to scale to plants with even a million points. In summary, our clustering method starts from a random seed point and the cluster is grown if any of three conditions are satisfied (Algorithm 1). Once the cluster is grown to exhaustion, a next random seed point is drawn from the points that have not been clustered yet, and this process is repeated until all points belong to some cluster.

Testing and validation framework

To test the accuracy of each lamina segmentation method, we only count a lamina as correctly segmented if all its points are correctly placed into the same subset. If a lamina’s points are split into two subsets, we call this oversegmentation; if two lamina are merged, we call this undersegmentation. Each ground-truth lamina is counted independently and is not weighted by its size.

Stem Skeletonization Algorithms

The next challenge is to transform the stem points into an underlying skeleton (a graph-theoretic tree) that captures the essential shape of the plant. Deriving skeleton graphs of plants is commonly used to perform morphological analysis of plant architectures (Bucksch et al., 2017; Conn et al., 2017b; Prusinkiewicz and Lindenmayer, 1996).

To extract a skeleton graph, we extended upon the PypeTree method (Delagrange et al., 2014), which improves upon the Verroust and Lazarus method (Verroust and Lazarus, 2000) and other prior work (Bucksch, 2014). The Verroust and Lazarus method assumed that cloud points are sampled uniformly or near uniformly; PypeTree removed this assumption to better deal with point clouds with inconsistent density and outliers. Prior work has also studied skeletonization of botanical trees (Bucksch et al., 2009; Bucksch et al., 2010; Delagrange et al., 2014); one challenge with our dataset, however, are the numerous small stems, where a more scale-invariant method would be desired.

Our proposed method builds upon PypeTree and is designed to better capture skeleton curvature by adding two new features: (1) More accurate tip detection and (2) Enhanced root selection.

First, we briefly outline the five main steps of the PypeTree algorithm.

• (1) A point cloud is turned into a neighborhood graph by connecting each point with its neighbors. Specifically, each cloud point corresponds to a node , and there exists an edge between and if is within radius from .

• (2) The neighborhood graph is converted into a geodesic graph, with respect to some root point, . The edges of the geodesic graph are found by computing the shortest path from to every vertex using Dijkstra’s algorithm, and then taking the union of all the edges in these paths.

• (3) The nodes of the geodesic graph are divided into “levels” based on the geodesic distance from the root. In other words, all nodes within distance 1 from the root are placed into the first level; all nodes with distance between 1 and 2 are placed into the second level; etc. Figure 4, A and B, illustrates nodes assigned to different levels with different colors. Each level will ultimately represent one or more nodes in the final skeleton graph (see Step 5 below).

• (4) A subgraph is created for each level . The nodes of are all the points in level . Each node in is connected to its two nearest neighbors in .

• (5) A subgraph may contain multiple connected components; for example, after a branch point (Y-junction) in the preceding level, there may be two connected components in the same level, representing the left and right sides of the stem. Thus, we create one node in the final skeleton graph per connected component, and this node is created at the centroid of the connected component. The final step is to create edges between centroid nodes in level and level . A centroid in level is connected to a centroid in level if a (random) point in the former’s connected component goes through some point in the latter’s connected component along the way to the root in the geodesic graph. Intuitively, this will avoid stems from “crossing over” across levels.

Figure 4.

Improvements in stem skeletonization. A, Visualization of the levels (colored bands) outputted by the PypeTree method. The default root point shown in Level 1 can be off-center when selected as the point with the lowest y-value. This can cause tilting of the architecture and other deformities in the resulting skeletonization (black line), especially when holes are present in the point cloud data. B, Visualization of the levels and root point outputted by our method. The balanced levels and centered root point improve the quality of the skeleton graph. C, Visualization of first (most dominant) eigenvalues for each point. Points with low eigenvalues, as indicated by the scale bar, are predominantly located at tips (the root and regions where lamina emerge).

PypeTree includes additional filtering steps to reduce noise and other implementation optimizations to reduce running time, which we do not describe here and instead refer the reader to the original publication (Delagrange et al., 2014).

We next describe two additional features we added to improve skeletonization.

Improved detection of tips

A critical component of skeletonization is finding tips, which are degree-one nodes of the skeleton where lamina emerge. Accurately identifying tips is challenging due to impartial capture, holes, and small features, such as hairs on the stem. The tips that PypeTree identifies are highly sensitive to the size of the levels: if the level size is set very large, then tips connected to small stems and petioles will be missed. On the other hand, if the level size is set too small, then false tips may emerge and resulting skeleton may be too sensitive to variation.

Our approach to finding tips is independent of connected components or level size, thus making it independent of the resolution of the skeleton graph. Tips are located at the boundary (edges) of a point cloud, which have a natural discontinuity compared with other points. Thus, we can detect edge points by using principal components analysis. Specifically, for every point in the point cloud, we compute its first eigenvalue based on its neighboring points within some radius r. The first eigenvalue of points close to the boundary will be smaller than that of points far from a boundary because the neighbors of boundary points will not be spread around the point but rather will be biased to some side (Fig. 4C). We then perform Euclidean clustering on all points whose first eigenvalue is among the smallest 15% of all points, such that points that belong to the same tip should reside in the same cluster. Clusters that contain less than 50 points are filtered out as noise. The centroids of each cluster become the tip nodes of the skeleton graph. Finally, a tip is attached to the skeleton graph by picking a random point from its cluster, finding which connected component it falls in (Step 5 above), and adding an edge from the centroid of this cluster to the tip. Finding tips in this manner allows for the detection of even small stems emerging during development.

Improved detection of the root

The stem base (hereafter simply called the “root”) is a special case of a tip and is critical for grounding the skeleton in the correct location. The goal is to find a root point that lies at the base and center of the stem. This location of the plant, however, is often difficult to scan because of its proximity to the soil and because of holes caused by scanning errors. The default root point of PypeTree is the point with the lowest y-value. This point can be highly skewed to one side of the stem (Fig. 4A), which can cause artificial curvature in the skeleton.

Our solution starts by noticing that the level closest to the default root—i.e. all the points within distance 1 to the root—often contains points that do not wrap evenly around the base of the stem (Level 1, Fig. 4A). This is because these distances are highly sensitive to noise, holes, or other small variations around the root. However, as the level number increases, the points appear more evenly balanced around the stem, forming even rings (Level 11, Fig. 4A).

To form even rings around the root, then, our idea is to build the geodesic graph starting from the tip that is furthest away from the root and trace backward (Level 1, Fig. 4B). In this way, the level closest to the root now will be less sensitive to variability. Specifically, let be PypeTree’s default root, and let be the point that is furthest away from . Then we find all points such that:

where is a parameter that adjusts amount of points to be sampled in the vicinity of the root. Then, for each we compute:

Intuitively, all points with will represent an evenly balanced ring around the true root (Level 1, Fig. 4B). We reset the root to be the centroid of these points, and then add edges from to all such points (with equal weight) to the neighborhood graph . We then proceed with Step 2 of the PypeTree algorithm above.

Testing and validation framework

We tested the accuracy of our method to detect tips on the 12 oldest tomato plants, which had the most branch points. We manually identified and counted the tips for each plant and compared these counts with those predicted by each method based on precision and recall.

Application and Validation

First, we describe results of classifying between lamina and stem points on our dataset of 54 shoot architectures, including generalization tests to assess how classifiers perform on an additional Solanaceous species and on growth conditions for which no training data are used. Second, we use the lamina points as input to compare our lamina segmentation/counting approach versus classic methods. Third, we use the stem points as input to compare our algorithm to generate skeleton graphs versus prior methods. Finally, we combine these three tasks to isolate individual tomato leaves and their connecting stem tissue.

Improved Classification of Lamina and Stem Points in 3D Shoot Architectures

Here we ask two questions: What combination of features and classifiers leads to robust classification results? How do these models perform when tested on plants grown in never-before-seen conditions or on plants from a different species?

Features

Using 6-fold cross-validation on the 36 tomato plants, we found that point feature histograms (PFH, and their faster version, FPFH) achieved the highest accuracy in classifying between lamina and stem points: 98.12% ± 0.39% and 97.77% ± 0.50%, respectively (Fig. 2A).

Although PFH was slightly more accurate than FPFH (by 0.35%), PFH’s complexity is (Rusu, 2010) as opposed for FPFH. This difference corresponds to days to perform cross-validation versus a few hours for FPFH, and thus FPFH achieves a better trade-off between scalability and accuracy.

Because our dataset is not balanced between classes (≈ 85% of points are lamina, ≈ 15% are stems) we computed the accuracy for each feature by class (Fig. 2A). Here again, PFH and FPFH perform similarly and dominate the other features. For lamina points, the accuracies per class was 98.94% and 98.85%, respectively for PFH and FPFH; for stem points, the accuracies were 92.86% and 93.60%, respectively. The next best feature, SHOT, performs comparable for lamina, but substantially worse (85.69%) for stem points. We see the largest variation in performance for stem point classification, ranging from 93.60% (FPFH) to 54.1% (spin image features). We note that an improvement in even 1% accuracy corresponds to potentially thousands of cloud points, and thus can be substantial.

It might be reasonable to say that the structure of a shoot architecture is roughly composed of two types of primitive shapes: planes (lamina) and cylinders (stems). Thus, simple and fast geometric descriptors, such as normal vectors and curvature values, should in principle be sufficient to accurately segment lamina and stems. However, our empirical results above show that these features do not in practice achieve high segmentation accuracy, likely due to irregularities in these basic shapes and noise. For example, principal curvatures achieved an overall accuracy of only 88.1% ± 5.2. Different stressors, such as temperature and drought, can also affect lamina shape, which makes these primitives somewhat simplistic. Thus, in the results described below, we consider only FPFH.

Classifiers

The classification results above were achieved using a deep learning classifier. Prior work has used a variety of classifiers for segmentation. For example, Paulus et al. (2013) used FPFH in conjunction with SVMs, and Wahabzada et al. (2015) used FPFH with k-means. Here, we study the effect of using different classifiers in conjunction with FPFH features for classification accuracy. We tested the following classifiers: KNN, random forest, SVM, and deep learning (DL).

Several of these classifiers performed comparably (Fig. 2B); however, when comparing the nature of the errors of each method, we found that DL made the most “reasonable” mistakes. Specifically, we visualized the errors made by each method and noticed qualitatively that most of the errors made by DL were in ambiguous regions of the plant where lamina emerge from a stem. In these regions, it is difficult, even for humans, to agree on the precise boundary between a stem and a lamina. The other methods (RF, KNN) made more mistakes in less ambiguous locations (e.g. the middle of a lamina or stem). Thus, for the remaining experiments, we used the DL classifier.

Generalization across different environments

Plants architectures are highly plastic and often modify their structure (including lamina shape, stem thickness, and stem angles) in response to the environment. In practice, training data may not be available for all possible conditions, and thus in practical applications, it is important that classifiers can accurately classify lamina and stems when applied to architectures grown in never-seen-before conditions.

To test this, we trained a DL model using a set of 12 control tomato plants (ranging from developmental days 5–20) and tested the model on plants grown in four other conditions: shade, drought, high-heat, and high-light (Conn et al., 2017b). This condition produced many variations in architectural features (Fig. 3, left). For example, in high-light, we observed thickening of the lamina and stem compared with all other conditions, and many more irregularities in lamina shape. In drought, we observed a weaker plant with fewer and thinner lamina and stems.

When compared with training a DL classifier with other features, we found that FPFH features again performed best, achieving over 95% accuracy on all new conditions (Fig. 3, A–D). For example, in high-light, FPFH achieved an overall accuracy of 95.88%, compared with 93.55% for SHOT and 72.91% for principal curvature (PC). In high-heat, FPFH achieved 97.48% accuracy, compared with 94.64% for SHOT and 90.35% for PC. In drought, FPFH achieved 98.2% accuracy compared with 96.01% for SHOT and 91.96% for PC. Principal curvatures has the largest variance across conditions, perhaps because this feature only consists of two values. On the other hand, SHOT has over 10 times as many feature values as FPFH (352 versus 33), but it still performed worse than FPFH.

While prior results have also proposed using FPFH features for lamina/stem classification (Paulus et al., 2013; Wahabzada et al., 2015), here we quantitatively demonstrate its accuracy, coupled with deep learning, across new conditions.

Generalization from tomato to Nicotiana benthamiana plants

A related challenge is to build classifiers that can accurately classify lamina and stems on species that have not been trained on. Indeed, there are thousands of plant species, some with very different branching and lamina patterns, and it is unreasonable to assume that training data will be available for all of them. Here, we tested how well a DL classifier trained with FPFH features from tomato plants generalized to another dicot plant: Nicotiana benthamiana. The Nicotiana benthamiana plants were also collected under three growth environments (control, shade, and high heat) and across three developmental time points (days 6, 12, and 20; Conn et al., 2017b).

Overall, we find >95.89% accuracy, averaged over all Nicotiana benthamiana plants (Fig. 3E). Older plants are classified more accurately regardless of the environment (e.g. 98.6% control day 20 versus 94.1% control day 6), whereas plants grown in shade had a slightly lower accuracy. Nonetheless, these results suggest that some cross-species generalization from tomato to Nicotiana benthamiana is possible with our framework and sets a benchmark for future studies to improve upon.

Improvement in Lamina Counting

The lamina points identified after classification will be “free flowing” islands of points in 3D space. The next task is to cluster these points, where each cluster corresponds to an individual lamina.

When analyzed on 291 total ground-truth lamina from the tomato plants, our enhanced lamina counting method achieved an overall accuracy of 86.60%, with 13 undersegmented and 45 oversegmented lamina (Table 2). In contrast, the Euclidean clustering method achieved 69.42% accuracy, with 55 undersegmentations and 2 oversegmentations. The performance of Euclidean clustering decline substantially as plants grow and more lamina started to become in contact with each other. Finally, segmentation with the smoothness constraint performed better than Euclidean clustering, achieving 83.16% accuracy, with 23 undersegmentations and 23 oversegmentations, although still slightly worse than our method. Thus, adding FPFH features to the growing criteria helped the algorithm deal better with natural variations in lamina structure, including hair, ripples, wrinkles, and veination. Although not reported here, we also tested other eigenvalue-based curvature metrics, such as planarity and surface variation (Hackel et al., 2016), which did not perform competitively.

Table 2. Lamina segmentation and counting results.

Comparison of three methods: Euclidean Clustering, Segmentation with Smoothed Curvature, and our method. ‘Under’ and ‘Over’ indicate the number of under- and oversegmented lamina, respectively. Summary on the bottom row shows overall performance over all plants.

Plant	Day	Euclidean Clustering			Segmentation using SC			Our Method
		Under	Correct	Over	Under	Correct	Over	Under	Correct	Over
Control-A	5	0	4/4	0	0	4/4	0	0	4/4	0
Control-A	12	1	10/12	0	0	11/12	1	0	11/12	2
Control-A	20	6	11/21	0	0	20/21	1	1	18/21	2
Control-B	5	0	4/4	0	0	4/4	0	0	4/4	0
Control-B	12	2	5/10	1	1	6/10	2	0	9/10	1
Control-B	20	5	12/18	1	1	14/18	4	1	12/18	13
Control-C	5	0	4/4	0	1	3/4	0	0	4/4	0
Control-C	12	1	8/11	0	1	9/11	0	0	8/11	3
Control-C	20	6	10/18	0	1	15/18	3	1	16/18	5
Control-d	5	0	3/3	0	0	3/3	0	0	3/3	0
Control-d	12	2	7/10	0	0	10/10	0	0	10/10	0
Control-d	20	6	12/18	0	1	16/18	1	0	16/18	7
Heat-A	5	1	1/3	0	1	2/3	0	0	3/3	0
Heat-A	12	1	3/6	0	1	4/6	0	1	4/6	0
Heat-A	20	0	6/6	0	0	6/6	0	0	6/6	0
Heat-B	5	0	2/2	0	0	2/2	0	0	2/2	0
Heat-B	12	0	3/3	0	0	3/3	0	0	3/3	0
Heat-B	20	1	2/4	0	1	2/4	1	0	4/4	0
Shade-A	5	0	4/4	0	0	4/4	0	0	4/4	0
Shade-A	12	0	6/8	0	1	7/8	0	1	7/8	0
Shade-A	20	0	9/12	0	2	10/12	0	2	10/12	0
Shade-B	5	0	4/4	0	0	4/4	0	0	4/4	0
Shade-B	12	1	6/8	0	1	7/8	0	0	7/8	1
Shade-B	20	2	12/16	0	2	13/16	0	3	11/16	0
Drought-A	5	0	4/4	0	1	3/4	0	0	4/4	0
Drought-A	12	1	7/9	0	2	6/9	0	1	8/9	0
Drought-A	20	2	12/15	0	0	15/15	0	1	14/15	0
Drought-B	5	2	1/4	0	1	3/4	0	0	4/4	0
Drought-B	12	1	5/7	0	1	4/7	1	0	7/7	0
Drought-B	20	2	8/12	0	1	9/12	1	1	9/12	1
High light-A	5	1	2/4	0	1	2/4	0	0	4/4	0
High light-A	12	7	3/11	0	1	8/11	2	0	10/11	2
High light-B	5	0	4/4	0	0	4/4	0	0	4/4	0
High light-B	12	4	8/13	0	0	9/13	6	0	8/13	8
Summary	–	55	202/291	2	23	242/291	23	13	252/291	45
Percent	–	–	69.42	–	–	83.16	–	–	86.60	–

Across all methods, the bulk of oversegmentation errors occurred for lamina with scanning holes or with poor registration from multiple scans. All of these methods include parameters that can be tuned to optimize under- and oversegmentation, and future work may try to introduce a learning procedure to set these parameters automatically given data.

Improvements in Stem Skeletonization

Using only the stem points as input, we next compared the quality of our algorithm versus PypeTree in forming a skeleton graph of the stem points.

First, we find that our algorithm generated skeletons that are visually better aligned with the underlying cloud points (Fig. 5). PypeTree consistently struggled identifying small stems (Fig. 5A), corresponding to where new lamina are emerging. PypeTree also often off-centered the root point, which caused the underlying skeleton to incorrectly lean. Although PypeTree does include parameters to adjust the level size that can improve these errors in some cases, it was difficult to set this parameter in such a way that it generalized across species, plant size, and branching patterns. In contrast, our method, which is independent of level size parameters, defined a root point that was much more centered along the main axis of the stem and better dealt with more ambiguous regions of the point cloud (Fig. 5B).

Figure 5.

Stem skeletonization comparison for three plants. The stem skeletonization produced by PypeTree (A) and our method (B), on three tomato plants. The red points are the stem cloud points, and the black line is the resulting skeleton graph. Dotted circles highlight areas of improved tip detection for our method; dotted rectangles highlight instances where the root tip was off-center according to PypeTree and better centered by our method.

Second, we quantified how accurately each method identified tips on a set of 12 tomato plants from day 20 across the five environmental conditions (Table 3). Day 20 plants were selected because these plants have the largest size and most branch points. Our method achieved higher or equivalent precision on 11 of the 12 plants, and higher or equivalent recall on all 12 plants, compared with PypeTree. Averaged over all 12 plants, our method achieved a precision of 95.0% ± 7.0% compared with 89.0% ± 12.0% for PypeTree. The reduction in false positive and false negative tips is important, as such errors can alter the underlying shape of the skeleton, which can affect downstream morphological analyses.

Table 3. Improvement in skeleton tips identification.

Precision vs. recall of PypeTree versus our tip identification method on 12 tomato plants at day 20. Overall, our method achieves higher precision and recall and is over an order of magnitude faster.

Plant	PypeTree			Our Method
	Precision	Recall	Time (s)	Precision	Recall	Time (s)
Control-A	0.87	0.93	677	0.88	1.00	23.56
Control-B	0.83	0.83	764	0.80	1.00	25.92
Control-C	0.67	0.80	427	0.83	1.00	16.89
Control-d	0.64	0.82	596	1.00	1.00	14.94
Drought-A	0.86	0.75	163	0.89	0.89	7.66
Drought-B	1.00	0.57	570	1.00	0.86	8.03
Heat-A	1.00	1.00	25	1.00	1.00	4.14
Heat-B	1.00	1.00	12	1.00	1.00	2.27
High light-A	0.86	0.75	1132	1.00	1.00	2.29
High light-B	0.93	0.72	919	1.00	0.94	17.51
Shade-A	1.00	1.00	137	1.00	1.00	10.40
Shade-B	1.00	0.75	217	1.00	1.00	14.85
Average	0.89	0.83	469.92	0.95	0.97	12.37

Third, we evaluated the run-time of each method and found 1 to 2 orders of magnitude improvement in our algorithm versus the provided PypeTree implementation (Table 3). For example, on the Highlight-B plant, PypeTree took 919 s, whereas our method only took 17.51 s. This improvement is subsantial, especially when dealing with large collections of plants scanned in real-time applications.

Isolating True Leaves of Tomato Plants

Here, we show how to isolate true leaves of tomato plants by building off the three pipelines developed above. This requires solving two tasks. First, the lamina classification and counting steps together identified a set of individual lamina, which need to be grouped into individual leaves. Second, the stem classification step identified petioles and petiolules, which also need to be associated with individual leaves based on their branching patterns (as identified by the skeletonization algorithm).

Algorithmically, we first assign each lamina to the closest tip node on the skeleton graph. This gives us a full skeleton graph with nodes corresponding to the root, graph-theoretic branch points, or lamina. Finding the cotyledon blades requires iterating through each level starting at level 0 (root) and finding the two closest lamina nodes that have similar geodesic distance from the root and that are oriented in nearly the opposite direction. Specifically, we mark each node u with the number of tips in the subtree rooted at u. Nodes marked with a value of 1 and satisfying the conditions above indicate a cotyledon blade. We then group the cotyledon petioles with the cotyledon blades by finding the first and second branch point off the main stem, respectively. A similar procedure is used to identify individual leaves, by grouping together individual lamina and their connecting petioles and petiolules. Each node in the full skeleton graph is associated with a set of points from the original point cloud when it was sectioned into levels. Thus, each individual cloud point can be assigned to either the stem or to an individual leaf.

Figure 6 shows examples of six tomato plants with the main stem and individual leaves high- lighted. The stem is shown in brown; the cotyledons are shown in purple; and each individual leaf (including associated lamina, petioles, and petiolules) are shown in a different color. The six plants were selected from different days and different conditions: highlight, heat, drought, shade, and control. Overall, we find that our pipelines can be used to accurately segment individual leaves, which is an important component of downstream leaf analyses. This procedure to isolate leaves takes specific advantage of the structure of tomato plants; other logic may be needed to isolate the leaves of other, especially non-Solanaceous, species.

Figure 6.

Isolating true leaves of tomato plants. Combining the three pipelines to segment individual tomato leaves. The main stem is shown in brown; the cotyledons are shown in purple; and each individual leaf, including associated lamina, petioles, and petiolules, are shown in a different color (red, purple, green). A, Control. B, Drought. C, Shade. D, Control. E, Highlight. F, Heat.

CONCLUSIONS

High-throughput phenotyping requires fast and accurate analysis methods that can handle large, noisy, and heterogeneous point cloud data. Here, we tackled three basic problems of point cloud analysis: lamina versus stem classification, lamina counting/segmentation, and stem skeletonization.

First, we showed how a deep learning classifier using fast point feature histogram features can achieve over 95% accuracy in classifying lamina versus stem points, outperforming many other features and classifier types. We also tested how well this approach generalized to instances where training data are not available for a test condition of interest, for example, testing on a new species or new growth condition. This is an important yet at times understudied component of validating phenotyping algorithms, especially because many different architectures can be formed within the same genotype. Second, we extended a classic region growing algorithm to improve segmentation and counting of individual lamina. Third, we developed an enhanced stem skeletonization algorithm with more visually aligned skeletons, improved tip identification, and faster run-time.

When compared with prior approaches that addressed these problems only on a small number of plants or plants from only a single species (Paulus et al., 2013; Wahabzada et al., 2015; McCormick et al., 2016; Gélard et al., 2017), our dataset encompassed 3 time points, 3–5 growth environments, and 2 species, which allowed us to test some generalization ability of our algorithms. Of course, the two Solanaceous species we selected (tomato and Nicotiana benthamiana) both demonstrate phyllotaxis and other similar growth habits, and it remains to be seen how well our algorithm generalizes to the many thousands of other plant species. Further, unlike some prior work (McCormick et al., 2016; Gélard et al., 2017a, 2017b), we purposefully did not perform any filtering, manual clean up, or other such preprocessing steps to the point cloud data to mimick the type of data expected in real-time scanning applications. Overall, these techniques can help improve downstream analyses, such as those based on analyzing lamina shape and size, self-shading, developmental growth, and morphological properties of stem patterning. Moreover, the accuracy of our methods may help uncover new phenotypes that have not yet been quantified, and may be used to perform quantitative genetic analyses to identify genes involved in regulating shoot architectures (Zhou et al., 2019).

These are several avenues for future work. First, advances in point cloud analysis using deep learning without feature extraction have been recently proposed (Charles et al., 2017; Qi et al., 2017); however, there remain challenges in using this technique when plant size varies. Second, we tested our stem skeletonization algorithm on shoot architectures, but it is possible that a very similar method can also be applied to skeletonizing root system architectures to study foraging behavior (De Kroon et al., 2009; Zhu et al., 2011; Flavel et al., 2017; Morris et al., 2017; Shahzad and Amtmann, 2017; Atkinson et al., 2019). Third, because lamina structure can vary across environments, prior methods (Dey et al., 2012) have proposed using conditional random field spatial smoothing to reduce noise and improve lamina/stem classification, although this may impose additional computational overhead. Fourth, extracting lamina veins from individually identified lamina may provide another useful lamina characteristic to aid segmentation, and may enable study of vasculature (Dengler and Kang, 2001; Nelson and Dengler, 1997). Fifth, while architecture data captured using 3D point clouds may be expected to depict fewer overlapping lamina compared to using 2D imaging, segmenting lamina that are touching remains an outstanding challenge in leaf analysis. Prior work has proposed methods tailored to particular leaf structures (e.g. sunflowers; Gélard et al., 2017) or particular shapes of blades (e.g. unlobed; McCormick et al., 2016); however more general approaches are currently lacking. Finally, although our approach can isolate individual tomato leaves, a next step would be to classify each tomato leaf into subsequent categories: primary, intercalary, secondary, and tertiary.