Table 2.

Ablation study of the PEN + CellPose (base) instance segmentation network. To evaluate the effects of ablation, each model was retrained from an initialized set of random weights. We evaluated removal of the $K=1$ and $K=11$ kernel sizes, thereby removing an individual branch of PEN shown in Figure 1A. The subsequent convolution in each branch was replaced with a max-pooling operation in the axial dimension in the Branch Max model. The final convolution in PEN was replaced with a max-pooling operation in Collect Max model. Finally, the ground-truth assignment strategy to the available $N_{out}$ channels of our modified CellPose algorithm was set to randomly assign cell labels to $N_{out}=3$ channels in the Random GT model, and to a single $N_{out}=1$ channel in the $N_{out}=1$ model. All models were evaluated on a low-density cell dataset $(\mathrm{N}=4082)$ with fewer than 0.5% of cells overlapping axially at an average of just 7.2% intersection over union in the 2D projection, and a high-density cell dataset $(\mathrm{N}=111)$ where 36.8% of cells were overlapping axially at an average 12.7% intersection over union. Metrics are measured at a minimum intersection over union of 50% for true-positive detections. See Supplementary S4 for details regarding metrics.

Model	Low-Density				High-Density
	Jaccard	Precision	Recall	Quality	Jaccard	Precision	Recall	Quality
Base	0.523	0.574	0.854	0.807	0.518	0.616	0.766	0.782
− K = 1	0.502	0.546	0.864	8103	0.518	0.610	0.775	0.782
− K = 11	0.489	0.527	0.871	0.812	0.449	0.512	0.784	0.762
Branch Max	0.5202	0.564	0.870	0.818	0.4785	0.600	0.703	0.746
Collect Max	0.485	0.520	0.877	0.816	0.5298	0.619	0.802	0.771
Random GT	0.009	0.023	0.014	0.686	0.0759	0.125	0.162	0.592
$N_{out}=1$	0.6578	0.734	0.863	0.840	0.480	0.811	0.541	0.710