\section*{Methods} %1850Words

\subsection*{Ethics Statement}
Prior to enrollment to the study and RCM imaging, all patients signed an informed consent to an institutional review board (IRB) approved protocol at Memorial Sloan Kettering Cancer Center. The study and the experiments were conducted in accordance with the MSKCC IRB regulations and the Declaration of Helsinki principles.

\subsection*{Algorithm Overview}
The algorithm is composed of a series of steps (Fig.~\ref{Flowchart})  designed to handle each of the three challenges listed in the introduction.  We started with a classical image mosaicking pipeline from the multiview 3D reconstruction literature \cite{Hartley2004} and added methods to \textit{identify discontinuities} (called ``scene cuts") between frames, requiring separation into individual "subvideos" and to \textit{stitch} with maximal preservation of image fidelity, both specifically tailored for application to video-microscopy.

Overall, the algorithm takes an RCM video as an input, registers the consecutive small FOV frames, detects any rapid motions indicating the need for scene cuts, and finally stitches all frames into a large FOV mosaic, automatically within subvideos and automatically or semi-automatically across scene cuts (Fig.~\ref{Flowchart}). The registration procedure itself is composed of 2 sub-routines, keypoint extraction \cite{Hartley2004,Lowe} and keypoint matching \cite{Hartley2004}. During \textit{keypoint extraction}, uniquely identifiable pixel locations in each frame are determined. Then, through \textit{keypoint matching}, keypoints are matched between consecutive frames and their inter-frame displacement is tracked. These displacement parameters are then used to estimate registration parameters to map the pixels of  each successive  frame to the reference frame of the mosaic. These registration parameters also provide quantitative estimates of  inter-frame motion. Therefore, the algorithm uses them to identify rapid and/or tilted motion in the videos where inter-frame registration is either not feasible or unreliable; at those frames the algorithm automatically imposes scene cuts (Fig.~\ref{Artifact}). Each resulting contiguous segment is stitched into an individual video-mosaic. Since consecutive frames overlap significantly, rather than using common algorithms based on blending, which blur details and coarsen resolution, we adopted a graph-cuts based stitching algorithm \cite{Agarwala2004} which retains the original values for each pixel in the mosaic, critical to preserve cellular detail. In the following sections, we briefly describe each of these steps;  details are in the supplementary material.

\begin{figure}[ht]
\centering
\includegraphics[width = \textwidth]{./FlowChart_v3.eps}
\caption{Flowchart of video-mosaicking process. The individual steps are further described in the methods section, with further details in the supplementary material.}
\label{Flowchart}
\end{figure}

\subsection*{Frame Registration}\label{Sec:FrameReg}
Since the user moves the microscope freely over the region of interest without any positioning feedback, motion information must be extracted by analyzing the image frames themselves. We use a two step approach (Fig. \ref{Flowchart}) to estimate the microscope motion information: (i) keypoint extraction, and (ii) keypoint matching.

\subsubsection*{Keypoint extraction}\label{Sec:KeypointExt}
We use an optical flow estimator that tracks the inter-frame motion of uniquely identifiable spatial locations, referred as ``keypoints". In macroscopic images of the natural world images, keypoints are usually found at high-contrast locations such as corners and local extrema. Here they are not as clearly identifiable, which necessitates some further processing as described below. Keypoints are detected by their relation with their neighbors, and matched between consecutive frames using their descriptors. 

Due to changes in microscope-to-skin contact parameters (angle, pressure, velocity) during video capture, the scale, orientation, and intensity value of keypoints may change  slightly between consecutive frames. To improve robustness, we use ``Scale-Invariant Feature Transform" (\textit{SIFT}) \cite{Lowe} keypoints and descriptors, among several available descriptors in the literature \cite{Hartley2004}. SIFT is invariant to commonly encountered variation in images such as uniform scaling and changes in orientation, and also robust against affine distortion and changes in intensity. The first step in the SIFT algorithm is keypoint identification. Keypoints are typically high contrast pixels within their neighbors. SIFT uses a difference-of-Gaussian (DoG) representation (Sup. Fig. S1)%\Supp%ref{DOG}) of the image at octave-based scales, followed by  extrema detection, to locate uniquely identifiable pixel locations. The multi-octave DoG analysis  enhances scale-invariance of keypoint detection. A detailed explanation of the complete SIFT feature extraction procedure, including how the DoG representation is obtained at different octaves and how keypoint descriptors are composed, is provided in the supplementary material %(Sec.~\ref{SuppSec:SIFT}).

At the end of this keypoint extraction stage, we typically acquire $\sim 3000$ keypoints per frame. Each keypoint is composed of two components, a spatial location (coming from the extrema detection step) and a $128$-length descriptor vector (histogram representing the distribution of the gradients coming from the SIFT algorithm (Sup. Fig. S2)%Supp%\ref{HOG})).

\subsubsection*{Keypoint Matching}\label{Sec:KeypointMat}

In order to estimate a geometric transformation between frames, corresponding keypoints must be reliably matched despite varying intensity and scale. The keypoint descriptor used by SIFT is robust to such changes because it relies on the distribution of the gradients (histogram) in the local neighborhood of the keypoint; moreover we normalize orientation using the direction of the maximum gradient \cite{Lowe}) in the keypoint's neighborhood  to provide invariance against rotational motion. 

We used the two step keypoint matching procedure described in \cite{Hartley2004}  to estimate displacement between consecutive frames. Two requirements are considered during the matching procedure: (i) descriptors of the keypoints should be similar, and (ii) relative displacement of the matched keypoints should be similar. The second condition follows from the fact that all keypoints should move in a tandem fashion, consistent with translation, rotation, and change of (projection) angle of the microscope, but not  a general non-rigid morphing. Specifically, the global displacement of the keypoints should be well approximated using a single registration transformation.

Since the keypoint estimator itself is noisy with many outliers, we systematically eliminate unreliable matches that do not comply with this restricted type of motion. All keypoints in a subsequent frame are provisionally matched with the two "most similar" keypoints in the previous frame as determined by the $\ell_2$ (Euclidean) distances between their SIFT descriptors, which we use as our keypoint similarity metric. A keypoint is then considered non-uniquely matched, and therefore the match is designated as unreliable, if the two most similar  keypoints in the previous frame are nearly equally similar to the given keypoint, as measured by their similarity metric ratio being below a threshold (heuristically set in our experiments to 1.2). For example, a keypoint located in a region with similar or repeating textures is typically not unique. We prune our set of keypoint matches by discarding matches deemed unreliable by this procedure. 

Next, the pruned matches are used to model the transformation that best registers consecutive frames. In principle, only 3 keypoint matches are needed to determine the three registration parameters, and those parameters should be consistent across the choice of any three keypoints. However, in practice there will still be outlier correspondences, even after the previous refinement step, since even after refinement we typically have more than $1000$ pairs of keypoint matches between frames. Thus, we find a consistent single transformation using the well-known ``RANdom SAmple Consensus (RANSAC)" algorithm \cite{RANSAC}. RANSAC iteratively  estimates models from randomly chosen subsets of the set of observations and then chooses the model that generalizes best to the whole set. In each iteration, an affine transformation between consecutive frames is estimated using a randomly chosen subset of three matches and an error measure is calculated by generalizing this  transformation to the rest of the matches. We then chose the affine transformation that minimizes the generalization error over the random subsets. In our experiments we  observed that 3000 trials gave a good balance between computational complexity and generalizability. The list of matches is then pruned again by discarding keypoints whose displacement cannot be estimated with at least 50 pixel ($\ell_1$ distance based threshold that is determined experimentally) accuracy by the chosen transformation matrix.  

We then run a final optimization over all matches that remain in this refined set and find the best transformation that describes their motion. The cost that we minimize is the sum of the $\ell_2$ distances between the estimated location of the keypoint matches in the latter frame and their real locations. Since we are matching consecutive frames that were collected through smooth motion of the microscope, the scale and warping introduced through the final registration transformation should be relatively small. However, when the set of matches remaining after RANSAC pruning happen to be spatially concentrated in a relatively small spatial portion of the frames, the transformation matrix is susceptible to over-fitting to that region, leading to distortions in the rest of the frame. In order to avoid such distortions, we added an additional penalty term to the cost that penalizes scale change and shear in the final registration, and thus avoids distortions due to over-fitting. The effect of this new penalty parameter is minimal as long as the matches are relatively homogeneously spread across the frames. 

\subsection*{Scene Cut Identification}\label{Sec:SceneCut}

\begin{figure}[htb]
\centering
\includegraphics[width = \textwidth]{./RapidMotion+Jump.eps}
\newline
\caption{Common problems in imaging that will lead to discontinuities or ``scene-cuts'' in the RCM video-mosaics; (i) Blur due to rapid motion (\textbf{Row-1}), (ii) Lack of overlap between frames (\textbf{Row-2})}
\label{Artifact}
\end{figure}

To avoid distortion of the mosaic due to artifacts such as warping/distortion in frames or abrupt motion between them (Fig. \ref{Artifact}), we cut the image sequence when an artifact is detected  and start a new sequence from the subsequent frame. We automatically determine where to locate the cuts based on the calculated registration transformations. In particular, the diagonal coefficients of the transformation matrix control the amount of warping and distortion. Since a pure translation would result in those coefficients all being equal to $1$, we place cuts when the trace of the matrix,ideally equal to $3$, is above an experimentally determined threshold (in the work presented here the threshold used was $3.5$). Once the cut locations are determined, each resulting subsequence is mosaicked separately. These subsequences can then be assembled manually or semi-automatically, and a mosaic covering the whole imaging area can be obtained. An example of this is shown in Fig. \ref{Case:LMM} (scene cut locations are shown by white arrows)

\subsection*{Graph-Cut Based Stitching}\label{Sec:Graphcut} 
After registration, given the large overlap between consecutive frames, we need to ``merge" frames to construct the mosaic. As noted above, typical blending methods such as averaging or multi-resolution pyramid blending will lead to blurring, while the common alternative of choosing the maximum (or minimum) pixel value across the overlapped frames is not appropriate here since there is no reason to prefer dark versus bright intensities. In addition, the highly textured nature of RCM images will cause a loss of coherent structure and important cellular detail if extremal methods are used. Thus, we needed a method that keeps the actual pixel values while maintaining image fidelity. Simply choosing acquisition frame boundaries will lead to artifacts as well, due to imaging aberrations at frame edges as well as illumination changes between frames. Thus we devised an approach that meets this criteria by objectively choosing a single continuous boundary through the overlap area, driven by the pixel values themselves.

For simplicity of exposition, we concentrate here on the case where only 2 frames overlap; the extension to multi-frame overlap is straightforward. Since we want to retain the original pixel intensities, we have two candidate pixels to choose between, one from the previous and one from the subsequent frame. Our stitching algorithm determines a contiguous boundary through the overlap region, with the mosaic on one side coming from the previous frame and on the other from the subsequent frame (Fig. \ref{GCutIllustration}). From a graph cut perspective the optimal boundary traverses the overlap region along the trajectory on which variation in the pixel values is minimum. To accomplish this, we assign a cost of having the stitching border go through each pixel, defined as the sum of (i) the absolute intensity difference between the candidate pixels from the two frames, and (ii) the absolute intensity differences between neighboring pixel locations in the same frame. We model the problem of estimating the optimal stitching border as a graph-cuts based labeling problem \cite{Boykov2011}, an approach which has been applied to many computer vision applications including photomontage \cite{Agarwala2004}, texture synthesis \cite{Kwatra2003} and segmentation \cite{Felzenszwalb2004}, resulting in a data-driven optimized boundary. (For in-depth technical description of our implementation, please refer to supplementary materials) % Sec.~\ref{SuppSec:Graphcut} in supplementary materials).

\begin{figure}[htb]
\centering
\includegraphics[width = \textwidth]{./UNARY3.eps}
\newline
\caption{Example illustration of graph-cuts based stitching. In the overlapping areas of the registered images, the algorithm finds a stitching boundary by minimizing a local pixel variation based cost metric, which is composed of 2 parts a unary and  binary cost. The unary costs (upper middle) depends on the intensity difference between overlapping images. The binary costs (lower middle and left) is calculated using intensity variations between neighboring pixel values within the images. The graph-cut algorithm finds a continuous stitching path (upper right) that goes through pixels in overlap area, whose accumulated cost is minimum among all possible such paths. The two images are then stitched together using this boundary (lower right).}
\label{GCutIllustration}
\end{figure}