Table 3.

Testing Precision using CT Volumetry for Illustration^*

STEP	DESCRIPTION
1. Make measurements on N cases	For each case, measure the tumor volume at time point 1 (denoted Yi1) and at time point 2 (Yi2) where i denotes the i-th case (i=1, 2, …N).
2. For each case, calculate mean and wSD2	For each case, calculate the mean and within-tumor SD: Ȳi = (Yi1 + Yi2)/2 and $wS{D}_i^2={(Y_{i1}-Y_{i2})}^2/2$. (Note that some authors suggest a correction to this wSD estimate [11] or a model-based estimate [12] to account for the small number of replicate measurements.)
3. Estimate wSD or wCV	From the N cases, estimate within-tumor SD (wSD) or CV (wCV): $wSD=\sqrt{{\sum }_{i=1}^{N}wS{D}_i^2/N}$ and $wCV=\sqrt{{\sum }_{i=1}^N(wS{D}_i^2/{\overline{Y}}_i^2)/N}$. (Note that averaging over the N cases is appropriate when we can assume that the wSD is constant over the range of tumor volume values.)
4. Estimate RC or %RC*	Estimate the Repeatability Coefficient (RC) or %RC: $\widehat{RC}=2.77\times wSD$ and $\widehat{\%RC}=2.77\times wCV\times 100$. For the CT volumetry example, %RC is used.
5. Calculate test statistic and assess compliance	The null hypothesis is that the RC does not satisfy the requirement in the Profile (i.e. the RC is too large); the alternative hypothesis is that the RC does satisfy the requirement. The test statistic T is: $T=\frac{N\times ({\widehat{\%RC}}^2)}{{\delta }^2}$, where δ is the performance value from the Profile claim statement (i.e. δ = 40%). Compliance with the claim is shown if $T<{\chi }_{\alpha ,N}^2$, where ${\chi }_{\alpha ,N}^2$ is the α-th percentile of a chi square distribution** with N dfs (for a one-sided test with α type I error rate).
6. Construct precision profile	Estimate %RC as a function of tumor size and check that all %RC ≤ δ.

^*The process above is applicable for testing a reader’s conformance using a specific algorithm or for testing a fully-automated algorithm (no reader interaction). For testing an algorithm that requires manipulation by human readers, usually, 3–5 independent readers not involved in developing the algorithm should be included in the conformance study. Steps 1–4 are repeated for each reader separately. Thus, in step 4 there will be an estimate of the RC for each reader. Instead of using the test in step 5, a different statistical approach is used, which assesses whether the average readers’ RC satisfies the performance requirements in the Profile. A generalized linear model can be built for the RC, treating readers as a random effect nested in cases [5]. From the model, a 95% CI for the mean RC is constructed and used to evaluate the actor’s performance relative to the requirement in the Profile.

^**A chi square distribution is a commonly used probability distribution that is used when constructing a CI for a population standard deviation of a normal distribution, e.g. wSD or RC, from a standard deviation estimated from a sample of size N.