Hypothesis testing procedure for binary and multi-class $F_1$-scores in the paired design

Abstract

In modern medicine, medical tests are used for various purposes including diagnosis, disease screening, prognosis, and risk prediction. To quantify the performance of the binary medical test, we often use sensitivity, specificity, and negative and positive predictive values as measures. Additionally, the $F_1$-score, which is defined as the harmonic mean of precision (positive predictive value) and recall (sensitivity), has come to be used in the medical field due to its favorable characteristics. The $F_1$-score has been extended for multi-class classification, and two types of $F_1$-scores have been proposed for multi-class classification: a micro-averaged $F_1$-score and a macro-averaged $F_1$-score. The micro-averaged $F_1$-score pools per-sample classifications across classes and then calculates the overall $F_1$-score, whereas the macro-averaged $F_1$-score computes an arithmetic mean of the $F_1$-scores for each class. Additionally, Sokolova and Lapalme¹ gave an alternative definition of the macro-averaged $F_1$-score as the harmonic mean of the arithmetic means of the precision and recall over classes. Although some statistical methods of inference for binary and multi-class $F_1$-scores have been proposed, the methodology development of hypothesis testing procedure for them has not been fully progressing yet. Therefore, we aim to develop hypothesis testing procedure for comparing two $F_1$-scores in paired study design based on the large sample multivariate central limit theorem.

1 |. INTRODUCTION

Medical tests are important for the early detection and treatment of disease in modern medicine. Tests are used for various purposes including diagnosis, disease screening, prognosis, and risk prediction. Some measures exist to quantify the test performance; sensitivity, specificity, and positive and negative predictive values are commonly used for binary tests. Additionally, the $F_1$-score for binary data (binary $F_1$-score), which is defined as the harmonic mean of precision (positive predictive value) and recall (sensitivity), has been used in the medical field.^1,2

The binary $F_1$-score is especially useful when evaluation of true negatives is relatively unimportant because true negatives are not included in computation of either precision or recall. In addition, the binary $F_1$-score performs well for a poor diagnostic test that identifies majority of the data as positive. In this situation, a simple arithmetic mean of precision and recall may be as high as 0.50 because recall will be 1.00 if all the data are diagnosed as positive. However, the binary $F_1$-score will be appropriately low in these instances: it will be 0.18 and 0.02 when the precision is 0.10 and 0.01, respectively, even if recall is 1.00. Therefore, $F_1$-score is a better statistic to report.²

Most of measures for performance of medical tests are only applicable to binary classification data, and multi-class classification data need to be dichotomized to compute these measures. In the motivating example,³ for instance, skin cancer images were originally classified into six categories (malignant melanoma (MM), basal cell carcinoma (BCC), nevus, seborrheic keratosis (SK), senile lentigo (SL) and hematoma/hemangioma (H/H)), and the classification performances of board-certified dermatologists and dermatologic trainees were compared. The classification performance was assessed by accuracy, sensitivity, specificity, false negative rate, false positive rate, and positive predictive value after dichotomizing the six categories (MM and BCC vs. nevus, SK, SL, and H/H). However, evaluating the performance with the original six categories would have been preferable because dichotomization led to loss of information regarding the performance of the this classification.^4,5

As measures of multi-class classification performance, a micro-averaged $F_1$-score and a macro-averaged $F_1$-score have been proposed.² The micro-averaged $F_1$-score calculates the overall $F_1$-score by pooling per-sample classifications across classes. Contrarily, the macro-averaged $F_1$-score computes an arithmetic mean of the $F_1$-scores for each class. In addition, Sokolova and Lapalme⁶ proposed an alternative macro-averaged $F_1$-score as the harmonic mean of the arithmetic mean of the precisions and recalls for each class.

Although $F_1$-scores for binary and multi-class classifications have been originally used for measuring the performance of text classification in the field of information retrieval or of a classifier in machine learning, it has become frequently used in medicine.^7–14 Some statistical methods for inference have been proposed for the binary $F_1$-score,¹⁵ and the methods for estimating confidence intervals of the micro-averaged $F_1$-scores and macro-averaged $F_1$-scores has been developed.^16,17 However, these previous methods are for inference from one-sample. To our knowledge, no method is available for hypothesis testing of $F_1$-scores for paired samples as in our motivating example or two independent samples. Thus, we aim to provide the methods for comparing the binary $F_1$-scores, micro-averaged $F_1$-scores and macro-averaged $F_1$-scores in the paired-design setting. For two-independent-sample setting, the proposed method is readily applicable by setting the covariance part of the test statistics to 0.

The layout of this article is as follows: In Section 2, the definitions of the binary $F_1$-score, micro-averaged $F_1$-score and macro-averaged $F_1$-score are reviewed. Test statistics for comparing those scores are derived in Section 3. Then, the simulation results of the proposed statistics and the application to the motivating example are presented in Sections 4 and 5, respectively. Finally, our brief discussions are provided in Section 6.

2 |. REVIEW OF F1-SCORES

This section introduces notations and definitions of binary $F_1$-score ($biF$), micro-averaged $F_1$-score ($miF$), and macro-averaged $F_1$-score ($maF$). Consider an $r\times r\times r$ table of data for a nominal categorical variable with $r$ levels $(r\ge 2)$. Each true class $1,\dots ,r$ has an $r\times r$ table representing prediction frequencies of the two tests to be compared.

This arrangement of data represents the binary classification when $r=2$, and the multi-class classification when $r>2$. Table 1 shows general notations for each cell probability $p_{ijk}$, where $i$ indicates the class of Test 1, $j$ indicates the class of Test 2, and $k$ indicates the true condition. Let Test 1 be a new medical test and Test 2 be an existing medical test. We consider a hypothesis testing to compare $F_1$-scores of Test 1 and Test 2. Using these notations, the true positive rate $\left(T{P}_a\right)$, the false positive rate $\left(F{P}_a\right)$, and the false negative rate $\left(F{N}_a\right)$ for each class $a(a=1,\dots ,r)$ in Test 1 are defined as follows:

$$T{P}_{1a}=p_{a.a},F{P}_{1a}=\sum _{\begin{array}{c}k=1\\ k\ne a\end{array}}^r{p}_{a.k},F{N}_{1a}=\sum _{\begin{array}{c}i=1\\ i\ne a\end{array}}^r{p}_{i.a}.$$

TABLE 1.

General notations.

		True condition = 1					⋯	True condition = r
		Test 2						Test 2
		1	2	⋯	r			1	2	⋯	r
Test 1	1	p 111	p 121	⋯	p 1r1	p 1.1		p 11r	p 12r	⋯	p 1rr	p 1.r	p 1..
	2	p 211	p 221	⋯	p 2r1	p 2.1		p 21r	p 22r	⋯	p 2rr	p 2.r	p 2..
	⋮	⋮	⋮	⋱	⋮	⋮		⋮	⋮	⋱	⋮	⋮	⋮
	r	p r11	p r21	⋯	p rr1	p r.1		p r1r	p r2r	⋯	prrr	pr.r	p r..
		p .11	p .21	⋯	p .r1	p ..1		p .1r	p .2r	⋯	p .rr	p ..r	1

Note that $T{P}_{1a}+F{P}_{1a}=p_{a..}$, and $T{P}_{1a}+F{N}_{1a}=p_{..a}$. Similarly, $T{P}_a,F{P}_a,F{N}_a$ for each class $a(a=1,\dots ,r)$ for Test 2 are defined as follows:

$$T{P}_{2a}=p_{.aa},F{P}_{2a}=\sum _{\begin{array}{c}k=1\\ k\ne a\end{array}}^r{p}_{.ak},F{N}_{2a}=\sum _{\begin{array}{c}j=1\\ j\ne a\end{array}}^r{p}_{.ja}.$$

Note that $T{P}_{2a}+F{P}_{2a}=p_{.a.}$, and $T{P}_{2a}+F{N}_{2a}=p_{..a}$.

2.1 |. Binary ${\mathit{F}}_1$-score

When $r=2$, we consider the following precision $\left(biP\right)$ and recall $\left(biR\right)$ for Test 1 as:

$$bi{P}_1=\frac{T{P}_{11}}{T{P}_{11}+F{P}_{11}}=p_{1.1}/p_{1..},$$

$$bi{R}_1=\frac{T{P}_{11}}{T{P}_{11}+F{N}_{11}}=p_{1.1}/p_{..1}.$$

And binary $F_1$-score for Test 1 $\left(bi{F}_1\right)$ is defined as the harmonic mean of $bi{P}_1$ and $bi{R}_1$, that is,

$$bi{F}_1=2\frac{bi{P}_1\times bi{R}_1}{bi{P}_1+bi{R}_1}=2\frac{p_{1.1}}{p_{1..}+p_{..1}}.$$ (1)

Similarly, the binary $F_1$-score for Test 2 $\left(bi{F}_2\right)$ is as follows:

$$bi{P}_2=\frac{T{P}_{21}}{T{P}_{21}+F{P}_{21}}=p_{.11}/p_{.1.},$$

$$bi{R}_2=\frac{T{P}_{21}}{T{P}_{21}+F{N}_{21}}=p_{.11}/p_{..1},$$

$$bi{F}_2=2\frac{bi{P}_2\times bi{R}_2}{bi{P}_2+bi{R}_2}=2\frac{p_{.11}}{p_{.1.}+p_{..1}}.$$ (2)

2.2 |. Micro-averaged ${\mathit{F}}_1$-score

When $r>2$ the micro-averaged precision (miP) and micro-averaged recall (miR) are obtained from the sum of each class of TP_i, FP_i, FN_i. miP and miR for Test 1 can be written as

$$mi{P}_1=\frac{\sum _{a=1}^{r}T{P}_{1a}}{\sum _{a=1}^r\left(T{P}_{1a}+F{P}_{1a}\right)}=\frac{\sum p_{a.a}}{\sum p_{a..}}=\sum _{a=1}^r{p}_{a.a},$$

$$mi{R}_1=\frac{\sum _{a=1}^{r}T{P}_{1a}}{\sum _{a=1}^r\left(T{P}_{1a}+F{N}_{1a}\right)}=\frac{\sum p_{a.a}}{\sum p_{..a}}=\sum _{a=1}^r{p}_{a.a}.$$

Finally, as the harmonic mean of $mi{P}_1$ and $mi{R}_1$, we have the micro-averaged $F_1$-score for Test 1 $\left(mi{F}_1\right)$ as

$$mi{F}_1=2\frac{mi{P}_1\times mi{R}_1}{mi{P}_1+mi{R}_1}=\sum _{a=1}^r{p}_{a.a}.$$ (3)

Similarly, the micro-averaged $F_1$-score for Test 2 $\left(mi{F}_2\right)$ is

$$mi{P}_2=\frac{\sum _{a=1}^{r}T{P}_{2a}}{\sum _{a=1}^r\left(T{P}_{2a}+F{P}_{2a}\right)}=\frac{\sum p_{.aa}}{\sum p_{.a.}}=\sum _{a=1}^r{p}_{.aa},$$

$$mi{R}_2=\frac{\sum _{a=1}^{r}T{P}_{2a}}{\sum _{a=1}^r\left(T{P}_{2a}+F{N}_{2a}\right)}=\frac{\sum p_{.aa}}{\sum p_{..a}}=\sum _{a=1}^r{p}_{.aa}.$$

$$mi{F}_2=2\frac{mi{P}_2\times mi{R}_2}{mi{P}_2+mi{R}_2}=\sum _{a=1}^r{p}_{.aa}.$$ (4)

2.3 |. Macro-averaged ${\mathit{F}}_1$-score

When $r>2$, to define the macro-averaged $F_1$-score for Test 1 $\left(ma{F}_1\right)$, first consider the following precision $\left(P_{1a}\right)$ and recall ($R_{1a}$) within each class, $a=1,\dots ,r$:

$$P_{1a}=\frac{T{P}_{1a}}{T{P}_{1a}+F{P}_{1a}}=p_{a.a}/p_{a..},$$ (5)

$$R_{1a}=\frac{T{P}_{1a}}{T{P}_{1a}+F{N}_{1a}}=p_{a.a}/p_{..a}.$$ (6)

And $F_1$-score within each class for Test 1 $\left(F_{1a}\right)$ is defined as the harmonic mean of $P_{1a}$ and $R_{1a}$, that is,

$$F_{1a}=2\frac{P_{1a}\times R_{1a}}{P_{1a}+R_{1a}}=2\frac{p_{a.a}}{p_{a..}+p_{..a}}.$$

The macro-averaged $F_1$-score for Test 1 $\left(ma{F}_1\right)$ is the simple arithmetic mean of $F_{1a}$:

$$ma{F}_1=\frac{1}{r}\sum _{a=1}^r{F}_{1a}=\frac{2}{r}\sum _{a=1}^r\frac{p_{a.a}}{p_{a\mathrm{..}}+p_{..a}}.$$ (7)

Similarly, the macro-averaged $F_1$-score for Test 2 $\left(ma{F}_2\right)$ is

$$P_{2a}=\frac{T{P}_{2a}}{\left(T{P}_{2a}+F{P}_{2a}\right)}=p_{.aa}/p_{.a.},$$

$$R_{2a}=\frac{T{P}_{2a}}{\left(T{P}_{2a}+F{N}_{2a}\right)}=p_{.aa}/p_{..a}.$$

$$F_{2a}=2\frac{P_{2a}\times R_{2a}}{P_{2a}+R_{2a}}=2\frac{p_{.aa}}{p_{.a.}+p_{..a}}.$$

$$ma{F}_2=\frac{1}{r}\sum _{a=1}^r{F}_{2a}=\frac{2}{r}\sum _{a=1}^r\frac{p_{.aa}}{p_{.a.}+p_{..a}}.$$ (8)

2.4 |. Alternate definition of macro-averaged ${\mathit{F}}_1$-score

Sokolova and Lapalme⁶ gave an alternative definition of the macro-averaged $F_1$. First, macro-averaged precision $\left(maP\right)$ and macro-averaged recall $\left(maR\right)$ for Test 1 are defined as simple arithmetic means of the within-class precision and within-class recall in (5) and (6), respectively.

$$ma{P}_1=\frac{1}{r}\sum _{a=1}^r{P}_{1a}=\frac{1}{r}\sum _{a=1}^r\frac{p_{a.a}}{p_{a..}},$$

$${maR}_1=\frac{1}{r}\sum _{a=1}^r{R}_{1a}=\frac{1}{r}\sum _{a=1}^r\frac{p_{a.a}}{p_{..a}}.$$

And the alternate definition of macro-averaged $F_1$-score for Test 1 $(ma{F}_1^{\mathrm{*}})$ is the harmonic mean of these quantities.

$$ma{F}_1^{\mathrm{*}}=2\frac{ma{P}_1\times ma{R}_1}{ma{P}_1+ma{R}_1}.$$ (9)

Similarly, the alternate definition of macro-averaged $F_1$-score for Test 2 $(ma{F}_2^{\mathrm{*}})$ is

$$ma{P}_2=\frac{1}{r}\sum _{a=1}^r{P}_{2a}=\frac{1}{r}\sum _{a=1}^r\frac{p_{.aa}}{p_{.a.}},$$

$$ma{R}_2=\frac{1}{r}\sum _{a=1}^r{R}_{2a}=\frac{1}{r}\sum _{a=1}^r\frac{p_{.aa}}{p_{..a}}.$$

$$ma{F}_2^{\mathrm{*}}=2\frac{ma{P}_2\times ma{R}_2}{ma{P}_2+ma{R}_2}.$$ (10)

3 |. PROPOSED HYPOTHESIS TESTING PROCEDURE

In this section, we derive the test statistics for comparing two $F_1$-scores ($bi{F}_1$ and $bi{F}_2;mi{F}_1$ and $mi{F}_2;ma{F}_1$ and $ma{F}_2$; and $ma{F}_1^{\mathrm{*}}$ and $ma{F}_2^{\mathrm{*}}$). We assume that the observed frequencies, $n_{ijk}$ for $1\le i\le r,1\le j\le r,1\le k\le r$, have a multinomial distribution with overall sample size $N={\sum }_{i,j,k}{n}_{ijk}$ and probabilities $\mathit{p}={\left[p_{111},\dots ,p_{1r1},\cdots ,p_{rr1},\dots ,p_{rrr}\right]}^T$, where $i$ indicates the class of Test 1, $j$ indicates the class of Test 2, $k$ indicates the true condition, and “T” represents the transpose. The maximum likelihood estimate (MLE) of $p_{ijk}$ is ${\hat{p}}_{ijk}=n_{ijk}/N$. That is

$$\left(n_{111},n_{121},\dots ,n_{\text{rrr}}\right)\sim Multinomial\left(N;\mathit{p}\right).$$

By invariance property of MLE’s, the maximum likelihood estimate of $biF,miF,maF,ma{F}^{\mathrm{*}}$, and other quantities in the previous section can be obtained by substituting $p_{ijk}$ by ${\hat{p}}_{ijk}$.

3.1 |. Test statistic for comparing two $\mathit{b}\mathit{i}\mathit{F}\mathit{s}$

Let $\mathit{b}\mathit{i}\mathit{F}={\left(bi{F}_1,bi{F}_2\right)}^T$ be a vector whose components are the $biFs$ of the two medical tests, and let $\widehat{\mathit{b}\mathit{i}\mathit{F}}$ be the MLE of $\mathit{b}\mathit{i}\mathit{F}.\widehat{\mathit{b}\mathit{i}\mathit{F}}$ can be obtained by substituting $p_{ijk}$ by their MLE’s in (1) and (2).

$${\widehat{biF}}_1=2\frac{{\hat{p}}_{1.1}}{{\hat{p}}_{1..}+{\hat{p}}_{..1}}=2\frac{n_{1.1}}{n_{1..}+n_{..1}},\widehat{bi{F}_2}=2\frac{{\hat{p}}_{.11}}{{\hat{p}}_{.1.}+{\hat{p}}_{..1}}=2\frac{n_{.11}}{n_{.1.}+n_{..1}}.$$

Using the delta-method and the multivariate central limit theorem, we have

$$\sqrt{N}\left(\widehat{\mathit{b}\mathit{i}\mathit{F}}-\mathit{b}\mathit{i}\mathit{F}\right)\dot{\sim }Normal\left(0,{\left[\frac{\partial \left(\mathit{b}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]}^T\left[diag\left(\mathit{p}\right)-\mathit{p}{\mathit{p}}^T\right]\left[\frac{\partial \left(\mathit{b}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]\right),$$

where $diag\left(\mathit{p}\right)$ is an $r^2\times r^2\times r^2$ diagonal matrix whose elements are the diagonal elements of $\mathit{p}$, and “$\dot{\sim }$” represents “approximately distributed as”. The Wald statistic for testing $H_0:bi{F}_1=bi{F}_2$ vs. $H_1:bi{F}_1\ne bi{F}_2$, therefore, is

$$T_{biF}^W=\frac{{\left({\widehat{biF}}_1-{\widehat{biF}}_2\right)}^2}{{\widehat{Var}}_{biFd}},$$

where ${\widehat{Var}}_{biFd}$ is the variance of $\left({\widehat{biF}}_1-{\widehat{biF}}_2\right)$ with $\left\{p_{ijk}\right\}$ replaced by $\left\{{\hat{p}}_{ijk}\right\}$. Derivation of the variance of $\left({\widehat{biF}}_1-{\widehat{biF}}_2\right)$ appear in Appendix A.1. The test statistic is distributed asymptotically as a ${\chi }^2$ distribution with one degree of freedom under the null hypothesis.

As a side note, the confidence interval of $biF$ for each test can be derived in the same way. $\mathrm{A}(1–\alpha )\times 100\mathrm{\%}$ confidence interval of $bi{F}_1$ and $bi{F}_2$ is

$${\widehat{biF}}_1\pm Z_{1-\alpha /2}\times \sqrt{{\widehat{Var}}_{b{F}_1},}$$

$${\widehat{biF}}_2\pm Z_{1-\alpha /2}\times \sqrt{{\widehat{Var}}_{bi{F}_2}},$$

where $Z_p$ denote the $100p$-th percentile of the standard normal distribution, and ${\widehat{Var}}_{bi{F}_1}$ and ${\widehat{Var}}_{bi{F}_2}$ are the variance of ${\widehat{biF}}_1$ and the variance of ${\widehat{biF}}_2$ with $\left\{p_{ijk}\right\}$ replaced by $\left\{{\hat{p}}_{ijk}\right\}$. These simple formulas based on the multinomial distribution have not been proposed yet. Wang et al. proposed a confidence interval of $biF$ based on the beta prime distribution and associated calculations using the bootstrap method.^15,18

For the score statistic, we consider the MLE of $\left\{p_{ijk}\right\}$ under the null hypothesis that could be obtained, for example by applying the Newton-Raphson method to the log-likelihood equations. The score statistic for testing $H_0:bi{F}_1=bi{F}_2$ vs. $H_1:bi{F}_1\ne bi{F}_2$ is

$$T_{biF}^S=\frac{{\left({\widehat{biF}}_1-{\widehat{biF}}_2\right)}^2}{{\widetilde{Var}}_{biFd}},$$

where ${\widetilde{Var}}_{biFd}$ is the variance of $\left({\widehat{biF}}_1-{\widehat{biF}}_2\right)$ with $\left\{p_{ijk}\right\}$ replaced by $\left\{{\tilde{p}}_{ijk}\right\}$, that is calculated from the MLE of $\left\{p_{ijk}\right\}$ under the null hypothesis.

3.2 |. Test statistic for comparing two $\mathit{m}\mathit{i}\mathit{F}\mathit{s}$

As shown in (3) and (4), $mi{F}_1=\sum p_{a.a},mi{F}_2=\sum p_{.aa}$, and the MLE of $mi{F}_1$ and $mi{F}_2$ are

$$\widehat{mi{F}_1}=\sum _{a=1}^r{\hat{p}}_{a.a}=\sum _{a=1}^r\frac{n_{a.a}}{N},\widehat{mi{F}_2}=\sum _{a=1}^r{\hat{p}}_{.aa}=\sum _{a=1}^r\frac{n_{.aa}}{N}.$$

Again by the delta-method and multivariate central limit theorem (Appendix A.2), the Wald statistic for testing $H_0:mi{F}_1=mi{F}_2$ versus $H_1:mi{F}_1\ne mi{F}_2$ is

$$T_{\mathit{miF}}^W=\frac{{\left({\widehat{miF}}_1-{\widehat{miF}}_2\right)}^2}{{\widehat{Var}}_{miFd}},$$

where ${\widehat{Var}}_{\mathit{miFd}}$ is the variance of $\left({\widehat{miF}}_1-{\widehat{miF}}_2\right)$ with $\left\{p_{ijk}\right\}$ replaced by $\left\{{\hat{p}}_{ijk}\right\}$. The test statistic is distributed a symptotically as a ${\chi }^2$ distribution with one degree of freedom under the null hypothesis.

Again to develop the score statistic, we consider the MLE of $\left\{p_{ijk}\right\}$ under the null hypothesis as in the case of $biF$. The score statistic for testing $H_0:mi{F}_1=mi{F}_2$ versus $H_1:mi{F}_1\ne mi{F}_2$ is

$$T_{miF}^S=\frac{{\left({\widehat{miF}}_1-{\widehat{miF}}_2\right)}^2}{{\widetilde{Var}}_{miFd}},$$

where ${\widetilde{Var}}_{\mathit{miFd}}$ is the variance of $\left({\widehat{miF}}_1-{\widehat{miF}}_2\right)$ with $\left\{p_{ijk}\right\}$ replaced by $\left\{{\tilde{p}}_{ijk}\right\}$, that is calculated from the MLE of $\left\{p_{ijk}\right\}$ under the null hypothesis.

3.3 |. Test statistic for comparing two $\mathit{m}\mathit{a}\mathit{F}\mathit{s}$

The MLE of $ma{F}_1$ and $ma{F}_2$ can be obtained by substituting $p_{a.a},p_{.aa},p_{a..},p_{.a.}$ and $p_{..a}$ by their MLE’s in (7) and (8).

$${\widehat{maF}}_1=\frac{2}{r}\sum _{a=1}^r\frac{{\hat{p}}_{a.a}}{{\hat{p}}_{a..}+{\hat{p}}_{..a}}=\frac{2}{r}\sum _{a=1}^r\frac{n_{a.a}}{n_{a..}+n_{..a}},{\widehat{maF}}_2=\frac{2}{r}\sum _{a=1}^r\frac{{\hat{p}}_{.aa}}{{\hat{p}}_{.a.}+{\hat{p}}_{..a}}=\frac{2}{r}\sum _{a=1}^r\frac{n_{.aa}}{n_{.a.}+n_{..a}}.$$

Again by the delta-method and multivariate central limit theorem (Appendix A.3), we have the Wald statistic for testing $H_0:ma{F}_1=ma{F}_2$ versus $H_1:ma{F}_1\ne ma{F}_2$ as

$$T_{maF}^W=\frac{{\left({\widehat{maF}}_1-{\widehat{maF}}_2\right)}^2}{{\widehat{Var}}_{maFd}},$$

where ${\widehat{Var}}_{maFd}$ is the variance of $\left({\widehat{maF}}_1-{\widehat{maF}}_2\right)$ with $\left\{p_{ijk}\right\}$ replaced by $\left\{{\hat{p}}_{ijk}\right\}$. The test statistic is distributed asymptotically as a ${\chi }^2$ distribution with one degree of freedom under the null hypothesis.

For the score statistic, we consider the MLE of $\left\{p_{ijk}\right\}$ under the null hypothesis as in the case of $biF$ and $miF$. The score statistic for testing $H_0:ma{F}_1=ma{F}_2$ versus $H_1:ma{F}_1\ne ma{F}_2$ is

$$T_{maF}^S=\frac{{\left({\widehat{maF}}_1-{\widehat{maF}}_2\right)}^2}{{\widetilde{Var}}_{maFd}},$$

where ${\widetilde{Var}}_{maFd}$ is the variance of $\left({\widehat{maF}}_1-{\widehat{maF}}_2\right)$ replaced by $\left\{{\tilde{p}}_{ijk}\right\}$, that is calculated from the MLE of $\left\{p_{ijk}\right\}$ under the null hypothesis.

3.4 |. Test statistic for comparing two $\mathit{m}\mathit{a}{\mathit{F}}^{\mathrm{*}}\mathbf{s}$

To obtain the MLEs of $ma{F}_1^{\mathrm{*}}$ and $ma{F}_2^{\mathrm{*}}$, we first substitute $p_{a.a},p_{.aa},p_{a..},p_{.a.}$ and $p_{..a}$ by their MLE’s to get MLE’s of $maP$ and $maR$ and use these in (9) and (10):

$${\widehat{maF}}_1^{\mathrm{*}}=2\frac{\widehat{ma{P}_1}\times \widehat{ma{R}_1}}{\widehat{ma{P}_1}+\widehat{ma{R}_1}},{\widehat{maF}}_2^{\mathrm{*}}=2\frac{\widehat{ma{P}_2}\times \widehat{ma{R}_2}}{\widehat{ma{P}_2}+\widehat{ma{R}_2}}.$$

Using the delta-method and multivariate central limit theorem (Appendix A.4), we have the Wald statistic for testing $H_0:ma{F}_1^{\mathrm{*}}=ma{F}_2^{\mathrm{*}}$ versus $H_1:ma{F}_1^{\mathrm{*}}\ne ma{F}_2^{\mathrm{*}}$ as

$$T_{maF}^W=\frac{{\left({\widehat{maF}}_1^{\mathrm{*}}-{\widehat{maF}}_2^{\mathrm{*}}\right)}^2}{{\widehat{Var}}_{maF{d}^{\mathrm{*}}}},$$

Again to get ${\widehat{Var}}_{maF{d}^{\mathrm{*}}}$, all components of the variance of $\left({\widehat{maF}}_1^{\mathrm{*}}-{\widehat{maF}}_2^{\mathrm{*}}\right)$ are replaced by their respective MLE’s. The test statistic is distributed asymptotically as a ${\chi }^2$ distribution with one degree of freedom under the null hypothesis.

On the other hand, for the score statistic, we consider the MLE of $\left\{p_{ijk}\right\}$ under the null hypothesis as in the case of $biF,miF$, and $maF$. The score statistic for testing $H_0:ma{F}_1^{\mathrm{*}}=ma{F}_2^{\mathrm{*}}$ versus $H_1:ma{F}_1^{\mathrm{*}}\ne ma{F}_2^{\mathrm{*}}$ is

$$T_{maF}^S=\frac{{\left({\widehat{maF}}_1^{\mathrm{*}}-{\widehat{maF}}_2^{\mathrm{*}}\right)}^2}{{\widetilde{Var}}_{maF{d}^{\mathrm{*}}}},$$

where ${\widetilde{Var}}_{maF{d}^{\mathrm{*}}}$ is the variance of $\left({\widehat{maF}}_1^{\mathrm{*}}-{\widehat{maF}}_2^{\mathrm{*}}\right)$ replaced by $\left\{{\tilde{p}}_{ijk}\right\}$, that is calculated from the MLE of $\left\{p_{ijk}\right\}$ under the null hypothesis.

4 |. SIMULATION

4.1 |. Simulation setup

A simulation study was conducted to evaluate the performance of the test statistics proposed in Section 3. We set $r=3$ (class 1, 2, 3), and generated data according to the multinomial distributions with $\mathit{p}$ shown in Table 2. Classes 2 and 3 were combined when calculating $biF$. The total sample size, $N$, was set to 100, 300, 500, and 1,000. The nominal type I error rate was set to 0.05 (two-sided test). We used the empirical type I error rate and empirical power as performance measures. For each combination of the scenario and sample size, we performed 100,000 repeated simulations.

TABLE 2.

Simulation study: True cell probabilities.

		True class = 1			True class = 2			True class = 3
		Test 2			Test 2			Test 2
Scenario 1		1	2	3	1	2	3	1	2	3
Test 1	1	40/300	10/300	10/300	5/300	10/300	5/300	5/300	5/300	10/300
	2	10/300	5/300	5/300	10/300	40/300	10/300	5/300	5/300	10/300
	3	10/300	5/300	5/300	5/300	10/300	5/300	10/300	10/300	40/300
$bi{F}_1=mi{F}_1=ma{F}_1=ma{F}_1^{\text{*}}=0.60$
$bi{F}_2=mi{F}_2=ma{F}_2=ma{F}_2^{\text{*}}=0.60$
		True class = 1			True class = 2			True class = 3
		Test 2			Test 2			Test 2
Scenario 2		1	2	3	1	2	3	1	2	3
Test 1	1	120/500	30/500	30/500	5/500	10/500	5/500	5/500	5/500	10/500
	2	30/500	15/500	15/300	10/500	40/500	10/500	5/500	5/500	10/500
	3	30/500	15/500	15/500	5/500	10/500	5/500	10/500	10/500	40/500
$bi{F}_1=0.69,mi{F}_1=0.60,ma{F}_1=0.56,ma{F}_1^{\text{*}}=0.58$
$bi{F}_2=0.69,mi{F}_2=0.60,ma{F}_2=0.56,ma{F}_2^{\text{*}}=0.58$
		True class = 1			True class = 2			True class = 3
		Test 2			Test 2			Test 2
Scenario 3		1	2	3	1	2	3	1	2	3
Test 1	1	30/300	15/300	15/300	5/300	10/300	5/300	5/300	5/300	10/300
	2	10/300	5/300	5/300	15/300	30/300	15/300	5/300	5/300	10/300
	3	10/300	5/300	5/300	5/300	10/300	5/300	15/300	15/300	30/300
$bi{F}_1=mi{F}_1=ma{F}_1=ma{F}_1^{\text{*}}=0.60$
$bi{F}_2=mi{F}_2=ma{F}_2=ma{F}_2^{\text{*}}=0.50$
		True class = 1			True class = 2			True class = 3
		Test 2			Test 2			Test 2
Scenario 4		1	2	3	1	2	3	1	2	3
Test 1	1	90/500	45/500	45/500	5/500	10/500	5/500	5/500	5/500	10/500
	2	30/500	15/500	15/300	15/500	30/500	15/500	5/500	5/500	10/500
	3	30/500	15/500	15/500	5/500	10/500	5/500	15/500	15/500	30/500
$bi{F}_1=0.69,mi{F}_1=0.60,ma{F}_1=0.56,ma{F}_1^{\text{*}}=0.58$
$bi{F}_2=0.60,mi{F}_2=0.50,ma{F}_2=0.47,ma{F}_2^{\text{*}}=0.49$

Scenarios 1 and 2 are set up to evaluate the empirical type I error rate of the proposed test statistics, while scenario 3 and 4 are designed to assess their empirical power. In scenario 1, the true conditions of classes 1, 2, and 3 have the same probability (1∕3), and the recalls and precisions within each class are equal in the two tests (60%). Thus, ${maR}_1=ma{R}_2=ma{P}_1=ma{P}_2=0.60$, and $F_{1a}=F_{2a}=0.60$ for each class, $a=1,2,3$. Then, $ma{F}_1=ma{F}_2=ma{F}_1^{\mathrm{*}}=ma{F}_2^{\mathrm{*}}=0.60$. Because classes 2 and 3 are combined to calculate $bi{F}_1=F_{11}=0.60$ and $bi{F}_2=F_{21}=0.60$. Also, $p_{a.a}=p_{.aa}=0.20$ for each class $a=1,2,3$, and $mi{F}_1=mi{F}_2=0.60$.

In scenario 2, the true condition of class 1 has higher probability than the others (60% vs. 20%), and performances of two tests are equal: $bi{F}_1=bi{F}_2=0.69,mi{F}_1=mi{F}_2=0.60,ma{F}_1=ma{F}_2=0.56$, and $ma{F}_1^{\mathrm{*}}=ma{F}_2^{\mathrm{*}}=0.58$. Although the distributions in scenario 2 are the same as those in scenario 1 for each class, the value of $biF,maF$ and $ma{F}^{\mathrm{*}}$ are different between scenarios because $T{P}_a/\left(T{P}_a+F{P}_a\right)$ is large in the true class = 1 and, conversely, relatively small in the true classes 2 and 3. In contrast, $miF$ in scenario 2 is the same as that in scenario 1 because $p_{a.a}$ and $p_{.aa}$ for each class $a=1,2,3$ in scenarios 1 and 2 are equal.

The true conditions of classes 1, 2, and 3 have the same probability (1∕3) in scenario 3. However, $maR$ and $maP$ of Test 2 are lower than Test 1 (60% vs. 50%), $F_{2a}$ are lower than $F_{1a}$ (60% vs. 50%), and $p_{.aa}$ is lower than $p_{a.a}$ for each class $a=1,2,3$ (20% vs. 17%). Therefore, $bi{F}_1=mi{F}_1=ma{F}_1=ma{F}_1^{\mathrm{*}}=0.60$, whereas $bi{F}_2=mi{F}_2=ma{F}_2=ma{F}_2^{\mathrm{*}}=0.50$.

In scenario 4, the true condition of class 1 has higher probability than the others (60% vs. 20%) as in scenario 2. However, the performance of two tests are different: $bi{F}_1=0.69$ versus $bi{F}_2=0.60,mi{F}_1=0.60$ versus $mi{F}_2=0.50,ma{F}_1=0.56$ versus $ma{F}_2=0.47$, and $ma{F}_1^{\mathrm{*}}=0.58$ versus $ma{F}_2^{\mathrm{*}}=0.49$.

4.2 |. Simulation result

Table 3 shows the empirical type I error rates of the proposed tests for scenarios 1 and 2. The empirical type I error rates for both test statistics were close to nominal type I error rate of 0.05 when the sample size is large (300, 500, 1000). When $N$ is relatively small (100), the empirical type I error rates tended to be slightly larger than 0.05, especially for Wald statistics. Contrarily, the empirical type I error rates with score statistics are close to the nominal type I error rate of 0.05 for all sample sizes. Table 4 shows the empirical power of the proposed tests for scenarios 3 and 4. As shown in Table 4, the empirical powers increase with the sample size. The empirical powers of Wald statistics and score statistics are similar, especially when the sample size is large.

TABLE 3.

Simulation study: Empirical type I error rates.

Scenario	N	$T_{biF}^W$	$T_{biF}^S$	$T_{miF}^W$	$T_{miF}^S$	$T_{maF}^W$	$T_{maF}^S$	$T_{maF*}^W$	$T_{maF*}^S$
1	100	0.057	0.050	0.053	0.049	0.055	0.051	0.057	0.053
	300	0.052	0.050	0.051	0.050	0.052	0.051	0.052	0.051
	500	0.051	0.050	0.050	0.050	0.051	0.050	0.051	0.050
	1000	0.050	0.049	0.051	0.050	0.051	0.050	0.051	0.051
2	100	0.052	0.049	0.054	0.049	0.058	0.053	0.061	0.055
	300	0.052	0.050	0.051	0.050	0.054	0.052	0.054	0.052
	500	0.051	0.050	0.051	0.050	0.051	0.050	0.052	0.051
	1000	0.050	0.050	0.051	0.051	0.052	0.051	0.051	0.051

TABLE 4.

Simulation study: Empirical power.

Scenario	N	$T_{biF}^W$	$T_{biF}^S$	$T_{miF}^W$	$T_{miF}^S$	$T_{maF}^W$	$T_{maF}^S$	$T_{maF*}^W$	$T_{maF*}^S$
3	100	0.192	0.174	0.304	0.289	0.309	0.297	0.310	0.300
	300	0.438	0.429	0.694	0.689	0.696	0.692	0.696	0.692
	500	0.641	0.635	0.890	0.888	0.889	0.888	0.889	0.888
	1000	0.905	0.904	0.995	0.995	0.995	0.995	0.995	0.995
4	100	0.235	0.226	0.305	0.291	0.291	0.278	0.271	0.256
	300	0.560	0.556	0.695	0.690	0.662	0.657	0.615	0.609
	500	0.773	0.771	0.889	0.887	0.865	0.863	0.826	0.824
	1000	0.969	0.969	0.995	0.995	0.992	0.992	0.984	0.984

5 |. EXAMPLE

We describe an application of the proposed hypothesis testing procedure to the motivating example.³ In this study, a skin cancer classification system with faster, region-based convolutional neural network algorithm (FRCNN) for brown to black pigmented skin lesions was developed using a deep learning method. The target diseases were malignant tumors (malignant melanoma (MM) and basal cell carcinoma (BCC)) and benign tumors (nevus, seborrheic keratosis (SK), senile lentigo (SL) and hematoma/hemangioma (H/H)), and 2000 images were evaluated. The 2000 images were obtained by randomly sampling 200 images from the 666 images 10 times. For illustration, all images were treated as independent in this study. The data are shown in Tables B1–B3, Appendix B. Although images were classified into six categories (MM, BCC, nevus, SK, SL, H/H), accuracy was the only performance measure computed for six-class classification data in the motivating example. Other performance measures, sensitivity, specificity, false negative, false positive, and positive predictive value, were calculated for two-class classification data after combining malignant tumors (MM and BCC) and benign tumors (nevus, SK, SL, and H/H). The accuracy of six-class classification by the FRCNN (86.2% ± 2.95%) was statistically higher than that of board-certified dermatologists (BCD) (79.5% ± 5.27%, p = 0.0081) and that of dermatologic trainees (75.1% ± 2.18%, p < 0.0001).

We compared the performance of skin cancer classification between the FRCNN and BCD using $biF\mathrm{s},miF\mathrm{s}$, $maF\mathrm{s}$, and $ma{F}^{\mathrm{*}}\mathrm{s}$ with the proposed hypothesis testing procedures. $miF\mathrm{s},maF\mathrm{s}$, and $ma{F}^{\mathrm{*}}\mathrm{s}$ were calculated from six-class classification data, while $biF\mathrm{s}$ were calculated from two-class classification data (malignant tumors vs. benign tumors). The results are shown in Table 5. All $biF\mathrm{s},miF\mathrm{s},maF\mathrm{s},$ and $ma{F}^{\mathrm{*}}\mathrm{s}$ of six-class classification by FRCNN were significantly higher than those by BCD.

TABLE 5.

Example.

	FRCNN	BCD	Difference	Test statistics	p-value
biF (Wald)	0.840	0.776	0.064	19.4	< 0.001
biF (score)	0.840	0.776	0.064	18.9	< 0.001
miF (Wald)	0.862	0.795	0.067	41.9	< 0.001
miF (score)	0.862	0.795	0.067	41.0	< 0.001
maF (Wald)	0.846	0.768	0.078	26.2	< 0.001
maF (score)	0.846	0.768	0.078	24.5	< 0.001
maF* (Wald)	0.848	0.772	0.076	26.4	< 0.001
maF* (score)	0.848	0.772	0.076	23.0	< 0.001

6 |. DISCUSSION

We developed hypothesis testing procedures for comparing two $F_1$-scores $\left(bi{F}_1\right.$ and $bi{F}_2,mi{F}_1$ and $mi{F}_2,ma{F}_1$ and $ma{F}_2$, and $ma{F}_1^{\mathrm{*}}$ and $ma{F}_2^{\mathrm{*}}$) in paired study design. Through the simulation study and motivating example, we assessed the performance and feasibility of those testing procedures. We conclude that the method based on the score statistics ( $T_{biF}^S,T_{miF}^S,T_{maF}^S$, and $T_{ma{F}^{\mathrm{*}}}^S$) is slightly better compared to the method based on the Wald statistics ($T_{biF}^W$, $T_{miF}^W$, $T_{maF}^W$, and $T_{ma{F}^{\mathrm{*}}}^W$) because the empirical type I error rate is closer to the nominal level even when the sample size is small. However, when multi-class classification is considered, typical sample size is much larger than 100, and both approaches perform equally well in such scenarios.

We did not observe a substantial disparity in the empirical powers of the two approaches.

At present, others have not studied hypothesis testing procedure of $biF\mathrm{s},miF\mathrm{s}$, $maF\mathrm{s}$, and $ma{F}^{\mathrm{*}}\mathrm{s}$, and only the point estimates of these scores were reported in most studies. Han et al¹⁹ applied one sample t-test for comparison of $biF\mathrm{s}$; however, this approach may not be appropriate because $biF$ is the harmonic mean of precision and recall, and the distribution of the difference between two $biF\mathrm{s}$ is unlikely to follow a Student’s t-distribution.

A limitation of this work is that the proposed procedures are based on the large sample theory, and thus require a large sample size to provide strict control of the type I error rate. For future works, we are working on the exact test for comparing two $biF\mathrm{s},miF\mathrm{s}$, $maF\mathrm{s}$, and $ma{F}^{\mathrm{*}}\mathrm{s}$ based on the methods presented in this article.

An R code for computing point estimates, Wald statistics, score statistics, and p-values for $biF\mathrm{s},miF\mathrm{s}$, $maF\mathrm{s}$, and $maF\mathrm{s}$ of each statistic in the paired design, is available on the lead author’s GitHub page: https://github.com/kanaet52/f1score/blob/main/R/F1score_test.R. For two-sample designs, the $F_1$-scores can be compared by setting the covariance part of the test statistic ($C_{\mathit{biF}}^W,C_{\mathit{miF}}^W,C_{\mathit{maF}}^W,C_{\mathit{maF}\mathrm{*}}^W$, see Appendix A) to 0. Note that for the score statistics, the MLE of $\left\{p_{ijk}\right\}$ under each null hypothesis is obtained by applying the Newton-Raphson method to the log-likelihood equations in the code.

APPENDIX A. DERIVATION OF VARIANCES

A.1. Variance of $\mathit{b}\mathit{i}\mathit{F}$

The derivation of the variance of $\left({\widehat{biF}}_1-{\widehat{biF}}_2\right)$ is as follows:

$$\begin{array}{l}\mathit{b}\mathit{i}\mathit{F}=\left[\begin{array}{c}bi{F}_1\\ bi{F}_2\end{array}\right],\\ {\left[\frac{\partial \left(\mathit{b}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]}^T\left(\mathit{p}{\mathit{p}}^T\right)\left[\frac{\partial \left(\mathit{b}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]=\mathbf{0},\\ {\left[\frac{\partial \left(\mathit{b}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]}^T\left(diag\left(\mathit{p}\right)-\mathit{p}{\mathit{p}}^T\right)\left[\frac{\partial \left(\mathit{b}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]={\left[\frac{\partial \left(\mathit{b}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]}^T\left(diag\left(\mathit{p}\right)\right)\left[\frac{\partial \left(\mathit{b}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]=\left[\begin{array}{ll}{A}_{biF}^W& C_{biF}^W\\ C_{biF}^W& B_{biF}^W\end{array}\right],\end{array}$$

with

$$A_{biF}^W=\frac{1}{{\left(p_{1..}+p_{..1}\right)}^2}\left[p_{1.1}{\left(2\left(1-bi{F}_1\right)\right)}^2+\left(p_{1.2}+p_{2.1}\right)bi{F}_1^2\right]$$

$$B_{biF}^W=\frac{1}{{\left(p_{.1.}+p_{..1}\right)}^2}\left[p_{.11}{\left(2\left(1-bi{F}_2\right)\right)}^2+\left(p_{.12}+p_{.21}\right)bi{F}_2^2\right]$$

$$\begin{gathered} C_{biF}^W=\frac{1}{\left(p_{1..}+p_{..1}\right)\left(p_{.1.}+p_{..1}\right)}\left[2^2{p}_{111}\left(1-bi{F}_1\right)\left(1-bi{F}_2\right)-2p_{121}\left(1-bi{F}_1\right)bi{F}_2\right. \\ \left.-2p_{211}bi{F}_1(1-biF2)+\left(p_{221}+p_{112}\right)bi{F}_{1}bi{F}_2\right]. \end{gathered}$$

Therefore, the variance of $\widehat{bi{F}_1}$ is

$$\frac{1}{{\left(p_{1..}+p_{..1}\right)}^2}\left[p_{1.1}{\left(2\left(1-bi{F}_1\right)\right)}^2+\left(p_{1.2}+p_{2.1}\right)bi{F}_1^2\right]/n,$$

the variance of $\widehat{bi{F}_2}$ is

$$\frac{1}{{\left(p_{.1.}+p_{..1}\right)}^2}\left[p_{.11}{\left(2\left(1-bi{F}_2\right)\right)}^2+\left(p_{.12}+p_{.21}\right)bi{F}_2^2\right]/n,$$

the variance of $\left(\widehat{bi{F}_1}-\widehat{bi{F}_2}\right)$ is

$$\left(A_{biF}^W+B_{biF}^W-2C_{biF}^W\right)/n.$$

A.2. Variance of $\mathit{m}\mathit{i}\mathit{F}$

The derivation of the variance of $\left({\widehat{miF}}_1-{\widehat{miF}}_2\right)$ is as follows:

$$\begin{array}{l}\mathit{m}\mathit{i}\mathit{F}=\left[\begin{array}{c}mi{F}_1\\ mi{F}_2\end{array}\right],\\ {\left[\frac{\partial \left(\mathit{m}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]}^T\left(\mathit{p}{\mathit{p}}^T\right)\left[\frac{\partial \left(\mathit{m}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]=\left[\begin{array}{cc}{\left(\sum _{a=1}^r{p}_{a.a}\right)}^2& \left(\sum _{a=1}^r{p}_{a.a}\right)\left(\sum _{a=1}^r{p}_{.aa}\right)\\ \left(\sum _{a=1}^r{p}_{a.a}\right)\left(\sum _{a=1}^r{p}_{.aa}\right)& {\left(\sum _{a=1}^r{p}_{.aa}\right)}^2\end{array}\right]=\left[\begin{array}{cc}mi{F}_1^2& mi{F}_{1}mi{F}_2\\ mi{F}_{1}mi{F}_2& mi{F}_2^2\end{array}\right],\\ {\left[\frac{\partial (\mathit{m}\mathit{i}\mathit{F}}{\partial \left(\mathit{p}\right)}\right]}^T\left(diag\left(\mathit{p}\right)\right)\left[\frac{\partial \left(\mathit{m}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]=\left[\begin{array}{cc}\sum _{a=1}^r{p}_{a.a}& \sum _{a=1}^r{p}_{aaa}\\ \sum _{a=1}^r{p}_{aaa}& \sum _{a=1}^r{p}_{.aa}\end{array}\right]=\left[\begin{array}{cc}mi{F}_1& \sum _{a=1}^r{p}_{aaa}\\ \sum _{a=1}^r{p}_{aaa}& mi{F}_2\end{array}\right],\\ {\left[\frac{\partial \left(\mathit{m}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]}^T\left(diag\left(\mathit{p}\right)-\mathit{p}{\mathit{p}}^T\right)\left[\frac{\partial \left(\mathit{m}\mathit{i}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]=\left[\begin{array}{cc}{A}_{miF}^W& C_{miF}^W\\ C_{miF}^W& B_{miF}^W\end{array}\right],\end{array}$$

with

$$A_{miF}^W=mi{F}_1\left(1-mi{F}_1\right),$$

$$B_{miF}^W=mi{F}_2\left(1-mi{F}_2\right),$$

$$C_{miF}^W=\sum _{a=1}^r{p}_{aaa}-mi{F}_{1}mi{F}_2.$$

Therefore, the variance of $\left({\widehat{miF}}_1-{\widehat{miF}}_2\right)$ is

$$\left(A_{miF}^W+B_{miF}^W-2C_{miF}^W\right)/n.$$

A.3. Variance of $\mathit{m}\mathit{a}\mathit{F}$

The derivation of the variance of $\left({\widehat{maF}}_1-{\widehat{maF}}_2\right)$ is as follows.

$$\begin{array}{l}\mathit{m}\mathit{a}\mathit{F}=\left[\begin{array}{c}ma{F}_1\\ ma{F}_2\end{array}\right],\\ {\left[\frac{\partial \left(\mathit{m}\mathit{a}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]}^T\left(\mathit{p}{\mathit{p}}^T\right)\left[\frac{\partial \left(\mathit{m}\mathit{a}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]=\mathbf{0},\\ {\left[\frac{\partial \left(\mathit{m}\mathit{a}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]}^T\left(diag\left(\mathit{p}\right)-\mathit{p}{\mathit{p}}^T\right)\left[\frac{\partial \left(\mathit{m}\mathit{a}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]={\left[\frac{\partial \left(\mathit{m}\mathit{a}\mathit{F}\right)}{\partial \left(\mathit{p}\right)}\right]}^T(diag(\mathit{p}))\left[\frac{\partial (\mathit{m}\mathit{a}\mathit{F})}{\partial (\mathit{p})}\right]\\ =\frac{1}{r^2}\left[\begin{array}{ll}{A}_{maF}^W& C_{maF}^W\\ C_{maF}^W& B_{maF}^W\end{array}\right],\end{array}$$

with

$$\begin{array}{l}{A}_{maF}^W=\sum _{a=1}^r{p}_{a.a}{\left(\frac{2\left(1-F_{1a}\right)}{p_{a..}+p_{..a}}\right)}^2+\sum _{a=1}^r\sum _{b\ne a}{p}_{a.b}{\left(\frac{F_{1a}}{p_{a..}+p_{..a}}+\frac{F_{1b}}{p_{b..}+p_{..b}}\right)}^2,\\ B_{maF}^W=\sum _{a=1}^r{p}_{.aa}{\left(\frac{2\left(1-F_{2a}\right)}{p_{.a.}+p_{..a}}\right)}^2+\sum _{a=1}^r\sum _{b\ne a}{p}_{.ab}{\left(\frac{F_{2a}}{p_{.a.}+p_{..a}}+\frac{F_{2b}}{p_{.b.}+p_{..b}}\right)}^2,\\ C_{maF}^W=\sum _{a=1}^r{p}_{aaa}\frac{2^2\left(1-F_{1a}\right)\left(1-F_{2a}\right)}{\left(p_{a..}+p_{..a}\right)\left(p_{.a.}+p_{..a}\right)}\\ -\sum _{a=1}^r\sum _{b\ne a}\left\{p_{aba}\frac{2\left(1-F_{1a}\right)}{\left(p_{a..}+p_{..a}\right)}\left(\frac{F_{2a}}{\left(p_{.a.}+p_{..a}\right)}+\frac{F_{2b}}{\left(p_{.b.}+p_{..b}\right)}\right)+p_{baa}\frac{2\left(1-F_{2a}\right)}{\left(p_{.a.}+p_{..i}\right)}\left(\frac{F_{1a}}{\left(p_{a..}+p_{..a}\right)}+\frac{F_{1b}}{\left(p_{b..}+p_{..b}\right)}\right)\right\}\\ +\sum _{a=1}^r\sum _{b\ne a}\sum _{c\ne a}{p}_{bca}\left(\frac{F_{1a}}{\left(p_{a..}+p_{..a}\right)}+\frac{F_{1b}}{\left(p_{b..}+p_{..b}\right)}\right)\left(\frac{F_{2a}}{\left(p_{.a.}+p_{..a}\right)}+\frac{F_{2c}}{\left(p_{.c.}+p_{..c}\right)}\right).\end{array}$$

Therefore, the variance of $\left({\widehat{maF}}_1-{\widehat{maF}}_2\right)$ is

$$\frac{1}{r^2}\left(A_{\text{maF}}^W+B_{\text{maF}}^W-2C_{\text{maF}}^W\right)/n.$$

A.4. Variance of $\mathit{m}\mathit{a}{\mathit{F}}^{\mathbf{*}}$

The derivation of the variance of $\left({\widehat{ma{F}^{\mathrm{*}}}}_1-{\widehat{ma{F}^{\mathrm{*}}}}_2\right)$ is as follows.

$$\begin{array}{l}\mathit{m}\mathit{a}{\mathit{F}}^{\mathrm{*}}=\left(\genfrac{}{}{0pt}{}{ma{F}_1^{\mathrm{*}}}{ma{F}_2^{\mathrm{*}}}\right),\\ {\left[\frac{\partial (\mathit{m}\mathit{a}{\mathit{F}}^{\mathrm{*}})}{\partial (\mathit{p})}\right]}^T\left(\mathit{p}{\mathit{p}}^T\right)\left[\frac{\partial (\mathit{m}\mathit{a}{\mathit{F}}^{\mathrm{*}})}{\partial (\mathit{p})}\right]=\mathbf{0},\\ {\left[\frac{\partial (\mathit{m}\mathit{a}{\mathit{F}}^{\mathrm{*}})}{\partial (\mathit{p})}\right]}^T\left(\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(\mathit{p})-\mathit{p}{\mathit{p}}^T\right)\left[\frac{\partial (\mathit{m}\mathit{a}{\mathit{F}}^{\mathrm{*}})}{\partial (\mathit{p})}\right]={\left[\frac{\partial (\mathit{m}\mathit{a}{\mathit{F}}^{\mathrm{*}})}{\partial (\mathit{p})}\right]}^T(\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{g}(\mathit{p}))\left[\frac{\partial (\mathit{m}\mathit{a}{\mathit{F}}^{\mathrm{*}})}{\partial (\mathit{p})}\right]\\ =\left(\begin{array}{ll}{A}_{ma{F}^{\mathrm{*}}}^W& C_{ma{F}^{\mathrm{*}}}^W\\ C_{ma{F}^{\mathrm{*}}}^W& B_{ma{F}^{\mathrm{*}}}^W\end{array}\right),\end{array}$$

with

$$\begin{gathered} A_{ma{F}^{\mathrm{*}}}^W=\frac{2^2}{r^2{\left(ma{P}_1+ma{R}_1\right)}^4}\left[\sum _{a=1}^r{p}_{a.a}{\left(\frac{\left(p_{a..}-p_{a.a}\right)ma{R}_1^2}{p_{a..}^2}+\frac{\left(p_{..a}-p_{a.a}\right)ma{P}_1^2}{p_{..a}^2}\right)}^2\right. \\ \left.+\sum _{a=1}^r\sum _{b\ne a}{p}_{a.b}{\left(\frac{p_{a.a}ma{R}_1^2}{p_{a..}^2}+\frac{p_{b.b}ma{P}_1^2}{p_{..b}^2}\right)}^2\right], \end{gathered}$$

$$\begin{gathered} B_{ma{F}^{\mathrm{*}}}^W=\frac{2^2}{r^2{\left(ma{P}_2+ma{R}_2\right)}^4}\left[{\sum }_{a=1}^r{p}_{.aa}{\left(\frac{\left(p_{.a.}-p_{.aa}\right)ma{R}_2^2}{p_{.a.}^2}+\frac{\left(p_{..a}-p_{.aa}\right)ma{P}_2^2}{p_{..a}^2}\right)}^2\right. \\ \left.+\sum _{a=1}^r\sum _{b\ne a}{p}_{.ab}{\left(\frac{p_{.aa}ma{R}_2^2}{p_{.a.}^2}+\frac{p_{.bb}ma{P}_2^2}{p_{..b}^2}\right)}^2\right], \end{gathered}$$

$$\begin{array}{l}{C}_{ma{F}^{\mathrm{*}}}^W=\frac{2^2}{r^2{\left(ma{P}_1+ma{R}_1\right)}^2{\left(ma{P}_2+ma{R}_2\right)}^2}\\ & \times \left[\sum _{a=1}^r{p}_{aaa}\left(\frac{p_{a..}-p_{a.a}}{p_{a..}^2}ma{R}_1^2+\frac{p_{..a}-p_{a.a}}{p_{..a}^2}ma{P}_1^2\right)\left(\frac{p_{.a.}-p_{.aa}}{p_{.a.}^2}ma{R}_2^2+\frac{p_{..a}-p_{.aa}}{p_{..a}^2}ma{P}_2^2\right)\right.\\ & -\sum _{a=1}^r\sum _{b\ne a}\left\{p_{aba}\left(\frac{p_{a..}-p_{a.a}}{p_{a..}^2}ma{R}_1^2+\frac{p_{..a}-p_{a.a}}{p_{..a}^2}ma{P}_1^2\right)\left(\frac{p_{.bb}}{p_{.b.}^2}ma{R}_2^2+\frac{p_{.bb}}{p_{..b}^2}ma{P}_2^2\right)\right.\\ & \left.+p_{baa}\left(\frac{p_{.a.}-p_{.aa}}{p_{.a.}^2}ma{R}_2^2+\frac{p_{..a}-p_{.aa}}{p_{..a}^2}ma{P}_2^2\right)\left(\frac{p_{b.b}}{p_{b..}^2}ma{R}_1^2+\frac{p_{b.b}}{p_{..b}^2}ma{P}_1^2\right)\right\}\\ & \left.+\sum _{a=1}^r\sum _{b\ne a}\sum _{c\ne a}{p}_{bca}\left(\frac{p_{b.b}}{p_{b..}^2}ma{R}_1^2+\frac{p_{a.a}}{p_{..a}^2}ma{P}_1^2\right)\left(\frac{p_{.cc}}{p_{.c.}^2}ma{R}_2^2+\frac{p_{.aa}}{p_{..a}^2}ma{P}_2^2\right)\right].\end{array}$$

Therefore, the variance of $\left({\widehat{ma{F}^{\mathrm{*}}}}_1-{\widehat{ma{F}^{\mathrm{*}}}}_2\right)$ is

$$\left({\mathit{A}}_{\mathit{m}\mathit{a}{\mathit{F}}^{\mathrm{*}}}^{\mathit{W}}+{\mathit{B}}_{\mathit{m}\mathit{a}{\mathit{F}}^{\mathrm{*}}}^{\mathit{W}}-2{\mathit{C}}_{\mathit{m}\mathit{a}{\mathit{F}}^{\mathrm{*}}}^{\mathit{W}}\right)/\mathit{n}.$$

APPENDIX B. EXAMPLE DATA

Tables B1, B2, and B3 here.

TABLE B1.

Example data.

		True condition
FRCNN	BCD	MM	BCC	Nevus	SK	H/H	SL
MM	MM	289	2	20	6	2	0
	BCC	9	2	2	5	0	0
	Nevus	10	0	14	0	0	0
	SK	14	2	4	10	0	0
	H/H	3	0	2	0	1	0
	SL	2	0	0	0	0	0
BCC	MM	6	6	0	1	0	0
	BCC	2	95	0	6	0	0
	Nevus	0	2	6	0	0	0
	SK	1	5	0	2	0	0
	H/H	0	0	0	0	0	0
	SL	0	0	0	0	0	0
Nevus	MM	32	1	108	0	1	0
	BCC	1	6	8	1	0	0
	Nevus	11	1	789	8	1	0
	SK	3	3	50	27	0	0
	H/H	0	1	9	0	16	0
	SL	1	0	3	0	0	0
SK	MM	13	1	3	11	0	0
	BCC	0	1	1	12	0	0
	Nevus	1	0	11	9	0	0
	SK	7	4	14	186	0	1
	H/H	0	0	0	0	0	0
	SL	0	0	1	5	0	2
H/H	MM	0	0	0	0	6	0
	BCC	0	0	0	0	1	0
	Nevus	0	0	3	0	5	0
	SK	0	0	0	0	1	0
	H/H	0	0	0	0	44	0
	SL	0	0	0	0	0	0
SL	MM	0	0	0	0	0	0
	BCC	0	0	0	0	0	1
	Nevus	0	0	0	0	0	0
	SK	1	0	0	0	0	6
	H/H	0	0	0	0	0	0
	SL	2	0	0	0	0	35

TABLE B2.

Example data (FRCNN only).

	True condition
FRCNN	MM	BCC	Nevus	SK	H/H	SL
MM	327	6	42	21	3	0
BCC	9	108	6	9	0	0
Nevus	48	12	967	36	18	0
SK	21	6	30	223	0	3
H/H	0	0	3	0	57	0
SL	3	0	0	0	0	42

TABLE B3.

Example data (BCD only).

	True condition
BCD	MM	BCC	Nevus	SK	H/H	SL
MM	340	10	131	18	9	0
BCC	12	104	11	24	1	1
Nevus	22	3	823	17	6	0
SK	26	14	68	225	1	7
H/H	3	1	11	0	61	0
SL	5	0	4	5	0	37

DATA AVAILABILITY STATEMENT

Data sharing is not applicable to this article as no new data were created or analyzed in this study.