A simple two-step procedure using the Fellegi–Sunter model for frequency-based record linkage

Abstract

The widely used Fellegi–Sunter model for probabilistic record linkage does not leverage information contained in field values and consequently leads to identical classification of match status regardless of whether records agree on rare or common values. Since agreement on rare values is less likely to occur by chance than agreement on common values, records agreeing on rare values are more likely to be matches. Existing frequency-based methods typically rely on knowledge of error probabilities associated with field values and frequencies of agreed field values among matches, often derived using prior studies or training data. When such information is unavailable, applications of these methods are challenging. In this paper, we propose a simple two-step procedure for frequency-based matching using the Fellegi–Sunter framework to overcome these challenges. Matching weights are adjusted based on frequency distributions of the agreed field values among matches and non-matches, estimated by the Fellegi–Sunter model without relying on prior studies or training data. Through a real-world application and simulation, our method is found to produce comparable or better performance than the unadjusted method. Furthermore, frequency-based matching provides greater improvement in matching accuracy when using poorly discriminating fields with diminished benefit as the discriminating power of matching fields increases.

1. Introduction

Record linkage identifies and matches records belonging to the same entity from disparate data sources [2]. This is not a trivial task due to the lack of unique identifiers across data sources. As a result, fields such as names, address, birth date, and phone number have been used in record linkage. Record pairs are formed by comparing the field values of two records and the comparison results are used to determine if two records belong to the same entity. Record pairs that belong to the same entity are called matches; otherwise, they are called non-matches. Challenges arise for record linkage due to the use of these ambiguous and changeable fields that often contain typographical errors. Many record linkage algorithms have been developed with the model proposed by Fellegi and Sunter as the theoretical foundation [6]. The Fellegi-Sunter (FS) model is an unsupervised classification algorithm that assigns field-specific weights based on the agreement/disagreemnt between the corresponding fields. Due to its reasonable performance without the need for training data, the FS model has become widely used in many applications.

The FS model, however, does not leverage the information contained in specific values of the fields. For example, when two records agree on last name, a common name such as Smith and a rare name such as Harezlak receive the same weight. Agreement on a rare last name might be less likely to occur by chance than agreement on a common last name and as such, it is reasonable to assume that records with agreement on rare names are more likely to be a match that those with agrement on common last names.

1.1. Challenges in frequency-based matching

Efforts for frequency-based matching have been focusing on value-specific methods to modify the matching weights. Specifically, field values are assigned different weights based on their frequencies: rare values are given larger weights and common values are given smaller weights. In their seminal work, Fellegi and Sunter proposed a method by assuming that files to be linked contain no duplicate records [6]. The magnitude of the weight for a specific field is dependent on two important sets of parameters: the probability of errors associated with the field values, and frequencies of agreed field values among truly matched record pairs. The errors include misreported or unreported values in either files, or correctly reported values changing from one file to the other, reflecting variations that may arise over time. For example, address might change after relocation or last name might change for women after getting married. The true frequencies reflect the number of truly matched record pairs that agree on a specific field value. Successful implementation of this approach relies on accurately estimating both sets of parameters. When used in the linkage of Census data to its Post Enumeration Survey, the error probabilities were determined based on past experience and the frequency of specific field values occurring among true matches was assumed to equal to the minimum of the corresponding frequencies in the two files [19]. Specifically, if a field value appeared f times in one file and g times in the other file, then among the $f\times g$ record pairs that agree on the specific field value, it was assumed that the number of true matches $h=min(f,g)$, which is the maximum possible number of matches assuming no duplicate records in each file. These assumptions may not be valid in practice.

In a later investigation of Fellegi and Sunter's frequency-based matching, Winkler noted that the error probabilities can be estimated using the FS model with binary agreement/disagreement of matching fields. This avoids the calculation of the error probabilities but still requires the knowledge of matching frequencies h [17]. Yancey proposed to adjust the matching weights based on the frequency distributions of field values among true matches and among true non-matches [21], where the frequency distributions were estimated based on an assumption that a fixed proportion of the maximum possible number of matches are true matches. That is $h=\rho min(f,g)(0\le \rho \le 1)$ record pairs are true matches and the remaining $f\times g-\rho min(f,g)$ record pairs are non-matches. The value of ρ was assumed to be close to 1 in the application of Census file linkage. Such an assumption may not be valid, especially because ρ could be dependent on specific field values. An inaccurate value of ρ could again have a negative impact on the match performance. More recently, Zhu et al. proposed an empirical modification to the original matching weights based on the binary agreement/disagreement of fields in the FS model [22]. The matching weights are scaled by a factor determined by the relative magnitude of the value-specific frequency to the average field frequency to assign smaller matching weights for values with frequencies higher than the average frequency and greater weights for values with frequencies lower than the average frequency. This method is rather ad hoc and lacks theoretical justification. Performance of these value-specific frequency-based matching methods was demonstrated using real-world applications. In some applications, these methods were found to identify more matches or probable matches [17,22], while in others, there were no significant changes in matching results [21]. Systematic evaluation of the frequency-based matching, however, has not been conducted.

1.2. A motivating example

In order for frequency-based methods to provide discriminating power to identify matches from non-matches, it is important to have different frequency distributions of field values between true matches and true non-matches. To see this, assume that M is the true match status, taking values 1 (true match) or 0 (true non-match), and y is the binary agreement of the field under consideration, taking values 1 (agreement) or 0 (disagreement). Also assume two strings $s_1$ and $s_2$ are values of the field, with $s_1$ more common than $s_2$. The intuition behind the frequency-based matching is that a record pair with agreement on a more common value $s_1$ has a lower probability to be a true match than a record pair with agreement on a less common value $s_2$, or $P(M=1|y=1,s_1)<P(M=1|y=1,s_2)$. Using the Bayes formula, we could see that it implies

$$\frac{P(s_1|y=1,M=1)}{P(s_1|y=1,M=0)}<\frac{P(s_2|y=1,M=1)}{P(s_2|y=1,M=0)},$$

suggesting that the common value $s_1$ occurs more frequently among true non-matches while the rare value $s_2$ occurs more frequently among true matches.

With this intuition, we examine the frequency distributions of last names in a real-world data deduplication example and evaluate the possibility of improved matching accuracy when frequency-based matching is employed. In this application, a total of 765,814 Health Level 7 (HL7) messages sent to the state public health program for newborn screening (NBS) results of patients less than two months of age in 2017 were extracted from a local Health Information Exchange (HIE). Fields in the NBS data include medical record number (MRN), patient's first and last name, sex, telephone number, address (street address, city, state, and zip code), date of birth (year, month, and day), and next of kin's first and last name. More than 300 billion record pairs need to be formed to compare all records for deduplication. To reduce the number of comparisons, three blocking schemes were employed, where only record pairs with agreement on blocking variables were examined. In the first blocking scheme, day of birth, month of birth, and zip code (DB-MB-ZIP) were used as the blocking variables, resulting in a total of 8,083,396 record pairs. A random sample of 12,643 record pairs were selected and manually reviewed, of which 48.6% were found to be matches. In the second blocking scheme, a total of 2,644,454 record pairs were formed using the telephone number (TEL) as the blocking variable. Approximately 84% of the 4,138 manually reviewed record pairs were matches. The third blocking scheme utilized the next-of-kin's last name and first name (NKLN-NKFN) as blocking variables, which formed 1,217,736 record pairs. Among the 1,904 record pairs manually reviewed, 76.5% were true matches.

Approximately 49% of the record pairs in the DB-MB-ZIP blocking scheme agree on a total of 28,950 unique last names. In the TEL blocking scheme, 84% of the record pairs agree on a total of 27,122 unique last names, while in the in the NKLN-NKFN blocking scheme, 80% of the record pairs agree on 25,106 unique last names. Separately for each blocking scheme, we derive the last name frequency as the number of record pairs sharing a specific last name. Smith is the most frequent last name in all three blocking schemes and occurs for more than 1% of the record pairs with last name agreement. Other common last names include Johnson, Miller, Jones, Williams, and Brown. Using the manual review samples, density histograms of the last name frequencies are plotted for true matches and true non-matches (Figure 1, panels (a), (c), and (e)). It can be seen clearly that matches are more likely to have low-frequency names while non-matches are more likely to have high-frequency names in all blocking schemes, suggesting the potential benefit for considering frequency-based matching for last name.

Figure 1.

Density histograms of last name frequency by true match status among record pairs that agree on the last name and ROC curves with and without frequency-based agreement weight adjustment for the newborn screening data deduplication.

1.3. Overview of the paper

The purpose of this paper is two-fold: (i) to propose a weight adjustment method that overcomes the difficulty in estimating the error probabilities and the frequencies of agreed values that are true matches; (ii) to evaluate systematically the improvement in matching performance using the frequency-based method. The new method that we propose is based on the difference in frequency distributions of field values between true matches and true non-matches. These distributions are easily estimated using the FS model without the need for prior study or training data. The rest of the paper is organized as follows. The proposed frequency-based matching method is described in Section 2 and applied to a real-world example of deduplicating newborn screening data in Section 3. Section 4 presents a simulation study to evaluate the performance of the proposed method. This is followed by a brief discussion in Section 5.

2. Frequency-based record linkage

2.1. The Fellegi–Sunter model

Let $\mathit{y}=(y_1,y_2,\dots ,y_J)$ be the agreement vector of the J fields on a record pair, where $y_j=1$ if the two records agree on the $j^{th}$ field and 0 otherwise. Given the observed agreement pattern $\mathit{y}$, the contribution of the record pair to the likelihood is as follows:

$$P(\mathit{y})=P(\mathit{y}|M=1)P(M=1)+P(\mathit{y}|M=0)P(M=0),$$

where $P(M=1)$, denoted as π, is the prevalence of the true matches and $P(\mathit{y}|M)$ is the conditional distribution of the agreement vector $\mathit{y}$ given the true match status M. The decision to classify a record pair as match or non-match is then made based on the match probability

$$P(M=1|\mathit{y})=\frac{P(\mathit{y}|M=1)P(M=1)}{P(\mathit{y}|M=1)P(M=1)+P(\mathit{y}|M=0)P(M=0)},$$

or equivalently, the logarithm of the ratio of conditional probabilities $P(\mathit{y}|M=1)/P(\mathit{y}|M=0)$,

$$R=\log \frac{P(\mathit{y}|M=1)}{P(\mathit{y}|M=0)}.$$

The FS model assumes that the agreement patterns on multiple fields of the same record pair are independent conditional on the true match status. In other words, $P(\mathit{y}|M)=\prod _{j=1}^{J}P(y_j|M)$, where the conditional probabilities $P(y_j|M)$ define the m- and u-probabilities $m_j=P(y_j=1|M=1)$ and $u_j=P(y_j=1|M=0)$. Maximum likelihood (ML) estimates of the parameters, including the match prevalence π, the m-probabilities $m_1,m_2,\dots ,m_J$, and the u-probabilities $u_1,u_2,\dots ,u_J$, can be estimated using the Expectation–Maximization (EM) algorithm or other iterative optimization algorithms such as the Newton Raphson approach. Under the conditional independence assumption, the logarithm of the ratio of conditional probabilities, termed as the matching weight, can be easily computed by adding weights from individual fields:

$$R=\sum _{j=1}^J\{y_j\log \frac{m_j}{u_j}+(1-y_j)\log \frac{1-m_j}{1-u_j}\}.$$ (1)

Here ${\displaystyle \log \frac{m_j}{u_j}}$ is known as the agreement weight and ${\displaystyle \log \frac{1-m_j}{1-u_j}}$ as the disagreement weight for field j. The matching weights can then be used to classify record pairs as matches or non-matches.

2.2. Proposed frequency-based matching method

Assume two files A and B are being linked and the field j agrees on K distinct values $s_1,s_2,\dots ,s_K$. If the field j agrees on the $k^{th}$ value $s_k$, denoted as $G_{jk}$, the match probability is

$$\begin{array}{rl}& P(M=1|\mathit{y}:y_j=1,G_{jk})\\ & ={\displaystyle \frac{P(\mathit{y}:y_j=1,G_{jk}|M=1)P(M=1)}{P(\mathit{y}:y_j=1,G_{jk}|M=1)P(M=1)+P(\mathit{y}:y_j=1,G_{jk}|M=0)P(M=0)},}\end{array}$$

where

$$P(\mathit{y}:y_j=1,G_{jk}|M=1)=P(G_{jk}|\mathit{y}:y_j=1,M=1)P(\mathit{y}:y_j=1|M=1).$$

If the field j does not agree, the match probability remains the same:

$$P(M=1|\mathit{y}:y_j=0)=\frac{P(\mathit{y}:y_j=0|M=1)P(M=1)}{P(\mathit{y}:y_j=0|M=1)P(M=1)+P(\mathit{y}:y_j=0|M=0)P(M=0)}.$$

This implies that the matching weight is the same as in (1) if $y_j=0$. While $y_j=1$, the matching weight is adjusted as follows:

$$R=\log \frac{P(\mathit{y}:y_j=1|M=1)}{P(\mathit{y}:y_j=1|M=0)}+\log \frac{P(G_{jk}|\mathit{y}:y_j=1,M=1)}{P(G_{jk}|\mathit{y}:y_j=1,M=0)}.$$

The adjustment to the matching weight, ${\displaystyle \log \frac{P(G_{jk}|\mathit{y}:y_j=1,M=1)}{P(G_{jk}|\mathit{y}:y_j=1,M=0)}}$, depends on the difference in the frequency distributions of the agreed value $s_k$ between matches and non-matches, as well as the agreement pattern of other matching fields ${\mathit{y}}_{-j}$. We will simplify this by assuming the conditional independence of $G_{jk}$ and ${\mathit{y}}_{-j}$ given the true match status. This is a reasonable assumption since the distribution of the agreed values for one field conditional on the match status is unlikely related to whether another field agrees. Under this assumption, the adjustment to the matching weight is

$$\log \frac{P(G_{jk}|\mathit{y}:y_j=1,M=1)}{P(G_{jk}|\mathit{y}:y_j=1,M=0)}=\log \frac{P(G_{jk}|y_j=1,M=1)}{P(G_{jk}|y_j=1,M=0)}.$$

Denote $p_{1k}=P(G_{jk}|M=1,y_j=1)$ and $p_{0k}=P(G_{jk}|M=0,y_j=1)$. This is equivalent to adjusting the agreement weight for field j from ${\displaystyle \log \frac{m_j}{u_j}}$ to ${\displaystyle \log \frac{m_j}{u_j}+\log \frac{p_{1k}}{p_{0k}}}$. The conditional probabilities $p_{1k}$ and $p_{0k}$ describe the distributions of individual field values $s_k$ among true matches and among true non-matches. The frequency of each value $s_k$ is denoted as $f_k$ ( $f_k>0$), denoting the number of record pairs in $A\times B$ that agree on $s_k$. When blocking is employed to reduce the number of record pairs under consideration due to the extremely large number of record pairs in $A\times B$, these frequencies could be derived based on record pairs within the blocking scheme. Alternatively, if files A and B can be considered as random samples from a large population such as the U.S. census, these frequencies could be derived to represent the frequencies in the population.

A reasonable assumption for frequency-based matching is that field values with the same frequency have the same discriminating power. In other words, $p_{1k}=p_{1k^{\prime }}$ and $p_{0k}=p_{0k^{\prime }}$ when $f_k=f_{k^{\prime }}$. Therefore, we group field values with the same frequency into the same frequency level. For each frequency level l, we denote $q_{1l}$ and $q_{0l}$ as probabilities of agreeing on a value with a specific frequency level l among true matches and true non-matches. That is,

$$\begin{array}{lll}{q}_{1l}& =& P(\bigcup _{f_k=l}{G}_{jk}|M=1,y_j=1)=\sum _{k=1}^K{p}_{1k}I(f_k=l),\\ q_{0l}& =& P(\bigcup _{f_k=l}{G}_{jk}|M=0,y_j=1)=\sum _{k=1}^K{p}_{0k}I(f_k=l).\end{array}$$

The frequency-specific agreement weight is then defined as

$$\log \frac{m_j}{u_j}+\log \frac{q_{1l}}{q_{0l}}.$$ (2)

The frequency-specific adjustment to the agreement weight, ${\displaystyle \log \frac{q_{1l}}{q_{0l}}}$, is equal to value-specific weight adjustment ${\displaystyle \log \frac{p_{1k}}{p_{0k}}}$ for $s_k$'s with $f_k=l$.

In order to compute the frequency-specific weights (2), we need to estimate $q_{1l}$ and $q_{0l}$, which requires the information on the number of record pairs agreeing on values with frequency l that are true matches. We propose to estimate the number of true matches (and number of true non-matches) that agree on values with different frequency levels by weighting the record pairs using the match probability derived based on the FS model. Specifically, the match probability, the probability that a record pair i ( $i=1,2,\dots ,n$) is a true match given its binary agreement vector ${\mathit{y}}_i=(y_{i1},y_{i2},\dots ,y_{iJ})$, is:

$${\omega }_i=P(M=1|{\mathit{y}}_i)=\frac{\pi \prod _{j=1}^J{m}_j^{y_{ij}}(1-m_j{)}^{1-y_{ij}}}{\pi \prod _{j=1}^J{m}_j^{y_{ij}}(1-m_j{)}^{1-y_{ij}}+(1-\pi )\prod _{j=1}^J{u}_j^{y_{ij}}(1-u_j{)}^{1-y_{ij}}}.$$

The number of true match record pairs agreeing on a field value with frequency l, denoted as $g_{1l}$, and the number of true non-match record pairs agreeing on a field value with frequency l, denoted as $g_{0l}$, are estimated by probability weighting as follows:

$$\begin{array}{lll}{g}_{1l}& =& \sum _{i=1}^n{\omega }_{i}I(y_{ij=1},\text{field value frequency}=l),\\ g_{0l}& =& \sum _{i=1}^n(1-{\omega }_i)I(y_{ij=1},\text{field value frequency}=l).\end{array}$$

Note that $g_{1l}+g_{0l}=g_l$, the observed number of record pairs agreeing on a value with frequency l. Dividing $g_{1l}$ by the total number of matches among record pairs with field agreement, estimated by $\sum _{i=1}^n{\omega }_{i}I(y_{ij}=1)$, and dividing $g_{0l}$ by the total number of non-matches among record pairs with field agreement, estimated by $\sum _{i=1}^n(1-{\omega }_i)I(y_{ij}=1)$, we obtain the estimates of $q_{1l}$ and $q_{0l}$:

$$\begin{array}{lll}{q}_{1l}& =& {\displaystyle \frac{\sum _{i=1}^n{\omega }_{i}I(y_{ij=1},\text{field value frequency}=l)}{\sum _{i=1}^n{\omega }_{i}I(y_{ij=1})},}\\ q_{0l}& =& {\displaystyle \frac{\sum _{i=1}^n(1-{\omega }_i)I(y_{ij=1},\text{field value frequency}=l)}{\sum _{i=1}^n(1-{\omega }_i)I(y_{ij=1})}.}\end{array}$$

The adjustment to the agreement weight, ${\displaystyle \log \frac{q_{1l}}{q_{0l}}}$, can vary over a large range. Record pairs agreeing on field values that are more frequent among true matches than among true non-matches receive a positive adjustment of the agreement weight, leading to a greater likelihood to be a match. On the other hand, record pairs with agreement on field values that are less frequent among true matches than among true non-matches receives a negative adjustment of the agreement weight, resulting in a lower likelihood to be a match. For field values that are extremely rare among true non-matches ( $q_{0l}\approx 0$), the adjustment could potentially be extremely large, leading to the domination of the field over the importance of other fields in determining the match status. In addition, when field values are extremely rare among matches ( $q_{1l}\approx 0$), the negative adjustment could result in an agreement weight lower than the disagreement weight ${\displaystyle \log \frac{1-m_{jl}}{1-u_{jl}}}$. This is counterintuitive since record pairs with field agreement, regardless of how frequent the agreed value is, should be more likely to be a match than those with field disagreement. To ensure this, the adjustment needs to satisfy the following:

$$\log \frac{q_{1l}}{q_{0l}}\ge \log \frac{(1-m_j)/m_j}{(1-u_j)/u_j}.$$

To make sure that the magnitude of the adjustment to the agreement weight is not too large, we recommend that field value frequencies be categorized into a small number of levels. With categorization, the estimated $q_{1l}$ and $q_{0l}$ will move away from zero, making it less likely to have extreme values of the adjustment. Another benefit of categorizing the frequencies is that it could partially alleviate the adverse impact of typographical errors in the field values. Categorization could be performed using percentiles with 5-10 levels.

3. Newborn screening data deduplication

In this section, we will apply the proposed frequency-based matching method to the NBS data deduplication example. Matching weights are adjusted based on the last name frequency. We do not consider adjusting weights based on the first name frequency because of the large amount of missing data for newborns and the relatively high level of typographical errors in the data. Specifically, 41% of the HL7 messages extracted contain missing values in first name. In addition, large amount of missing data are seen for middle initial (54%) and telephone number (32%). In record linkage applications, missing data happen quite often due to the incompleteness of the data. In literature, missing values are often treated as disagreement [9,12]. However, this is not a reasonable assumption because it is possible for missing values to be agreement. Following [5,10,11], we consider the missing data as missing at random (MAR) and, therefore, utilize the full-information maximum likelihood method for the FS model estimation.

Fields available for matching include MRN, last name, first name, day and month of birth, telephone number, street address, and zip code. Year of birth is not considered as a matching field since all records have the same birth year. Street address and zip code are not used in the TEL blocking scheme since more than 95% of record pairs in this blocking scheme agree on these fields, resulting in little to no discriminating power. Usefulness of the last name frequency in adjusting match weights is investigated in each of the three blocking schemes separately. We consider two FS models in each blocking scheme: Model 1 includes all fields except blocking variables and Model 2 contains three selected fields with relatively poor discriminating power. These two models are used to demonstrate the impact of frequency-based weight adjustment on the match performance when fields with different levels of discriminating power are used. Parameter estimation of the FS models is performed using the SAS NLMIXED procedure, with results shown in Table 1. Frequency-based adjustment of matching weights is calculated after parameter estimates are obtained. SAS codes for fitting the FS model and implementing the frequency-based matching are available at http://huipxu.pages.iu.edu/publications.html.

Table 1.

Estimates of match prevalence and m- and u-probabilities of FS models for the newborn screening data deduplication.

		DB-MB-ZIP scheme		TEL scheme		NKLN-NKFN scheme
		Model 2	Model 1	Model 2	Model 1	Model 2
Prevalence		0.5495	0.5589	0.9195	0.9808	0.9238	0.9174
m-probabilities
	MRN	0.7701	–	0.8898	–	0.8507	–
	Last name	0.8808	0.8670	0.8622	0.8559	0.8258	0.8298
	First name	0.9414	–	0.9719	–	0.8901	–
	Sex	0.9733	0.9722	0.9995	–	0.9703	0.8298
	Middle initial	0.8139	–	0.8420	–	0.7524	–
	Day of birth	–	–	0.9992	0.9977	0.9974	–
	Month of birth	–	–	1.0000	1	0.9997	–
	Telephone	0.9238	0.9296	–	–	0.9438	0.9609
	Street address	0.9647	–	–	–	0.9594	–
	ZIP code	–	–	–	–	0.9824	–
u-probabilities
	MRN	0.0003	–	0.0522	–	0.0016	–
	Last name	0.0084	0.0075	0.6312	0.2159	0.5397	0.5173
	First name	0.0077	–	0.0182	–	0.0098	–
	Sex	0.5078	0.4993	0.5560	–	0.5132	0.5856
	Middle initial	0.0265	–	0.1023	–	0.0336	–
	Day of birth	–	–	0.7528	0.0384	0.0624	–
	Month of birth	–	–	0.7985	0.1505	0.1343	–
	Telephone	0.0118	0.0054	–	–	0.0395	0.0000
	Street address	0.0132	–	–	–	0.0344	–
	ZIP code	–	–	–	–	0.0729	–

Based on the quantiles of the frequency, last names are collapsed into six levels: the lowest 20%, 20–90% , 90–95% , 95–99% , 99–99.5%, and above 99.5%. With this categorization, we consider the top 0.5% of the last names as the most common names and the bottom 20% of the last names as the rarest names. Estimates of the conditional probabilities $q_{1l}$ and $q_{0l}$, representing the probabilities of grouped frequency levels in matches and non-matches, are shown in Table 2. Across all blocking schemes and all models, it can be seen clearly that truly matched record pairs are more likely to have low-frequency names and less likely to have high-frequency names, compared to the truly non-matched record pairs. For example, in the DB-MB-ZIP scheme, it is estimated based on Model 1 that 24% of truly matched record pairs with the same last name agree on names with the lowest two frequency levels, in comparison to only 11% of truly non-matched record pairs. On the other hand, 27% of truly matched record pairs and 37% of truly non-matched record pairs with the same last name agree on names with the highest frequency level. These differences result in a positive adjustment to agreement weights for low-frequency levels and a negative adjustment for high-frequency levels. Receiver operating characteristics (ROC) curves with and without frequency-based weight adjustments are shown in Figure 1 (panels (b), (d), and (f)). It can be seen that ROC curves with and without adjustments are mostly indistinguishable when the more complex and better performing Model 1 is used, resulting in comparable AUCs. However, weight adjustment based on Model 2 leads to improved matching accuracy, especially in the TEL and NKLN-NKFN blocking schemes. The improvement in AUC is 0.027 (from 0.806 to 0.835) for the TEL scheme and 0.013 (from 0.818 to 0.831) for the NKLN-NKFN scheme.

Table 2.

Frequency-based adjustment of last name agreement weights for the newborn screening data deduplication.

Frequency		DB-MB-ZIP scheme			TEL scheme			NKLN-NKFN scheme
Level		$q_1$	$q_0$	Adjustment	$q_1$	$q_0$	Adjustment	$q_1$	$q_0$	Adjustment
Model 1
	1: Low	0.0029	0.0003	2.1586	0.0065	0.0010	1.9046	0.0210	0.0004	3.8678
	2	0.2365	0.1130	0.7384	0.2676	0.1053	0.9329	0.2674	0.0216	2.5147
	3	0.1241	0.0923	0.2954	0.1258	0.0809	0.4413	0.1287	0.0262	1.5898
	4	0.2712	0.3069	−0.1236	0.2667	0.2759	−0.0338	0.2573	0.1143	0.8115
	5	0.0947	0.1139	−0.1841	0.0908	0.1258	−0.3265	0.0851	0.0712	0.1793
	6: High	0.2707	0.3736	−0.3223	0.2425	0.4110	−0.5277	0.2405	0.7662	−1.1587
Model 2
	1: Low	0.0029	0.0009	1.2241	0.0062	0.0025	0.9029	0.0207	0.0061	1.2164
	2	0.2363	0.1229	0.6537	0.2582	0.1856	0.3301	0.2623	0.1230	0.7572
	3	0.1241	0.0768	0.4796	0.1233	0.0840	0.3841	0.1260	0.0784	0.4744
	4	0.2718	0.2204	0.2094	0.2675	0.2240	0.1776	0.2534	0.1886	0.2956
	5	0.0949	0.0853	0.1074	0.0931	0.0519	0.5847	0.0846	0.0808	0.0470
	6: High	0.2699	0.4937	−0.6037	0.2517	0.4520	−0.5856	0.2530	0.5232	−0.7265

4. Simulation study

Results in the NBS data deduplication example show that frequency-based weight adjustment produces comparable or improved matching accuracy, with greater benefits for poorly discriminating models. In this section, we perform a simulation study to further evaluate when frequency-based matching provides improved accuracy. Binary field agreement data are simulated based on the FS model assuming conditional independence of agreement pattern across fields. These data can be thought as vector patterns in real applications and are used directly in FS models. Three scenarios are considered, each with six matching fields. Scenario I has fields with relatively poor discriminating power (m-probabilities are 0.75, 0.25, 0.65, 0.75, 0.78, 0.79, and u-probabilities are 0.6, 0.15, 0.4, 0.4, 0.65, 0.5), Scenario II has fields with moderate discriminating power (m-probabilities are 0.85, 0.35, 0.75, 0.85, 0.88, 0.89, and u-probabilities are 0.5, 0.05, 0.3, 0.3, 0.55, 0.4), and Scenario III has fields with high discriminating power (m-probabilities are 0.95, 0.45, 0.85, 0.95, 0.98, 0.99, and u-probabilities are 0.4, 0.05, 0.2, 0.2, 0.45, 0.3). For each scenario, we consider eight values of true match prevalence (0.01, 0.05, 0.1, 0.2, 0.5, 0.8, 0.9, and 0.95). These values are often seen in real-world record linkage applications with loose to tight blocking schemes.

The first field is considered to have frequencies with five levels. When the field agrees, the frequency level is generated based on the conditional probabilities $q_{1l}$ and $q_{0l}$ ( $l=1,2,\dots ,5$). We consider four sets of conditional distributions that imply small, moderate, large, and extremely large differences between true matches and true non-matches. The first set has $q_{1l}=0.25,0.2,0.2,0.2,0.15$ and $q_{0l}=0.15,0.2,0.2,0.2,0.25$ with a small standardized mean difference (SMD) of 0.29. The second set has $q_{1l}=0.3,0.2,0.2,0.2,0.1$ and $q_{0l}=0.1,0.2,0.2,0.2,0.3$ with a moderate SMD of 0.59. The third set has $q_{1l}=0.4,0.3,0.1,0.1,0.1$ and $q_{0l}=0.1,0.1,0.1,0.3,0.4$ with a large SMD of 1.2. The fourth set has $q_{1l}=0.5,0.3,0.1,0.05,0.05$ and $q_{0l}=0.05,0.05,0.1,0.3,0.5$. With a very large SMD of 2.08, this set represents extremely large difference between the two conditional distributions $q_{1l}$ and $q_{0l}$. These conditional distributions are informed by the real application of the NBS data deduplication, where the last name frequency distributions among true matches and true non-matches in the manual review samples have a small SMD of 0.37 in the DB-MB-ZIP scheme, a moderate SMD of 0.62 in the TEL scheme, and a large SMD of 0.93 in the NKLN-NKFN scheme. All four sets of conditional probabilities satisfy the common intuition that rare names occur more often among true matches than among true non-matches.

For each combination of the above settings, we generate 100 Monte Carlo data sets with 50,000,000 record pairs, where binary agreement patterns are simulated based on the FS model and frequency levels of the first field are based on the conditional distributions $q_{1l}$ and $q_{0l}$. We then fit the FS model to obtain the ML estimates of the parameters and derive the matching weights with and without weight adjustments. Comparing these matching weights to the true match status, we construct the ROC curves and calculate the AUCs. Boxplots of the unadjusted and adjusted AUCs when field frequency distributions have small, moderate, large, and extremely large differences for each of the three scenarios are shown in Figure 2. It can be seen that frequency-based weight adjustment produces comparable or improved match performance in all settings.

Figure 2.

Boxplots of AUCs without frequency-based weight adjustment (unadjusted) and with frequency-based weight adjustment (small, moderate, large, and extremely large difference) over 100 simulated data sets in Scenario I, II, and III with N = 50,000,000 record pairs in the simulation study.

Further comparison of the unadjusted and adjusted AUCs for each Monte Carlo dataset revealed that weight adjustment yields a higher AUCs in all settings, although the magnitude of the improvement varies. More prominent improvements are observed when the conditional distributions of field frequency have larger differences between true matches and true non-matches, when the match prevalence is 0.5, and when matching fields have relatively poor discriminating power as seen in Scenario I. Across the three scenarios and eight match prevalence values, the average improvement in AUC is 0.1% when there is a small difference and 0.4% when there is a moderate difference in frequency distributions between matches and non-matches. As the difference becomes larger, the average improvement in AUC increases to 1% (large difference) and 1.9% (extremely large difference). The largest improvement occurs when the true match prevalence is 0.5. Smaller improvements are observed as the true match prevalence moves closer to 0 or 1. For example, in Scenario I with poorly discriminating fields and extremely large difference in frequency distributions, weight adjustment results in an average 5.2% increase in AUC when the true match prevalence is 0.5. This decreases to 3.2% when the match prevalence is 0.05 or 0.95 and further decreases to 1.6% when the match prevalence is 0.01. The benefit of frequency-based weight adjustment is higher when poorly discriminating fields are used for matching and diminishes as the discriminating power increases. This is seen by the average 1.7% improvement in AUC across all Monte Carlo data sets in Scenario I with poorly discriminating fields, 0.8% improvement in Scenario II, but less than 0.1% in Scenario III with highly discriminating fields.

5. Discussion

The widely used FS model for probabilistic record linkage does not leverage information on the frequencies of field values and hence results in the same matching weights for record pairs agreeing on common values and those agreeing on rare values. Prior efforts in accounting for field value frequencies typically relied on knowledge of error probabilities and frequency distributions of field values, often derived from prior studies or from training data. When such information is not available, applications of these approaches become quite challenging. In this paper, we propose a two-step procedure for frequency-based matching to overcome these challenges. Our approach utilizes the FS model to estimate the frequency distributions of the field values among true matches and among true non-matches for record pairs with field agreement without the need of training data. Depending on the estimated difference in frequency distributions, the agreement weights are adjusted positively or negatively. Values that occur more often among true matches than among true non-matches receive a positive adjustment, leading to a larger matching weight, while those that occur more often among true non-matches than among true matches receive a negative adjustment, leading to a smaller matching weight. Our proposed approach can be easily implemented and produces comparable or better performance than the unadjusted method across many different settings. We therefore recommend our approach in record linkage applications.

Our study further shows that our proposed frequency-based matching provides greater improvement in matching accuracy when the match prevalence is 50% and when there is a large difference in the frequency distributions between true matches and true non-matches. In addition, substantial matching accuracy improvement is seen when matching fields have poor discriminating power. As the discriminating power of matching fields increases, the benefit in frequency-based matching diminishes. These findings highlight the importance of using highly discriminating fields for record linkage whenever possible. When such fields are not available, additional information in the data, such as frequencies of field values, should be incorporated to improve the accuracy of the matching algorithm.

The proposed frequency-based weight adjustment is implemented based on the standard FS model. It is known that the FS model with ML parameter estimation does not perform well when there is a lack of overlap between the two linked data files [16,18]. In our simulation study, the FS model produced reliable ML parameter estimates when the match prevalence is as low as 0.01. However, when the match prevalence further decreased to 0.001 or even lower, parameter estimates of the ML model displayed substantial variation, especially when poorly discriminating fields were used for matching. Effective blocking has been recognized as an essential step when there is a relatively low number of matches [5]. In situations with extremely low match prevalence due to the use of no blocking criteria or very loose blocking schemes, FS model parameters are usually estimated iteratively through manual review [16]. Although frequency-based weight adjustment can be similarly implemented after model parameters are estimated, there is little improvement in matching performance, seen by the negligible differences in AUC produced by weight adjustment in all three scenarios of the simulation study when the FS model parameters are set to equal to the true values (results not shown). This is consistent to our previous finding that weight adjustment is less beneficial as the match prevalence approaches 0 or 1.

The FS model is also known to have poor performance when the conditional independence assumption is invalid. Conditional dependence latent class models for record linkage have been investigated by many researchers [1,3,4,7,8,13–15,20]. Using these models, our proposed frequency-based weight adjustment method can be easily generalized. We will use the conditional dependence models to estimate the match probability and the conditional distributions of the field frequency levels, which are then similarly used to adjust the matching weights.

In addition, our weight adjustment method is based on the assumption that the frequency distribution of agreed values of one field is independent of the agreement status of other fields given the true match status. Although we have found no violation in our NBS data deduplication example (evaluated using the manual review samples), this assumption can be easily relaxed so that the weight adjustment depends on the agreement status of other matching fields. The probabilities $q_{1l}$ and $q_{0l}$ will be calculated among record pairs with a specific agreement pattern of other matching fields. However, due to the smaller number of record pairs involved in this calculation, we recommend categorizing the frequency levels into a smaller number of groups to avoid the issue of extreme adjustments.

One limitation of frequency-based matching methods, including the proposed method, is the assumption that typographical errors are low [18]. When the rate of typographical errors is high, many observed field values are invalid. For example, if a rare name such as Harazlak has five different spellings in the five records in which it appears, accurate frequency-based weight adjustment is not possible. Although categorization of field values with similar frequencies alleviates some of the adverse impact of typographical errors, reformatting or standardization of field values before linkage could be helpful in boosting the benefit of frequency-based matching.

Funding Statement

This project was supported from the Agency for Healthcare Research and Quality [grant number R01HS023808] and from the Patient-Centered Outcomes Research Institute [grant number ME-2017C1-6425]. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Agency for Healthcare Research and Quality or the Patient-Centered Outcomes Research Institute.

Disclosure statement

No potential conflict of interest was reported by the author(s).