Estimating racial and ethnic healthcare quality disparities using exploratory item response theory and latent class item response theory models

Abstract

Healthcare quality metrics refer to a variety of measures used to characterize what should have been done or not done for a patient or the health consequences of what was or was not done. When estimating healthcare quality, many metrics are measured and combined to provide an overall estimate either at the patient level or at higher levels, such as the provider organization or insurer. Racial and ethnic disparities are defined as the mean difference in quality between minorities and Whites not justified by underlying health conditions or patient preferences. Several statistical features of healthcare quality data have been ignored: quality is a theoretical construct not directly observable; quality metrics are measured on different scales or, if measured on the same scale, have different baseline rates; the construct may be multidimensional; and metrics are correlated within-individuals. Balancing health differences across race and ethnicity groups is challenging due to confounding. We provide an approach addressing these features, utilizing exploratory multidimensional item response theory (IRT) models and latent class IRT models to estimate quality, and optimization-based matching to adjust for confounding among the race and ethnicity groups. Quality metrics measured on 93,000 adults with schizophrenia residing in five US states illustrate approaches.

1. Introduction

The US Institute of Medicine defines the quality of healthcare as the degree to which healthcare services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge (Institute of Medicine, 1990). Numerous professional organizations have promulgated quality measures for a variety of uses, including improving care, selective contracting with healthcare providers, and reimbursement of services delivered (Kronick et al., 2000; Winkelman & Damler, 2008). Healthcare quality measurement is not confined to the U.S.—for instance, the UK (Minchin et al., 2018), Belgium (Gerkens et al., 2023), France (de Pouvourville, 1997), and Portugal (Ramalho et al., 2022) also measure and report quality. In the U.S., the National Committee for Quality Assurance routinely collects administrative billing data to create the healthcare Effectiveness Data and Information Set (HEDIS) indicators on behalf of State Medicaid programs and use these measures to describe the quality of publicly funded healthcare services (National Committee for Quality Assurance, 2010). Medicaid is a joint state and federal US program that provides health coverage to low-income adults, children, and people with disabilities. The program is large—in 2020, Medicaid included 77.8 million enrollees with total health expenditures of 666.3 billion dollars (Center for Medicaid & CHIP Services, 2023).

The types of quality metrics fall broadly into three categories: process measures that characterize whether a patient received recommended care, such as receiving a needed medication; outcome measures that characterize the outcomes of healthcare, such as health status or patient satisfaction; and structural measures that describe the healthcare provider’s capacity to deliver healthcare, such as the size of nursing staff.

Healthcare disparities refer to the unequal treatment of patients based on race or ethnicity, gender, geography, or other patient characteristics. The U.S. Institute of Medicine (2002) defines a racial and ethnic disparity as a difference in treatment provided to members of different racial and ethnic groups that is not justified by the underlying health conditions or treatment preferences of patients. This definition purposefully excludes adjustment for socioeconomic differences when measuring racial and ethnic disparities. Rather, socioeconomic differences are viewed as potential mediators of racial and ethnic disparities.

In this article, we assess the quality of care for individuals with schizophrenia, a disabling mental illness, associated with chronic medical comorbidities and premature mortality (Weinberger & Harrison, 2010). These chronic medical comorbidities also complicate treatment, compound disability, and increase healthcare utilization (Druss et al., 2000; Parks et al., 2006). In the U.S., Medicaid is a key payer for care received by individuals with schizophrenia because most individuals qualify for Medicaid due to the disability caused by the disease (Frank & Glied, 2006; Khaykin et al., 2010). County governments also play a prominent role in the healthcare of people with schizophrenia—state funds directed to counties and other local governments accounted for 24% of all mental health spending in the U.S. during 2015 (Substance Abuse & Mental Health Services Administration, 2019). County variation in treatment patterns for serious mental illness have been reported (Horvitz-Lennon et al., 2009) and may be related to the strength of the county’s Medicaid healthcare infrastructure, including availability of and accessibility to specialty care providers, such as psychiatrists (Cummings & Druss, 2013).

Table 1 summarizes quality information on 93,000 Medicaid beneficiaries aged between 18 and 64 diagnosed with schizophrenia in five US states between 2010 and 2013. Black, Latinx, and White beneficiaries comprise 44%, 15%, and 40%, respectively, of the cohort. Care for individuals is assessed over 12 months. Twenty-three mental health quality metrics are reported and cover pharmacological treatments, acute care services, and ambulatory care services. Each measure is recommended by professional societies or other healthcare entities such as the National Committee for Quality Assurance. All metrics are reported such that a higher percentage corresponds to a more desirable outcome with some exceptions. The exceptions provide denominators for other measures (e.g. items 7, 10). Some metrics are binary, some are ordinal, and some are nominal. While virtually all adults had some ambulatory care, <3/4 had quarterly visits. Some variation across race and ethnicity groups is apparent; for instance, quarterly ambulatory visits range from 58% for Black beneficiaries to 66% for White and Latinx beneficiaries.

Table 1.

Quality measures, overall and by race and ethnicity

Number of Persons	All 93,311	Black 41,297	Latinx 14,301	White 37,713
1. Any antipsychotic drug use	90	90	93	90
2. Any log-acting injectable antipsychotic drug	15	17	14	13
3. Any clozapine use	4.6	2.5	4.7	7.0
4. Antipsychotic adherencea ≥ 80%	49	39	51	59
5. Antipsychotic adherencea ≥ 50%	72	64	76	80
6. No antipsychotic polypharmacy	79	83	78	76
7. Any schizophrenia inpatient discharge	19	20	21	18
8. Follow-up within 7 days	24	23	26	24
9. Follow-up care 30 days	36	34	38	36
10. Any schizophrenia ED visit	3.6	4.4	2.3	3.1
11. Follow-up care within 7 days	37	35	38	39
12. Follow-up care within 30 days	47	44	51	51
13. Any ambulatory mental healthcare visit	99	98	98	99
14. Quarterly ambulatory MH care visits	63	58	66	66
No excessive acute care utilization for:
15. schizophrenia	95	94	95	95
16. mental health	91	91	91	91
17. Any schizophrenia hospitalization	20	20	22	19
18. No 30-day re-admission	77	77	76	77
19. Any MH hospitalization	23	23	24	22
20. No 30-day re-admission	74	74	73	73
21. Any MH psychosocial services	84	85	83	83
22. Quarterly psychosocial services	41	40	37	43
23. Any individual, group, or family psychotherapy	28	36	12	26
Mean number met (SD)	10.0 (2.14)	9.89 (2.21)	10.0 (2.02)	10.2 (2.08)

Note. Entries are percentages except the last row which reports the mean number (SD) of quality measures met. Ordinal metrics include antipsychotic adherence level (items 4 and 5), follow-up after discharge (items 7 + 8 + 9), and follow-up after an emergency department visit (items 10 + 11 + 12); nominal metrics include no re-admission following a schizophrenia hospitalization (items 17 + 18) and no re-admission following a mental health hospitalization (items 19 + 20). ED = emergency department, MH = mental health.

^aMeasured as a percentage of days covered.

The most common approach to estimate healthcare quality when multiple indicators are measured uses a compensatory scoring algorithm, either averaging or summing the numerical values assigned to the metrics for each individual, typically weighing all quality metrics equally (Higashi et al., 2007). Although less common, conjunctive scoring of the observed responses, which requires a passing performance on each quality metric, has also been used (Nolan & Berwick, 2006). To identify variables associated with quality, investigators have also fitted separate regression models to each quality indicator (Song et al., 2014), examining regression coefficients. Findings focused on the quality of care received by mentally ill individuals have typically reported results separately for each measure (Busch et al., 2008; Cully et al., 2008). We are aware of only two studies that provided a summary measure capturing multiple domains of care (Hepner et al., 2007; Kilbourne et al., 2010) but neither used a psychometric-based approach. Few approaches adopt a measurement framework for quality estimation, that is, viewing quality as a theoretical construct that is not directly observable. In a measurement framework, each quality metric provides observable information about the theoretical quality construct(s) and that construct needs to be inferred. The absence of a measurement approach may introduce bias into summary estimates of quality and quality disparities. Given the policy implications of erroneously describing the quality of care received by some patient subgroups as poor, the adoption of models commensurate with the data-generating process and the theoretical underpinnings of quality is critical.

Approaches to measuring constructs exist—there is a large psychometric literature for measure development. The text by Skrondal and Rabe-Hesketh (2004) and references therein focus on statistical issues as does the text by Baker (1994). Of particular relevance are approaches that utilize item response theory (IRT) modelling in which the metrics (or items) are discrete (Embretson & Reise, 2000). For measuring healthcare quality specifically, Landrum et al. (2000), Normand et al. (2007), Teixeira-Pinto and Normand (2008), and He et al. (2010) employ IRT models to assess hospital quality using aggregated patient data, while Agniel et al. (2020) use IRT models on aggregated ambulatory service measures to evaluate hospital systems. Landrum et al. (2003) modelled clustered mixed (continuous and discrete) patient-level quality data using latent variables to connect outcomes; Daniels and Normand (2006) proposed methods for clustered longitudinal mixed patient-level quality data. However, both approaches relied on parametric modelling to risk-adjust for underlying health differences.

The goal of this paper is to derive an approach for the estimation of race and ethnicity healthcare quality disparities for categorical data, possibly having a multidimensional structure and requiring risk adjustment.

2. Approaches

Let $Y_{ij}$ denote the observed outcome associated with quality metric (or item) $i$ for individual $j$. Response options for the items can be binary or ordinal such that larger values correspond to better quality, or nominal where there is no natural ordering. We assume there are $I$ items and the total number of individuals is $N$. Not all metrics will apply to an individual; for instance, follow-up within 30 days of a hospital admission is only applicable to those discharged alive from a hospital. We let $n_j$ denote the number of measures for which individual $j$ is eligible.

2.1. Common approaches

The most frequently used estimator of quality received by individual $j$ is

$$\begin{array}{c}{Z}_j=\frac{{\bar{Y}}_{·j}-{\bar{Y}}_{..}}{s}\text{where}{\bar{Y}}_{·j}=\frac{1}{n_j}\sum _{i=1}^{n_j}{Y}_{ij},\\ {\bar{Y}}_{..}=\frac{1}{N}\sum _{j=1}^N{\bar{Y}}_{·j},\text{and}s=\sqrt{\frac{\sum _{j=1}^N{\left({\bar{Y}}_{·j}-{\bar{Y}}_{..}\right)}^2}{N}.}\end{array}$$ (1)

When individuals are eligible for different numbers of metrics, ${\bar{Y}}_{·j}$ has been referred to as an opportunity score (Anderson et al., 2016) because it reflects the number of opportunities available to a healthcare provider to deliver guideline-concordant or recommended care. Using these approaches, the minority-White quality disparity estimator is

$$\frac{1}{\sum _j{x}_j}\sum _{j=1}^N{Z}_j{x}_j-\frac{1}{N-\sum _{j=1}^N{x}_j}\sum _j{Z}_j\left(1-x_j\right)$$ (2)

where $x_j=1$ if the individual is a minority individual and 0 if the individual is White individual. To adjust for differences in health status across groups, the disparity is estimated using a linear regression model, e.g.

$$Z_j={\mathbf{w}}_j{\mathit{\gamma }}_r+{\mathbf{v}}_j{\mathit{\gamma }}_b+{ϵ}_j$$ (3)

where ${\mathbf{w}}_j$ is a $1\times q$ vector of race and ethnicity indicators, ${\mathit{\gamma }}_r$ is a $q\times 1$ coefficient vector that characterizes quality disparities, ${\mathbf{v}}_j$ is a $1\times p$ vector of health status variables, ${\mathit{\gamma }}_b$ is a $p\times 1$ coefficient vector meant to balance health status differences among race and ethnicity groups, and ${ϵ}_j\sim N\left(0,{\sigma }^2\right)$.

While this approach is simple to interpret and easy to calculate, there are several problems. The approach assumes all metrics use the same response scale which is often unlikely. The continuous or categorical responses could be converted to binary responses. Even when the metrics have the same response options, the underlying frequency of each metric may differ. The optimal (in terms of variance) algorithm for pooling information depends upon the type of measurement error present, which in turn depends on the underlying frequency of the metric. Moreover, researchers assume that the $\mathrm{V}\mathrm{a}\mathrm{r}\left(Z_j\right)$ is constant across individuals and that the items used to construct $Z_j$ are independent within an individual. Both assumptions are incorrect and conflict with the conceptual model of healthcare quality. Pooling algorithms are not clearly defined if some items are missing. Although multiple imputation could be used to fill in missing values at the metric level, typically single-based imputation, such as using the mean response, is used. If separate regression models are estimated, the same methodological issues remain, global tests of overall race and ethnicity effects are not possible, and missing data complicates inferences (Daskalakis et al., 2002; Yoon et al., 2011). Finally, adjusting for baseline differences in health status using a parametric model may fail to appropriately balance observed differences. A unified multivariate model avoids many of these issues.

Our point of departure treats the observed quality metrics as manifestations of a single or a few latent variables, and conditional on the latent variables, assumes that the observed outcomes are independent. The latent variables characterize the relationship among the observed quality metrics within an individual. Using algorithms from the causal inference literature, a model-free matching approach balances observed health status differences across groups and a structural model estimates racial and ethnic disparities in the matched sample. Throughout we use trait, dimension, and construct interchangeably.

2.2. IRT model

An IRT model connects categorical responses to item-specific parameters and a latent trait (or traits). Assuming a dichotomous item and a single latent variable, a commonly used model is the 2-parameter logistic (2PL) model defined as

$$P_i\left({\theta }_j\right)=P\left(Y_{ij}=1\mid {\theta }_j\right)=\frac{1}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left(-a_i\left({\theta }_j-b_i\right)\right)}$$ (4)

where ${\theta }_j$ is the underlying latent trait (quality in our setting) for individual $j,a_i$ is an item discrimination parameter, and $b_i$ is an item difficulty parameter. While we use a logit link, a probit link would work as well. Assumptions about ${\theta }_j$ are discussed later. The larger the value of $a_i$, the more discriminating item $i$ is as it reflects the rate of change in the probability of receiving guideline-concordant care to changes in the latent trait. The item difficulty parameter, $b_i$, is the point on the latent trait continuum in which an individual has a 0.50 probability of receiving recommended care. In our setting, the difficulty parameter measures how difficult it is to provide or to receive guideline-concordant care. If $a_1=a_2=\cdots a_I=1$, then the IRT model has a random intercept only (referred to as a Rasch model), and ${\bar{Y}}_{·j}$ is a sufficient statistic for ${\theta }_j$ conditional on $b_i$. Only when the item discrimination parameters are equal is an unweighted average (or sum) of the binary metrics (e.g. equation 1) suitable for estimating the latent construct. If some of the discrimination parameters differ across items, then the pattern of outcomes, rather than the total or average outcome, is important.

Without loss of generality, we reparameterize the 2-parameter logistic IRT model as

$$P_i\left({\theta }_j\right)=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left(a_i{\theta }_j+d_i\right)}{1+\mathrm{e}\mathrm{x}\mathrm{p}\left(a_i{\theta }_j+d_i\right)}.$$ (5)

In this parameterization, $d_i$ is referred to as item easiness so that $b_i=-d_i/a_i$. For quality measures having polytomous responses, a nominal IRT model is given by

$$P_{ik}\left({\theta }_j\right)=P\left(Y_{ij}=k\mid {\theta }_j\right)=\frac{\mathrm{e}\mathrm{x}\mathrm{p}(a_i^k{\theta }_j+d_i^k)}{\sum _{p=0}^{K-1}\mathrm{e}\mathrm{x}\mathrm{p}\left(a_i^p{\theta }_j+d_i^p\right)}\text{with}{a}_i^0=d_i^0=0$$ (6)

where quality metric $i$ has response options $k=0,1,\dots ,K_i-1$. Here, the discrimination and easiness parameters vary not only by quality metric but also by the response option associated with the metric. The point on the latent trait scale where an individual is equally likely to have response options to adjacent categories is

$$\frac{d_i^{k-1}-d_i^k}{a_i^k-a_i^{k-1}}.$$

For ordinal measures, we consider the graded IRT model,

$$P\left(Y_{ij}\ge 0\mid {\theta }_j\right)=1$$

$$P\left(Y_{ij}\ge k\mid {\theta }_j\right)=\frac{\mathrm{e}\mathrm{x}\mathrm{p}(a_i{\theta }_j+d_i^k)}{1+\mathrm{e}\mathrm{x}\mathrm{p}(a_i{\theta }_j+d_i^k)};k=1,\dots ,K_i-1$$

$$P\left(Y_{ij}\ge K_i\mid {\theta }_j\right)=0\text{so}\text{that}$$

$$P\left(Y_{ij}=k\mid {\theta }_j\right)=P\left(Y_{ij}\ge k\mid {\theta }_j\right)-P\left(Y_{ij}\ge k+1\mid {\theta }_j\right).$$

Examination of the item easiness parameters provides some insight into the appropriateness of the ordinal assumption. Other IRT models are available that vary by the type of restrictions placed on the $d_i^k$ but we do not consider these in this manuscript.

2.2.1. Assumptions

Several assumptions are made when using IRT models.

Assumption 1 (local independence).

Responses within individuals are independent conditional on the latent variable, ${\theta }_j$

$$P\left({\mathbf{Y}}_j={\mathbf{y}}_j\mid \mathbf{a},\mathbf{d},{\theta }_j\right)=\prod _{i=1}^{n_j}P\left(Y_{ij}=y_{ij}\mid a_i,d_i,{\theta }_j\right)$$

where $y_{ij}$ is the observed response, ${\mathbf{y}}_j=\{y_{1j},y_{2j},\dots ,y_{n_{j}j}\},\mathbf{a}=\left\{a_1,a_2,\dots ,a_I\right\}$, and $\mathbf{d}=\left\{d_1,d_2,\dots ,d_I\right\}$. This assumption is common in random effects models and could be relaxed by permitting conditional associations among sets of items.

Assumption 2 (monotonicity).

Increasing ${\theta }_j$ implies that the probability of receiving care consistent with the guideline does not decrease. With $a_i>0$, Assumption 2 implies that the probability distribution of $Y_{ij}$ moves toward higher values as ${\theta }_j$ increases.

Assumption 3 (latent trait distribution).

Some assumptions about the distribution of latent traits must be made to provide a metric for the trait scale. The most common assumption is that of Normal latent traits.

3a: Normal Latent Trait:

${\theta }_j\stackrel{iid}{\sim }N(\mu ,\mathrm{\Sigma })$, $j=1,\dots ,N$. This is likely the most familiar latent trait model, and under this assumption, the 2-PL and graded IRT model result. Other assumptions about the latent trait can be adopted, such as mixtures of Normal distributions, log-linear, or kernel smoothing. We also assume a discrete distribution.

3b: Latent Class (Goodman (1974)):

${\theta }_j,j=1,2,\dots ,N$ are iid with $C$ support points ${\xi }_1,{\xi }_2,\dots ,{\xi }_C$ with corresponding probabilities ${\pi }_1,{\pi }_2,\dots ,{\pi }_C$ having

$${\pi }_{cj}=p\left({\theta }_j={\xi }_c\right),\text{such}\text{that}\sum _{c=1}^C{\pi }_{cj}=1.$$

The ${\xi }_c\mathrm{s}$ are locations that depend on the class. In the latent class model, individuals belong to one of $C$ classes and class membership is unknown. Within each class, individuals are homogeneous with respect to the underlying construct. The model is semi-parametric and thus offers some robustness protection. Classes are categories of a latent variable so that the prior probability that beneficiary $j$ belongs to class $c,{\pi }_{cj}$, is a model parameter.

2.2.2. Multidimensionality

A unidimensional model assumes that the variation in each quality metric is only affected by the variation in a single construct and not in any other related construct. In our setting, quality can be multidimensional given that metrics cover a wide range of healthcare services: pharmacological treatments, acute care services, and ambulatory care services. There are various modelling options, each depending on what is hypothesized about quality. Suppose that there are $s=1,2,...,S$ constructs. A multidimensional 2-PL (M2PL) is given by

$$P_i\left({\mathit{\theta }}_j\right)=\frac{\mathrm{e}\mathrm{x}\mathrm{p}(\sum _{s=1}^S{a}_{is}{\theta }_j^s+d_i)}{1+\mathrm{e}\mathrm{x}\mathrm{p}(\sum _{s=1}^S{a}_{is}{\theta }_j^s+d_i)}$$ (7)

where ${\mathit{\theta }}_j={({\theta }_j^1,{\theta }_j^2,\dots ,{\theta }_j^S)}^{\mathrm{\top }}$ is an $S\times 1$ vector of quality constructs, ${\mathbf{a}}_i={\left(a_{i1},a_{i2},\dots ,a_{iS}\right)}^{\mathrm{\top }}$ is an $S\times 1$ vector of discrimination parameters, and $d_i$ is an easiness parameter for item $i$. This model assumes that the variance in each quality metric is a weighted function of up to $S$ common traits so that quality is what the primary traits have in common. In the multidimensional setting, the a parameters indicate how well an item discriminates but this depends on the direction that is being measured in the $\theta$-space. Model (7) is referred to as a compensatory multidimensional IRT model because it permits individuals’ higher placement on one trait to overcome low positions on another trait in the probability of receiving guideline-concordant care associated with a metric. Partially compensatory multidimensional IRT models that do not have this property exist but will not be discussed further. The interested reader should consult Reckase (2009).

A multidimensional latent class IRT model assumes that each item is related to only one latent trait in the set of latent traits. This assumption is not imposed by the multidimensional IRT models described above. Let ${ℐ}_s$ denote disjoint subsets of items, $i=1,...,I$ describing the items relating to trait $s$ and let

$${\delta }_{is}=\left\{\begin{array}{ll}1& \text{if}i\in {ℐ}_s\\ 0& \text{otherwise.}\end{array}\right.$$

Then for binary items, for instance, the multidimensional latent class IRT model takes the form

$$P_i\left({\mathit{\theta }}_j\right)=\frac{1}{1+\mathrm{e}\mathrm{x}\mathrm{p}(-a_i(\sum _{s=1}^S{\delta }_{is}{\theta }_j^S-b_i))}.$$ (8)

Assumptions.

The assumptions in the multidimensional setting for the models considered in this paper are similar to those made in the unidimensional setting.

Assumption M1: (local independence).

Responses within individuals are independent conditional on ${\mathit{\theta }}_j$.

Assumption M2: (monotonicity).

The probability of the response occurring increases as any element in ${\mathit{\theta }}_j$ increases.

Assumption M3: (latent trait distribution).

For Normal latent traits, ${\mathit{\theta }}_j\sim N_S(\mathit{\mu },\mathrm{\Sigma })$ for $j=1,\dots ,N$. For the discrete setting, ${\mathit{\theta }}_j$ has a discrete distribution with support points ${\mathit{\xi }}_1,{\mathit{\xi }}_2,\dots ,{\mathit{\xi }}_C$ and corresponding probabilities ${\pi }_1,{\pi }_2,\dots ,{\pi }_C$ such that

$${\pi }_{cj}=p\left({\mathit{\theta }}_j={\mathit{\xi }}_c\right),\text{such}\text{that}\sum _{c=1}^C{\pi }_{cj}=1.$$

2.2.3. Including covariates

Quality disparities.

Rather than assuming that all the latent variables have the same distribution, covariates may be introduced. In our setting, summaries of ${\theta }_j$ by race and ethnicity groups are examined to detect quality disparities. More formal associations are estimated using a structural regression model. In the Normal latent trait model, the latent trait is linked to covariates using

$${\theta }_j={\mathbf{w}}_j\mathit{\gamma }+{ϵ}_j$$ (9)

where ${\mathbf{w}}_j$ is a $1\times q$ vector of dummy variables indicating if the individual is from a Black or Latinx race and ethnicity group, and ${ϵ}_j$ a vector of errors from a $N(0,1)$ distribution. Negative values of vector components $\mathit{\gamma }$ correspond to a lower quality for minorities compared to White individuals. If ${\mathit{\theta }}_j$ is multidimensional, then a multivariate model can be estimated, allowing the relationship of ${\mathbf{w}}_j$ with each trait ${\theta }_j^s$ to differ.

In contrast, in the latent class model, the class probabilities, ${\pi }_{cj}=p\left({\theta }_j={\xi }_c\mid {\mathbf{w}}_j\right)$, are modelled. For instance, a multinomial logit model assumes

$$\begin{gathered} {\pi }_{cj}=\frac{\mathrm{e}\mathrm{x}\mathrm{p}\left({\mathbf{w}}_j{\mathit{\gamma }}_c\right)}{1+\sum _{c^{\mathrm{*}}=2}^C\mathrm{e}\mathrm{x}\mathrm{p}({\mathbf{w}}_j{\mathit{\gamma }}_{c^{\mathrm{*}}})}c=2,\dots ,C \\ {\pi }_{1j}=\frac{1}{1+\sum _{c^{\mathrm{*}}=2}^C\mathrm{e}\mathrm{x}\mathrm{p}({\mathbf{w}}_j{\mathit{\gamma }}_{c^{\mathrm{*}}})} \end{gathered}$$ (10)

which links covariates to the probability of class $c$ membership. Alternatively, a cumulative logit model imposes ordering constraints on intercepts within ${\mathit{\gamma }}_c$ while forcing the other components of ${\mathit{\gamma }}_c$ to be the same across all classes (Bartolucci et al., 2016). Without loss of generality, order ${\xi }_1,{\xi }_2,\dots ,{\xi }_C$, , and

$$\begin{gathered} \frac{\sum _{c>c}{\pi }_{cj}}{\sum _{c\le c}{\pi }_{cj}},c=1,2,\dots ,C-1 \\ \text{where}\mathrm{l}\mathrm{o}\mathrm{g}\left(\frac{\sum _{c>c}{\pi }_{cj}}{\sum _{c\le c}{\pi }_{cj}}\right)={\varphi }_c+{\mathbf{w}}_j^{\mathrm{*}}{\mathit{\gamma }}^{\mathrm{*}} \end{gathered}$$ (11)

such that ${\varphi }_1<{\varphi }_2<\cdots <{\varphi }_{C-1}$, with ${\mathbf{w}}_j^{\mathrm{*}}$ and ${\mathit{\gamma }}^{\mathrm{*}}$ having dimension $1\times \mathit{q}-1$. If ${\mathit{\theta }}_j$ is multidimensional, the multinomial logit model is adopted.

Adjusting for health status.

Health differences are likely between race and ethnicity groups and balancing these differences can be challenging. To avoid misspecification of the regression model and to strengthen the balance of patient health status between race and ethnicity groups, approaches other than regression could be adopted. In template matching (Bennett et al., 2020), a simple random sample of individuals is taken to create a template that reflects the target population of interest. Within each race and ethnicity group, $r$, individuals are matched to the characteristics in the template using a constrained optimization procedure. The goal is to

$$\underset{c,z}{\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{z}\mathrm{e}}\sum _v\sum _p{c}_{v,p}\text{such}\text{that}\left|\sum _{s=1}^{N_r}{z}_s-N_{v,p}\right|\le c_{v,p}\text{and}\sum _{s=1}^{N_r}{z}_s=T\forall v\in 𝒱$$

where $c_{v,p}$ is a measure of imbalance for category $p$ of covariate $v;z_s\in \{0,1\}$ is a binary indicator assuming a value of 1 if the sth observation in a racial and ethnic group $r$ having sample size $N_r$ is matched to the template sample and 0 otherwise; $N_{v,p}$ is the number of observations in the template sample having category $p$ for the $v$th covariate; $T$ is the size of the template, and all covariates are discretized. This approach is attractive because it is model-free and specific constraints can be selected. Using the resulting matched sample, the simpler structural model described in equations (9) and (10) provides estimates of racial and ethnic disparities. Regression estimates based on the matched template reduce the dependence of the validity of the estimated disparities on the correct specification of the model.

2.2.4. Identifiability

Identifiability of the model is generally approached in one of two ways: (1) The diagonal elements of $\mathrm{\Sigma }$ are set to 1 and $\mu$ is equated to 0. If a structural regression model is estimated, this implies $\mu =\sum _j{\mathbf{w}}_j\mathit{\gamma }=0$. Otherwise, (2) for each latent trait, one discriminating parameter is set to 1 and one difficulty parameter is set to 0. Other constraints are also necessary; for unidimensional models, at least three items are needed. For multidimensional traits, an orthogonal rotation is used to address rotational indeterminancies. For a comprehensive discussion of identification in latent variable models from an analytical perspective, see Skrondal and Rabe-Hesketh (2004) [Chapter 5]. Reckase (2009) provides guidance on rotations to facilitate interpretation of multidimensional Normal latent trait structure once estimates have been obtained.

2.2.5. Inference

Assuming the traits are independent and identically distributed (possibly within subsets defined by covariates $\mathbf{w}$) from a population with a Normal distribution, the marginal-likelihood function for the data is

$$L(\mathrm{\Theta },\mathbf{a},\mathbf{d},\mathrm{\Sigma },\mathit{\gamma }\mid \mathbf{y})=\prod _{j=1}^N\left\{{\int }_{\theta }\prod _{i=1}^{n_j}P\left(Y_{ij}=y_{ij}\mid {\mathbf{a}}_i,{\mathbf{d}}_i,{\theta }_j\right)N\left({\theta }_j\mid \mathit{\gamma },\mathrm{\Sigma }\right)d\theta \right\}$$ (12)

where $\mathrm{\Theta }=\left\{{\theta }_1,{\theta }_2,\dots ,{\theta }_N\right\}$. Estimation of model parameters uses the expectation–maximization (EM) algorithm via Gaussian–Hermite quadrature for each unique response vector. In models with >3 traits, Quasi-Monte Carlo EM (QMCEM) estimation methods are implemented (Jank, 2005). The QMCEM algorithm is a stochastic version of the EM algorithm that replaces the E-Step by a Monte Carlo approximation and uses a sequence of points rather than purely random sampling.

For the latent class model, the marginal likelihood is a mixture distribution

$$L(\mathrm{\Theta },\mathit{\xi },\mathbf{a},\mathbf{d},\mathit{\pi },\mathit{\gamma }\mid \mathbf{y})=\prod _{j=1}^N\left\{\sum _{c=1}^{\mathrm{C}}{\pi }_{cj}(\mathit{\gamma })\prod _{i=1}^{n_j}P\left(Y_{ij}=y_{ij}\mid {\theta }_j={\xi }_c,\mathbf{a},\mathbf{d}\right)\right\}$$ (13)

where ${\pi }_c\left(\gamma \right)$ denotes that the probability depends on a regression function. As with the Normal latent trait model, the EM algorithm is used for estimation.

For multidimensional models, a multivariate Normal distribution is used in equation (12). For the latent class model, equation (13) becomes

$$\prod _{j=1}^N\left\{\sum _{c=1}^C{\pi }_{cj}(\mathit{\gamma })\prod _{s=1}^S\prod _{i\in {ℐ}_s}P\left(Y_{ij}=y_{ij}\mid {\theta }_j={\xi }_c,\mathbf{a},\mathbf{d}\right)\right\}.$$ (14)

To avoid multimodality of the likelihood in latent class models, the EM algorithm is initialized by a deterministic rule and by a multistart strategy based on random starting values: the class weights are drawn from a continuous uniform distribution between 0 and 1, normalized to sum up to 1; the other parameters are generated from independent standard normal distributions.

2.2.6. Distribution of the latent traits or class probabilities

The distribution of latent traits provides the key information on quality of healthcare care for our application. For Normal latent traits, the mean

$$E(\theta \mid \mathbf{y})=\int \theta P(\theta \mid \mathbf{y})\mathrm{d}\theta$$ (15)

for each individual provides information regarding where, on the quality scale, an individual resides. Higher mean values support receipt of guideline-concordant care. For the latent class models, we can assign individuals to a single latent class $h\mathrm{*}$ using

$$h\mathrm{*}=\underset{c=1,\dots C}{\mathrm{m}\mathrm{a}\mathrm{x}}p\left({\theta }_j={\xi }_c\mid {\mathbf{y}}_j\right)$$ (16)

and obtain a distribution of quality for each enrollee in the sample. Within each class, the race and ethnicity distribution can be summarized.

2.2.7. Model fit

Normal latent trait IRT model.

Dimensionality and general model fit are assessed using the Aikaike Information Criteria (AIC) and the Bayesian Information Criteria (BIC) statistics by estimating models of increased complexity, and selecting models with the lowest statistic. Overall model fit is also assessed using the root mean squared error of approximation (RMSEA) proposed by Maydeu-Olivares and Joe (2014) with values of <0.05 desirable. The normality of the latent traits is assessed through quantile–quantile plots. Local independence is examined by correlating the residuals (deviation of the observed and model-predicted item scores) for pairs of items with values near zero desirable.

Latent class IRT model.

In the latent class models, the number of traits and item allocation to each trait are required. Factor analysis or hierarchical clustering (Bartolucci, 2007) can be used to determine these. The number of classes is determined using AIC and BIC; with a fixed number of classes, the number of dimensions can be tested using BIC or the likelihood ratio test (Bartolucci et al., 2014). Local independence may be assessed by fitting extended IRT models that relax the monotonicity assumption. For instance, a model that included a set of association parameters for specific pairs of responses after conditioning on the latent trait could be tested against the standard model that assumes standard local independence.

2.2.8. Validation

If an outcome or item is available, the model is validated by correlating the predicted item scores or predicted class assignments with the outcome. If the relationships are in the expected directions, then model validation is supported.

3. Application

Using healthcare billing data, we identified all Medicaid beneficiaries aged 18–64 years of age residing in one of five US states (California, Georgia, Michigan, Mississippi, New Jersey) having (1) at least two outpatient visits from two different dates during 12 months with a primary or secondary diagnosis of schizophrenia, or at least one inpatient discharge with a primary schizophrenia diagnosis; (2) continuous enrollment 6 months before their qualifying schizophrenia diagnosis; and (3) continuous enrollment in Medicaid for 9 out of 12 months of enrollment, with no more than a month’s gap after the qualifying diagnosis. These states were selected to ensure the inclusion of sufficient Black and Latinx enrollees. We refer to the first date of the qualifying schizophrenia diagnosis as the index date and quality of care was assessed in the 12 months following the index date.

Columns 1 through 4 in Table 2 summarize the cohort, overall, and by race and ethnicity group. Approximately half of the cohort lived in California, had a mean age of 42 years, a quarter had other serious psychiatric comorbidities, 1 in 10 had substance use disorder comorbidity, and close to half had cardiometabolic morbidity. Most individuals received supplemental security income due to a physical or mental health disability which we use to characterize their health status along with other comorbidities. As expected, race and ethnicity distributions varied by state, with many Latinx enrollees residing in California. Some differences in comorbidities between the races and ethnic groups are evident, such as cardiometabolic comorbidity and cognitive comorbidity.

Table 2.

Cohort characteristics, by race and ethnicity group

Characteristic	(1) All	(2) Black	(3) Latinx	(4) White	(5) Template
Number (%)	93,311 (100)	41,297 (44)	14,301 (15)	37,713 (40)	24,000 (26)
Female	46	47	41	47	46
Mean age, (SD)	42 (13)	42 (13)	37 (13)	43 (13)	42 (13)
Index year
2010	52	50	52	54	53
2011	20	21	19	20	20
2012	15	16	15	13	14
2013	13	14	14	13	13
State of residence
California	54	38	91	58	62
Georgia	13	21	1.0	9.5	11
Michigan	18	22	2.1	21	14
Mississippi	5.7	9.8	0.15	3.5	4.6
New Jersey	8.8	9.7	6.0	8.9	8.4
SSI	91	92	89	92	92
aComorbidity
Cognitive	3.9	3.7	2.5	4.7	3.9
bPsychiatric disorders	25	24	26	37	25
Substance use disorder	11	11	10	11	11
Diabetes	10	11	9.9	9.0	10
Cardiometabolic	44	45	38	44	43
Other medical	21	20	17	24	21
aSchizophrenia inpatient days
0	85	85	84	86	85
1 day	11	11	12	10	11
≥2 days	4.0	4.0	4.4	3.8	3.9
aSchizophrenia ED visits
0	98	97	98	98	98
1 visit	1.6	1.8	1.3	1.4	1.6
≥2 visits	0.7	0.9	0.4	0.5	0.6

Note. ED = emergency department, SSI = supplemental security income.

^aMeasured in the 6 months before the index observation. All entries are percentages except for age.

^bIncludes affective disorders, anxiety disorders, obsessive-compulsive disorders, post-traumatic stress disorders, and other psychoses.

We combined the response options for the measures in Table 1 that occur in <2% of the cohort with the closest response option and, if not possible, eliminated the metric from further consideration. Items that are conditional on another item, e.g. responses on one affect the responses on others, were combined to help with the monotonicity assumption. While some indicators have ordinal responses, nominal response options were also used. We excluded quarterly receipt of psychosocial services (item 22) from all models to use as a validation measure. To adjust for health status, we created a template sample by randomly sampling 24,000 enrollees to match 8,000 enrollees per race and ethnicity group to the template. This number of observations should be sufficient to estimate multidimensional IRT models. Using the cardmatch function in the designmatch R package, we matched individuals from each race and ethnicity group to the template by constraining the means of the covariates to be equal across race and ethnicity groups. The balance of observed confounders was examined by comparing the distribution of the covariates across race and ethnicity covariates. We used the covariates presented in Table 2 except for the state of residence because the state may be on the causal pathway, as it reflects the availability of social and healthcare resources. Year was included to capture cohort effects and age was categorized into quartiles.

3.1. Exploratory normal latent trait IRT model

We initially estimated 2-parameter logistic, ordinal, and nominal response IRT models according to the original item response type, assuming between 1 and 3 traits using the mirt function in the mirt R package (Chalmers, 2012), with $\mathit{\theta }\sim {\mathit{N}}_S(\mathit{\mu },\mathrm{\Sigma })$, setting the diagonal elements of $\mathrm{\Sigma }$ to 1 and $\mathit{\mu }=0$. We determined the nature of each trait by examining the relative loadings across items. We used an orthogonal rotation, varimax, constrained the $\mathbf{a}$ parameters to be positive, and considered loadings >|0.20 meaningful. Local independence was assessed by examining pairwise item residual correlations. Items were eliminated if the fit was poor and models re-estimated.

3.2. Exploratory latent class IRT model

Latent class models were estimated using the MultiLCIRT package in R (Bartolucci et al., 2014); the EM algorithm was initialized using the recommended strategies of a deterministic rule and again using random starting values. We first determined the number of classes, examining AIC and BIC statistics, fitting models with 1 through 5 classes. Next, we determined if unidimensionality held by comparing values of BIC for unidimensionality to BIC values for higher dimensions, fixing the number of classes at the value selected in the first step. The number of traits, $S$, was guided by the earlier factor analysis, with item allocation, ${ℐ}_s$, determined by the factor loadings. If an item loaded on more than one trait, we selected the trait with the highest loading. For identifiability, we set the discrimination parameter to 1 and the difficulty parameter to 0 for the first item in each trait.

3.3. Quality disparities

We used equation (9) or (10) to estimate quality disparities depending on which latent trait distribution better fit the data.

3.3.1. Validity

The validity of the latent trait(s) was assessed by (a) examining the mean difference in the latent trait (quality) between those having quarterly psychosocial visits (our held-out quality metric) and those who did not for the Normal latent trait model and the (b) probability of quarterly psychosocial visits across class membership groups for the latent class model. Positive differences (Normal latent trait) or larger probabilities of quarterly visits in higher classes (latent class model) support the validity of the model.

3.4. Common approach

We followed the common approach, computing an observed score for each enrollee according to equation (1), estimating a linear regression model using the observed score as the response and the race and ethnicity variables as covariates. Scoring rules for $Y_j$ are provided in online supplementary material, Table 1.

3.5. Results

The final column in Table 2 describes the sample characteristics for the template sample of 24,000 individuals. Compared to the overall cohort (Column 1, Table 2), the template sample included more individuals from California; otherwise, all comorbid measures and health service use measures were no different from the full cohort. Covariates were balanced exactly across race and ethnicity groups (online supplementary material, Table 2). Racial and ethnic quality differences persisted, e.g. quarterly ambulatory visits remained lowest for Black individuals (59% vs 67% Latinx) and for some items, the differences grew, e.g. 30-day follow-up of a schizophrenia emergency department visit (42% Black vs 53% White individuals; online supplementary material, Table 3).

3.5.1. Exploratory normal latent trait IRT model

In our initial analysis, the 1-factor IRT model (upper half, Table 3) had factor loadings ranging from 0 to 0.99 that were weighted towards acute care quality. In the two-factor model, factor 1 describes acute care quality while factor 2 describes pharmacological and ambulatory care quality. In the three-factor model, pharmacological care quality and ambulatory care quality are separated, although a few items (14: quarterly ambulatory care and 10: follow-up after an ED visit) do not load cleanly on a single factor. In the 2-factor and 3-factor models, the average pairwise correlation between item residuals was small. However, the discrimination parameters were very large for some quality indicators, and quantile–quantile plots indicated the non-normality of the estimated traits (online supplementary material, Figure 1).

Table 3.

Factor loadings, VARIMAX rotation using a matched sample of 24,000 beneficiaries

No. of factors	One	Two		Three
Factor	1	1	2	1	2	3
(2.) Long-acting injectable	0.00	0.00	0.30	0.00	0.00	0.38
(3.) Clozapine use	0.08	0.00	0.24	0.02	0.00	0.33
(4., 5.) Adherence	0.12	0.07	0.25	0.11	0.05	0.29
(6.) No polypharmacy	0.00	0.00	0.00	0.00	0.00	0.00
(7.-9.) Follow-up after schizo. discharge	0.88	0.98	0.09	0.99	0.00	0.01
(10.-12.) Follow-up after MH ED visit	0.03	0.00	0.17	0.00	0.36	0.36
(14.) Quarterly ambulatory MH visits	0.24	0.09	0.99	0.23	0.00	0.86
(15.) No excessive schizo. acute care	0.92	0.91	0.10	0.83	0.56	0.00
(16.) No excessive MH acute care	0.87	0.87	0.08	0.75	0.65	0.00
(19., 20.) No re-admission after MH discharge	0.99	0.99	0.13	0.99	0.00	0.00
(21.) Psychosocial services	0.01	0.00	0.41	0.00	0.00	0.46
(23.) Psychotherapy	0.00	0.00	0.35	0.00	0.00	0.19
Cumulative % Variation	29	42			47
No. of parameters	30	46			53
AIC	255,243.5	258,851.1a			245,079.1
RMSEA	0.087	0.048			0.067
No. of factors	One	Two
Binary transformed indicators	1	1	2
(a.) Clozapine use	0.26	0.00	0.35
(b.) Adherence ≥80 %	0.32	0.12	0.38
(c.) Follow-up after schizo. discharge	0.66	0.59	0.34
(d.) Follow-up after MH ED visit	0.15	0.00	0.34
(e.) Quarterly ambulatory MH visits	0.38	0.13	0.93
(f.) No excessive schizo. acute care	0.90	0.88	0.01
(g.) No re-admission after MH discharge	0.94	1.00	0.00
(h.) Psychosocial services	0.13	0.00	0.42
Cumulative % Variation	31	46
No. of parameters	16	23
AIC	133,035.6	130,245.8
RMSEA	0.026	0.015

Note. Item 13 had no variation and was excluded. Item 22 was excluded and used as a validation measure. Loadings ≥ | 20 | considered meaningful. 0.00 items that were estimated as 0 by the model. ED=emergency department, MH=mental health, AIC=Aikaike Information Criteria, RMSEA=root mean squared error of approximation.

Bolded items are those with loadings at least | 0.20 |.

^aThe AIC is higher in the two factor model because some categories of items in the one factor models were combined.

We eliminated some items (1. long-acting injectable, 6. no polypharmacy, 16. no excessive mental health (MH) care, and 23. any psychotherapy) having low prevalence and poor item properties, and combined response options of some quality indicators. This resulted in 8 binary-valued indicators (bottom, Table 3). The distribution of these items are presented in online supplementary material, Table 4. The 2-factor solution continued to suggest acute care and ambulatory care quality traits. We fitted M2PL models (equation 7) and while the RMSEA decreased, the latent trait distributions continued to demonstrate non-normality. Initial fitting suggested combining categories within items. This ultimately yielded binary-only items. Because the assumptions for the Normal latent trait model were not met, we abandoned the Normal latent trait model and did not estimate a structural regression model.

3.5.2. Exploratory latent class IRT model

Using the 8 binary items, we next fitted a latent class IRT model assuming 1–5 classes. The BIC suggested 3 classes and, while the AIC suggested 4 classes (Table 4), the difference between the 3 and 4 class model was negligible.

Table 4.

Fit of exploratory unidimensional latent class IRT models by the number of classes. Eight binary quality indicators.

No. of classes	Estimated Log likelihood	No. of parms.	AIC	BIC
1	−69, 209.50	8	138,435.0	138,499.7
2	−66, 828.08	17	133,690.4	133,827.9
3	−65, 675.85	19	131,389.7	131,543.3
4	−65, 673.74	21	131,389.5	131,559.3
5	−65, 675.66	23	131,397.3	131,583.3

Bolded entries designate best model.

With 8 binary items, at most two dimensions were considered. We tested the two-dimensional model in equation (8) assuming an ambulatory care quality construct (quality indicators a, b, d, e, and h) and an acute care quality construct (quality indicators c, f, and g) against the unidimensional model. The constructs were identified by the earlier factor analysis. The BIC in Table 5 was lowest for 4-classes. Within a fixed number of classes, the likelihood ratio test rejected the assumption of unidimensionality.

Table 5.

Unidimensional versus two-dimensional exploratory latent class IRT models

model	Estimated Log likelihood	No. of parms.	AIC	BIC
3-classes
Unidimensional	−65, 675.85	19	131,389.7	131,543.3
Two-dimensional	−65, 505.40	20	131,050.8	131,212.5
−2LogL	340.9	1 dof
4-classes
Unidimensional	−65, 673.74	21	131,389.5	131,559.3
Two-dimensional	−65, 084.60	22	130,227.2	130,461.7
−2LogL	1,178.3	1 dof

The estimated class probabilities averaged over beneficiaries, and support points are presented in Table 6. Adjusting for race and ethnicity, the largest estimated class probability corresponds to the class receiving the highest care quality in the 3-class model (65% of the cohort) and only 9% of the cohort was characterized as belonging to the lowest quality class. The 4-class model included an additional higher class, removing a further 6% from the top 65% to form the highest quality group. The estimated support points reflect beneficiaries characterized by increasing quality, e.g. ${\widehat{\xi }}_2$ increases as ${\widehat{\pi }}_c$ increases. This is clear for both the 3 and 4-class models. However, the ambulatory care quality support points, ${\widehat{\xi }}_1$, do not increase smoothly as ${\widehat{\pi }}_c$ increases. In both the 3- and 4-class models, ${\widehat{\xi }}_1$ decreases as ${\widehat{\pi }}_1$ changes to ${\widehat{\pi }}_2$ although the size of the difference is smaller in the 4-class model.

Table 6.

Estimated class probabilities and support points for exploratory two-dimensional 3 and 4 class latent IRT models, adjusted for race and ethnicity group

Quality		Ambulatory care		Acute care
Class	$\widehat{\pi }$	${\widehat{\xi }}_1$	$\mathrm{s}\mathrm{e}({\widehat{\xi }}_1)$	${\widehat{\xi }}_2$	$\mathrm{s}\mathrm{e}({\widehat{\xi }}_2)$
1	0.09	−3.31	0.062	0.17	0.049
2	0.26	−4.28	0.163	1.37	0.044
3	0.65	−2.89	0.039	2.96	0.061
1	0.09	−3.48	0.051	0.18	0.048
2	0.25	−3.83	0.091	1.36	0.060
3	0.60	−3.43	0.060	2.82	0.047
4	0.06	−1.06	0.143	3.80	0.571

Examination of the IRT parameter estimates suggest better item fit for the 3-class model and we use that model for inferences (the IRT parameter estimates for the 3-class model are presented in online supplementary material, Table 5). Quarterly ambulatory visits best discriminate ambulatory care quality among individuals having the largest ${\widehat{a}}_i=3.49(\mathrm{s}\mathrm{e}=0.62)$ with antipsychotic (AP) adherence of at least 80% having the next largest discrimination 1.06 (0.12). For acute care quality, both discrimination parameters were high: no excessive schizophrenia acute care, 4.16 (0.35), and no re-admission following a mental health hospital discharge, 4.86 (0.53).

Figure 1 illustrates the estimated conditional response probabilities (y-axis) for each binary quality indicator. Class 3 had uniformly higher probabilities on the quality metrics relative to the other two classes. Relative to Class 1, individuals in Class 2 had higher probabilities on all acute care quality metrics but lower probabilities on all ambulatory care metrics.

Figure 1.

Estimated healthcare quality profiles for 24,000 Medicaid beneficiaries with schizophrenia. Estimates use a 3-class two-dimensional latent model adjusted for race and ethnicity using binary quality indicators listed in bottom half of Table 3.

3.5.3. Validation

While the percentage of individuals utilizing psychosocial services quarterly increased from 39% in Class 1 to 55% in Class 3, Class 2 had only 8%.

3.5.4. Estimated quality disparities

The estimated racial and ethnic quality disparities are presented in Table 7 using the linear regression model and the 3-class two-dimensional latent model using a multinomial logit model. The linear regression model indicates no differences in quality between Latinx and White beneficiaries. Black beneficiaries receive, on average, lower quality (0.092 SD) of care. The regression coefficients for the latent class model are the log odds of the probability of class membership relative to the lowest quality class (9% of the sample). There is no difference between Latinx and White beneficiaries in quality class membership probabilities. Relative to White beneficiaries, Black beneficiaries are 1.4 times more likely than White beneficiaries to belong to class 2 versus 1 but only $\frac{4}{5}$ as likely to belong to the highest versus the lowest quality class.

Table 7.

Racial and ethnic quality disparity estimates using a matched sample of 24,000 individuals

Approach	Black		Latinx
	${\hat{\gamma }}_1$	$\mathrm{s}\mathrm{e}\left({\hat{\gamma }}_1\right)$	${\hat{\gamma }}_2$	$\mathrm{s}\mathrm{e}\left({\hat{\gamma }}_2\right)$
Observed score regression	−0.092	0.020	0.010	0.020
	Class 2 vs 1		Class 3 vs 1
3-Class latent model	${\hat{\gamma }}_1$	$\mathrm{s}\mathrm{e}\left({\hat{\gamma }}_1\right)$	${\hat{\gamma }}_2$	$\mathrm{s}\mathrm{e}\left({\hat{\gamma }}_2\right)$
White	0.95	0.074	2.01	0.053
Black vs White	0.32	0.074	−0.18	0.064
Latinx vs White	0.03	0.073	0.07	0.063

Note. Observed score regression estimates are mean increases or decreases in quality relative to White beneficiaries on a linear scale. Latent class model estimates are changes in the log odds of the probability of a higher quality class versus Class 1 based on a two-dimensional model.

Figure 2 displays ${\widehat{\pi }}_c$ for the sample and illustrates no difference in the distribution of class membership between White and Latinx beneficiaries. Relative to White and Latinx beneficiaries, fewer Black beneficiaries belong to the highest quality class. Online supplementary material, Figure 2 displays assignment to a single latent class, by race and ethnicity, using the maximum estimated probability (Equation 16).

Figure 2.

Distribution of estimated latent class membership, by race and ethnicity status, for 24,000 Medicaid beneficiaries.

4. Conclusions

IRT models can provide a unifying framework to estimate healthcare quality. This approach reflects the conceptual model of quality, accommodate features of the data-generating process, and permit joint modelling of the data useful for global testing. In our study, as in many healthcare quality studies, the quality metrics cover a broad set of services, such as pharmacological treatments, acute care services, and ambulatory services, making multidimensional IRT models realistic alternatives to commonly used approaches that assume a single quality construct. While approaches for estimation of unidimensional and multidimensional IRT models are widely available and frequently used in educational and psychological assessment settings, their application in the healthcare quality is infrequent. Most often, researchers seek a unidimensional solution given its simplicity and ease of computation. However, they may be inappropriate because quality constructs are likely multidimensional.

Parametric adjustments for differences in health status are challenging in general, but even more difficult when using structural regression models combined with IRT models. Identifiability issues are common and the number of covariates included at the trait level is limited. The use of template matching eliminates many parametric assumptions, simplifies any subsequent modelling required, and permits the inclusion of more confounders than possible via standard regression. Nonetheless, the IRT models are parametric as is the structural measurement model linking race and ethnicity to healthcare quality. Many researchers have proposed approaches to assess the sensitivity of results to unmeasured confounders in matched sets (see, in particular, Rosenabum, 1987), but we are unaware of their use in IRT models combined with structural regression.

In our study, unidimensionality did not fit the data nor did the Normal latent trait IRT model. Some metrics had response options such that a subset of them could be naturally ordered and others did not. For instance, when assessing if someone was followed up after a hospital discharge, options included no follow-up within 30 days, follow-up between 7 and 30 days, follow-up within 7 days, and no hospital discharge. While the first 3-options order naturally, having or not having a hospitalization could be indicative of good or bad quality of care. In our application, we ultimately combined most categories and arrived at binary items. The latent class IRT model provides a robust alternative to the Normal latent trait model, making no parametric assumptions about the latent trait. However, the estimates of the support points were not ordered for the ambulatory care quality construct, with the estimate of the location for class 2 imprecise. Estimating the items from the ambulatory care construct separately did not resolve this problem. The number of quality items was small in our study compared to health and educational assessments, preventing an accurate assessment of healthcare quality and quality disparities. Some quality metrics may be perceived as measures of a person’s behaviour, such as antipsychotic drug adherence, rather than measures of the quality of health services delivered. However, arguably the onus is on the healthcare system to educate patients and reduce any external barriers they may face to adhere to recommended care. Moreover, each measure used in the article is recommended by professional societies or other healthcare entities such as the National Committee for Quality Assurance. Not all quality metrics used in the study require risk adjustment. For instance, virtually all schizophrenia patients should be prescribed an antipsychotic drug. Given our goal of estimating disparities, we balanced health status characteristics across the race and ethnicity groups using information observed in billing claims data, but unmeasured confounding is always a risk. Finally, we did not have any patient preference information.

In summary, using an exploratory IRT measurement model combined with structural regression and template matching within a cohort of adults with schizophrenia, we found that healthcare quality is likely multidimensional and race and ethnicity disparities in quality likely exist. About 2/3 of the cohort was categorized into the highest-quality class, a quarter into the middle-quality class, and less than a tenth in the lowest class. While there were no differences between Latinx and White beneficiaries in the probability of belonging to higher-quality groups, Black beneficiaries were more likely than White beneficiaries to belong to the middle-quality class relative to the lowest-quality class but less likely to belong to the highest-quality class. Using the common approach of estimating a regression model using a score derived from the observed responses, we found no differences between Latinx and White beneficiaries, and a small difference between Black and White beneficiaries, with Black beneficiaries receiving poorer quality of care. These findings are clear for acute care quality—more work is needed for the ambulatory care quality construct.

Data availability

The data used in this study are regulated by the Centers for Medicare and Medicaid Services (CMS) in the U.S. and are available for purchase under a data use agreement from CMS.