Patient Acceptable Symptom State versus Latent Class Analysis Outcome Classification: A Comparative Longitudinal Study of Knee Arthroplasty

Abstract

Objective:

To determine if Patient Acceptable Symptom State (PASS), a single-item deterministic binary measure of pain and function outcome satisfaction leads to better differentiation of outcome classification versus latent class analysis probability-based outcome subgroups one-year following knee arthroplasty (KA).

Methods:

We used data from KASTPain, a one-year no-effect multicenter randomized clinical trial of participants with KA along with prior work that developed and externally validated good and poor outcome trajectories. Confirmatory latent class analyses were conducted on two exemplar outcome measures (EQ VAS single-item self-rated health and 4-Item pain ratings) and compared to PASS scores. Separation of trajectories were used to compare good and poor latent class self-rated health/4-item pain trajectories and PASS score trajectories.

Results:

Prevalence rates for poor outcomes were 10% for self-rated health and 20% for 4-item pain and PASS. Probabilistic latent class derived classifications of self-rated health and 4-item pain outcomes outperformed PASS in separating growth trajectories. The effect size point estimates for 12-month 4-item pain scale score separation was approximately 3 times larger for latent class analyses as compared to PASS.

Conclusions:

When used for outcome classification, observed PASS scores consistently underperform relative to probabilistic latent class-derived subgroups of pain and self-rated health outcome. PASS is a weak substitute for probabilistic classification of other PROMs of KA outcome. Clinicians and researchers should rely on latent class analyses over PASS to differentiate between outcome subgroups following KA.

Two common types of patient-reported outcome measures (PROMS) to assess knee arthroplasty (KA) outcome are single/multi-item scales to quantify pain or health/functional status and single/multi-item scales of outcome satisfaction. (1,2). As psychometric instruments, however, pain, health status and satisfaction lack a true gold standard to aid interpretation (3).

The Patient Acceptable Symptom State (PASS) satisfaction measure was first reported in 2005 (4) and has become a commonly endorsed PROM for daily clinical practice and for clinical research. PASS is an appealing measure for many reasons. First, it is a single-item scale with a simple dichotomous response option and minimal patient burden. Second, it is designed to assess the patient’s current state though recent evidence indicates patients rate their satisfaction following KA by comparing their current state to their preoperative status (5,6). Third, it assesses an important albeit complex construct of satisfaction. Inquiring about satisfaction with symptom status is important because it asks patients to apply their own value systems in judging their current state, an attribute not directly addressed by self-reported pain and functional status measures. However, because different patients use different value systems and different priorities to rate their satisfaction with KA, interpretation of the meaning of satisfaction ratings is challenging. Additionally, many satisfaction measures have been used in KA which adds to the variability in satisfaction ratings. Klem and colleagues found, in a recent systematic review, that a large number of satisfaction measures assessing varying constructs (e.g. satisfaction with the surgical knee, pain, symptoms, function) are used in KA. Of the 43 studies included, only 15 provided a reference for the satisfaction instrument and none of the instruments, including PASS, included patient input during development as required by COSMIN criteria (7). Satisfaction estimates ranged from 39% to 99% depending on satisfaction construct and timeframe.

Much of the work published on PASS in KA has determined threshold scores for interpreting other self-report outcome measures. For example, Naal and colleagues used area under the curve (AUC) methods to determine threshold scores for a variety of PROM pain and function measures that best differentiated patients with satisfactory versus unsatisfactory PASS scores following arthroplasty (8). In a systematic review, Mackay and colleagues found substantial variation in PASS-based thresholds for WOMAC pain and function scores post knee and hip arthroplasty (1). Substantial variation in PASS estimates (9) also has been found for other hip and knee arthroplasty PROMS. As a single-item satisfaction scale, PASS cannot be considered as a gold standard (i.e., error free measure of satisfaction); thus, attempts to establish threshold values for multi-item instruments (e.g., WOMAC Pain) from PASS scores is psychometrically unsound (10).

We recently developed (11) and subsequently externally validated a probabilistic model-based approach for interpreting change over time using latent class growth curve modeling to differentiate persons with good versus poor pain and function outcome following KA (12). Latent class modeling has been used for decades (13) to study diagnosis in the absence of a gold standard, as is the case for PROMs. The purpose of the current study was to compare and contrast the performance of deterministic (PASS) and probabilistic latent class growth analysis approaches to good versus poor outcome classification using one-year trajectories of two PROMS. (14,15). Our prior work externally validated good and poor outcome trajectories for WOMAC Pain, and WOMAC Function, scores (11,12). In the current study, we used two measures independent of WOMAC Pain and Function, a 4-item pain measure (14) and EQ VAS self-rated health (15) to compare good versus poor outcome to PASS ratings. We hypothesized that outcome subgroups derived from latent class growth curve models would provide greater separation of subgroup outcome trajectories following KA as compared to subgroup trajectories using PASS.

Our intent was not to conduct a head-to-head comparison of PASS to 4-item pain/self-rated health scores to determine which is superior as these scales measure different constructs. Rather, we were interested in comparing two different methods for classifying outcome trajectories: PASS classification of satisfactory and unsatisfactory outcomes to probabilistic outcome classification using latent class modeling. Importantly, we used 4-item pain and self-rated health outcome measures and not outcomes that were used to externally validate good and poor latent classes (i.e., WOMAC) (12).

METHODS

Study Sample

We used data from our recently published no-effect randomized KASTPain clinical trial of a pain coping skills intervention for persons undergoing KA with moderate to high pain catastrophizing (16). A total of 384 participants from 5 sites had KA and participated in a three-arm National Institutes of Health funded randomized clinical trial (UM1AR062800) conducted at five sites (Virginia Commonwealth University, Duke University, New York University Medical Center, Southern Illinois University, and Wake Forest University). Of 384 participants, we included in the current study only those participants who had a one-year postoperative PASS score (n = 344). All participants signed an IRB approved consent form prior to KA and were followed for one year following surgery. KASTPain trial data also were used in a prior publication to develop a probabilistic latent class growth curve modeling method for classifying outcome as good or poor (11).

Outcome variables of interest

All participants completed 4-item pain measures and the EQ-5D-5L between one and eight weeks preoperatively and, 2-, 6- and 12-months postoperatively. The 4-item verbal rating pain scale asks participant to rate their pain on a 0 (no pain) to 10 (most intense pain) scale at the time of testing, and over the prior 2-weeks, at its best, worst and on average. Evidence supports the reliability and validity of this multi-item scale over single-item pain scales (14). The EQ-5D-5L, a commonly endorsed health measure, most commonly used in economic analyses, consists of two components, a 5-item pain, mental health and mobility scale, norm referenced to the US population (17) and a single-item overall health scale (EQ VAS) ranging from 0 to 100 with higher scores equating to better self-rated health (18). The EQ VAS scale score was used in the current study because, much like PASS, it also is a single-item scale designed to capture a complex construct, in this case, current overall health. The single-item self-rated health status scale is typically scored using a 20 centimeter vertical visual analogue scale. Because we collected follow-up data by phone, we asked participants for a verbal rating between 0 and 100, after providing the following instructions as stated by the developers (18): “We would like to know how good or bad your health is today. The scale is numbered from 0 to 100. 100 means the best health you can imagine and 0 means the worse health you can imagine.” Please rate how your health is today. Reliability for the EQ VAS self-rated health measure is in the moderate to high range (15).

PASS (4) scores were obtained at the one-year postoperative visit. We used the PASS measure originally described by Tubach et al (4). The measure asked the following question: “Taking into account all the activities you have during your daily life, your level of pain, and also your functional impairment, do you consider that your current state is satisfactory?”. Possible responses were “yes” or “no.”

Good versus Poor Outcome Latent Classes

We reported in a prior publication using KASTPain data, (11) the results from an exploratory piecewise latent class growth analysis that demonstrated WOMAC Pain and Function trajectories could be classified into two subgroups: good and poor latent classes with high entropy of 0.87. We externally validated this approach in an independent sample of 926 patients with KA from England (12). Approximately 20% of the samples in both studies were categorized as having poor outcome and approximately 80% of the sample as having good outcome.

Because we externally validated good versus poor outcome classification following KA, the current study is considered as confirmatory rather than exploratory. We estimated latent class good versus poor outcome trajectories of the 4-item pain scale and the single-item overall health scale for the 344 participants in the current study as a basis for comparison to PASS scores. We used the observed PASS score ratings of satisfactory or unsatisfactory outcome and applied these ratings to create satisfactory and unsatisfactory trajectories of 4-item pain scores and single-item self-rated health scores over the study period.

Statistical Analysis

Frequencies were used to describe the two subgroups for both outcomes (4-item pain and self-rated health) and PASS using 2 × 2 tables. Unconditional probability estimates were used to report frequencies for latent classes. As a deterministic classifier, satisfactory/unsatisfactory frequencies were readily available for the patient-reported binary PASS scores. Two-piece latent class growth models with individually varying times of observation were used to estimate self-rated health/4-item pain classes. The two-piece refers to two slopes, one from the baseline to 2-months post- surgery and another from two-months post-surgery to the end of study period (12-month post-surgery). The time scale was centered at the time of surgery. As it was impossible to collect outcome measurements exactly at 2-month pre, 2-, 6-, and 12-month post, we used individually varying times of observation to take account of variation at a given time point. As a probabilistic classifier, two-piece latent class growth modeling with two classes (based on our external validation study (12)) were used to classify outcome trajectories as either good or poor. Additional analytic detail from our prior latent class analyses and our MPlus coding for the current paper appears in Supplemental Text S1. For observed PASS ratings of satisfactory or unsatisfactory, a priori probabilities were either 0 or 1. The full information maximum likelihood method was used for handling missing data.

Entropy, a standardized index of classification accuracy ranging from 0 to 1 with higher scores indicating greater accuracy (19) was supplemented with the difference between the model-implied outcome scores of two subgroup and 95% confidence intervals around the difference at baseline and 12-month follow-up for PASS and for the latent class analyses. The difference between the two subgroups served as effect sizes indicating extent of class separation and to compare the two methods on both outcomes.

RESULTS

Baseline characteristics for the sample are reported in Table 1. The sample (n = 344) consisted of 230 females (67% of the sample) and 34.6% of the sample self-reported as African American using NIH terminology. Prevalence rates were similar, 19.8% and 20.6% for PASS unsatisfied ratings and for 4-item pain poor outcome ratings, respectfully. Prevalence of single-item overall health scores for poor outcome class was 10%. (see Figure 1). Frequencies for the various combinations of PASS observed ratings and 4-item pain and self-rated health latent class ratings are provided in Table 2.

Table 1.

Baseline and Follow-up Characteristics of the Study Sample

Characteristic	Full Sample N = 344	Missing Data
Age, mean (SD)	63.3 (8.1)	0
Sex (Female), N (%)	230 (67)	0
Body mass index (Kg/m2), mean (SD)	32.4 (6.1)	6
Race (African American), N (%)	119 (34.6)	0
Preop 4-item Pain Score, mean (SD)	6.06 (1.87)	0
2-month 4-item Pain Score, mean (SD)	3.21 (2.04)	17
6-month 4-item Pain Score, mean (SD)	2.29 (2.25)	26
12-month 4-item Pain score, mean (SD)	1.83 (2.15)	0
Preop overall health rating (mean, SD)	72.5 (17.6)	0
2-month overall health rating (mean, SD)	78.2 (14.7)	17
6-,month overall health rating (mean, SD)	78.9 (17.1)	26
12-month overall health rating (mean, SD)	78.9 (16.7)	0
WOMAC Pain (Good outcome) N (%)	287 (83.4%)	0
WOMAC Function (Good outcome) N (%)	281 (81.7%)	0
PASS# (Satisfactory), N (%)	273 (79.4%)	0

^*WOMAC = Western Ontario and McMaster Universities Osteoarthritis Index

^#PASS = Patient Acceptable Symptom State

Figure 1.

Frequency distributions of latent classes of good versus poor outcome for 4-item pain, self-rated health and observed PASS scores of satisfactory and unsatisfactory.

Table 2.

Ratings for Patient Acceptable Symptoms State (PASS), 4-item Pain and Self-rated Health combinations and Entropy values.

Combinations of PASS, 4-item Pain and Self-rated Health	Poor/Unsat1	Poor/Sat2	Good/Unsat	Good/Sat	Entropy
4-item Pain/PASS	28	40	43	233	4-item Pain= 0.91, PASS = 1.0
Self-rated Health/PASS	15	20	56	253	Self-rated Health = 0.90, PASS = 1.0
Combination of 4-item Pain and Self-rated Health	Poor/ Poor	Poor/Good	Good/Poor	Good/Good
4-item Pain/Self-rated Health	16	52	18	258	---

¹Unsatisfactory rating for PASS

²Satisfactory rating for PASS

Graphical representation of typical outcome trajectories by latent class and PASS for the two outcomes appear in Figure 2 and point estimates along with 95% confidence intervals for latent class outcomes and PASS appear in Table 3. Latent class growth analyses yielded accurate class assignments with entropy values of 0.91 and 0.90 for 4-item pain and single-item overall health outcomes, respectively. PASS has a perfect classification accuracy (entropy=1) for all outcomes, by definition (see Table 2). A visual inspection of typical trajectories, however, revealed that latent class better separates the trajectories than PASS for both outcomes. Table 4 provides the difference tests and 95% Cis between the model-implied scores for two methods (latent class versus PASS) across two outcomes (EQ VAS health/4-item pain) at two time points (baseline and 12 months). At the 12-month follow-up, compared to the unsatisfactory-satisfactory estimates using PASS, the poor-good estimates from LCA is, on average, 3 times larger for both 4-item pain (4.509/1.425) and self-rated health (38.527/12.678).

Figure 2.

In panel A. trajectories of latent classes of good versus poor 4-item pain and observed PASS scores over the study period. In panel B, trajectories of latent classes of good versus poor self-rated health outcome and observed PASS scores over the study period. In both panels, the 0 time point on the x-axis denotes the pre-operative visit and the vertical line denotes the surgery.

Table 3.

Point estimates and 95% confidence intervals of latent class growth curve generated 4-Item pain and self-rated health outcomes versus observed PASS classifications

Comparison	Subgroups	Baseline	2-month	6-month	12-month
Self-rated Health: LC vs. PASS	Poor	67.86	68.3	58.78	44.28
	95%CI Low	57.59	59.92	54.61	38.1
	95%CI High	78.13	76.68	62.95	50.46
	Good	73.02	79.51	80.82	82.82
	95%CI Low	69.92	77.26	78.4	80.01
	95%CI High	76.11	81.76	83.24	85.63
	Unsatisfactory	67.21	71.13	70.28	68.99
	95%CI Low	63.84	67.28	67.49	66.38
	95%CI High	70.58	74.97	73.07	71.59
	Satisfactory	73.88	80.27	80.82	81.65
	95%CI Low	71.43	78.93	79.17	79.14
	95%CI High	76.33	81.62	82.47	84.16
4-Item Pain: LC vs. PASS	Poor	7.06	5.36	5.38	5.42
	95%CI Low	6.56	4.35	4.5	4.64
	95%CI High	7.56	6.36	6.26	6.19
	Good	5.81	2.5	1.87	0.9
	95%CI Low	5.39	2.23	1.62	0.6
	95%CI High	6.23	2.76	2.11	1.2
	Unsatisfactory	6.27	3.93	3.51	2.87
	95%CI Low	5.71	3.28	2.83	2.07
	95%CI High	6.83	4.58	4.19	3.66
	Satisfactory	6	2.86	2.3	1.45
	95%CI Low	5.6	2.51	1.97	1.09
	95%CI High	6.41	3.21	2.63	1.81

Table 4.

Average distance between trajectory types by methods (PASS and LCA) and outcomes

Time	Method	Outcome	ΔY-hat	95% CI	p
Baseline	PASS*	4-Item Pain	.268	−0.279 – 0.815	0.336
		Self-Health	6.665	3.210 – 10.120	<0.001
	LCA+	4-Item Pain	1.248	0.638 – 1.858	<0.001
		Self-Health	5.172	−7.088 – 17.432	0.408
12-Month	PASS	4-Item Pain	1.425	0.813 – 2.037	<0.001
		Self-Health	12.678	8.800 – 16.476	<0.001
	LCA	4-Item Pain	4.509	3.931 – 5.087	<0.001
		Self-Health	38.527	34.005 – 43.049	<0.001

^*PASS = Patient acceptable symptom state,

⁺LCA = Latent class analysis

DISCUSSION

Our study had a narrow focus – to compare LCA to PASS ratings to determine which approach better differentiates between outcome subgroups following KA. We found that observed PASS ratings, and latent class 4-item pain and self-rated health measures indicated an approximate 10% to 20% unsatisfactory/poor outcome, an estimate that is consistent with prior evidence (20). Despite the similarities, we found in our trajectory analyses (see Figure 2) that PASS consistently overestimated benefit for the unsatisfactory outcome subgroup as compared to the two latent class analyses. In fact, latent class analyses outperformed PASS by approximately 3 times at 12-month follow-up in subgroup outcome separation. Our study suggests that latent class analyses are superior to PASS measures when the clinician/researcher is interested in differentiating among subgroups of KA patients with different outcomes (20–23).

It may be counter-intuitive that PASS has a perfect accuracy of classification (entropy=1) yet it underperforms compared to latent class analysis up to 4.6 times in outcome separation. PASS or any other categorical observed variable can be used to operationally define outcomes. The source of confusion originates from equating an operational definition (e.g. satisfied or not satisfied, with PASS) to a true gold standard. All categorical observed variables, including PASS, have an entropy of 1, by definition.

PASS ratings require patients to determine whether their current pain and functional status is satisfactory. Ratings of satisfaction reflect a complex, multidimensional construct (24) representing different phenomena and different ways of thinking about the construct for different patients, ranging from processes of care to outcomes of care (25). Patients may adopt different frames of reference, have their recollections facilitated or hampered by various transient factors, or simply differ in how much they think about whether they’re satisfied before answering. Use of a single item to quantify this complex multi-dimensional construct may well be inadequate. Consequently, responses across individuals may represent different underlying variables, which will likely erode associations between such a measure and a reference standard. However, much like a single-item satisfaction measure, the single-item EQ VAS also is a single-item scale designed to capture the complex multidimensional construct of current overall health. Rating overall health with a single item likely also leads to varying interpretations, much like PASS. When using LCA methods, the EQ VAS measure appreciably outperformed PASS which, in our view, further supports the superiority of LCA over PASS when the user wants to differentiate between KA outcome subgroups.

The 4-item pain scale and the EQ VAS health measure, in our opinion, served as unbiased outcome measures because these measures are independent of the outcomes used to derive the latent classes (i.e., WOMAC Pain and Function scores). Results indicated that the difference between poor versus good outcome as determined by latent class analysis was substantial with an approximate threefold greater separation at 12-month follow-up. In fact, the latent intercept difference between satisfactory and unsatisfactory was not statistically significant at baseline (p = 0.336) for 4-Pain scores leading to the conclusion that the data do not support satisfactory/unsatisfactory distinction.

Substantial variation in PASS estimates have been found, even among studies examining outcome in persons with KA (1,8). Investigators also have used PASS as the gold standard comparator to establish thresholds for a large variety of pain and function outcome measures designed for a number of different disorders (1,26,27). Keurentjes and colleagues, for example, determined PASS thresholds for the Oxford hip or knee scores in patients with hip or knee arthroplasty. The investigators contended that if Oxford threshold scores were met or exceeded, patients would be satisfied with their outcome (28). A PubMed search conducted on 2/22/21 and using keywords (“patient acceptable symptom state” AND (threshold OR cut point)) found 116 citations. Use of PASS to establish thresholds for other measures is becoming more common with most of these references published since 2020. The presumed purpose of establishing thresholds is to provide a clinical interpretation of outcome score meaning. In our opinion, three main problems with this approach are: 1) PASS is not a gold standard measure and should not be treated as a gold standard to interpret the meaning of other outcome measures, 2) PASS is a single-item scale with additional measurement error relative to multi-item scales (29), and 3) PASS results are inferior, with smaller or even non-significant outcome differentiations as compared to latent class methods, as our study demonstrates. For patient score interpretation, our data suggest that latent class methods are superior to PASS ratings at differentiating among patient subgroups with different outcomes. Because our probabilistic latent class model-based methods outperform PASS, satisfaction measures should not, in our view, be used for establishing thresholds for other types of outcome measures.

Satisfaction is considered a standard-of-care outcome assessment by the Centers for Medicare and Medicaid Services (30). As a measure of satisfaction, PASS measures a construct acknowledged as being complex and multifactorial (25). When considering PASS, in spite of the direct instruction to consider current health state, patients likely consider their pre-operative state (5,6) and processes of care delivery (e.g., costs, high care burden) and these varying sources may influence a participant’s PASS rating. With this said, assessment of satisfaction is a key outcome in KA. Both PASS and overall health single-item ratings reflect complex constructs that would benefit from a thorough understanding of the concepts that comprise these measures. While a thorough mapping of these concepts (31) is beyond the scope of the current paper, this is an important area for further study. Studies should determine whether the clinically attractive and efficient-to-obtain PASS is valid given the complexity of the construct of satisfaction.

Our study had several limitations. The measurement of satisfaction with outcome is endorsed as an important outcome but our study did not determine the utility of PASS relative to other satisfaction measures. We therefore cannot make direct judgements about the utility of PASS measures in isolation, nor can we directly compare PASS to self-rated health/4-item pain findings. Our study was designed only to compare the deterministic PASS classification to probabilistic latent class modeling classification using self-rated health/4-item pain ratings as exemplar measures. Approximately 10% of our sample had missing PASS data at 12 months and this could have influenced the findings though self-rated health/4-item pain ratings were not statistically different for those with and without missing 12-month data (data not shown). We studied only persons with moderate to high levels of pain catastrophizing and while the sample overall, demonstrated KA recovery that was similar to more heterogeneous samples of patients with KA (32–34), the presence of catastrophizing may have influenced our findings. While we were unable to determine why some participants in our study demonstrated discordant measures of PASS and WOMAC/4-item pain outcomes, our study supports the argument that a reasonably high number of participants demonstrated discordant PASS and WOMAC/4-item pain ratings. This discordance likely was attributable, at least in part, to a difference in constructs assessed by the measures. Our findings suggest that use of PASS-based thresholds for interpreting multi-item outcome scales will likely not be helpful at best and could be misleading at worst.

In conclusion, our findings challenge the increasingly common application of PASS for determining thresholds of other PROMS for the purposes of interpreting the meaning of PROM change scores in KA. Our data demonstrate that PASS actually underperforms as compared to latent class analyses in differentiating between outcome subgroups. Latent class analyses lead to greater subgroup separation compared to PASS ratings and therefore lead to superior prediction of subgroup outcome membership following KA.

Significance and Innovation.

• Latent class analyses are superior to PASS in differentiating between knee arthroplasty outcome subgroups.

• Probabilistic classification of knee arthroplasty outcome is psychometrically defensible in contrast to deterministic outcome classification such as PASS.

• Use of PASS to interpret the meaning of multi-item outcome scales in knee arthroplasty should be avoided.