Multidimensional Computerized Adaptive Testing: A Potential Path Toward the Efficient and Precise Assessment of Applied Cognition, Daily Activity, and Mobility for Hospitalized Patients

Abstract

Objective:

To develop and evaluate an efficient and precise variable-length functional assessment of Applied Cognition, Daily Activity, and Mobility to inform mobility preservation and rehabilitation service delivery among hospitalized patients.

Design:

A multidimensional item bank tapping into these dimensions was developed, with all items calibrated using a multidimensional graded response model (MGRM). The items were adaptively selected from an item bank to maximize the test information, and the test ended when a joint stopping rule was satisfied. A simulation study was conducted based on the completed instrument, the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT) to compare its measurement precision and efficiency capabilities relative to conventional unidimensional CAT. Precision was measured by the bias and root mean squared error between the estimated and true (i.e., simulated) θ estimates, whereas efficiency was measured by average test length. Data was collected by an interviewer reading questions from a tablet computer and entering patients’ responses.

Setting:

A large Midwestern hospital

Participants:

A total of 4,143 patients hospitalized with medical diagnosis and/or surgical complications with 2,060 in the calibration sample, and 2,083 in the validation cohort.

Intervention:

Not applicable.

Results:

Among the 2,083 patients in the validation sample, FAMCAT administration required an average of 6 (SD=3.11) minutes. Ninety-six per cent (96%) had their tests terminated by the standard error rule after responding to an average of 22.05 (SD = 7.98) items, whereas 15 were terminated by the change in θ rule, with an average test length of 45.27 (SD = 11.49). The remaining 76 responded until reaching the maximum test length of 60 items.

Conclusions:

The FAMCAT has the potential to satisfy the need for structured, frequent, and precise assessment of functional domains among hospitalized patients with medical diagnosis and/or surgical complications. The results are promising and may be informative for others who wish to develop similar instruments when concurrent assessment of correlated domains is required.

Introduction

Computerized adaptive testing (CAT) has been used in health measurement for more than a decade, and an appreciation of its benefits has spurred the Patient-Reported Outcome Measurement Information System (PROMIS) initiative. Compared to the administration of a full item bank or short-form assessment, in CAT individuals receive different scale items that are targeted to their specific trait level. During a CAT session, an individual’s successive responses are used to determine a provisional estimate on the measured trait (e.g., pain, fatigue, etc.), which then is used to select subsequent items. The provisional estimate is updated after each item response. An important advantage of CAT is it greatly reduces response burden and assessment duration—both uniquely desirable in clinical settings with busy, overworked providers and ill patients with many needs. As a consequence, CAT has become an increasingly common mode of clinical patient reported outcome measure (PROM) administration--a trend that has been facilitated by the migration of PROMIS CATs into several current generation electronic health records.

The importance of efficiently collecting clinical PROMs cannot be overstated as policymakers and payers increasingly endorse, and even mandate, their use as a means of gauging not only a treatment’s benefits, but also its appropriateness. Providers and provider organizations cannot fulfill these mandates if they lack the ability to collect high quality and complete data. Measurement efficiency is an important determinant of PROM response and completion rates. In addition, patients are overwhelmingly less likely to complete burdensome PROMs when they perceive the questions as lacking relevance to their situation. CAT reduces respondent burden while increasing the likelihood that an item will be relevant to a specific patient. Finally, clinical workflows are over-taxed and make the administration of inefficient and time-consuming PROMs untenable.

While traditional CATs offer a distinct advantage over fixed-length assessments, the traits they measure are unidimensional and must be assessed separately. Greater efficiency can be achieved with approaches that assess related traits concurrently. More specifically, interest is growing in the use of multidimensional computerized adaptive testing (MCAT) in health measurement, as the domains of interest are usually related, complex, and multifaceted. ^1–3

MCAT is an approach that unifies major elements of both traditional unidimensional CAT and multidimensional item response theory (MIRT) and, as a result, can naturally account for the multidimensionality of a patient’s latent (underlying) traits and maintain the high efficiency that CATs produce.⁴ This advantage is particularly desirable for hospital-based functional assessments when multiple domains are relevant to clinical decision making and only a limited number of items can be administered. Moreover, multi-trait assessment has emerged as a fundamental requirement for patient-centered decision making. Because respondent burden corresponds directly to the number of assessed traits, efficiencies achieved through MCAT may prove vital to realizing the goal of need- and preference-sensitive decision making.

While MIRT represents a broad family of multi-factor models with categorical outcomes, this study focuses on one specific type of MIRT model, namely, the between-item multidimensional models.⁵ These models are used to model a set of items in which each item principally measures one latent trait but may provide information relevant to the estimation of others. For example, an item that queries a patient’s ability to use a remote-control device can also not only elicit information regarding their ability to perform daily activities (manual dexterity), but give insight into their applied cognition (task sequencing).

Multidimensionality is modeled through the correlations among such latent dimensions. This structure is common among health measures as they usually are comprised of multiple subscales, each of which measures a unidimensional latent trait. The PROMIS emotional distress bank, for example, contains items measuring depression, anxiety, and anger separately, yet these three latent traits are moderately correlated. Subdomains of physical function, e.g., mobility and performance of daily activities, have been shown to be similarly correlated among the elderly and other patient subgroups.⁶ Recent studies have compared the performance of MCAT using between-item MIRT and separate unidimensional CATs (UCAT) in health measurement settings^7,8, and demonstrated improved measurement precision with the former approach. Specifically, between-item MCATs were 20–38% shorter than separate unidimensional CATs when between-dimension correlations were high (r > .76) ¹⁶. For moderate correlations (e.g., r = .56), MCATs were, on average, 17% shorter than unidimensional CATs.⁹

It is difficult to overstate MCAT’s potential to provide a more feasible means of gathering time sensitive PRO clinical data. The need to rapidly measure correlated clinical traits is ubiquitous. However, functional assessment in hospitals is among the most pressing and inconsistently fulfilled needs. Patients’ discharge and rehabilitation needs must be quickly determined to maintain hospital throughput. Yet, functional assessments, arguably the most critical determinant, are often delayed—leading to suboptimal discharge planning, costly reliance on post-acute care, and unplanned readmissions. ^10,11

Despite clear need, recent MCAT explorations have focused on simulation studies and, as far as we know, no MCAT data have been prospectively collected in a hospital setting. In addition, a newly proposed joint stopping rule ¹¹ appears to be promising but has not yet been fully evaluated in a MCAT and UCAT comparison.

The overall methodology of the development of the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT),¹² has been detailed previously.¹² This manuscript describes the development and evaluation of its application of the latest MCAT technology. Specifically, it describes FAMCAT item bank calibration, item selection and stopping rules design, a quality-control simulation check, as well as data collection and analysis. Specifics of the FAMCAT’s development, vetting and validation add to the growing literature describing MCAT in PRO measurement.

Methods

As, the general aspect of the FAMCAT’s development have been detailed previously,¹² the methodology outlined here pertains only to the MCAT aspects of the instrument’s development. As was noted in that document, the protocol was approved by the institution’s IRB. Participants provided oral informed consent and signed a HIPAA form during all data collection stages. Participants in both the calibration sample and validation sample are the same as those described in Cheville et al. (2021). The essential details are presented below.

Participants

Item Bank Calibration Sample

As reported in Cheville et al.,¹² a total of 2,341 patients hospitalized on medical services with at least one chronic condition or readmitted following a surgical complication from the [redacted for review] were recruited over a 13-month interval (May 2016 to June 2017). Among this sample, the data from 2,270 could be extracted from the EHR. The final sample size was 2,060 for item calibration. Due to the large item bank, data collection for item calibration was conducted in four batches using a common item linking design. Four batches were used to ensure that each patient did not respond to more than 100 items. Table 1 presents the sample size per batch before and after initial data cleaning. Approximately 10% of the patient data were deleted due to missing responses (i.e., if they had missing responses on 20 or more items). The final sample size was 2,060 for item calibration

Table 1.

Sample size per batch during data collection in the calibration phase

Batch	Original sample size	After cleaning	Reasons
1	630	563	Missing responses on at least 20 items
2	542	490	Missing responses on at least 10 items
3	555	500	Missing responses on at least 9 items
4	543	507	Missing responses on at least 20 items
Total	2270	2060

Validation Sample

Cheville et al.¹² reported that the FAMCAT’s validation initial sample consisted of 2,154 patients who were recruited using the same sampling and recruitment strategy as was used for the calibration sample. However, the sample’s size was reduced to 2,048 as 106 had not signed the institution’s data use agreement. Of these, the data of 1,930 was successfully extracted.¹² As shown in Cheville et al. (2021), ¹² the calibration and validation sample did not differ significantly in their demographic characteristics. There were some differences in their clinical characteristics depending on the time of sampling. For instance, the calibration sample had a larger proportion of patients diagnosed with neoplasms, whereas the validation cohort had a larger proportion of patients diagnosed with diseases of the circulatory system. 495 patients completed the FAMCAT on two or more occasions, with their information presented in Weiss et al. ¹⁰ The current paper analyzed 2,083 CAT sessions, which include repeated measures on the 1,930 patients. Because each CAT was administered independently, we also treated them as independent during our analysis, as we intend to merely document the features of FAMCAT such as average test length and testing time.

Instrument

The FAMCAT is a PRO measure that provides scores on Applied Cognition (107 Likert-scale items in the bank), Daily Activity (108 items), and Mobility (111 items) domains to help identify an appropriate rehabilitative care plan for post-acute care patients. Note that items in the Applied Cognition domain are self-report items rather than traditional cognitive performance items that are scored as correct or incorrect. All response options were coded as 1, 2, 3, and 4 for use in the item response theory (IRT) analyses.

The item banks for each scale were expanded from the Activity Measure for Post-Acute Care (AM-PAC) item banks¹¹. Details regarding item bank enrichment are described in Cheville et al.¹² In brief, 44 AM-PAC items were removed for lack of relevance to hospital settings, and 101 new items were added across three domains.

Data Collection

In the calibration stage, due to the large item bank, data collection for item calibration was conducted in four batches using a common item linking design, as illustrated in Table 2. That is, as the calibration sample was drawn over a period of time, the first 630 patients drawn in the sample formed batch 1, and the second 542 patients drawn in the sample formed batch 2, etc. Each batch of items was administered to a separate group of patients. The sample sizes per batch were close but not the same due to varying patient availability on different days in the hospital. The sample size per batch after data cleaning reached 500 except for batch 2, which aligns with the recommended sample size for psychometric analysis.⁵⁰ In IRT, linking is the process of placing test items on a common scale even though different items were administered to different groups of examinees. ^45,47 This is accomplished using a small set of items (the linking items) that are included in the data collection for each group. Eight linking items were selected for each subscale, and they were used in all batches of data collection. The numbers of unique items are also provided in Table 2. The percentage of linking items was 22% in batch 1 (i.e., 24 out of 109 items) or 25% in the remaining 3 batches, which is within the recommended range of 20 to 40% of linking items.⁴⁸ A good set of linking items per scale should provide an information curve of the same shape as the scale information. That is, the distances between the scale information curve (i.e., sum of item information from all items within the scale) and the total information of the linking items needed to be minimized. The methodology used for selecting linking items is described in Section 1 of the online Supplementary Material.

Table 2.

Illustration of common-item linking design

	Linking Items			Unique Items
Batch 1	8	8	8	28	28	30
Batch 2	8	8	8				24	24	24
Batch 3	8	8	8							24	24	24
Batch 4	8	8	8										23	24	25

Note. 𝜃₁, 𝜃₂, and denote Applied Cognition, Daily Activity, and Mobility, respectively.

Items were administered via Qualtrics^a interface. Research assistants read items to the patients and entered their responses into a tablet computer. Items within batches were categorized into blocks based on their content domain. Then the order of blocks within batches as well as the order of items within blocks were completely randomized to balance out order effects.

In the validation stage, items from FAMCAT were read to the patients by research coordinators, who then collected their oral responses and entered them into an iPad. FAMCAT items were administered through the FastTest System^b, a Web-based item banking and test delivery system designed to deliver conventional and IRT-based adaptive tests through any device that can access the Internet.

Exploratory and Confirmatory Item Factor Analysis

Although the items were written to cover three separate domains such that a three-factor structure was expected, an exploratory item factor analysis was conducted to provide empirical evidence on the factor structure of the item bank. Rotated 1-factor, 2-factor, and 3-factor solutions were produced using oblique Crawford-Ferguson Quartimax rotation available in flexMIRT.¹² This rotation assumes that factors are correlated, while intending to produce a near simple structure such that each item loads on one factor. Technical results from the factor analysis are presented in Section 5 of the online Supplementary Materials file.

After the number of factors was determined based on Akaike and Bayesian Information Criteria (AIC, BIC), a multidimensional graded response model (MGRM; see Section 2 of the Supplementary Material for the model used) was fitted to the combined data in a confirmatory approach. Full-information maximum likelihood estimation was used in flexMIRT to obtain model parameters and their standard errors.

By pooling together all patients’ response data from four batches, concurrent calibration was used, as it has been demonstrated to be more effective than separate per batch calibration followed by post-hoc linking ¹⁶; the latter has been identified as potentially suffering from linking errors. IRT item parameter estimation was conducted using flexMIRT.¹⁵ During this calibration process, when some item response options received no or low endorsement, the responses of that option, if any, were combined into the responses of the next higher option. This item-level response collapse greatly increased the stability of calibration without significantly affecting the resulting item parameter estimates.¹⁷

Variable-length FAMCAT Design

Two critical components of MCAT are the item selection rule and the stopping rule. The first determines how items will be selected sequentially from the bank for a patient, whereas the second determines when the assessment terminates (see Section 3 of the Supplementary Materials file for technical details of the MCAT design and implementation). In the FAMCAT design, the first three items were selected randomly with one from each domain. Then, the subsequent items were selected with the goal of simultaneously maximizing measurement precision across all three subscales. Specifically, the constraint-weighted D-optimal method,^{9, 18, 19} where “D” stands for the determinant of the Fisher information matrix, was used. The D-optimal method is computationally fastest among the available item selection methods for multidimensional CAT ^{18, 19}, and content balancing was added to maintain the content validity of the assessment by ensuring that the number of items selected from each subscale would fall between pre-specified lower and upper bounds.^{2, 20} Due to the imbalance of information across the three domains, adding content balancing constraints was necessary to avoid fewer items being selected from less informative domains (i.e., Applied Cognition), compromising the measurement precision from those domains. Maximum a posteriori (MAP) θ estimation with an informative prior was used to obtain interim latent trait estimates.²¹

A joint stopping rule ⁹ was used to terminate the FAMCAT. Within the joint rule, the primary stopping criterion is when the determinant of the Fisher information matrix exceeds a certain lower bound, implying that the volume of the confidence ellipsoid around the θ estimates $\widehat{(\theta })$ is sufficiently small (known as the SE-rule). This is equivalent to keeping the measurement error of a multidimensional trait satisfactorily low, hence the trait estimates can be precise and trustworthy. The secondary stopping criterion was when the change of two sequential estimates of $\widehat{\theta }$ was small, meaning that the interim update of $\widehat{\theta }$ during CAT reaches a stationary point such that administering additional items does not change the point estimates (known as the CT-rule). This secondary stopping rule serves to further improve the efficiency of the CAT by terminating the test when no more available items in the bank carry enough information to reduce measurement error and change the point estimates of θ.

For the operational FAMCAT, a cutoff of 477.25 was used for the SE-rule, and a cutoff of 0.01 was used for the CT-rule. That is, for a single patient’s FAMCAT, if the determinant of the three-dimensional Fisher information matrix exceeded 477.25, the test stopped; otherwise, if the maximum absolute change in two consecutive latent trait (θ) estimates was lower than .01, the test stopped; otherwise, the test continued until the maximum test length of 60 items was reached. 477.25 corresponds to the volume of the 95% confidence ellipsoid of θ that has a radius of 1.0 ^{9, 21}. This is roughly equivalent to a standard error of 0.411 for each unidimensional θ. These termination rules were implemented based on results from a series of Monte-Carlo simulations with the three MCAT item banks.⁹

To illustrate the fidelity of the item selection algorithm, a preliminary simulation study using the FAMCAT item bank was conducted. Persons’ true θ was simulated from a multivariate normal distribution with a mean vector of 0 and covariance matrix of

$$\text{Σ}=\left(\begin{array}{ccc}1& 0.6& 0.46\\ 0.6& 1& 0.83\\ 0.46& 0.83& 1\end{array}\right),$$ (1)

which was obtained from the calibration sample. As shown, Applied Cognition (the first of the three scales) correlated moderately with Daily Activity (r = 0.6) and Mobility (r = 0.46) whereas Daily Activity and Mobility were unsurprisingly highly correlated (r = 0.83). The simulated sample size was 3,000.

Results

Item bank Calibration

Table 3 presents the exploratory item factor analysis fit indices per batch. As shown, the 3-factor solution consistently produced the best relative fit indices across all four batches as it yielded the smallest AIC and BIC. Therefore, the three-dimensional GRM was used to estimate all item parameters.

Table 3.

Exploratory item factor analysis fit indices per batch

Data	Num. of Factors	AIC	BIC	−2LL
Batch 1	1	76469.98	78215.58	75663.98
	2	73559.44	75772.84	72537.44
	3	72793.42	75470.29	71557.42
Batch 2	1	58544.30	60045.9	57828.3
	2	55582.56	57482.62	54676.56
	3	54685.96	56980.3	53591.96
Batch 3	1	67249.37	68775.06	66525.37
	2	64199.57	66125.65	63285.57
	3	63197.44	65519.69	62095.44
Batch 4	1	64731.55	66241.13	64017.55
	2	61556.28	63463.34	60654.28
	3	60719.42	63019.73	59631.42

In terms of absolute model data fit, in the concurrent calibration using MGRM, the limited information M₂ statistic and M₂-based root mean square error of approximation (RMSEA) could not be computed because the contingency table and the marginal table were too large. This was due to the large item bank and the polytomous items used in the study. Instead, because each item loaded only on one factor, we fitted the GRM on different domains and batches of data separately, and successfully obtained M₂ and RMSEA statistics. As shown in Table 4, although the M₂ statistics were all significant at the .01 level, except the Daily Activity scale in batch 1, the M₂-based RMSEA were all in the acceptable range (≤ .05); ^{42, 43}. Hence, the significant M₂ statistic may be due to its inadequate chi-square distribution approximation. Prior research also found discrepancies in the conclusions based on M2 and M2-based RMSEA;⁴⁴ therefore, simulation studies are needed to thoroughly compare the performance of M2 and M₂-based RMSEA for the GRM with large item banks.

Table 4.

Absolute model fit results per scale and batch

	Scale	Items	M2 (df)	M2 based RMSEA
Batch 1	Applied cognitive	36	4509.05** (3471)	0.02
	Daily Activity	35	4170.71 (4438)	0.00
	Mobility	38	12717.63** (6178)	0.04
Batch 2	Applied cognitive	32	5510.67** (2762)	0.05
	Daily Activity	32	6668.10** (3889)	0.04
	Mobility	32	6738.15** (4432)	0.03
Batch 3	Applied cognitive	32	5052.85** (2762)	0.04
	Daily Activity	32	7764.49** (4339)	0.04
	Mobility	32	9349.76** (4339)	0.05
Batch 4	Applied cognitive	31	5478.25** (2474)	0.05
	Daily Activity	31	6462.15** (3975)	0.04
	Mobility	33	7548.668**(4719)	0.03

^**Statistically significant at p ≤.01.

In addition, we also checked two different item-level fit indices, marginal χ^{2 34} and S-χ^{2 35}. For the former case, none of the items were flagged as significant, implying that they all fit the model well. For the latter, about one third of the items showed significant S-χ². However, a recent study concluded that S-χ² had inflated false positive rates especially with a small sample size and a long test. ³⁶

To evaluate if there was excessive covariance between two or more items after accounting for the latent trait, the local dependence (LD) statistic was computed.³⁴ The LD diagnostic statistics approximately follow a chi-square distribution. Extreme values of 10 or higher imply violation of local independence, whereas borderline values between 5 to 10 may either indicate local dependence or be a result of sparseness in one or more response categories. ³⁷ Based on the cutoff, only 2% of item pairs had LD χ² larger than 10, and 2% of item pairs with LD χ² between 5 and 10. An inspection of the item content verified that none of those item pairs shared the same wording (i.e., “Difficult” vs. ‘Help”). The three items that had high LD χ² with all other items are:

• How much DIFFICULTY do you currently have understanding the things you read in books, magazines, or the newspaper? (Applied Cognition)

• How much DIFFICULTY do you currently have applying spreads to bread using a knife? (Daily Activity)

• How much DIFFICULTY do you currently have washing and drying your hands? (Daily Activity))

The exposure rates of these three items were all less than 5%, hence it is not concerning that the excess local dependence will adversely affect the final score reliability. However, future uses of the scale may consider eliminating these three items from further use.

Table 5 presents the mean, standard deviation, minimum, and maximum of all item parameters and their standard errors. They are all in a reasonable range. Following are four example items with the highest discrimination, lowest discrimination, highest average difficulty, and lowest average difficulty, respectively:

Table 5.

Descriptive statistics of FAMCAT item bank (items calibrated from a 3-dimensional graded response model)

	Test length				SE
Item parameters	α				SE(α)	SE ( β1 )	SE ( β2 )	SE ( β3 )
Mean	1.96	−4.89	−2.68	−0.83	0.19	0.41	0.23	0.17
Standard Deviation	0.60	1.59	1.69	1.72	0.06	0.20	0.12	0.07
Minimum	0.40	−9.08	−8.01	−4.98	0.06	0.12	0.08	0.06
Maximum	3.75	1.59	2.91	4.54	0.39	1.19	0.93	0.56

Note: The item discrimination (α) is aggregated over all three dimensions because each item has only has one non-zero α because of the simple multidimensional structure.

• How much DIFFICULTY do you currently have descending 3 to 5 steps without a handrail?

• How much DIFFICULTY do you currently have washing and drying your hands?

• How much DIFFICULTY do you currently have taking part in strenuous activities (e.g., running 3 miles, swimming half mile, etc.)?

• How much DIFFICULTY do you currently have washing and drying your hands?

Figure 1 shows the scale information functions for the three FAMCAT scales. Items in all three domains were more informative in the lower range of the θ continuum (i.e., −4 to 0), implying that these items can better differentiate patients with varying levels of low cognition, low mobility, and low daily activity, but they might not be able to differentiate normal persons well. Moreover, the scales differed substantially in the amount of information. Among the three subscales, items in the Mobility domain have highest psychometric information and precision, followed by the Daily Activity domain, whereas items in the Applied Cognition domain are least informative/precise. All three scale information functions deviated from the flat information function in the range θ = ±4 that is ideal for a CAT implementation.

Figure 1.

Scale information for each latent trait dimension

MCAT Simulation Results

Figure 2 shows the trajectories of the root mean squared error (RMSE) of θ as a function of test length, comparing D-optimal versus random selection (upper panel), and D-optimal with content balancing versus D-optimal without content balancing (lower panel). RMSE is defined as the square root of the average squared difference between estimated and true θ, and smaller RMSE indicates that estimates are closer to the true values. As shown from Figure 2, D-optimal item selection substantially outperformed random selection, and adding content balancing further reduced RMSE across all three dimensions.

Figure 2.

RMSE of θ s from three item selection methods using real item banks

(a) D-optimal (solid curves) vs. random selection (dashed curves)

(b) Constraint-weighted D-optimal (dashed curves) vs. D-optimal (solid curves)

Figure 3 presents the θ update history for two patients from administration of the FAMCAT to illustrate the logic of variable-length MCAT. The upper panel is the data from a patient with estimated $\widehat{\theta }$ = (0.44, 0.05, 0.96) whereas the lower panel is the data from a patient with $\widehat{\theta }$ (−1.86, −1.26, −0.71). In each panel, different marker types (and colors) denote the items selected from each of the three subscales, and the y-axis denotes the point estimated $\widehat{\theta }$, whereas the bar represents the 95% confidence interval. During an MCAT, items from three subscales are alternatively selected from the bank. When an item is selected from a certain subscale, the confidence interval of $\widehat{\theta }$ from that corresponding subscale drops quickly.

Figure 3.

Interim θ and its confidence interval update history for two real patients. “AC” denotes Applied Cognition, “DA” denotes Daily Activity, and “M” denotes Mobility.

(a) Patient 1 with $\widehat{\theta }$= (0.44, 0.05, 0.96)

(b) Patient 2 with $\widehat{\theta }$ = (−1.86, −1.26, −0.71)

However, because θ was estimated with Bayesian MAP with an informative prior, when an item from one subscale is administered, the $\widehat{\theta }$ from all subscales are also updated,⁹ although the confidence intervals from non-administered subscales do not decrease as quickly. Furthermore, the confidence intervals shrink much faster at the beginning and stay almost constant toward the end of the CAT. Interesting to note is that when $\widehat{\theta }$ = (0.44, 0.05, 0.96). it takes many more items to shrink the confidence interval to a small length. Hence the test length for someone with normal cognition, mobility, and daily activity is expected to be long, given the current characteristics of the item bank. Figure 3 shows that Patient 1 required 27 items whereas Patient 2 completed the FAMCAT in only 15 items. A simulation study was conducted to compare the performance of MCAT versus unidimensional CAT using the item bank described above; details are in Section 4 of the Supplementary Material.

Validation Sample Results

Among all the administered CATs, 95.6% (i.e., 1,992 out of 2,083) of patients had their CATs terminated by the primary SE-rule, implying that the pre-specified measurement precision was satisfied. 0.7% of patients (i.e., 15 out of 2,083) had their CATs terminated by the secondary CT-rule, implying that their latent trait estimate reached a stationary point such that administering additional items would not further improve precision. The remaining 3.6% had full length CAT (i.e., maximum length) administrations of 60 items. Table 6 presents the descriptive statistics of the test length and testing time, broken down by the termination criteria. On average, the FAMCAT stopped at around 24 items. For those tests stopped by the SE-rule, the average length was around 22 items (or an average of seven items per scale), whereas for those few tests stopped by the CT-rule, the average length was much longer--around 45 items. This is not surprising as the CT-rule is considered as a secondary rule, so it is much more difficult to satisfy, and it is only active when the primary SE-rule is not effective. Table 7 shows the descriptive statistics of the test length and standard error of measurement per scale, again broken down by the termination criteria. As shown, 7 to 8 items were administered to each patient on average. The measurement error reached 0.23–0.24 and 0.34 for Mobility and Daily Activity respectively, while it was higher for the Applied Cognition domain, 0.58. For patients whose tests were terminated by the CT rule, their average test length was much longer, and their measurement error was also much higher across all three domains.

Table 6.

Descriptive statistics of test length and testing time from live FAMCAT

Variable	N	Mean	SD	Min	Max
Test length
Total	2,083	23.60	10.76	15	60
SE-rule	1,992	22.05	7.98	15	59
CT-rule	15	45.27	11.49	27	59
Testing time (minutes)
Total	2,083	5.94	3.11	1.47	35.87
SE-rule	1,992	5.75	2.94	1.47	35.87
CT-rule	15	7.78	3.22	4.80	17.60
Maximum length	25	10.66	3.46	5.38	26.52

Table 7.

Mean and standard (in parentheses) of test lenght and standard error of measurement per scale from live FAMCAt

Variable	Test Length			SE
	Applied Cognition	Daily Activity	Mobility	Applied Cognition	Daily Activity	Mobility
Total	7.42 (3.65)	7.89 (3.56)	8.29 (3.77)	0.58 (0.17)	0.34 (0.09)	0.24 (0.07)
SE-rule	6.89 (2.68)	7.39 (2.68)	7.77 (2.91)	0.56 (0.14)	0.34 (0.05)	0.23 (0.04)
CT-rule	15.53 (3.68)	14.67 (3.89)	15.07 (4.08)	0.94 (0.52)	0.87 (0.41)	0.74 (0.41)
Maximum length	19.78 (0.67)	19.74 (0.44)	20.49 (0.50)	0.90 (0.35)	0.45 (0.19)	0.26 (0.13)

Figure 4 provides further evidence regarding the characteristics of patients whose tests were stopped by one of the three reasons. Different markers in Figure 4 denote patients in each of the three categories. As shown, consistent across all three subscales, those patients whose test either reached the maximum length or were stopped by the CT-rule were on the higher end of the θ scale. This is where the test information (see Figure 1) was lowest; hence the SE-rule could not be satisfied for them. This observation is reassuring, not only because the percentages of patient in these two categories (i.e., CT-rule and maximum length) were very small, but also because these groups of patients are not the target audience of the FAMCAT. Regarding testing time presented in Table 6, the average time was 5.94 minutes, which is quite efficient for in-hospital administration, compared to an average of 19.4 minutes if patients take roughly 96 to 109 items without adaptation.

Figure 4.

The determinant of Fisher information as a function of θ from the three subscales separately: Applied Cognition, Daily Activity, and Mobility

Discussion

This paper reviewed the development and implementation phases of the FAMCAT, the first multi-domain functional computerized PRO measure focused on medically ill patients in acute care settings. The primary goal of FAMCAT is to generate patients’ scores quickly and precisely on three domains—applied cognition, daily activity, and mobility—thereby providing timely assistance in matching patients’ needs with ability-matched care/discharge plans in what has historically been a fragmented and inefficient process.²³

Therefore, FAMCAT, if integrated into the EHR as a routinized functional measurement, has the potential to overcome the bottleneck to timely service provision and discharge planning because it may be used to replace human resource-intensive information collection and triage. An overview of the FAMCAT conceptual model and item development is described in Cheville et al.¹². The convergent validity of FAMCAT in terms of its score crosswalks with an 8-item PROMIS physical function short form, as well as its predictive validity in terms of predicting 30-day hospital readmission and discharge to institutional post-acute care, are presented in Marfeo et al.²⁴ and Keeney et al. ²⁵, respectively.

In the development and calibration phases, data collection was carefully planned such that patients’ response burden was not excessive. Each patient answered roughly 90 items, among which 24 were linking items to establish a common scale across four batches of items. A thorough psychometric analysis was conducted to evaluate the factor structure and to calibrate item parameters for use in the FAMCAT for scaling and scoring.

The FAMCAT was designed considering both precision and efficiency. A constraint weighted D-optimal item selection method was used to not only maximize aggregated Fisher information but also balance item selection for the three subscales. A quality-control simulation study was conducted to demonstrate that this item selection algorithm successfully reduced measurement errors compared to alternative methods (i.e., Figure 2). With respect to efficiency, a variable-length multidimensional CAT was considered and a joint stopping rule was implemented.⁹ Although from live FAMCAT results, it appears that the secondary CT-rule was only effective on 1% of patients, it affected about 5% of simulated patients in a prior simulation study.⁹ Hence, it is still recommended to include the secondary rule as a safeguard against unnecessary administration of additional items that provide little information, thereby improving test efficiency. A comparison study between MCAT and UCAT suggested that MCAT can, on average, save about 18% of items using the FAMCAT item bank. Note, however, that additional measurement precision brought by MCAT over UCAT only emerges when an informative prior is used in MAP θ estimation.

Further Research

The precision and efficiency of MCAT can be further enhanced by including collateral information, such as item response times.^{26, 27} In the health measurement domain, response time (RT) is often used to measure cognitive functioning, particularly in research on aging.^{28, 29} In those applications, RTs are usually collected from timed, target stimuli tasks in which respondents are instructed to respond as quickly as possible.³⁰ As a result, RTs are used as a proxy for the underlying latent trait such as cognitive ability. Despite these widespread applications, RT use has been mainly limited to the test level, but item-level RTs can also provide useful collateral information to improve measurement precision. In particular, in the CAT context, a maximum Fisher information per time unit (MFIT) method³¹ can be applied and the primary idea is to divide the Fisher information for an item by the expected RT on that item, i.e.,

$$\frac{\text{Fisher}\text{information}\text{for}\text{item}j}{\text{Expected}\text{response}\text{time}\text{on}\text{item}j}$$

By selecting an item that maximizes the above ratio, as much information as possible can be accumulated in a unit of time. Thus, if the average testing time is considered as an indicator of test efficiency instead of test length, this MFIT approach is preferred. In addition, given the potential correlation between an individual’s latent speed and a latent trait, RTs can also be used in the interim update of θ via a Bayesian MAP to further reduce the measurement error of θ. A recent study²⁷ using the real item bank showed that integrating RTs in the MCAT design would lead to 35% savings of testing time and 15% savings of test length.

Limitations

The limitations of the study include that it only considers a between-item multidimensional structure, and future studies should also consider the feasibility of MCAT with other types of multidimensional structures. Furthermore, the stopping rule considered herein focuses only on measurement precision. If the goal of assessment is to crudely classify patients into different categories, such as for fast screening, then classification CAT ³³ might be preferred to further improve test efficiency by reducing test length. In terms of the application of the MGRM for analyzing self-report items, the discrepancy between the M₂ and M₂-RMSEA for evaluating absolute model data fit needs to be further studied in scenarios when test length is long, sample size is moderately small, and there is systematic missing data. Moreover, as the S-χ² item fit index does not perform adequately, a more suitable item fit index for MGRM needs to be developed.

Another limitation worth highlighting is the that the MGRM used herein implicitly assumes the latent traits follow a normal distribution spanning from negative infinity to positive infinity although in our estimation we constrained the range of θ to be from −3 to 3. This implies that the measured constructs are bipolar, for instance, Applied Cognition falls along a continuum from far below average to far above average.³⁸ Some research has shown that the bipolar assumption may be less justified in clinical assessment because the lowest end of the continuum is not below average but the absence of symptom. ³⁹ Hence IRT models that assume a unipolar latent variable may be better suited for assessing clinical constructs⁴⁰ in a nonclinical sample where a large proportion of the sample will report absence of symptoms. The distinction between unipolar and bipolar constructs is less concerning in our application, however, because on one hand, our data were collected from hospitalized patients, most of whom show some level of symptoms. On the other hand, prior research demonstrated that the item parameters are rather robust to mild violations of the latent trait normality assumption.⁴⁶ That said, future research should still consider IRT models proposed specifically for unipolar traits ^{38, 41} and compare the relative model fit to the MGRM.

It should also be noted that the analyses of this study were based on a specific hospital population--those hospitalized on medical services with at least one chronic condition or readmitted following a surgical complication. While questions about the generalizability of this sample can be raised, the diagnoses of the patients--e.g., chronic disease, cancer, cardiac--make up a large portion of the population of all general hospitals.

Conclusions

The development of FAMCAT has the potential to have a major, positive impact on the clinical arenas as it provides a timely data-driven, standardized means to determine patients’ rehabilitative needs. FAMCAT scores offer a basis for ability-matching care plans such that mobility technicians or personal care assistants can implement simple but effective preservation care plans.^{32, 49} The psychometric evidence presented herein demonstrates that FAMCAT yields precise, high-quality information for care determination. It also imposes minimum burden as reflected by short test length and testing time. Hence, the tool has the potential to empower clinicians to screen for disablement across relevant domains to make appropriate treatment and referral decisions.

This is the first study to demonstrate the adequate precision and efficiency of FAMCAT in a hospital setting. The development included both the calibration and validation phases. From a psychometric perspective, our results are promising and may be informative for others who wish to develop similar MCATs. Clinically, FAMCAT scores can be used to quickly and precisely pinpoint patients’ rehabilitation care needs, and it has potential to revolutionize the treatment model for delivery of function-oriented care.

Explanation of Terminologies.

Unidimensional CAT:

An adaptive assessment assuming that the measured trait is unidimensional, i.e., a single latent trait. Items are selected sequentially from the item bank for each patient to maximize measurement precision and minimize response burden.

Multidimensional CAT:

An adaptive assessment assuming that the measured trait is multidimensional, i.e., multiple correlated latent traits, such as daily activity and mobility.

Multidimensional item response theory (MIRT):

In psychometrics, item response theory (IRT) is a paradigm for the design, analysis, and scoring of instruments measuring abilities, aptitudes, attitudes, and other latent traits. While classical IRT models assume the latent trait is unidimensional, MIRT extends to modeling multiple, correlated latent traits.

Between-item multidimensional models:

Also known as simple-structure MIRT models, they assume that each item measures only one latent trait.

Item bank:

The entire bank of items from which items covering all content domains are selected to form a CAT for each patient.

Abbreviations of Terms

(AM-PAC)

Activity Measure for Post-Acute Care

(CAT)

Computerized adaptive testing

(EHR)

Electronic health record

(FAMCAT)

Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test

(MCAT)

Multidimensional computerized adaptive testing

(UCAT)

Unidimensional computerized adaptive testing

(MFIT)

Maximum Fisher information per time unit

(IRT)

Item response theory

(MIRT)

Multidimensional item response theory

(MGRM)

Multidimensional graded response model

(SE-rule)

Standard error rule

(CT-rule)

Change of θ rule

(PRO)

Patient reported outcome

(PROMIS)

Patient-Reported Outcome Measurement Information System

(RMSE)

Root mean squared error

(MAP)

Maximum a posteriori trait estimation