Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2022 Oct 1.
Published in final edited form as: Clin Neuropsychol. 2021 Mar 17;36(7):1844–1859. doi: 10.1080/13854046.2021.1900400

Validation of the Personality Assessment Inventory (PAI) scale of scales in a mixed clinical sample

Kaley Boress a, Owen J Gaasedelen b, Anna Croghan a, Marcie King Johnson a,c, Kristen Caraher a, Michael R Basso d, Douglas M Whiteside e
PMCID: PMC8474121  NIHMSID: NIHMS1739082  PMID: 33730975

Abstract

Objective:

This exploratory study examined the classification accuracy of three derived scales aimed at detecting cognitive response bias in neuropsychological samples. The derived scales are composed of existing scales from the Personality Assessment Inventory (PAI). A mixed clinical sample of consecutive outpatients referred for neuropsychological assessment at a large Midwestern academic medical center was utilized.

Participants and Methods:

Participants included 332 patients who completed study’s embedded and free-standing performance validity tests (PVTs) and the PAI. PASS and FAIL groups were created based on PVT performance to evaluate the classification accuracy of the derived scales. Three new scales, Cognitive Bias Scale of Scales 1–3, (CB-SOS1-3) were derived by combining existing scales by either summing the scales together and dividing by the total number of scales summed, or by logistically deriving a variable from the contributions of several scales.

Results:

All of the newly derived scales significantly differentiated between PASS and FAIL groups. All of the derived SOS scales demonstrated acceptable classification accuracy (i.e. CB-SOS1 AUC = 0.72; CB-SOS2 AUC = 0.73; CB-SOS3 AUC = 0.75).

Conclusions:

This exploratory study demonstrates that attending to scale-level PAI data may be a promising area of research in improving prediction of PVT failure.

Keywords: Personality Assessment Inventory, Symptom Validity Test, Performance Validity Test, Neuropsychological Assessment, Scale of Scales

Performance validity tests (PVTs) allow neuropsychologists to evaluate the likelihood that a patient’s evaluation results reflect his/her true ability and thus play an important role in neuropsychological assessment. PVTs refer specifically to tasks designed to mimic traditional neuropsychological tests but provide information on patient task engagement (also referred to as “performance credibility”) rather than measuring a particular cognitive skill, while symptom validity tests (SVTs) help gauge whether an individual is accurately portraying his/her symptoms (Heilbronner et al., 2009; Larrabee, 2012). Use of validity measures are common practice in neuropsychological evaluations (Sweet et al., 2015); however, there is a dearth of SVTs that assess the exaggeration of cognitive symptoms.

The Personality Assessment Inventory (PAI; Morey, 2007) is widely used to evaluate psychiatric symptoms and personality characteristics and includes several embedded SVTs, including the Negative Impression Management (NIM), Positive Impression Management (PIM), Inconsistency (ICN), and Infrequency (INF) scales. While these SVTs are frequently employed by neuropsychologists, research suggests that they generally produce poor classification accuracy when used to differentiate between patients who passed and failed PVTs (Gaasedelen et al., 2017; Martin et al., 2015; Whiteside et al., 2009). One possible explanation for these results is that the current PAI SVT scales were created to assess psychiatric rather than cognitive response bias, thus highlighting a need for more specific cognitive response bias measures, as have been developed using other well-established tests of psychopathology like the Minnesota Multiphasic Personality Inventory-2-Restructured Form (MMPI-2-RF) (Ben-Porath & Tellegen, 2008; Tellegen & Ben-Porath, 2011). Another newer PAI scale named the Neuro-Item Sum scale, which was designed to be sensitive to genuine neurological complaints (Morey, 2020), has not been examined in the context of PVT failure. The scale was originally developed for a doctoral dissertation (Keiski, 2007) to attempt to determine if PAI items could detect self-reported neurological difficulties, and included items 3, 12, 38, 52, 92, 112, 115, 147, 152, 155, and 158. However, very limited research has been conducted on this scale (McCredie & Morey, 2018). McCredie and Morey (2018) reported that the scale distinguished individuals with “organic mental disorder” from other patients; however, it did not add incremental validity to the SOM scale.

Recently, Gaasedelen et al. (2019) developed the first measure of cognitive complaints, as defined by PVT failure, for the PAI, named the Cognitive Bias Scale (CBS). The CBS utilized item-level responses from PAI items that best discriminated between individuals who failed PVTs from those who passed PVTs. The 10-item scale was validated in a sample of primarily clinical, but also some forensic, neuropsychological referrals. The CBS was found to outperform other PAI symptom validity and clinical scales in predicting PVT failure (Gaasedelen et al., 2019). The CBS was further supported as a measure of cognitive response bias in a study utilizing a military sample, which found similar specificity and sensitivity rates as the original validation study (Armistead-Jehle et al., 2020) when predicting PVT failure.

A different scale development strategy is to develop a scale comprised of existing scales on the PAI rather than a scale comprised from item-level responses. For example, the Malingering Index (MAL; Morey, 2007) was constructed from several PAI scales and was designed to evaluate response bias on psychiatric issues. This scale development methodology has been successful for determining response bias for psychiatric conditions in simulated designs, with the MAL demonstrating excellent classification accuracy across studies (Hawes & Boccaccini, 2009; Liljequist et al., 1998; Morey & Lanier, 1998; Sumanti et al., 2006). Further, the MAL, among other PAI scales, has demonstrated initial evidence for validity as a measure of noncredible responding with regard to pain symptoms (Hopwood et al., 2010). This methodological approach of combining multiple PAI scales closely mirrors a PVT design approach in the PVT literature where several different cognitive measures are combined together to create embedded PVTs (e.g. Schutte et al., 2011; Suhr & Boyer, 1999; Whiteside et al., 2015; Wolfe et al., 2010). However, no studies have been published examining potential PAI cognitive bias scales based on this methodology of using scale-level data. Such a scale would expand validity testing options for neuropsychologists and could be used in conjunction with more traditional PVTs in order to assess symptom and performance validity in patients presenting with neuropsychological concerns.

The purpose of the present exploratory study was to derive one or more novel scales comprised of existing PAI scales (with initial analysis focused on the previously mentioned NIM, SOM, DEP, ANX, SCZ, and SUI scales) aimed at detecting cognitive bias. Both theory and empirical evidence were used to create composite scales that would ideally demonstrate adequate discriminability between individuals in the PASS (passed the PVTs per outlined criteria) and FAIL (individuals who failed the PVT per outlined criteria) groups.

Methods

Participants and procedure

The study was approved by institutional IRB. Retrospective data analysis identified 332 consecutively referred patients who completed the PAI and other study measures as part of a neuropsychological evaluation, including at least one free standing and one embedded PVT from a larger database (N = 408). The participants completed their outpatient neuropsychological evaluations at a large Midwestern academic medical center between 2014 and 2019. Exclusion criteria included failure to complete freestanding PVTs (n = 38), failure to complete PAI (n = 12), non-content based random responding on the PAI Infrequency (INF) and Inconsistency (ICN) scales (i.e. INF > 74 or ICN > 72, per PAI manual) (n = 25), and having a dementia diagnosis (n = 1).

All participants included in the study were at least 18 year of age. The participants consisted of individuals who were referred for a clinical neuropsychological evaluation due to concern for suspected neurological dysfunction, psychological conditions, learning disorders, genetic conditions, neurodevelopmental disorders, and memory/other cognitive concerns. The evaluation requests were typically for differential diagnosis, establishing a baseline, and providing treatment recommendations. None of the participants were forensic referrals or had clear external incentives. While we could not entirely rule out some type of external incentive that participants did not reveal (e.g. seeking stimulants), none of the participants in this sample acknowledged such an incentive. Further, none of the participants had potentially confounding diagnoses such as moderate to severe dementia or intellectual disability. In fact, these types of cases were typically not administered the PAI. The majority of the participants in the study were Caucasian (84%) and female (55%) (mean age = 37.28 SD = 15.96, mean education = 13.9 SD = 2.73). The sample included individuals who met diagnostic criteria for depression (33%), ADHD (15%), and anxiety disorders (14%) (American Psychiatric Association, 2013). Additionally, 11% of the sample had diagnoses of mild traumatic brain injury (TBI), 4% had diagnoses of chronic pain and 3% had diagnoses of severe TBI. There were no significant differences found for race, gender, age, or education, based on t-tests (age, education) and Chi-Square analysis (race, gender) between the PASS and FAIL groups (see Table 1). Additional Chi-Square analyses for the top three psychiatric (i.e. depression, ADHD, and anxiety) and medical diagnoses (i.e. mild TBI, chronic pain, severe TBI) between PASS and FAIL groups revealed no significant findings (p > .05). It is relevant to note, however, that these diagnoses were provided as a result of the neuropsychological evaluation and that it is plausible that such differences in failure rates may exist based on the initial referral question; however, the initial referral question was never included in the dataset and thus cannot be statistically analyzed. That being said, all referrals were from internal or external treating providers for clinical questions (e.g. differential diagnosis, treatment planning) and not from attorneys, workers compensation, or other forensic-type sources as these were screened out in the clinic’s triage process.

Table 1.

Participant characteristics.

Failed PVTs (N = 34) Passed PVTs (N = 298) Test statistic p-value
Demographics and other characteristics M (SD), Range M (SD), Range
Caucasian 73.53% 85.23% χ2 = 2.31 13
Male 41.18% 45.64% χ2 = 0.10  .75
Age 41.24 (15.45), 18–68 37.40 (15.74), 18–75 t = −1.37  .18
Education 13.70 (2.58) 13.90 (2.79) t = 0.42  .67
Open in a new tab

Note. N = Number of participants; PVT = Performance Validity Test; % = percentage of the total sample.

M = mean; SD = standard deviation.

Similar to prior research on MMPI-2-RF and PAI classification accuracy (Ben-Porath & Tellegen, 2008; Gaasedelen et al., 2019; Gervais et al., 2007; Tellegen & Ben-Porath, 2011), participants were divided into two groups based on whether or not they scored below established cutoffs on a specified number (2) of freestanding and embedded PVTs. The methodology chosen for the current study is supported in the literature as best practice for optimizing identification of invalid performance (Larrabee, 2003, 2008, 2014; Victor et al., 2009). The issue of determining criterion groups based on the number of failed PVTs continues to be an issue debated in the literature (Schroeder et al., 2019). Although various approaches could be used for determining noncredible performance (e.g. failure of one PVT, failure of 2+ PVTs, eliminating all 1 PVT failure participants), this study was designed to be consistent with the methodology of Gaasedelen et al. (2019), allowing for more direct comparison between newly developed PAI scales. Further, the common standard clinically for non-credible performance requires two failed PVTs (rather than one) and prior research suggests the classification accuracy of identifying non-credible performance increases with the number of failed PVTs (Bilder et al., 2014; Larrabee, 2008; Martin et al., 2015). Conversely, eliminating participants who fail one PVT risks spectrum bias and artificially overestimating sensitivity and specificity, thus limiting clinical utility (Lijmer et al., 1999; Schroeder et al., 2019). Thus, prior studies and recent PVT literature emphasize use of a two PVT failure approach for determining non-credible responding (Boone et al., 2002; Larrabee, 2008; Victor et al., 2009). Therefore, participants who failed two or more PVTs (including a free standing PVT) were placed in the FAIL group (n = 34) while all other participants were included in the PASS group (n = 298). The PASS group included 20 participants who failed only one standalone PVT and no other embedded PVTs, and 45 participants who failed one embedded PVTs but no freestanding PVTs. Out of the 20 participants who failed only one standalone PVT and no embedded PVTs, 15 failed the TOMM while 5 failed the Dot Counting Test.

Criterion measures

Test of Memory Malingering

The Test of Memory Malingering (TOMM) is a forced choice visually based performance validity test (Tombaugh, 1997) that is commonly used for assessing credible performance across a variety of populations (Donders, 2005; Love et al., 2014; Rees et al., 1998). Further the TOMM is commonly used as a criterion measure in PVT and SVT validation studies (e.g. Gervais et al., 2007; Whiteside et al., 2009; Young et al., 2011). For the purpose of the current study, cutoffs for Trial 2 and the retention trial were based off of the manual recommendations (cut off = < 45) (Tombaugh, 1997). However, the trial one cut off score (<42) was based on recommendations from Bauer et al. (2007) and O’Bryant et al. (2008) and supported by multiple studies which found that a cut score of <42 was optimal for predicting credible/non-credible performance (Denning, 2012; Hilsabeck et al., 2011; Martin et al., 2020; Perna & Loughan, 2013).

Dot counting task

The dot counting task is a freestanding performance validity measure developed by Andre Rey (Rey, 1941) and validated as a measure of noncredible performance by Boone et al. (2002). Similar to the TOMM, the Dot Counting Task is commonly used in SVT and PVT research and has been shown to be useful in detecting noncredible responding in both psychiatric and non-clinical populations (Boone et al., 2002; McCaul et al., 2018). For the purpose of the current study a cut off score of 14 was used based on previous research (McCaul et al., 2018).

Embedded PVTs

Several embedded PVTs were included in the study. Embedded PVTs are measures included within established neuropsychological tests that provide information about the credibility of the individual’s performance. The specific embedded PVTs included in the current study are Reliable Digit Span (cutoff <7; Greiffenstein et al., 1994); Wisconsin Card Sorting Loss of Set (cutoff > 2; Greve et al., 2009), Rey Complex Figure Test (RCFT)-Copy (< 23 raw score and RCFT-Recognition raw score < 16; Whiteside et al., 2011), Trail Making Test-Part B, (> 120 seconds; Busse & Whiteside, 2012), Continuous Performance Test-2nd edition (>30 omission and commission errors; Busse & Whiteside, 2012), Judgment of Line Orientation, (<18 total score; Whiteside et al., 2011), and California Verbal Learning Test-2nd edition, Forced Choice (<15 raw score; Delis et al., 2000).

Scale development

In order to determine which PAI scales are most likely to comprise a PAI Cognitive Bias Scale of Scales (CBS-SOS), we first examined the NIM, somatic complaints (SOM), depression (DEP), anxiety (ANX), schizophrenia (SCZ), and suicidal ideation (SUI) scales. These scales were chosen because recent research (Whiteside et al., 2020) found they best predicted performance levels on the Test of Memory Malingering (TOMM; Tombaugh, 1996). Further, analogous scales on the MMPI-2 which were identified by Larrabee (2003) previously, provided additional empirical support for these particular PAI scales. These results suggest that high levels of negative responding relate to reduced performance validity and are similar to outcomes reported by several earlier studies (Larrabee, 2003; Sumanti et al., 2006; Whiteside et al., 2010, 2012).

Exploratory analyses were conducted to determine if the FAIL group could be distinguished from the PASS group using different combinations of weighting techniques for existing PAI scales. There were three analyses conducted: two driven primarily by theory and prior research, and one driven completely by the data. The first approach identified the variables with highest classification accuracy from a previous study by this group of authors (Whiteside et al., 2020), summed them together and divided them by the total number of scales included to develop the Cognitive Bias Scale of Scales-1 (CB-SOS1). As noted above, the specific scales used in the first analysis were NIM, SOM, DEP, ANX, SCZ and SUI based on findings from Whiteside et al. (2020).

The second analysis used the same scales from the first analyses but ran them in a logistic regression equation and utilized the beta weights from that equation to create the second scale, called the Cognitive Bias Scale of Scales-2 (CB-SOS2). Such an approach could confer an advantage to the unit-weighted approach for the CB-SOS1 as a multiple logistic regression model accounts for the shared variance of the variables, which theoretically could improve the overall classification accuracy.

The final analysis started by examining the area under the curve (AUC) for all of the PAI scales and subscales to identify which scales to incorporate into the measure. In contrast with the first two scale selections, which were based on previous empirical findings, this approach utilized an exclusively empirical approach to the selection process. The scales with the highest individual AUC (i.e. AUC >0.68) included the SOM_C (Somatic-Conversion), DEP_P (Depression-Physiological), SOM_S (Somatic-Somatization), ANX_P (Anxiety-Physiological), SCZ (Schizophrenia), NIM (Negative Impression Management), and the PAR_R (Paranoia-Resentment) scales and subscales. Each of these scales were summed together and divided by the total number of scales included to the make the Cognitive Bias Scale of Scales 3 (CB-SOS3). Convergent, divergent, and diagnostic validation statistics were then calculated for each CB-SOS utilizing ROC analyses and correlations with NIM and the Neuro-Item Sum.

Results

Cognitive Bias Scale of Scales-1

Table 2 provides a summary description of each of the derived scales. For the first SOS, the T-scores from the following scales, NIM, SOM, DEP, ANX, SCZ and SUI, were simply summed together and divided by 6. A t-test comparing mean differences between PASS and FAIL groups indicated significant differences in groups on CB-SOS1 (Cohen’s d = 0.81; Table 3). An ROC curve examining classification accuracy into the FAIL group indicated adequate classification accuracy, with an area under the curve (AUC) of 0.72 (Table 4).

Table 2.

Description of derived scales.

Scale Names Description
CB-SOS1 Derived from the NIM, SOM, DEP, ANX, SCZ, and SUI scales in which all scales were summed together and divided by six.
CB-SOS2 Logistically derived variable from the NIM, SOM, DEP, ANX, SCZ, and SUI scales.
CB-SOS3 Derived from the SOM_C, DEP_P, SOM_S, ANX_P, SCZ, NIM, and the PAR_R in which all scales were summed together and divided by seven.
Open in a new tab

Note. CB-SOS = Cognitive Bias Scale of Scales; NIM = Negative Impression Management; SOM = Somatic Concerns; ANX = Anxiety; DEP = Depression; SCZ = Schizophrenia; SUI = Suicidal Ideation; SOM_C = Conversion subscale; SOM_S = Somatization subscale; DEP_P = Depression Physiological subscale; ANX_P = Anxiety Physiological subscale; PAR_P = Resentment subscale.

Table 3.

Descriptive statistics and group comparison tests for symptom validity tests.

Failed PVTs (N = 34) Passed PVTs (N = 298)
Measures M (SD) M (SD) t-statistic p-value
NIM** 66.38 (16.30) 56.64 (12.60) −3.37   .002
Neuro-Item Sum** 16.24 (4.87) 13.66 (4.33) −2.95   .005
SOM*** 72.97 (17.54) 61.04 (13.87) −3.83 <.001
DEP*** 76.91 (15.51) 65.33 (16.22) −4.10 <.001
ANX*** 72.26 (14.48) 62.95 (13.87) −3.56 <.001
SCZ** 68.85 (15.04) 60.90 (13.05) −2.96   .005
SUI 62.50 (18.71) 56.25 (17.06) −1.86  .07
CB-SOS1*** 69.99 (12.13) 60.52 (11.65) −4.33 <.001
CB-SOS2*** 4.99 (0.93) 4.21 (0.83) −4.67 <.001
CB-SOS3*** 69.96 (11.14) 59.81 (10.05) −5.08 <.001
Open in a new tab

Note. N = number of participants; CB-SOS = Cognitive Bias Scale of Scales; NIM = Negative Impression Management; SOM = Somatic Concerns; ANX = Anxiety; DEP = Depression; SCZ = Schizophrenia; SUI = Suicidal Ideation; M = Means; SD = Standard deviations; PVT = Performance Validity Test.

*

p < 0.05.

**

p < 0.01.

***

p < 0.001.

Table 4.

Area under the curve.

Measure AUC
NIM 0.68
SOM 0.70
ANX 0.68
DEP 0.70
SCZ 0.66
SUI 0.62
CBS-SOS1 0.72
CBS-SOS2 0.73
CBS-SOS3 0.75
Open in a new tab

Note. Number of participants (n) =342; CB-SOS = Cognitive Bias Scale of Scales; NIM = Negative Impression Management; SOM = Somatic Concerns; ANX = Anxiety; DEP = Depression; SCZ = Schizophrenia; SUI = Suicidal Ideation; AUC = Area Under the Curve.

Cognitive Bias Scale of Scales-2

For the second SOS, the same scales in the first analyses were entered into a logistic regression equation predicting group status (i.e. PASS or FAIL) and the beta weights for each scale in the regression equation were utilized to create a combined logistically derived variable, CB-SOS2. A t-test comparing mean differences between PASS and FAIL groups suggested significant differences in groups on CB-SOS2 (Cohen’s d = 0.93; Table 3). An ROC curve examining classification accuracy into the FAIL group indicated adequate classification accuracy (AUC = 0.74; Table 4).

Cognitive Bias Scale of Scales-3

For the final SOS, the T-scores from the following scales and subscales, SOM_C, DEP_P, SOM_S, ANX_P, SCZ, NIM, and the PAR_R, were simply summed together and divided by the total number of scales included (i.e. 7). A t-test comparing mean differences between PASS and FAIL groups suggested significant differences in groups on CB-SOS3 (Cohen’s d = 1.00; Table 3). An ROC curve examining classification accuracy into the FAIL group indicated adequate classification accuracy (AUC = 0.75; Table 4).

Diagnostic and convergent and divergent validity statistics

Table 5 provides a summary of the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). The base rate for PVT failure for PPV and NPV is 10%, which was the failure rate in the study. Data on a variety of cut-off scores can be found in Table 5, in addition to the recommended optimal cut-off scores (in bold print), based upon the highest sensitivity while maintaining at least 90% specificity. Briefly, a cut score of ≥78 for the CB-SOS1 (specificity = 0.90, sensitivity = 0.29), yielded high specificity while maintaining adequate sensitivity for the purposes of the scale. Using a cut score of ≥5.3 for the CB-SOS2 (specificity = 0.90, sensitivity = 0.41), also yielded high specificity; however, the sensitivity was mildly improved compared to the aforementioned sensitivity of the CB-SOS1. For the CB-SOS3, a cut score of ≥ 74 (specificity = 0.92, sensitivity =0.38) resulted in comparable specificity and sensitivity to the CB-SOS2. CB-SOS1 had a slightly higher cut score (T = 78) compared to CB-SOS3 (T = 74) while CB-SOS2 used the logistically derived cut off. All of the CB-SOS scales demonstrated high correlations with NIM (i.e. 0.79 to 0.85), and slightly lower correlations with the Neuro-ItemSum (i.e. 0.60 to 0.65), providing evidence that they are tapping into a negative response bias, as opposed to genuine neurological complaints. An ROC analysis for NIM predicting classification into the PASS and FAIL groups produced an AUC of 0.68, which is not above the 0.70 threshold for adequate classification. When an ROC analysis was conducted with the Neuro-Item Sum predicting classification into the PASS and FAIL groups, it demonstrated classification accuracy below the acceptable threshold, (AUC = 0.65), suggesting against its use as a measure of cognitive feigning. Thus, the CS-SOS scales all outperformance NIM and Neuro-Item Sum scales in our sample.

Table 5.

Sensitivity, specificity, positive predictive power, and negative predictive power.

Measure Cut score SN SP PPP NPP
CBS-SOS1 ≥70 0.44 0.79 0.19 0.93
≥75 0.38 0.86 0.23 0.92
≥78 0.29 0.90 0.24 0.92
≥82 0.18 0.96 0.33 0.91
≥86 0.12 0.99 0.50 0.91
CBS-SOS2 ≥4.9 0.49 0.79 0.21 0.93
≥5.1 0.47 0.84 0.25 0.93
≥5.3 0.41 0.90 0.32 0.93
≥5.7 0.29 0.95 0.40 0.92
≥6.2 0.06 0.99 0.33 0.90
CBS-SOS3 ≥69 0.53 0.80 0.23 0.94
≥71 0.44 0.84 0.24 0.93
≥73 0.38 0.89 0.29 0.93
≥74 0.38 0.92 0.35 0.93
≥77 0.35 0.96 0.48 0.93
≥84 0.12 0.99 0.57 0.91
Open in a new tab

Note. Number of participants (n) = 332; CBS-SOS = Cognitive Bias Scale of Scales; SN = Sensitivity; SP = Specificity; PPP = Positive Predictive Power; NPP = Negative Predictive Power; Base rate for calculating PPP and NPP is 10%, which is the base rate for failure in this sample; Bold Print = Recommended cutoff.

Discussion

The purpose of this paper was to examine the classification accuracy of three newly derived Personality Assessment Inventory (PAI) scales to assess cognitive response bias in neuropsychological patients. Since the CBS (Gaasedelen et al., 2019) recently demonstrated the feasibility of an item-level scale in the PAI to evaluate patients’ cognitive response bias, a next logical step would be to determine if a scale comprised of scale-level data would have equivalent or superior classification accuracy. To examine this question, this study developed three different scales based on different assumptions. The first PAI scales used in the construction of the Cognitive Bias Scale of Scales (CS-SOS1 and 2) were six measures previously found to have a significant relationship with TOMM performance (Whiteside et al., 2020), including NIM, SOM, ANX, DEP, SCZ, and SUI. As described above, these scales were conceptually similar to MMPI-2 scales found by Larrabee (2003) to relate to noncredible performance. Both weighted item and logical regression approaches were explored in this study.

The first scale used a simple equal weighting of the six PAI scales noted above (CB-SOS1). The second scale (CB-SOS2) used the same six PAI scales but utilized logistic regression analysis to create a measure based on the beta weights. The final scale (CB-SOS3) did not rely on previous research but instead calculated AUCs for each PAI scale and subscale and used the scales/subscales with the best individual classification accuracy based on this analysis (SOM_C, DEP_P, SOM_S, ANX_P, SCZ, NIM, and the PAR_R).

All three of the newly derived scales performed similarly. The final scale (CBS-SOS3) had a slightly higher AUC with the scales utilized in this construction being purely empirically derived from the available data. Of course, this begs the question of whether this would replicate using a different dataset. By a small margin, the second-best overall classification accuracy was CB-SOS2, which used logistic regression methodology to calculate beta weights to differentially weight the PAI scales. Previous PVT research has demonstrated the utility of logistic regression in creating new embedded PVTs that are difficult to coach for the CVLT-II (Persinger et al., 2018); Wisconsin Card Sorting Test (WCST; Suhr & Boyer, 1999), and the Wechsler Memory Scale-III (WMS-III; Ord et al., 2008; Schutte et al., 2011). It should also be noted that the CB-SOS scales outperformed existing PAI validity scales like the NIM and newly developed scales for detecting cognitive complaints (Neuro-Item Sum) in terms of classification accuracy for PVT failure. This finding also provides evidence that Neuro-Item Sum is measuring genuine symptom complaint, as opposed to non-credible symptoms. The current study provides support for this methodology, given the classification accuracy using this approach.

Interestingly, when setting specificity near 90% (Larrabee & Berry, 2007), the sensitivity of CB-SOS2 of 41% was highest in the current study even though the AUC was slightly lower than CB-SOS3, which had a sensitivity of 38% at this level of specificity (see Table 5). Further, the sensitivity of the CB-SOS1 was lower than ideal at only 29% even though it too had an adequate overall classification accuracy. Overall, the CB-SOS2 and 3 had similar or slightly higher sensitivity when compared to other embedded measures of performance and symptom validity (Gervais et al., 2007; Schroeder et al., 2012). For example, the Response Bias Scale from the MMPI-2-RF was found to have a sensitivity of 25% when specificity was set to 95% (Gervais et al., 2007). Similarly, the Improbable Frequency Scale from the Structured Interview of Reported Symptoms (SIRS), another measures of noncredible endorsement of psychiatric symptoms that is commonly used in forensic settings, demonstrated a sensitivity of 38% when specificity was set to 91% (Rogers et al., 2009). Further, when specificity was set to above 90% for Reliable Digit Span, an embedded measure of performance validity from the Weschler Adult Intelligence Scale, sensitivity ranged from 30 to 35% (Schroeder et al., 2012). In sum, the CB-SOS 2 and 3 scales displayed comparable or improved sensitivity in this study compared to existing embedded SVT measures.

When compared to other personality assessment validity measures, the classification accuracy of CB-SOS2 and CB-SOS3 are similar to or slightly higher than the original CBS (Gaasedelen et al., 2019), which has an AUC = .72 and a sensitivity of 37% when specificity was set at 90%. However, in a cross validation of the CBS in a military sample completed by Armistead-Jehle and colleagues, the classification accuracy (AUC = .79) was higher than the three CB-SOS scales (Armistead-Jehle et al., 2020). The CB-SOS3 slightly outperformed the CBS in overall classification accuracy when compared to the original CBS validation study (AUC = .75) while the CB-SOS2 slightly outperformed the CBS in sensitivity (SN = 41%) when specificity was 90% for the original CBS validation study. The sensitivity of the CB-SOS3 (SN = 38%) was very similar to that of the CBS. However, the CB-SOS2 &3 sensitivities were below the sensitivity of the CBS in the military cross-validation sample (sensitivity = 55%) when specificity was set to 92%. While CB-SOS1 had a similar overall classification accuracy to the CBS (AUC = .72), the sensitivity was noticeably lower (SN = 29%) when the specificity was set at 90%. It should be noted that high specificity is very important in PVTs because it minimizes the risk of false positive identification of cognitive response bias, thus sensitivity should be evaluated in the context of cutoff scores with high specificity. Thus, if a patient exceeds the cut-off, the clinician has reasonable data to suspect cognitively based response bias is occurring. When the cut-off is not exceeded, there is a greater risk of a false negative result, which is true of many embedded PVTs. Currently, given the exploratory nature of this study, additional research and validation of these scales is recommended before using them in clinical settings.

Limitations and future directions

There are several limitations associated with any study of this type. The first is the construction of the criterion groups, which is based on the PVTs utilized. Although the study used the standard for identification of non-credible responding with two or more PVTs failures (Larrabee, 2003, 2014; Victor et al., 2009), inclusion of other PVTs (e.g. Word Memory Test) may have resulted in differences in the group membership. A more liberal or conservative approach to the number of PVT failures as the criterion could alter classification accuracy. Notably, the inclusion of individuals who fail one PVT in the PASS group remains a point of debate in the literature, with some evidence that failure of one PVT can compromise neuropsychological test results (Fox, 2011) in at least some populations. However, others who note that the patient population needs to be considered when considering one PVT failure (e.g. Schroeder et al., 2019), and non-forensic populations such as the one in this study are advised to not consider one PVT failure evidence of noncredible performance. Using a criterion of one PVT failure to define noncredible performance may reduce false negatives, particularly in forensic populations, given the nature of the clinical sample and the non-forensic nature of the participants, attempting to minimize false positives was particularly desirable. Additionally, use of two PVT failures criterion reflects the clinical standard in neuropsychology, is broadly consistent with recent PVT literature and allows for a more direct comparison with the newly developed CBS scale (Bilder et al., 2014; Boone et al., 2002; Larrabee, 2003, 2008; Martin et al., 2015; Victor et al., 2009). Nonetheless, this remains an area of debate within the field, and other methodological choices may have led to differences in classification accuracy. Thus, further research with alternative criteria for PVT failures is recommended. Specific to this study, future studies may wish to examine the classification accuracy of the CB-SOS scale using more liberal or conservative PASS/FAIL criteria. The present sample also did not have any forensic referrals. Thus, future research designed to examine the CB-SOS scales in various forensic populations such as mild traumatic brain injury (mTBI), disability, academic accommodations, Veteran, and criminal cases would likely be beneficial. Additionally, future research should explore the utility of the current methodology for detecting non-credible responding in populations with high levels of psychological dysfunction or somatization (e.g. chronic illness, post-concussion syndrome, chronic pain etc.). Finally, future research comparing the CB-SOS scales with the existing supplemental scales noted in the PAI Plus manual (Morey, 2020), such as the Hong Malingering Scale (Hong & Kim, 2001) would likely be beneficial.

The current sample was primarily Caucasian, given its geographic location in a primarily rural midwestern state. Thus, further research examining individuals from different backgrounds would be beneficial. Finally, the specific criterion measures included were selected because of their common use as criterion measure in PVT and SVT validation studies (e.g. Gervais et al., 2007; Whiteside et al., 2009; Young et al., 2011). However, there are other commonly used PVTs in neuropsychological testing which potentially may have resulted in differential classification accuracy. Future studies should examine the classification accuracy of the CB-SOS measures using other PVT criterion measures (e.g. Word Memory Test). Finally, future studies could compare the CB-SOS scales to current MMPI-3 scales aimed at detecting cognitive bias (e.g. Response Bias Scale of the MMPI-3; Ben-Porath & Tellegen, 2020; Gervais et al., 2007).

In spite of these limitations, this exploratory study provides initial support for this methodology of deriving embedded cognitive bias scales on the PAI. The CB-SOS2 and 3 have sensitivity levels generally consistent with other embedded SVTs. Further, this study provides additional support for the use of logistic regression methods in developing SVTs that are combinations of other measures. The primary advantage of this approach is that this is difficult to coach for litigating patients (Larrabee, 2008). Additional research utilizing the methodology in this study to derive scale-based SVTs may be fruitful, based on the present study.

Footnotes

Disclosure statement

No potential conflict of interest was reported by the authors.

References

  1. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). American Psychiatric Publishing, Inc. [Google Scholar]
  2. Armistead-Jehle P, Ingram PB, & Morris NM (2020). Personality assessment inventory cognitive bias scale: Validation in a military sample. Archives of Clinical Neuropsychology, 35(7), 1154–1161. 10.1093/arclin/acaa049 [DOI] [PubMed] [Google Scholar]
  3. Bauer L, O’Bryant SE, Lynch JK, McCaffrey RJ, & Fisher JM (2007). Examining the Test of Memory Malingering trial 1 and Word Memory Test immediate recognition as screening tools for insufficient effort. Assessment, 14(3), 215–222. 10.1177/1073191106297617 [DOI] [PubMed] [Google Scholar]
  4. Ben-Porath YS, & Tellegen A (2008). MMPI-2-RF: Manual for administration, scoring and interpretation. University of Minnesota Press. [Google Scholar]
  5. Ben-Porath YS, & Tellegen A (2020). MMPI-3 technical manual. University of Minnesota Press. [Google Scholar]
  6. Bilder RM, Sugar CA, & Hellemann GS (2014). Cumulative false positive rates given multiple performance validity tests: Commentary on Davis and Millis (2014) and Larrabee (2014). The Clinical Neuropsychologist, 28(8), 1212–1223. 10.1080/13854046.2014.969774 [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Boone KB, Lu P, Back C, King C, Lee A, Philpott L, Shamieh E, & Warner-Chacon K (2002). Sensitivity and specificity of the Rey Dot Counting Test in patients with suspect effort and various clinical samples. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 17(7), 625–642. 10.1016/S0887-6177(01)00166-4 [DOI] [PubMed] [Google Scholar]
  8. Busse M, & Whiteside DM (2012). Detecting suboptimal cognitive effort: Classification accuracy of the Conner’s Continuous Performance Test, Brief Test of Attention, and Trail Making Test. The Clinical Neuropsychologist, 26(4), 675–613. 10.1080/13854046.2012.679623 [DOI] [PubMed] [Google Scholar]
  9. Delis DC, Kramer JH, Kaplan E, & Ober B (2000). The California Verbal Learning Test (2nd ed.). The Psychological Corporation. [Google Scholar]
  10. Denning JH (2012). The efficiency and accuracy of the Test of Memory Malingering Trial 1, errors on the first 10 items of the Test of Memory Malingering, and five embedded measures in predicting invalid test performance. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 27(4), 417–432. 10.1093/arclin/acs044 [DOI] [PubMed] [Google Scholar]
  11. Donders J (2005). Performance on the Test of Memory Malingering in a mixed pediatric sample. Child Neuropsychology, 11(2), 221–227. 10.1080/09297040490917298 [DOI] [PubMed] [Google Scholar]
  12. Fox DD (2011). Symptom validity test failure indicates invalidity of neuropsychological tests. The Clinical Neuropsychologist, 25(3), 488–495. 10.1080/13854046.2011.554443 [DOI] [PubMed] [Google Scholar]
  13. Gaasedelen OJ, Whiteside DM, Altmaier E, Welch C, & Basso MR (2019). The construction and the initial validation of the Cognitive Bias Scale for the Personality Assessment Inventory. The Clinical Neuropsychologist, 33(8),1467–1484. 10.1080/13854046.2019.1612947 [DOI] [PubMed] [Google Scholar]
  14. Gaasedelen OJ, Whiteside DM, & Basso M (2017). Exploring the sensitivity of the Personality Assessment Inventory symptom validity tests in detecting response bias in a mixed neuropsychological outpatient sample. The Clinical Neuropsychologist, 31(5), 844–856. 10.1080/13854046.2017.1312700 [DOI] [PubMed] [Google Scholar]
  15. Gervais RO, Ben-Porath YS, Wygant DB, & Green P (2007). Development and validation of a Response Bias Scale (RBS) for the MMPI-2. Assessment, 14(2), 196–208. 10.1177/1073191106295861 [DOI] [PubMed] [Google Scholar]
  16. Greiffenstein MF, Baker WJ, & Gola T (1994). Validation of malingered amnesia measures with a large clinical sample. Psychological Assessment, 6(3), 218–224. 10.1037/1040-3590.6.3.218 [DOI] [Google Scholar]
  17. Greve KW, Heinly MT, Bianchini KJ, & Love JM (2009). Malingering detection with the Wisconsin Card Sorting Test in mild traumatic brain injury. The Clinical Neuropsychologist, 23(2), 343–362. 10.1080/13854040802054169 [DOI] [PubMed] [Google Scholar]
  18. Hawes SW, & Boccaccini MT (2009). Detection of overreporting of psychopathology on the Personality Assessment Inventory: A meta-analytic review. Psychological Assessment, 21(1), 112–124. 10.1037/a0015036 [DOI] [PubMed] [Google Scholar]
  19. Heilbronner RL, Sweet JJ, Morgan JE, Larrabee GJ, & Millis SR (2009). American Academy of Clinical Neuropsychology Consensus Conference Statement on the neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist, 23(7), 1093–1129. 10.1080/13854040903155063 [DOI] [PubMed] [Google Scholar]
  20. Hilsabeck RC, Gordon SN, Hietpas-Wilson T, & Zartman AL (2011). Use of Trial 1 of the Test of Memory Malingering (TOMM) as a screening measure of effort: Suggested discontinuation rules. The Clinical Neuropsychologist, 25(7), 1228–1238. 10.1080/13854046.2011.589409 [DOI] [PubMed] [Google Scholar]
  21. Hong SH, & Kim YH (2001). Detection of random response and impression management in the PAI: II. Detection indices. Korean Journal of Clinical Psychology, 20(4), 751–761. [Google Scholar]
  22. Hopwood CJ, Orlando MJ, & Clark TS (2010). The detection of malingered pain-related disability with the Personality Assessment Inventory. Rehabilitation Psychology, 55(3), 307–310. 10.1037/a0020516 [DOI] [PubMed] [Google Scholar]
  23. Keiski MA (2007). Use of the Personality Assessment Inventory (PAI) following traumatic brain injury [Doctoral Dissertation, University of Windsor; ]. ProQuest Dissertations Publishing. https://scholar.uwindor.ca/etd/4710 [Google Scholar]
  24. Larrabee GJ (2003). Exaggerated MMPI-2 symptom report in personal injury litigants with malingered neurocognitive deficit. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 18(6), 673–686. 10.1016/S0887-6177(02)00157-9 [DOI] [PubMed] [Google Scholar]
  25. Larrabee GJ (2008). Aggregation across multiple indicators improves the detection of malingering: Relationship to likelihood ratios. The Clinical Neuropsychologist, 22(4), 666–679. 10.1080/13854040701494987 [DOI] [PubMed] [Google Scholar]
  26. Larrabee GJ (2012). Performance validity and symptoms validity in neuropsychological assessment. Journal of the International Neuropsychological Society, 18(4), 625–630. 10.1017/S1355617712000240 [DOI] [PubMed] [Google Scholar]
  27. Larrabee GJ (2014). Minimizing false positive error with multiple performance validity tests: Response to Bilder, Sugar, and Hellemann (2014 this issue). The Clinical Neuropsychologist, 28(8), 1230–1242. 10.1080/13854046.2014.988754 [DOI] [PubMed] [Google Scholar]
  28. Larrabee GL, & Berry D (2007). Diagnostic classification studies and diagnostic validity (Larrabee GL, Ed.). Oxford. [Google Scholar]
  29. Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, & Bossuyt PM (1999). Empirical evidence of design-related bias in studies of diagnostic tests. JAMA, 282(11), 1061–1066. 10.1001/jama.282.11.1061 [DOI] [PubMed] [Google Scholar]
  30. Liljequist L, Kinder BN, & Schinka JA (1998). An investigation of malingering posttraumatic stress disorder on the Personality Assessment Inventory. Journal of Personality Assessment, 71(3), 322–336. 10.1207/s15327752jpa7103_3 [DOI] [PubMed] [Google Scholar]
  31. Love CM, Glassmire DM, Zanolini SJ, & Wolf A (2014). Specificity and false positive rates of the test of memory malingering, Rey 15-item test, and Rey word recognition test among forensic inpatients with intellectual disabilities. Assessment, 21(5), 618–627. 10.1177/1073191114528028 [DOI] [PubMed] [Google Scholar]
  32. Martin PK, Schroeder RW, & Odland AP (2015). Neuropsychologists’ validity testing beliefs and practices: A survey of North American professionals. The Clinical Neuropsychologist, 29(6), 741–776. 10.1080/13854046.2015.1087597 [DOI] [PubMed] [Google Scholar]
  33. Martin PK, Schroeder RW, Olsen DH, Maloy H, Boettcher A, Ernst N, & Okut H (2020). A systematic review and meta-analysis of the Test of Memory Malingering in adults: Two decades of deception detection. The Clinical Neuropsychologist, 34(1), 88–119. 10.1080/13854046.2019.1637027 [DOI] [PubMed] [Google Scholar]
  34. McCaul C, Boone KB, Ermshar A, Cottingham M, Victor TL, Ziegler E, Zeller MA, & Wright M (2018). Cross-validation of the Dot Counting Test in a large sample of credible and non-credible patients referred for neuropsychological testing. The Clinical Neuropsychologist, 32(6), 1054–1067. 10.1080/13854046.2018.1425481 [DOI] [PubMed] [Google Scholar]
  35. McCredie MN, & Morey LC (2018). Evaluating new supplemental indicators for the Personality Assessment Inventory: Standardization and cross-validation. Psychological Assessment, 30(10), 1292–1299. 10.1037/pas0000574 [DOI] [PubMed] [Google Scholar]
  36. Morey LC (2007). The Personality Assessment Inventory professional manual (2nd ed.). Psychological Assessment Resources. [Google Scholar]
  37. Morey LC (2020). The Personality Assessment Inventory Plus (PAI Plus) professional manual supplement. Psychological Assessment Resources. [Google Scholar]
  38. Morey LC, & Lanier VW (1998). Operating characteristics of six response distortion indicators for the Personality Assessment Inventory. Assessment, 5(3), 203–214. 10.1177/107319119800500301 [DOI] [PubMed] [Google Scholar]
  39. O’Bryant SE, Gavett BE, McCaffrey RJ, O’Jile JR, Huerkamp JK, Smitherman TA, & Humphreys JD (2008). Clinical utility of Trial 1 of the Test of Memory Malingering (TOMM). Applied Neuropsychology, 15(2), 113–116. 10.1080/09084280802083921 [DOI] [PubMed] [Google Scholar]
  40. Ord JS, Greve KW, & Bianchini KJ (2008). Using the Wechsler Memory Scale-III to detect malingering in mild traumatic brain injury. The Clinical Neuropsychologist, 22(4), 689–704. 10.1080/13854040701425437 [DOI] [PubMed] [Google Scholar]
  41. Perna RB, & Loughan AR (2013). Children and the Test of Memory Malingering: Is one trial enough? Child Neuropsychology, 19(4), 438–447. 10.1080/09297049.2012.731500 [DOI] [PubMed] [Google Scholar]
  42. Persinger VC, Whiteside DM, Bobova L, Saigal S, Vannucci MJ, & Basso MR (2018). Using the California Verbal Learning Test, Second Edition as an embedded performance validity measure among individuals with TBI and individuals with psychiatric disorders. The Clinical Neuropsychologist, 32(6), 1039–1053. 10.1080/13854046.2017.1419507 [DOI] [PubMed] [Google Scholar]
  43. Rees LM, Tombaugh TN, Gansler DA, & Moczynski NP (1998). Five validation experiments of the test of memory malingering (TOMM). Psychological Assessment, 10(1), 10–20. 10.1037/1040-3590.10.1.10 [DOI] [Google Scholar]
  44. Rey A (1941). L’Examenpsychologie dans las casd’encephalopathietraumatique. Archives de Psychologie, 23(112), 286–340. [Google Scholar]
  45. Rogers R, Payne JW, Berry DT, & Granacher RP (2009). Use of the SIRS in compensation cases: An examination of its validity and generalizability. Law and Human Behavior, 33(3), 213–224. 10.1007/s10979-008-9145-9 [DOI] [PubMed] [Google Scholar]
  46. Schroeder RW, Martin PK, Heinrichs RJ, & Baade LE (2019). Research methods in performance validity testing studies: Criterion grouping approach impacts study outcomes. The Clinical Neuropsychologist, 33(3), 466–477. 10.1080/13854046.2018.1484517 [DOI] [PubMed] [Google Scholar]
  47. Schroeder RW, Twumasi-Ankrah P, Baade LE, & Marshall PS (2012). Reliable digit span: A systematic review and cross-validation study. Assessment, 19(1), 21–30. 10.1177/1073191111428764 [DOI] [PubMed] [Google Scholar]
  48. Schutte C, Millis S, Axelrod B, & VanDyke S (2011). Derivation of a composite measure of embedded symptom validity indices. The Clinical Neuropsychologist, 25(3), 454–462. 10.1080/13854046.2010.550635 [DOI] [PubMed] [Google Scholar]
  49. Suhr JA, & Boyer D (1999). Use of the Wisconsin Card Sorting Test in the detection of malingering in student simulator and patient samples. Journal of Clinical and Experimental Neuropsychology, 21(5), 701–708. 10.1076/jcen.21.5.701.868 [DOI] [PubMed] [Google Scholar]
  50. Sumanti M, Boone KB, Savodnik I, & Gorsuch R (2006). Noncredible psychiatric and cognitive symptoms in a workers’ compensation “stress” claim sample. The Clinical Neuropsychologist, 20(4), 754–765. 10.1080/13854040500428467 [DOI] [PubMed] [Google Scholar]
  51. Sweet JJ, Benson LM, Nelson NW, & Moberg PJ (2015). The American Academy of Clinical Neuropsychology, National Academy of Neuropsychology, and Society for Clinical Neuropsychology (APA Division 40) 2015 TCN professional practice and ‘Salary Survey’: Professional practices, beliefs, and incomes of U.S. neuropsychologists. The Clinical Neuropsychologist, 29(8), 1069–1162. 10.1080/13854046.2016.1140228 [DOI] [PubMed] [Google Scholar]
  52. Tellegen A, & Ben-Porath YS (2011). Minnesota Multiphasic Personality Inventory-2-Restructured Form (MMPI-2-RF). University of Minnesota Press. [Google Scholar]
  53. Tombaugh TN (1996). The test of memory malingering. Multi-Health System. [Google Scholar]
  54. Tombaugh TN (1997). The test of memory malingering (TOMM): Normative data from cognitively intact and cognitively impaired individuals. Psychological Assessment, 9(3), 260–268. 10.1037/1040-3590.9.3.260 [DOI] [Google Scholar]
  55. Victor TL, Boone KB, Serpa JG, Buehler J, & Ziegler EA (2009). Interpreting the meaning of multiple symptom validity test failure. The Clinical Neuropsychologist, 23(2), 297–313. 10.1080/13854040802232682 [DOI] [PubMed] [Google Scholar]
  56. Whiteside DM, Clinton C, Diamonti C, Stroemel J, White C, Zimberoff A, & Waters D (2010). Relationship between suboptimal cognitive effort and the clinical scales of the personality assessment inventory. The Clinical Neuropsychologist, 24(2), 315–325. 10.1080/13854040903482822 [DOI] [PubMed] [Google Scholar]
  57. Whiteside DM, Dunbar-Mayer P, & Waters DP (2009). Relationship between TOMM performance and PAI validity scales in a mixed clinical sample. The Clinical Neuropsychologist, 23(3), 523–533. 10.1080/13854040802389169 [DOI] [PubMed] [Google Scholar]
  58. Whiteside DM, Gaasedelen OJ, Hahn-Ketter AE, Luu H, Miller ML, Persinger V, Rice L, & Basso M (2015). Derivation of a cross-domain embedded performance validity measure in traumatic brain injury. The Clinical Neuropsychologist, 29(6), 788–803. 10.1080/13854046.2015.1093660 [DOI] [PubMed] [Google Scholar]
  59. Whiteside DM, Galbreath J, Brown M, & Turnbull J (2012). Differential response patterns on the Personality Assessment Inventory (PAI) in compensation-seeking and non-compensation-seeking mild traumatic brain injury patients. Journal of Clinical and Experimental Neuropsychology, 34(2), 172–182. 10.1080/13803395.2011.630648 [DOI] [PubMed] [Google Scholar]
  60. Whiteside DM, Hunt I, Choate A, Caraher K, & Basso MR (2020). Stratified performance on the Test of Memory Malingering (TOMM) is associated with differential responding on the Personality Assessment Inventory (PAI). Journal of Clinical and Experimental Neuropsychology, 42(2), 131–141. 10.1080/13803395.2019.1695749 [DOI] [PubMed] [Google Scholar]
  61. Whiteside DM, Wald D, & Busse M (2011). Classification accuracy of multiple visual spatial measures in the detection of suspect effort. The Clinical Neuropsychologist, 25(2), 287–301. 10.1080/13854046.2010.538436 [DOI] [PubMed] [Google Scholar]
  62. Wolfe PL, Millis SR, Hanks R, Fichtenberg N, Larrabee GJ, & Sweet JJ (2010). Effort indicators within the California Verbal Learning Test-II (CVLT-II). The Clinical Neuropsychologist, 24(1), 153–168. 10.1080/13854040903107791 [DOI] [PubMed] [Google Scholar]
  63. Young JC, Kearns LA, & Roper BL (2011). Validation of the MMPI-2 Response Bias Scale and Henry-Heilbronner Index in a U.S. veteran population. Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists, 26(3), 194–204. 10.1093/arclin/acr015 [DOI] [PubMed] [Google Scholar]

RESOURCES