Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2015 Mar 1.
Published in final edited form as: Schizophr Res Cogn. 2014 Mar;1(1):e21–e26. doi: 10.1016/j.scog.2014.01.004

Virtual Reality Functional Capacity Assessment In Schizophrenia: Preliminary Data Regarding Feasibility and Correlations with Cognitive and Functional Capacity Performance

Stacy A Ruse 1, Philip D Harvey 2,3, Vicki G Davis 1, Alexandra S Atkins 1, Kolleen H Fox 4, Richard SE Keefe 1,5
PMCID: PMC4113005  NIHMSID: NIHMS577465  PMID: 25083416

Abstract

Introduction

Assessment of functional capacity is an intrinsic part of determining the functional relevance of response to treatment of cognitive impairment in schizophrenia. Existing methods are highly and consistently correlated with performance on neuropsychological tests, but most current assessments of functional capacity are still paper and pencil simulations. We developed a computerized virtual reality assessment that contains all of the components of a shopping trip.

Methods

We administered the Virtual Reality Functional Capacity Assessment Tool (VRFCAT) to 54 healthy controls and to 51 people with schizophrenia to test its feasibility. Dependent variables for the VRFCAT included time to completion and errors on 12 objectives and the number of times that an individual failed to complete an objective. The MATRICS Consensus Cognitive Battery (MCCB) and a standard functional capacity measure, the UCSD Performance-Based Skills Assessment-Brief (UPSA-B) were administered to the patients with schizophrenia.

Results

Patients with schizophrenia performed more poorly than healthy controls on 10/11 of the time variables, as well as 2/12 error scores and 2/12 failed objectives. Pearson correlations for 7 of 15 VRFCAT variables with MCCB composite scores were statistically significant.

Conclusion

These results provide support for the possibility of computerized functional capacity assessment, but more substantial studies are required.

Keywords: Virtual Reality, Cognitive Assessment, Computer Based Assessment, Schizophrenia, Functional Capacity

Introduction

Schizophrenia is marked by substantial impairments in everyday functioning in multiple domains (Harvey and Bowie, 2005). Achievement of functional milestones in areas such as full-time employment, independence in residence, and social functioning is reduced compared to both healthy people and other severe mental illnesses such as bipolar disorder (Harvey et al., 2010). Candidates for the causes of these impairments include cognitive deficits, impairments in the ability to perform the skills required to achieve success in everyday, negative symptoms and depression, health variables, and a variety of social, cultural, and environmental factors.

A recent development in research on the determinants of disability in schizophrenia has been performance- (Harvey et al., 2007) and interview-based (Keefe et al., 2006) measures of Functional Capacity (FC). Studies of performance-based assessments of FC have found that impairments on these measures predict failures to achieve milestones in vocational, residential, and social domains (Mausbach et al., 2010; Mausbach et al., 2013) in schizophrenia and bipolar disorder populations (Bowie et al., 2010; Depp et al., 2012). Whether everyday functioning is defined either by milestone achievement (Gould et al., 2012) or by ratings generated by high-contact informants (Bowie et al., 2008), impairments on measures of FC have typically been found to be more proximal to everyday functional deficits than cognitive impairments. Further, the correlation between performance on FC measures and neuropsychological (NP) tests has been remarkably consistent and substantial, typically r=0.60 or greater (Leifker et al., 2011).

The importance of valid and efficient assessment of FC has been increased by the requirement of the US Food and Drug Administration (FDA) that all studies attempting to demonstrate improvements in cognitive functioning induced by pharmacological or cognitive remediation means also provide evidence of functional relevance by “co-primary” measures (Buchanan et al., 2005; Buchanan et al., 2010). In a study of people with schizophrenia, performance-based assessments of FC, specifically the UCSD Performance-Based Skills Assessment (UPSA; Patterson et al., 2001) and the Test of Adaptive Behavior in Schizophrenia (TABS; Velligan et al., 2007), demonstrated substantially higher correlations with NP test performance than patient self-reports derived from interview-based measures (Green et al., 2011). Although use of high-contact informants, particularly clinicians, yields correlations with NP performance consistent with performance-based measures of FC (Keefe et al., 2006), many people with schizophrenia may not have access to high-contact clinicians (Patterson et al., 1997) and the use of informants other than close-contact caregivers or clinicians may yield questionable results (Poletti et al., 2012; Sabbag et al., 2011).

Functional capacity measures have also demonstrated high levels of test-retest reliability, minimal practice effects, and minimal missing data in large-scale clinical trials (Keefe et al., 2011). Despite these multiple strong features, there are some limitations to the current set of FC measures. These measures are delivered in a paper and pencil format, which is not practical for remote delivery or for simultaneous assessment of multiple cases. With the advent of remotely deliverable cognitive remediation therapy (CRT), in-person assessment of functional gains may not always be possible. Further, these measures are comprised of several functional tasks that are not required consistently across different cultures and do not have alternate forms (Velligan et al., 2012).

In an attempt to enhance the assessment of FC we have developed a computerized, immersive, and potentially remotely deliverable FC assessment referred to as the Virtual Reality Functional Capacity Assessment Tool (VRFCAT). The VRFCAT consists of a tutorial and 6 versions of 4 mini scenarios that include navigating a kitchen and planning a trip to the grocery store, catching a bus to a grocery store (selecting the correct bus and paying the correct fare), purchasing food at the grocery store, and returning home on a bus. Thus, this assessment strategy captures several of the functional domains of other FC measures: transportation, finances, household management, and planning. Further, the alternate forms are a unique feature of this assessment and the scenarios have the potential to be rapidly updated and cross-culturally adapted.

There have been previous efforts made to create computerized FC assessments. There is a long history of these tasks being used in aging populations in order to simulate functional demands that include either use of computer or interactive voice menus (Czaja and Sharit, 2003). Computerized FC assessments have also previously been employed in schizophrenia. For instance, a computerized version of the UPSA was recently developed, although this assessment currently requires an in-person examiner (Moore et al., 2013). Virtual reality assessments aimed at delusions of persecution have been developed as well. (Freeman, 2008). Further, Kurtz et al. (2007) developed a medication management assessment with a virtual apartment. These previous assessments are different from the current one because of the sequential, multi-task demands of a simulated shopping trip and the goal of wide coverage of functional domains in the VRFCAT.

In this paper we present the preliminary results from the development and initial feasibility study of the VRFCAT. The VRFCAT was administered to both healthy individuals and people with schizophrenia. In addition, schizophrenia patients were assessed with the MATRICS Consensus Cognitive Battery (MCCB) and the UCSD Performance-Based Skills Assessment-Brief (UPSA-B). This study was conducted in sequence, with the healthy control (HC) group assessed first in order to understand the feasibility of the task and the schizophrenia patients examined later. Therefore, the samples were not selected to be “matched” on demographics and there are some differences between the samples. Our analyses examined the differences in performance between the HC group and people with schizophrenia, as well as the correlations between a standard paper and pencil functional capacity measure (UPSA-B), cognitive performance (composite scores on a modified version of the MCCB), and performance on the VRFCAT in people with schizophrenia.

Methods

Subjects

Two different samples of subjects were compared. During the development phase, 102 healthy controls from Durham, North Carolina, were recruited. As this was a feasibility study only, a formal assessment of psychopathology was not performed although subjects were asked if they had received previous mental health treatment. The subjects were tested with 1 of 6 randomly selected versions of the assessment and then asked to return for re-test with a different randomly selected version 7 to 14 days later. All research participants provided signed, informed consent, and this research study was approved by the Western IRB. Healthy control subjects received $20.00 per visit for their time and effort in completing the VRFCAT. Ninety of those 102 returned for testing with a different version of the application. Due to an initial data management problem that was later rectified, only 69 of the 90 who returned had complete data sets. During our initial analysis of the data, two application errors were identified: First, we observed that one of the versions of the VRFCAT yielded significantly outlying data and did not perform in an equitable way to the other versions. As a result, we excluded this version from all subsequent analyses. Second, we identified a programing error resulting in inaccurate collection of the time to complete Objective 12. This variable was therefore excluded from subsequent analyses. In addition, two significant outliers were discovered and removed from data analysis, resulting in a HC sample size of n=54.

The patient sample was collected from one of the two sites participating in the Validation of Everyday Real-world Outcomes (VALERO) study, phase 2. All patients were recruited and assessed at the University of Miami Miller School of Medicine. All patients provided signed, informed consent, and this research study was approved by the local IRB. During the initial analysis of the data, one significant outlier was discovered and removed. In addition, four patients were removed from analysis due to rater administration errors, resulting in a patient sample size of n=51.

All patients with schizophrenia were administered the Structured Clinical Interview for the DSM-IV (SCID; First et al., 1995) by a trained interviewer, and diagnoses were subjected to a consensus procedure. Participants were excluded if they had a history of traumatic brain injury with unconsciousness >10 minutes, brain disease such as seizure disorder or neurodegenerative condition, or the presence of any DSM-IV-TR diagnosis on Axis I that would exclude the diagnosis of schizophrenia. None of the patients were experiencing their first psychotic episode. In order to capture a broad array of patients, substance abuse was not an exclusion criterion for patients but anyone who appeared intoxicated was rescheduled. Inpatients were not recruited, but patients resided in an assortment of unsupported, supported, or supervised residential locations. Patients received $25.00 for their time and effort above and beyond their compensation for participation in VALERO-II.

Procedure

All participants were examined with the VRFCAT. All patients were also examined with a performance-based assessment of NP abilities and FC. The VRFCAT was administered to patients after completion of a comprehensive assessment of a variety of aspects of cognition and everyday functioning, which is partially reported here.

VRFCAT Description

The VRFCAT was developed in order to measure four different functional abilities: checking for the availability of items to complete a recipe, taking a bus, shopping in a store, and managing currency. These scenarios were developed using immersive virtual reality (VR) technology. All participants received a brief tutorial, which included sample items similar to those from the test and practice in using the mouse and computer prior to their assessment. There were 12 different objectives, presented in Table 1. For each objective, the dependent variables were accuracy of performance and time to completion. For all objectives, participants who were unable to complete the objective within a pre-specified time period or without making a certain number of errors were automatically progressed to the next objective.

Table 1. VRFCAT Objectives.
Objective Description
1 Pick-up the Recipe
2 Search for Ingredients
3 Cross Off Correct Ingredients & Pick-up Bus Schedule
4 Pick-up the Billfold
5 Exit the Apartment
6 Get on the Bus to the Grocery Store
7 Pay for the Bus
8 Select an Aisle
9 Shop for Groceries
10 Pay for Groceries
11 Get on the Bus to go Home
12 Pay for the Bus
Open in a new tab

Performance-based Assessment for Patients

Neurocognition

We examined NP performance with a modified version of the MCCB (Nuechterlein et al., 2008). For this study, the MSCEIT social cognition measure was not included, as the VALERO investigators were interested in variables that were purely neurocognitive, and there is evidence that social cognition and neurocognition may be separate constructs (Ventura et al., 2013). We calculated the composite score generated by the MCCB computer program, which is based on 6 of the 7 domain scores, excluding social cognition. See Kern et al. (2011) for a description of the typical patterns of impairment seen in schizophrenia patients on the MCCB.

In addition to the modified version of the MCCB, participants in the patient group completed the Wide-Range Achievement Test, 3rd edition (WRAT-3; Wilkinson, 1993). This was done to ensure that all patients could read adequately to be assessed with the cognitive battery and also as an approximate index of intellectual functioning. Cases who received scores at less than the 6th grade level were excluded prior to any other assessments.

Functional Capacity

Patients' FC was assessed using the Brief version of the UCSD Performance-Based Skills Assessment (UPSA-B; Mausbach et al., 2008). The UPSA-B is a measure of FC in which patients are asked to perform everyday tasks related to communication and finances. During the Communication subtest, participants are required to role-play exercises using an unplugged telephone (e.g., emergency call; dialing a number from memory; calling to reschedule a doctor's appointment). For the Finance subtest, participants are required to count change, read a utility bill, and write a check for the bill. The UPSA-B requires approximately 10-15 minutes to complete, and raw scores are converted into a total score ranging from 0-100, with higher scores indicating better FC. Recently Green et al. (2011) reported that the UPSA-B was determined to be the most suitable short form of the available FC assessments as a co-primary measure in clinical treatment trials.

Results

The HC sample had more females than the schizophrenia sample (65% versus 31%). The schizophrenia patients were 51% African American, 35% Hispanic, and 12% Caucasian, while the HC sample were 39% African American and 59% Caucasian. Each sample had one participant of Asian descent. The ages of the samples were similar, with the schizophrenia sample's age averaging 39.7 (SD=11.95) and the HC group averaging 37.6 (SD=12.48).

There are three aspects of performance on the 12 objectives that were examined and compared across groups: time to complete each objective, errors made on each objective, and number of times that the task was progressed to the next objective because the previous one was not completed in a timely manner. As can be seen in Figure 1, the schizophrenia patients performed significantly more slowly than controls on 10 of the 11 objectives with valid data and made significantly more errors on 2 of the 8 objectives with total errors greater than 0. As seen in the figure, the distributions of times and errors were quite similar for the HC group and patients, with no evidence of a global performance deficit across all objectives on the part of the schizophrenia patients. Table 2 presents the statistical tests of the differences between healthy controls and patients across the 12 objectives for time, errors, and forced progressions. Note that for every variable on which the HC and SCZ groups differed by p<.05, the differences would have met the Bonferroni correction for multiple corrections.

Figure 1.

Figure 1

Figure 1

Open in a new tab

Table 2. Differences between Healthy Controls and Schizophrenia Patients on the Twelve VRFCAT Objectives.

Time to Completion # of Errors Forced Progression Frequency
T statistic p-value T Statistic p-value Healthy Control Patient p-value*
1. Pick-up the Recipe 3.66 <0.001 1.07 0.285 0 0 ---
2. Search for Ingredients 3.52 <0.001 1.03 0.306 0 1 0.4857
3. Cross Off Correct Ingredients & Pick-up Bus Schedule 5.65 <0.001 3.60 <0.001 7 21 0.0017
4. Pick-up the Billfold 5.81 <0.001 0.04 0.968 0 0 ---
5. Exit the Apartment 3.31 0.001 --- --- 0 0 ---
6. Get on the Bus to the Grocery Store 4.76 <0.001 1.71 0.090 0 0 ---
7. Pay for the Bus 3.73 <0.001 0.72 0.472 0 0 ---
8. Select an Aisle 2.96 0.004 --- --- 0 0 ---
9. Shop for Groceries 7.00 <0.001 2.66 0.009 3 13 0.0059
10. Pay for Groceries 3.19 0.002 1.57 0.120 0 2 0.2334
11. Get on the Bus to go Home 9.04 <0.001 1.85 0.072 0 0 ---
12. Pay for the Bus ** ** 1.12 0.264 1 2 0.611
Open in a new tab
**

Data not collected

*

Based on Fisher's exact test

Next we examined the extent to which performance on different objectives discriminated the groups. For this analysis, which is intrinsically exploratory, we entered the variables which significantly discriminated the two groups in the domains of times, errors, and forced progressions from Table 2, as well as the numbers of times the recipe was accessed, with α-level for entry set to 0.15 and α-level for remaining in the model set to 0.10. We selected items that were significant in a forward entry stepwise regression analysis and examined classification accuracy as well as variance contributed to discrimination. The final model yielded the 8 variables presented in Table 3. Time and errors for getting on the bus home as well as time to pick up the billfold each contributed more than 10% variance to the canonical discriminant function. Classification accuracy was outstanding, with all 51 of the schizophrenia patients correctly classified and 53 out of 54 HC participants correctly classified as well.

Table 3. Results of the Discriminant Function Comparing the Two Groups of Study Subjects.

Order of Entry Objective Partial R-Square F p-value
1 Time to Catch the Bus Home 0.442 81.64 <0.001
2 # Errors made Catching the Bus Home 0.308 45.29 <0.001
3 Time to Pick up the Billfold 0.132 15.40 <0.001
4 Time to Pick up the Recipe 0.081 8.84 0.004
5 Time to Exit the Apartment 0.046 4.75 0.032
6 # Errors made Catching the Bus to the Grocery Store 0.040 4.13 0.045
7 Forced Progression while Shopping for Groceries 0.045 4.53 0.036
8 # Times the Recipe was Accessed 0.036 3.55 0.063
Open in a new tab

In order to examine other potential influences on performance, we correlated age with summary variables from errors, time, and forced progressions on the VRFCAT in each sample separately. For the schizophrenia patients, total errors (r=0.47), total time (r=0.53) and total progressions (r=0.52) were all statistically significant with p-values<0.001. In the HC sample, the correlations were noticeably smaller, total errors (r=0.10), total time (r=0.17), and total progressions (r=0.09), and did not achieve statistical significance (all p-values>0.20).

We also examined the correlations between MCCB scores, UPSA-B scores, and three aspects of performance on the VRFCAT objectives. In this analysis, we selected the individual VFRCAT variables that discriminated between HC and schizophrenia patients, as well as the three VRFCAT summary variables, and correlated them with the MCCB cognitive composite and the UPSA-B total score. These correlations are presented in Table 4. As can be seen in the table, a number of the individual variables and summary scores for errors, time, and progression were correlated with the MCCB cognitive composite. Interestingly, in this preliminary study, there were no significant correlations between VRFCAT performance and any of the scores on the UPSA-B.

Table 4. Pearson Correlations between VRFCAT Variables and MCCB and UPSA-B performance in Schizophrenia Patients.

MCCB Cognitive Composite UPSA-B Total Score
Time to Pick-up the Recipe -0.40 (0.004) -0.22 (0.123)
Time to Search for Ingredients 0.10 (0.497) -0.05 (0.748)
Time to Cross Off Correct Ingredients & Pick-up Bus Schedule -0.24 (0.088) -0.15 (0.307)
Time to Pick-up the Billfold -0.00 (0.981) 0.27 (0.055)
Time to Exit the Apartment -0.35 (0.011) -0.22 (0.128)
Time to Get on the Bus to the Grocery Store -0.04 (0.760) -0.02 (0.895)
Time to Pay for the Bus -0.23 (0.109) -0.08 (0.596)
Time to Select an Aisle 0.05 (0.718) 0.09 (0.513)
Time to Shop for Groceries -0.33 (0.018) -0.15 (0.293)
Time to Pay for Groceries -0.01 (0.936) 0.06 (0.699)
Time to Get on the Bus to go Home -0.21 (0.148) -0.22 (0.114)
Time to Pay for the Bus -0.37 (0.007) -0.19 (0.180)
# Errors Crossing Off Correct Ingredients & Picking-up Bus Schedule -0.16 (0.277) -0.13 (0.366)
# Errors Getting on the Bus to the Grocery Store -0.32 (0.021) -0.11 (0.440)
# Errors Shopping for Groceries -0.41 (0.003) -0.20 (0.164)
# Errors Getting on the Bus to go Home -0.21 (0.138) -0.18 (0.205)
# Forced Progressions when Crossing Off Correct Ingredients & Picking-up Bus Schedule -0.23 (0.111) -0.06 (0.662)
# Forced Progressions when Shopping for Groceries -0.41 (0.003) -0.14 (0.325)
# Times the Recipe was Accessed 0.09 (0.526) -0.22 (0.116)
Total Time -0.28 (0.045) -0.14 (0.322)
Total Errors -0.36 (0.009) -0.21 (0.142)
Total Forced Progressions -0.29 (0.036) -0.07 (0.605)
Global Performance Composite 0.32 (0.023) 0.14 (0.335)
Open in a new tab

In a final descriptive analysis, we examined the total variance shared between the VRFCAT items and the MCCB cognitive composite. Using a forward entry stepwise regression, with a liberal criterion of p<0.10 to enter, we found the following variables entered, in order, from the biggest to smallest contribution to total variance accounted for: errors while shopping, time to pick up the recipe, time to pay for groceries, and time to pay for the bus. The overall regression was statistically significant with these four variables entered, F(4,46)=6.82, p<0.001, and the shared variance between the MCCB cognitive composite and these four outcome variables was R2=0.37.

Discussion

The assessment of functional capacity with virtual reality methodology in people with schizophrenia is feasible with the proportion of patients able to complete the assessment in a valid manner consistent with healthy controls. There is evidence that patients perform more poorly on several aspects of the task than healthy controls, without demonstrating evidence of global deficits on every element of the task. The parts of the task that were performed most poorly suggest slowing in the performance of information processing and working memory. In line with this finding, VRFCAT performance in schizophrenia patients was correlated with performance on the MCCB. Thus, the VRFCAT tentatively meets criteria for being a co-primary measure. There were no significant correlations of the VRFCAT total scores with performance on the UPSA-B, indicating that the VRFCAT and UPSA-B seem to be measuring different aspects of functional deficits. Further research will more systematically evaluate the relationships between the VRFCAT and other potential co-primary measures, as well as performing a more suitable normative understanding of the differences between HC and patients with schizophrenia on this task.

There are some limitations in this study. Our healthy controls were tested first, as part of the development process of the VRFCAT, and as a result some participants were excluded because of problems with an earlier, immature version of the task. Educational attainment for the healthy controls was not measured systematically. The schizophrenia patients were not selected for representativeness to any pre-specified group of patients, other than all participants being willing and able to participate in an extended assessment similar to those seen in a schizophrenia treatment trial. Gender and racial ethnic differences in the samples were present. Given the small sample sizes, there are no normative statements about performance that can be made and we cannot inquire about test-retest stability or the stability of correlations between the MCCB, UPSA-B, and VRFCAT in the current data set. These issues are being addressed comprehensively in an on-going study with substantial statistical power.

Our study does suggest that computerized functional capacity tests are feasible in schizophrenia patients and correlated with MCCB scores. The VRFCAT runs on standard operating systems and does not require specialized equipment such as a touch screen. Thus, the potential for remote deliverability is clear. Research is currently ongoing to more systematically examine the VRFCAT's psychometric properties in a much larger sample, including test-retest stability, deliverability across multiple sites, more extensive normative comparisons to healthy controls, and stability of correlations between MCCB and VRFCAT performance over time.

Acknowledgments

We thank the staff at the University of Miami Miller School of Medicine who completed the data collection for the VALERO study.

Footnotes

Contributors: Ruse, S. assisted in the development of the VRFCAT, collected data, and drafted the manuscript. Harvey, P. assisted in the development of the VRFCAT and in editing and writing this paper. Davis, V. analyzed the data for this paper. Atkins, A. assisted in editing this paper. Fox, K. assisted in the development of the VRFCAT, supervised data collection, and editing this paper. Keefe, R. supervised the development of the VRFCAT, oversaw data collection and assisted in editing this paper.

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

References

  1. Bowie CR, Depp C, McGrath JA, Wolyniec P, Mausbach BT, Thornquist MH, Luke J, Patterson TL, Harvey PD, Pulver AE. Prediction of real-world functional disability in chronic mental disorders: a comparison of schizophrenia and bipolar disorder. Am J Psychiatry. 2010;167(9):1116–1124. doi: 10.1176/appi.ajp.2010.09101406. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bowie CR, Leung WW, Reichenberg A, McClure MM, Patterson TL, Heaton RK, Harvey PD. Predicting schizophrenia patients' real-world behavior with specific neuropsychological and functional capacity measures. Biol Psychiatry. 2008;63(5):505–511. doi: 10.1016/j.biopsych.2007.05.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Buchanan RW, Davis M, Goff D, Green MF, Keefe RS, Leon AC, Nuechterlein KH, Laughren T, Levin R, Stover E. A summary of the FDA-NIMH-MATRICS workshop on clinical trial design for neurocognitive drugs for schizophrenia. Schizophr Bull. 2005;31(1):5–19. doi: 10.1093/schbul/sbi020. [DOI] [PubMed] [Google Scholar]
  4. Buchanan RW, Kreyenbuhl J, Kelly DL, Noel JM, Boggs DL, Fischer BA, Himelhoch S, Fang B, Peterson E, Aquino PR. The 2009 schizophrenia PORT psychopharmacological treatment recommendations and summary statements. Schizophr Bull. 2010;36(1):71–93. doi: 10.1093/schbul/sbp116. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Czaja SJ, Sharit J. Practically relevant research: Capturing real world tasks, environments, and outcomes. Gerontologist. 2003;43(suppl 1):9–18. doi: 10.1093/geront/43.suppl_1.9. [DOI] [PubMed] [Google Scholar]
  6. Depp CA, Mausbach BT, Harmell AL, Savla GN, Bowie CR, Harvey PD, Patterson TL. Meta-analysis of the association between cognitive abilities and everyday functioning in bipolar disorder. Bipolar Disord. 2012;14(3):217–226. doi: 10.1111/j.1399-5618.2012.01011.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. First MB, Spitzer RL, Gibbon M, Williams JB. The structured clinical interview for DSM-III-R personality disorders (SCID-II) Part I: Description. Journal of Personality Disorders. 1995;9(2):83–91. [Google Scholar]
  8. Freeman D. Studying and treating schizophrenia using virtual reality: a new paradigm. Schizophr Bull. 2008;34(4):605–610. doi: 10.1093/schbul/sbn020. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Gould F, Bowie CR, Harvey PD. The influence of demographic factors on functional capacity and everyday functional outcomes in schizophrenia. J Clin Exp Neuropsychol. 2012;34(5):467–475. doi: 10.1080/13803395.2011.651102. [DOI] [PubMed] [Google Scholar]
  10. Green MF, Schooler NR, Kern RS, Frese FJ, Granberry W, Harvey PD, Karson CN, Peters N, Stewart M, Seidman LJ. Evaluation of functionally meaningful measures for clinical trials of cognition enhancement in schizophrenia. Am J Psychiatry. 2011;168(4):400–407. doi: 10.1176/appi.ajp.2010.10030414. [DOI] [PubMed] [Google Scholar]
  11. Harvey P, Bowie C. Late-life schizophrenia. What providers need to know. Director. 2005;13(2):90, 93–94. [PubMed] [Google Scholar]
  12. Harvey PD, Velligan DI, Bellack AS. Performance-based measures of functional skills: usefulness in clinical treatment studies. Schizophr Bull. 2007;33(5):1138–1148. doi: 10.1093/schbul/sbm040. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Harvey PD, Wingo AP, Burdick KE, Baldessarini RJ. Cognition and disability in bipolar disorder: lessons from schizophrenia research. Bipolar Disord. 2010;12(4):364–375. doi: 10.1111/j.1399-5618.2010.00831.x. [DOI] [PubMed] [Google Scholar]
  14. Keefe RS, Fox KH, Harvey PD, Cucchiaro J, Siu C, Loebel A. Characteristics of the MATRICS Consensus Cognitive Battery in a 29-site antipsychotic schizophrenia clinical trial. Schizophr Res. 2011;125(2):161–168. doi: 10.1016/j.schres.2010.09.015. [DOI] [PubMed] [Google Scholar]
  15. Keefe RS, Poe M, Walker TM, Harvey PD. The relationship of the Brief Assessment of Cognition in Schizophrenia (BACS) to functional capacity and real-world functional outcome. J Clin Exp Neuropsychol. 2006;28(2):260–269. doi: 10.1080/13803390500360539. [DOI] [PubMed] [Google Scholar]
  16. Kern RS, Gold JM, Dickinson D, Green MF, Nuechterlein KH, Baade LE, Keefe RS, Mesholam-Gately RI, Seidman LJ, Lee C. The MCCB impairment profile for schizophrenia outpatients: Results from the MATRICS psychometric and standardization study. Schizophr Res. 2011;126(1):124–131. doi: 10.1016/j.schres.2010.11.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Kurtz MM, Baker E, Pearlson GD, Astur RS. A virtual reality apartment as a measure of medication management skills in patients with schizophrenia: A pilot study. Schizophr Bull. 2007;33(5):1162–1170. doi: 10.1093/schbul/sbl039. [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Leifker FR, Patterson TL, Heaton RK, Harvey PD. Validating measures of real-world outcome: The results of the VALERO expert survey and RAND panel. Schizophr Bull. 2011;37(2):334–343. doi: 10.1093/schbul/sbp044. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Mausbach BT, Bowie CR, Harvey PD, Twamley EW, Goldman SR, Jeste DV, Patterson TL. Usefulness of the UCSD Performance-Based Skills Assessment (UPSA) for predicting residential independence in patients with chronic schizophrenia. J Psychiatr Res. 2008;42(4):320–327. doi: 10.1016/j.jpsychires.2006.12.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Mausbach BT, Harvey PD, Pulver AE, Depp CA, Wolyniec PS, Thornquist MH, Luke JR, McGrath JA, Bowie CR, Patterson TL. Relationship of the Brief UCSD Performance-based Skills Assessment (UPSA-B) to multiple indicators of functioning in people with schizophrenia and bipolar disorder. Bipolar Disord. 2010;12(1):45–55. doi: 10.1111/j.1399-5618.2009.00787.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Mausbach BT, Moore RC, Davine T, Cardenas V, Bowie CR, Ho J, Jeste DV, Patterson TL. The use of the Theory of Planned Behavior to predict engagement in functional behaviors in schizophrenia. Psychiatry Res. 2013;205(1-2):36–42. doi: 10.1016/j.psychres.2012.09.016. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Moore RC, Harmell AL, Ho J, Patterson TL, Eyler LT, Jeste DV, Mausbach BT. Initial validation of a computerized version of the UCSD Performance-Based Skills Assessment (C-UPSA) for assessing functioning in schizophrenia. Psychiatry Res. 2013;144(1-3):87–92. doi: 10.1016/j.schres.2012.12.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Nuechterlein K, Green M, Kern R, Baade L, Barch D, Cohen J, Essock S, Fenton W, Frese F, Gold J. The MATRICS Consensus Cognitive Battery, part 1: test selection, reliability, and validity. Am J Psychiatry. 2008;165(2):203–213. doi: 10.1176/appi.ajp.2007.07010042. [DOI] [PubMed] [Google Scholar]
  24. Patterson TL, Goldman S, McKibbin CL, Hughs T, Jeste DV. UCSD Performance-Based Skills Assessment: Development of a new measure of everyday functioning for severely mentally ill adults. Schizophr Bull. 2001;27(2):235–245. doi: 10.1093/oxfordjournals.schbul.a006870. [DOI] [PubMed] [Google Scholar]
  25. Patterson TL, Semple SJ, Shaw WS, Halpain M, Moscona S, Grant I, Jeste DV. Self-reported social functioning among older patients with schizophrenia. Schizophr Res. 1997;27(2):199–210. doi: 10.1016/S0920-9964(97)00078-9. [DOI] [PubMed] [Google Scholar]
  26. Poletti S, Anselmetti S, Riccaboni R, Bosia M, Buonocore M, Smeraldi E, Cavallaro R. Self-awareness of cognitive functioning in schizophrenia: Patients and their relatives. Psychiatry Res. 2012;198(2):207–211. doi: 10.1016/j.psychres.2011.12.040. [DOI] [PubMed] [Google Scholar]
  27. Sabbag S, Twamley EM, Vella L, Heaton RK, Patterson TL, Harvey PD. Assessing everyday functioning in schizophrenia: Not all informants seem equally informative. Schizophr Res. 2011;131(1):250–255. doi: 10.1016/j.schres.2011.05.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Velligan DI, Diamond P, Glahn DC, Ritch J, Maples N, Castillo D, Miller AL. The reliability and validity of the Test of Adaptive Behavior in Schizophrenia (TABS) Psychiatry Res. 2007;151(1-2):55–66. doi: 10.1016/j.psychres.2006.10.007. [DOI] [PubMed] [Google Scholar]
  29. Velligan DI, Rubin M, Fredrick MM, Mintz J, Nuechterlein KH, Schooler NR, Jaeger J, Peters NM, Buller R, Marder SR. The cultural adaptability of intermediate measures of functional outcome in schizophrenia. Schizophr Bull. 2012;38(3):630–641. doi: 10.1093/schbul/sbq136. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Ventura J, Wood RC, Hellemann GS. Symptom domains and neurocognitive functioning can help differentiate social cognitive processes in schizophrenia: a meta-analysis. Schizophr Bull. 2013;39(1):102–111. doi: 10.1093/schbul/sbr067. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Wilkinson GS. Wide Range Achievement Test: Third Edition. Wide Range; Wilmington, DE: 1993. [Google Scholar]

RESOURCES