PTSD Disability Examination Reports: A Comparison of Veterans Health Administration and Contract Examiners

Abstract

Background

An enormous increase in disability claims for posttraumatic stress disorder (PTSD) has occurred over the past decade. To meet the demand for examinations required to determine diagnosis, causation, and impairment, the US Department of Veterans Affairs Veterans Health Administration (VHA) has increasingly relied on contract examiners. Despite anecdotal reports of poor-quality examinations by contractors, no systematic study comparing VA and contract examinations has been reported.

Methods

Data from 113 initial PTSD examination reports were coded and rated on variables related to content and quality. Administrative disability decisions rendered by VHA were identified and coded independently.

Results

Contract examinations reported more symptoms and a greater degree of impairment, resulting in higher VHA disability ratings compared with VHA examiner reports. Contractor examinations were rated as having poorer quality than were VHA examinations on 2 of 3 metrics and included several examination reports that contained no relevant history or discussion required to support opinions about diagnosis or impairment.

Conclusions

The findings provide the first systematic evidence of greater symptom/impairment reporting and poorer overall quality in contract examinations for PTSD disability claims compared with those conducted by VHA examiners, with resulting differential outcomes in VHA disability ratings. The findings have implications for the quality, integrity, and reliability of the VHA PTSD disability claims process and support the need for program oversight, examiner training, and quality assurance.

The US Department of Veterans Affairs (VA) provides health care for > 9 million military veterans, nearly half of all former service members.¹ Over the past 15 years, there has been a steady and substantial increase in the frequency of disability awards for veterans with post-9/11 military service. Recent data from the Bureau of Labor Statistics indicate that 41% of veterans who served after 9/11 receive service-connected disability benefits compared with 28% of veterans overall.² More than 5 million veterans receive VA service-related disability benefits.^2,3 More than half of the VA $243 billion budget for fiscal year (FY) 2021 ($135.5 billion) was allocated to the Veterans Benefits Administration (VBA), of which $115.7 billion (85%) was allocated specifically for service-related compensation claims payments.⁴

The VA predicted that VBA will have completed 1.4 million ratings for disability claims in 2021.⁵ A substantial percentage of these claims will be for mental disorders, specifically posttraumatic stress disorder (PTSD). VA officials testifying before Congress in 2017 noted that the number of PTSD claims had nearly tripled in the previous 10 years.⁶ As far back as 2013, McNally and Frueh analyzed “the skyrocketing of disability claims,” particularly for PTSD, among veterans who served in Iraq and Afghanistan.⁷

This large increase has placed an unprecedented burden on the VBA to expand its capacity to conduct initial PTSD disability evaluations that by regulations are completed by psychologists or psychiatrists. This need has led the VBA to make significant changes in the compensation and pension (C&P) process, including a reduced role for Veterans Health Administration (VHA) examiners and increased reliance on non-VA (contract) examiners through the Contract Medical Disability Examination (MDE) program. In 2019, the MDE budget was $1.23 billion; in 2020, it was increased to $1.79 billion, and for 2021, it was $2.23 billion, reflecting the increasing investment of resources in non-VA examiners, ostensibly to both increase capacity and save costs.⁵

Anecdotally, concerns have been raised regarding inadequate training of contract examiners as well as inadequate reports by these examiners. A 2018 Government Accountability Office (GAO) report concluded that VA lacked the data to determine whether contract examiners were meeting standards for quality, timeliness, and accuracy.⁸ The GAO report noted that VA required 92% of contractor reports contained no obvious errors, a relatively low target; however, in the first half of 2017, only 1 contractor group met that target. The report noted further that “VBA does not verify if examiners have completed training nor does it collect information to assess training effectiveness in preparing examiners.”⁸ A subsequent analysis of contract examinations completed by the VA Office of the Inspector General (OIG) in 2019 concluded that the MDE program was “hampered in their ability to provide oversight because of limitations with VBA’s electronic examination management systems, the lack of reliable data, and inadequate staffing of the program.”⁹

These reports have focused almost exclusively on simple performance metrics, such as timeliness of examination completion. However, the 2018 GAO report referenced isolated “focused reviews” of complaints about the quality of examinations by contract examiners and gave as an example an isolated “review of one contracted examiner who had high rates of diagnosing severe posttraumatic stress disorder.”⁸ After review indicated the examiner’s reports were of poor quality, the VBA discontinued the examiner’s contract.

Unfortunately, despite such anecdotal reports and isolated actions, to date there are no published reports examining and comparing the quality of PTSD examination reports completed by VHA and contract examiners or the subsequent disability determinations made by the VBA as a result of these evaluations. In a November 2020 letter to the VA Secretary, 11 US Senators expressed “grave concerns” regarding the VA decision to privatize C&P programs noting, among other concerns, that there were “no clinical quality measurement for, or evaluation of, contractor examinations.”¹⁰ The letter cited anecdotal evidence of contract examiners not reviewing veteran’s medical records and diagnosing conditions “without supporting evidence.”¹⁰

The purpose of the present evaluation was to provide a systematic comparison of the content and quality of initial PTSD disability examinations conducted by VHA and non-VA contract examiners. In addition, this study compared the disability rating decisions resulting from VHA and contract examinations.

METHODS

A random sample of 100 Initial PTSD Disability Benefits Questionnaires (DBQs)— structured forms completed by all examiners—were obtained from a list supplied by the VA Office of Performance Analysis and Integrity. All examinations were from the Veterans Integrated Service Network (VISN) 1, encompassing the New England region and were conducted in 2019 and 2020. Two of the 100 cases were excluded for technical reasons, resulting in 98 examination reports. However, the final pool yielded 62 contract examinations and only 36 VHA examinations. To make the sample sizes more comparable, an additional 15 examinations were randomly selected from the local examination database (also VISN 1) to complement the original examination pool.

Once DBQs were retrieved, all identifying information was deleted, and cases were analyzed using assigned record numbers. All coding was completed by the 2 principal investigators, both VA psychologists with extensive training and experience in C&P evaluation and treatment of veterans with PTSD. Due to inherent structural differences between the forms used for VA and contract examinations, raters could not be masked/blinded to the source of the report.

A number of measures were taken to reduce bias and enhance objectivity of rating. First, objectively coded variables (eg, age and sex of veteran, period of service, trauma type, diagnoses rendered by the examiner, impairment category endorsed, number and type of symptoms) were transcribed directly from the DBQ as recorded by the examiner. Second, to rate report quality, an initial categorical rating scale was developed based on predetermined elements of examination quality that were considered essential. After refinement and preliminary analysis of interrater reliabilities, 3 quality-related indices were identified: (1) level of detail in description of key content areas (history before service, service trauma, after service social and vocational history, mental health history, substance use); (2) synthesis of history and findings in explaining opinion rendered; and (3) clarity of opinion regarding causation required “at least as likely as not” degree of confidence. The first 2 quality ratings were based on a 3-point scale (poor, fair, good), and the third variable was coded as yes or no. (eAppendix available at doi:10.12788/fp.0225). Interrater reliabilities calculated based on a subsample of 18 cases, randomly selected and rated by both raters, yielded Cohen κ in the acceptable range (.61, .72, and .89 for detail, synthesis, and clarity, respectively). Finally, for information regarding VBA decision making, rating decision documents contained in the Veterans Benefit Management System database were reviewed to determine whether the veteran was granted service connection for PTSD or another mental disorder based on the examination report in question and, if so, the disability rating percentage awarded. These were recorded independently after all other coding had been completed.

RESULTS

Comparison of VHA and contract examinations revealed no significant differences between groups on relevant sociodemographic and other measures (Table). Missing data were not obtained from other records or sources, and for this study, reflect only what is recorded in the examination reports except for age, which was calculated using veteran’s date of birth and the date of examination.

TABLE.

Examinee Demographics

	Veterans Health Administration	Contract
Age, mean, y	46.9	46.1
Sex, No. (%)
Female	5 (10)	13 (21)
Male	46 (90)	49 (79)
Military era, No. (%)
Korean War	1 (2)	0 (0)
Vietnam War	7 (14)	8 (13)
Post-Vietnam	6 (11)	5 (9)
Desert Storm	1 (2)	4 (6)
1992–2001	6 (12)	7 (11)
Post-9/11	30 (59)	37 (59)
Unanswered	0 (0)	1 (2)
Trauma type, No. (%)
Combat	33 (64)	30 (48)
Military sexual trauma	9 (17)	12 (20)
Other	9 (19)	18 (29)
No trauma reported	0 (0)	2 (3)
Employment, No. (%)
Full-time	14 (27)	16 (26)
Part-time	5 (10)	0 (0)
Work status unclear	16 (31)	24 (39)
In school	2 (4)	2 (3)
Work and school	0 (0)	1 (2)
Unemployed	9 (18)	12 (19)
Retired	4 (8)	4 (6)
Unreported	1 (2)	3 (5)

To examine differences between VHA and contract examinations, the groups were first compared on a set of predetermined objectively coded variables taken directly from the DBQ. The frequency of PTSD diagnoses by VHA (57%) and contract (71%) examiners was not significantly different nor were rates of non-PTSD diagnoses by VHA (51%) and contract (73%) examiners. There also was no difference in the mean number of PTSD symptoms endorsed across PTSD diagnostic criteria B, C, D, and E (maximum of 20) recorded by VHA (9.4) and contract (10.9) examiners.

Contract examiners recorded a significantly greater mean number of “other symptoms” on a checklist of 31 possible symptoms as compared to VHA examiners: 7.3 vs 5.8, respectively (t[104] = 2.27, P < .05). An initial analysis of overall social/vocational impairment ratings coded by examiners did not reveal significant differences between examiner groups. However, when the 2 most severe impairment categories were combined to create a pooled “severe” category, 31% of contract examiners rated veterans as severely impaired compared with only 12% of VHA examiners (χ² = 5.79, 1 df, P < .05) (Figure 1).

FIGURE 1.

Examinations Indicating Severe Impairment, %

VHA and contract examinations were compared on 3 measures of report quality. Significant differences were found for both level of detail (χ² = 16.44, 2 df, P < .01) and synthesis (χ² = 6.68, 2 df, P < .05). Contract examinations were more likely to be rated as poor and less likely to be rated good, with a similar proportion of fair ratings for the 2 examination types (Figures 2 and 3). There was no significant difference in the proportion of VHA and contract examinations providing clear statement of opinion regarding causation (ie, whether or not the diagnosed condition was service related), with the majority rendering an adequate opinion in both examiner groups (VHA, 78%; contract, 79%).

FIGURE 2.

Examination Ratings for Detail

FIGURE 3.

Examination Ratings for Synthesis

Qualitative review revealed examples of markedly deficient examinations among contract examinations, including several reports that contained no review of records, no report of relevant background, and no mention or assessment of social and vocational function needed to inform opinions about diagnosis and impairment.

Finally, the VBA database was used to compare the resulting disability award decisions made by VBA based on the examination reports in question. Examination by contractors resulted in significantly higher mean service-connected disability ratings for examinees compared with VHA examiners (46.8 vs 33.5, respectively; t[108] = 2.3, P < .05).

DISCUSSION

The present study provides the first reported systematic comparison of VA disability examinations for PTSD completed by examiners employed by the VHA and those hired as contract examiners through the MDE program. Although the frequency of PTSD diagnoses by contract examiners was higher than that of VHA examiners (71% vs 57%, respectively), the difference was not statistically significant. However, contract examiners recorded significantly more symptoms for examinees and rated them as severely impaired more frequently than did their VHA counterparts. In keeping with rating guidelines used by the VBA, these differences in examination content resulted in higher disability ratings for veterans seen by contract examiners.

Along with these elevated reports in symptom and severity ratings, contract examiners were less likely to provide adequate detail in the narrative sections of their reports and less frequently provided a satisfactory explanation and synthesis of relevant history and findings in support of their conclusions. Although not reflected in the statistical analysis, case-by-case review revealed some startlingly inadequate examination reports by contract examiners, several of which contained no review of records, no report or discussion of relevant background, and no discussion or analysis of social and vocational function to inform and support their opinion about level of impairment. None of the VHA examination reports reviewed lacked information to that degree.

Such deficiencies in detail and synthesis run counter to accepted guidelines for the adequate assessment of psychological injury in general and in VA disability claims specifically.^11,12 For example, Watson and colleagues proposed that a minimum of 3 hours was required to conduct an initial PTSD examination, with more complex cases possibly taking longer.¹¹ There is no information available about how long contract examiners take to complete their examinations and how that compares with the time taken by VA examiners. The VBA failure to monitor whether or not examiners follow accepted guidelines for PTSD examination has not previously been evaluated. Historically, a large number of clinicians, researchers, and policy critics have raised concerns about the potential for exaggeration or malingering among VA PTSD disability claimants and have urged the need to adequately assess for unreliable reporting and presentation.^13,14 However, the possibility of systematic examiner deficiency and/or bias increasing the frequency of false or inflated claims being approved has received little empirical attention.

Although contract examiners did not diagnose PTSD significantly more frequently than VHA examiners (71% vs 57%, respectively), the overall frequency of PTSD diagnosis across both groups (65%) was substantially higher than previous figures that have, on average, estimated the lifetime prevalence of PTSD in trauma-exposed veterans to be about 31%.¹⁵ A re-analysis of the same National Vietnam Veterans Readjustment Survey data, but applying more conservative diagnostic criteria, reduced the lifetime prevalence to just under 19%, with point prevalence estimates even lower.^15,16

In a study of concordance rates between service connection for PTSD and both current and lifetime diagnosis by independent, structured assessment, Marx and colleagues found that a “significant minority” of veterans who were already receiving serviceconnected disability for PTSD did not meet lifetime and/or current diagnostic criteria.¹⁷ Although it is possible that the group of veterans who were applying for disability benefits in our study had a higher rate of PTSD, it also is possible if not likely that the PTSD examination process overall yields inflated rates of diagnosis and levels of impairment. This speaks to the concern raised by Marx and colleagues who found that veterans with service connection for PTSD who received related benefits “may not have the disorder.”¹⁷

Limitations

A methodological limitation of the present study was that, due to structural differences in the DBQ forms used for VHA and contract examinations, the reports could not be de-identified as to examiner type and thus raters could not be masked/blinded. To mitigate bias, a predetermined, piloted, and refined coding and rating plan for report quality metrics was adhered to strictly, and interrater reliabilities were acceptable. Future study is suggested in which all report content is standardized for coding using the same format, which at present would require a complete rewriting of the entire report; this problem could be resolved by having the VBA adopt a more coherent system in which all reports, regardless of examiner type, use a single, standardized template. Further study using larger data sets and expanding to other VA regions also is needed.

CONCLUSIONS

The present study suggests that poor examination and report quality—by contract examiners and to a lesser degree VHA examiners—are not uncommon. The findings confirm and extend previous anecdotal reports of deficiencies in PTSD examinations performed by contract examiners and provide empirical support for concerns raised of global deficiencies in the VBA oversight of the MDE program. Such deficiencies have significant implications for the quality and integrity of the VA disability determination process for veterans claiming PTSD related to military service.

The current findings support and strengthen the call for development and management of a structured and enforced training and quality assurance/improvement program for VA PTSD disability examinations. Such training and oversight will be critical to improve the quality and integrity of these examinations, reduce error and waste in VBA’s Compensation and Pension process, and in doing so optimize VA financial resources to best serve veterans’ benefits and health care needs.

eAPPENDIX.

Rating Definitions

Criteria	Poor	Fair	Good
Level of detail	Missing or cursory information that provides no qualitative understanding of relevant background; individual or multiple subsections of report (psychosocial, trauma history, educational, vocational, mental health, substance use, legal) with little or no detail to inform conclusion and opinion.	Qualitative information that provides understanding of relevant background; individual subsections of report (psychosocial, trauma history, educational, vocational, mental health, substance use, legal) provide sufficient individualized detail to inform conclusion and opinion.	Report subsections contain detailed, qualitative information that provides comprehensive understanding of relevant background; subsections of report (psychosocial, trauma history, educational, vocational, mental health, substance use, legal) provide individualized detail with history that is informative and relevant to diagnosis and causation.
Synthesis	Does not provide explanation of a causal nexus; conclusion and opinion not supported by evidence in the report; opinion does not reconcile conflicting/inconsistent evidence within the report; problems with differential diagnosis, causation, and/or reliability of history and symptoms not addressed.	Some causal nexus articulated; no glaring inconsistencies between content and conclusion; no obvious conflicting/inconsistent evidence within the report that would raise concern about differential diagnosis, causation, and/or reliability of history and symptoms reported.	Provides clear explanation of causal nexus; conclusion and opinion clearly supported by evidence in the report; reconciles any conflicting/inconsistent evidence within the report with clear explanation to support opinion.
Clarity of opinion
Present	Opinion regarding diagnosis and causation is articulated with Veterans Benefits Administration required language, indicating it is “at least as likely as not” or a “50/50 or greater probability.”
Not present	Opinion regarding diagnosis and causation is not articulated; Veterans Benefits Administration required language indicating it is “at least as likely as not” or is a “50/50 or greater probability” is not used.