Garbage in, garbage out: The tenuous state of research on PTSD in the context of the COVID-19 pandemic and infodemic

Despite contemporary classification of posttraumatic stress disorder (PTSD) as a trauma- and stressor-related disorder in the DSM-5 (American Psychiatric Association, 2013), the Journal of Anxiety Disorders continues to publish work pertaining to this condition. The journal receives a large number of PTSD-relevant submissions and, as might be expected, many recent submissions have purported to address the issue of PTSD in the context of the COVID-19 pandemic. While this is certainly an issue of critical importance, and a sound understanding of COVID-related traumatic stress reactions is warranted in order to facilitate appropriate assessment and treatment, much of the research submitted to this journal and published in other journals has missed the mark. Specifically, much of the published work on PTSD related to the pandemic is flawed with respect adequate consideration of PTSD criteria, particularly the assessment of Criterion A, conceptualization of Criterion A events, use of outdated measures, and the timeframe of reported symptoms.

According to the DSM-5 (American Psychiatric Association, 2013), in order to meet the diagnostic criteria for PTSD a person must directly experience (Criterion A1) or witness (Criterion A2) a traumatic event, including actual or threatened death, serious injury, or sexual violence, except in the case of indirect exposure (e.g., learning that a loved one has suddenly died in a violent or accidental way; Criterion A3) or exposure to the traumatic experiences of others (e.g., first responders gathering human remains; Criterion A4). Likewise, the ICD-11 (World Health Organization, 2020) requires exposure to a threatening or horrific event or experience that would be likely to promote pervasive distress in most people. In addition, the requisite number of symptoms within each symptom cluster must be endorsed, the duration of these symptoms must be more than one month, there must be clinically significant distress or functional impairment, and, in the DSM-5, symptoms cannot be attributable to the physiological effects of substances or a general medical condition. There are some differences between DSM-5 and ICD-11 regarding the diagnostic criteria for PTSD (Stein et al., 2014). Nevertheless, for replication and other reasons it is important that PTSD researchers select one of these systems and ensure that their PTSD participants actually meet the diagnostic criteria for whatever system they are using. Ideally a diagnosis of PTSD is determined via careful assessment using a structured clinical interview; although, for research purposes, and as might be expected for research conducted within the context of public health measures designed to thwart viral spread, PTSD caseness is often determined using self-reports that assess the various DSM-5 or ICD-11 criteria.

Recent meta-analyses suggest that the prevalence of PTSD among the general population and others affected by COVID-19 ranges from as low as 5% to more than 50 % (Arora et al., 2020; Cenat et al., 2021; Xiong et al., 2020). To assess the veracity of this and related research we reviewed more than 60 empirical articles identified in recent meta-analyses as well as a PsychInfo search conducted on February 3, 2021 using the keywords “COVID-19 or coronavirus or 2019-nCoV or SARS-CoV-2 or COV-19” and “PTSD or post traumatic stress disorder or posttraumatic stress disorder or post-traumatic stress disorder”. Unfortunately, we found that much of the published work purporting to address PTSD in response to the COVID-19 pandemic is significantly flawed in that an assessment of Criterion A was typically lacking; indeed, a majority of studies did not include an assessment of Criterion A events and failed to anchor self-reports of PTSD symptoms to any event. An assessment of PTSD symptoms in the absence of identifying a target event or experience to which the symptoms are anchored provides nothing more than an indication of levels of general distress. It does not, and cannot, be used to establish PTSD caseness. As such, this research has the potential to lead the field astray and create a host of problems that may arise from erroneously attributing PTSD to COVID-19 and overestimating its prevalence, including planning for and delivering specialized mental health services at present and following the pandemic.

Of the studies that did assess PTSD symptoms relative to an anchor event, that event was, in almost all cases, some aspect of the COVID-19 pandemic. This approach is reasonable within the context of assessing pandemic-related traumatic stress responses. However, the approach poses challenges to current diagnostic conceptualizations of PTSD that require direct or, under specific circumstances, indirect exposure to actual or threatened death, serious injury, or sexual violence. While most people will have had some experience with COVID-19, the majority of these experiences are unlikely to qualify as a Criterion A event. Bridgland et al. (2021) have shown that most COVID-related stress is in response to indirect or future events, such as media coverage of the pandemic and fears of contracting the virus. Consequently, investigations limited to these sorts of stressors are not studies of PTSD. They are, instead, studies of stress reactions to the pandemic. Few studies (e.g., Bridgland et al., 2021; Boyraz, Legros, & Tigerhtrom, 2020; Liu et al., 2020) presented their findings in a manner that acknowledges this issue, referring to traumatic or posttraumatic stress symptoms as opposed to PTSD. Even here, however, there can be conceptual problems. In the midst of an ongoing pandemic, stress reactions are not necessarily “post” traumatic and not necessarily indicative of full-blown or even subclinical PTSD. Other conceptualizations of the stress associated with COVID-19, such as the COVID Stress Syndrome (Taylor et al., 2020a, 2020b), may provide a solid foundation from which this line of research can progress. In this syndrome, traumatic stress symptoms are a prominent feature, but severe forms of the syndrome are conceptualized as a pandemic-related adjustment disorder, not a posttraumatic stress syndrome (Asmundson & Taylor, 2020; Taylor, 2021).

Beyond issues with Criterion A assessment, anchoring, and conceptualization, it is of great concern that current research efforts designed to address and inform our understanding of traumatic stress responses and, presumably, PTSD, in relation to COVID-19 are utilizing outdated measures. Yet, many of the studies identified in our PsychInfo search used measures designed to assess DSM-IV (American Psychiatric Association, 2000) rather than DSM-5 PTSD symptoms. While there are certainly similarities and overlap between former and contemporary PTSD symptoms, corresponding self-report measures (e.g., PCL-C and PCL-5), and the conclusions that can be drawn from each (Rosellini et al., 2015), there is no sound justification for using outdated measures where the goal is to study PTSD relative to circumstances and events that have developed only over the past year. Contemporary measures, including those modified to assess traumatic stress responses to COVID-19 (Vanaken, Scheveneels, Belmans, & Hermans, 2020), are free and readily accessible. It is also noteworthy that a number of studies did not specify the timeframe over which respondents were asked to rate traumatic stress symptoms or, rather than assessing symptoms experienced over the past month, focused on the past week. These issues further illustrate the problematic nature of much of the research in this area, suggesting that some contributing researchers may be generally uninformed regarding contemporary conceptualization and assessment of PTSD, or that they are otherwise interested in acute as opposed to persistent stress responses.

The problematic nature of COVID-related PTSD research can be understood in the context of the broader problem of COVID-19 junk science. By the end of the first 13 months of the COVID-19 pandemic, the amount of published psychological research on the pandemic has exceeded the combined amount of psychological research conducted for all past pandemics. This is illustrated by a PsychInfo search using the keywords “pandemic” or “pandemics” for the pre-COVID-19 period (1957 to 2019; n = 1,824 citations) as compared to a search in which “COVID-19” was added as a third keyword (2020 to January 15, 2021; n = 2,615 citations). Similarly, a search of mental health articles from the Embase, PubMed, and Scopus databases, up to August 2020, found more articles on COVID-19 (n = 3,070) as compared to articles on the H1N1 (“Swine flu”) pandemic (n = 327) or the Ebola outbreak (n = 127) (Maalouf, Medawar, Meho, & Akl, 2020).

With the proliferation of COVID-related research comes a proliferation of junk science, not only in the field of mental health but also in biomedical fields. During the COVID-19 pandemic there has been a surge of poorly conducted, low quality, and hastily conducted research, as illustrated above in the context of PTSD. Across research disciplines, the large amount of slipshod research was signalled by the notable number of retractions of journal articles during 2020, which were withdrawn for various reasons including flawed methodology and failure to provide evidence of the adequacy of data collection (see www.retractionwatch.com). A recent review of the retracted COVID-19 research articles concluded that the main reason for retractions was the rush to quickly publish the articles, whether by the authors or the journal editors and reviewers (Soltani & Patini, 2020). The problem with rushed research has been widely discussed in various places, including journals such as the British Medical Journal and Nature (e.g., Lakens, 2020; Mathew, 2020; Steinberg, 2020). Quotations from some of these commentaries are instructive:

I am both exhilarated and worried as I watch the unprecedented pace and implementation of medical research currently being done. Speed is, of course, important when a crisis such as COVID-19 is at hand. But speed – in research, the interpretation and the implementation of science – is a risky endeavor. The faster science is published and implemented, the greater the chances it is unsound. (Steinberg, 2020).

Early in the COVID-19 pandemic, the WHO raised concerns about the “infodemic”, a term coined by Rothkopf (2003), who was writing about SARS. In the context of COVID-19, the infodemic refers to the deluge of information about the pandemic, consisting of a mix of accurate information, misinformation, and deliberately misleading disinformation. In the field of mental health research, including PTSD research, we are facing an infodemic where journals are flooded with a mix of methodologically sound articles and flawed submissions from researchers racing to make contributions to the psychology of COVID-19. Eventually the wheat will be sorted from the chaff, as systematic reviews in the months and years to come flag the important studies that contribute relevant new knowledge and reject the flawed investigations. In the meantime, researchers have to sort through the research infodemic themselves, trying to sort out the important work from junk science. Journal editors and reviewers face a similar challenge. Collectively, we all need to work on the problem of the research infodemic by ensuring that our work is methodologically rigorous. The urgency of the pandemic is no reason to cut corners. Cutting corners leads to unsound research, journal retractions, and undermines the public’s confidence in the work of researchers, including that of mental health investigators. Anti-science attitudes among the general public have become disturbingly prevalent in recent years (Hotez, 2020). The publication of flawed research adds to the problem.

As researchers or research practitioners, we should strive to publish sound research that encourages trust in science within the general public. In the context of PTSD and related traumatic stress research relevant to COVID-19 this will, at minimum, necessitate rigorous methods that incorporate the assessment of traumatic stress symptoms relative to a target event or experience, using contemporary conceptual frameworks and measures, a specified timeframe over which symptoms are assessed, and, where feasible, some indication of impact on functional ability. Research meeting these minimum standards may serve to advance rather than mislead the field and related efforts to plan, evaluate, and deliver appropriately tailored mental health services to those in need.