How to measure mental pain: a systematic review assessing measures of mental pain

Abstract

Question

Although mental pain is present in many mental disorders and is a predictor of suicide, it is rarely investigated in research or treated in care. A valid tool to measure it is a necessary first step towards better understanding, predicting and ultimately relieving this pain.

Study selection and analysis

Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we performed a systematic review to identify all published standardised measures of mental pain. We used qualitative content analysis to evaluate the similarity of each measure, quantified via Jaccard Index scores ranging from no similarity (0) to full similarity (1). Finally, using the Consensus-based Standards for the selection of health Measurement INstruments (COSMIN) methodology, we evaluated each measure’s development (assessing 35 features), its content validity (31 features) and if the latter was rated at least adequate, its other psychometric properties.

Findings

We identified 10 self-reported scales of mental pain in 2658 screened studies relying on diverse definitions of this construct. The highest average similarity coefficient for any given measure was 0.24, indicative of weak similarity (individual pairwise coefficients from 0 to 0.5). Little to no information was provided regarding the development and the content validity of all 10 scales. Therefore, their development and content validity were rated ‘inadequate’ or ‘doubtful’.

Conclusions and clinical implications

There is not enough evidence of validity to recommend using one measure over others in research or clinical practice. Heterogeneous use of disparate measures across studies limits comparison and combination of their results in meta-analyses. Development by all stakeholders (especially patients) of a consensual patient-reported measure for mental pain is needed.

PROSPERO registration number

CRD42021242679.

Summary box.

What is already known on this topic

• Mental pain is a life-threatening outcome important to patients, found in a variety of mental disorders, yet rarely evaluated or treated in routine clinical practice.

• A valid tool to measure mental pain is a necessary step to improve both research and clinical practice.

What this study adds

• Our systematic review identified 10 patient-reported measures of mental pain which vary substantially in their theoretical frameworks, definitions of mental pain, semantics and content.

• We cannot currently recommend one scale over any other for research or clinical purposes, mainly because of the lack of available data to evaluate the tools’ development and content validity.

How this study might affect research, practice or policy

• The existing measures can however be understood as important first efforts to gain an understanding of mental pain paving the way towards improving the conceptualisation of mental pain and its measurement in research and practice.

Introduction

Mental pain is a risk factor for suicidal behaviour, independently of other constructs such as depression or hopelessness.^1–5 In a recent qualitative study, 1912 patients, 627 healthcare professionals and 464 informal caregivers in 52 countries listed mental pain as the fifth most important outcome in depression, describing it as ‘unbearable’ and similar to ‘torture’.⁶ This stresses the importance of mental pain as a treatment target. Moreover, mental pain has been reported as part of other mental disorders, including obsessive-compulsive disorder, post-traumatic stress disorder and borderline personality disorder, as well as somatic conditions such as migraine.^7–12 During the COVID-19 pandemic, mental pain was one of the negative psychological outcomes observed in general population settings.¹³ Neuroscientific investigations of mental pain tend to show common pathways with physical pain (eg, activation of the anterior cingulate cortex and the insula, mostly when confronted with social rejection, as well as involvement of opioid µ-receptors).^14–17 Given the ubiquity of mental pain, it is no surprise that it has been proposed as a transdiagnostic construct that should be assessed via patient-reported outcome measures (PROMs).¹⁸ Despite growing evidence of the need for its routine assessment, it is rarely measured in clinical trials or evaluated in clinical practice.^{6 7}

Identifying a valid measure of mental pain is a first step in developing research to help address this outcome in clinical practice. Interestingly, like other constructs such as depression, no consensus exists for the definition of mental pain.¹⁹ In the Diagnostic and Statistical Manual of Mental Disorders (DSM-5), mental pain is defined in the introductory section as a synonym for the ’distress associated with the symptoms’.²⁰ This makes mental pain a relevant component of every diagnosis with a diagnostic criterion including ‘significant clinical distress’.²⁰ The history of this construct reveals that mental pain emerged in the psychiatric literature early during the 19th century as one of the three core symptoms of melancholia with depressed mood and suicidal ideas/behaviours in the work of Krafft-Ebing, Séglas, Guislain, Clouston, Maudsley and Griesinger.^21–23 These authors used different and not completely overlapping terminology, including ‘psychischer Schmerz’, ‘psychalgia’, ‘douleur morale’ and ‘phrénalgie’.^{22 23} In the 1990s, Shneidman qualitatively analysed hundreds of suicide notes and proposed a model for suicide based on frustrated psychological needs.²⁴ He coined the term ‘psychache’ defined as ‘the hurt, anguish, soreness, aching, psychological pain in the psyche, the mind’.²⁵ Other authors have proposed newer definitions, for example, ‘a lasting, unsustainable and unpleasant feeling resulting from negative appraisal of an inability or deficiency of the self’ (Meerwijk) or ‘stemming from the discrepancy between the ideal and the actual perception of self, accompanied by the awareness of one’s role in the experience of emotional pain’ (Tossani).^{7 26}

Simultaneously, there is no consensus about what measures should be used in scientific research or clinical practice. We found no study that systematically evaluated and compared measures of mental pain to help clinicians and researchers choose appropriate scales for specific uses. A few narrative reviews cover some measures of mental pain, but these studies have three limitations.^{2 7 27} First, they were not systematic and might have missed relevant measures. Second, the theoretical conceptualisations of mental pain underlying these definitions and hence their operationalisation in these measures may not be similar or even convertible, an important point because the lack of standardisation of measures across studies prevents their comparison and combination, thereby causing research waste.^{7 28–30} Third, previous reviews have not assessed the development of the measures, nor have they compared their psychometric properties. The frequency at which measurement instruments of psychological dimensions lack validity suggests that this may also be true of mental pain scales.^{19 31}

These issues call for a systematic effort to review mental pain measures, and the evidence of their validity, and collect the definitions that support them—the aim of this study.

Objective

The aim of this study was to (1) identify all existing measures stated by their authors to assess mental pain and the definitions that underlie them, (2) evaluate the similarities of the measures and (3) evaluate and compare their development and psychometric properties.

Study selection and analysis

Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) checklist, we conducted a systematic review to identify all existing tools measuring mental pain. We then determined the content overlap of each measure and evaluated their similarity by both qualitative and quantitative methods. Finally, we evaluated the development of each measure and its psychometric properties by applying the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN).^{32 33} For a global overview of the study design, see online supplemental material 1. The study protocol is available on Zenodo (4603560) and PROSPERO (CRD42021242679).

Systematic review to identify measures of mental pain

Eligibility criteria

We defined a measure of mental pain as any standardised tool whose authors claim specifically that it measures ‘mental pain’ or ‘psychological pain’ or ‘psychache’ or ‘psychic pain’. Since our aim was to identify tools to measure mental pain for the purpose of making recommendations about their use in clinical research or practice, we chose to focus on tools that have been used in peer-reviewed published studies. When there were several versions of a single tool (eg, shorter scales), we only included the first version that had at least one development and validation publication. We mention the other versions but we do not count them as distinct tools.

We included peer-reviewed scientific publications (eg, clinical studies, comments, letters to editors and reviews) that either mentioned the names of standardised measures of mental pain OR reported mental pain as an outcome measured in the study, without restriction of date or language or target population (eg, age, gender, country or disorder). We included all mental and physical health conditions. We excluded studies not involving humans, not using standardised instruments and using measures that assessed mental pain only in a subscale or by a few items among a larger set of items measuring other constructs. We excluded studies using measures designed for contexts other than clinical research or practice (such as the determination of non-material damage in legal actions). We excluded studies that stated they measured mental pain but used tools purporting to measure other constructs such as distress or depression.

Information sources and search strategy

As recommended by Cochrane, we systematically searched three databases relevant for psychiatry and psychological studies: PubMed, EMBASE and PsychINFO, from their dates of inception through 8 February 2022.³⁴ We searched for ‘mental pain’, as well as ‘psychological pain’, ‘psychache’ and ‘psychic pain’.⁷ Online supplemental material 2 presents the full search queries.

Selection of the relevant scientific publications

Two investigators (either CC or CLB and AC) independently screened the articles by title and abstract, using Rayyan, a free software program for systematic reviews. Disagreements were resolved by consensus. Online supplemental material 3 lists all the included papers.

Data extraction process

The full text of all studies selected by title and abstract was examined by one investigator (either CC or CLB) to identify potential measures of mental pain and their development/validation studies. A second investigator checked a randomly selected 30% of all screened studies (AC).

Data extraction form

The data charting form (online supplemental material 4) included all information recommended by COSMIN to evaluate the development, content validity and further measurement properties of each tool.³³

To provide a picture of the scientific use of the measures of mental pain, we also extracted the field of study, the study design (review, observational study, development and validation study, effectiveness study) and the population in which the measure was used. Finally, we evaluated whether mental pain was the main outcome of the study and if it involved suicidal behaviours.

Analysis and synthesis of results

All identified measures are presented in table 1 reporting their name with the first development and validation publication, whether it is a PROM, a brief description of the tool, any different versions of it, the underlying conceptual framework, its definition of mental pain and the initial purpose for which the measure was developed.

Table 1.

Description of the 10 measures of mental pain

	Self-reported scale	Description	Other versions of the scale	Conceptual framework of the measure	Initial purpose for which the measure was developed	Number of studies citing the scale*
The Psychological Pain Assessment ScalePPAS) Shneidman 1999	Yes	11 items VAS Rating of TAT-like pictures History of suicide Free essay No final scoring		Shneidman’s theory of the suicidal mind	To explore the relation between a suicidal act and mental pain	9
The Psychache Scale Holden 2001	Yes	13 items scored on a Likert scale		Shneidman’s theory of the suicidal mind	To explore the relation between a suicidal act and mental pain	85
The Orbach and Mikulincer Mental Pain Scale Orbach and Mikulincer 2003	Yes	44 items scored on a Likert scale	41 items (cited by Levinger 2015 but not validated) 40 items (Orbach, unpublished) 31 items (Tossani, 2019) 8 items (Casanova, 2021)	No a priori conceptual framework. Items developed from a survey based on grounded theory principles and qualitative content analysis to propose a comprehensive definition of this experience	To explore the relation between a suicidal act and mental pain	44
The Tolerance of Mental Pain Scale Orbach 2004	Yes	20 items scored on a Likert scale	10 items (Meerwijk, 2019)	No conceptual framework	To explore the relation between a suicidal act and mental pain	11
The Physical and Psychological Pain-VAS Olié 2009	Yes	3 dimensions of physical pain and 3 dimensions of psychological pain each evaluated by a VAS (scored from 0 to 10) +1 VAS evaluating suicidal ideation and one Likert scale evaluating the frequency of suicidal ideation No final scoring		Inspired by the measurement of physical pain	To explore the relation between a suicidal act and mental pain	21
The Mee-Bunney Psychological Assessment Pain Scale Mee and Bunney 2011	Yes	10 items scored on a Likert scale		Inspired by the measurement of physical pain and the Shneidman theory of the suicidal mind	To predict and prevent of suicide in routine clinical practice	11
The Three-dimensional Psychological Pain scale Li 2014	Yes	17 items scored on a Likert scale		No conceptual framework	To predict and prevent suicide in depressed patients	16
The Mental Pain Questionnaire Fava 2016	Yes	10 questions with dichotomous response format (ie, yes/no)		No conceptual framework	To explore the experience of mental pain in patients with mental disorder	7
The Unbearable Psychache Scale Pachkowski 2019	Yes	3 items scored on a Likert scale		Shneidman’s theory of the suicidal mind	To predict and prevent suicide in routine clinical practice	2
The Psychic Pain Scale Lewis 2020	Yes	12 items scored on a Likert scale	Original version of 20 items (Fowler, 2007, unpublished)	Malstberger’s theory of the suicidal mind	To explore the relation between a suicidal act and mental pain	2

*One study could have used several measures.

TAT, thematic apperception test; VAS, Visual Analogue Scale.

Content overlap and similarity of the measures

To investigate the similarity between the measures, we first conducted a semantic analysis to identify all the different words used to denominate the pain and its location, then performed a qualitative content analysis to identify the different domains assessed by each measure. Then we represented the content overlap graphically, and finally calculated a Jaccard Similarity Index. This method is explained extensively elsewhere and has been duplicated in several studies with, for example, depression scales, anxiety scales and mania scales.^{28 35 36}

Two researchers with professional experience with mental pain (AC and CC) and the patient with lived experience of mental pain (ST) conducted the semantic and qualitative content analyses. For the semantic analysis, they identified each word used to name or describe the pain and its location. In the qualitative content analysis, they coded all the different domains measured by each item of each scale by using their professional and experiential knowledge (each domain was coded with a specific number). Similarly worded items, as well as reverse worded items, were coded as the same domains. For example, “I cope with the pain even though it is difficult to bear” (Tolerance of Mental Pain Scale) and “I can’t take my pain anymore” (Psychache Scale) were classified as ‘unbearable’. Metaphors were coded according to their latent content; for instance, “my pain makes my life seem dark” (Psychache Scale) was coded as ‘hopelessness’ and “it feels like my heart crunched up” (Three-Dimensional Psychological Pain Scale) as ‘to be destroyed’. Some scale items were placed in multiple domains simultaneously (one item received several codes). For example, the item “When my feelings are intense, I can’t think straight” (Psychic Pain Scale) measures both ‘overflooded’ and ‘cognitive impairment’. Our conservative approach to avoid overestimating heterogeneity among and between scales assumed, when we hesitated between categories, that the items were similar rather than distinct.²⁸

We then represented the content overlap of the measures in figure 1.

Figure 1.

Content overlap of the different measures of mental pain. Each grey concentric circle represents a measurement tool (from outer circle to inner circle: Orbach and Mikulincer Mental Pain Scale (OMMPS), Psychological Pain Assessment Scale (PPAS), Tolerance of Mental Pain Scale (TMPS), Psychache Pain Scale (PPS), Three-Dimensional Psychological Pain Scale (TDPPS), Mental Pain Questionnaire (MPQ), Psychache Scale (PAS), Mee-Bunney Psychological Assessment Pain Scale (MBPPS), Physical and Psychological Pain Visual Analogue Scale (PPP-VAS) and Unbearable Psychache Scale (UP3)). The qualitative content analysis of the 112 items of the 10 measures retrieved 35 domains. A coloured dot on a grey concentric circle means that the domain is measured by the corresponding scale. For instance, 'anxiety' is measured by the PPP-VAS and the PPS. The 35 domains are presented anticlockwise around the circle, from the most frequently measured (seven measures assess pain intensity) to the least often measured (one measure assesses anger).

Finally, to determine the similarity of the content of the measures, we calculated a Jaccard Similarity Index; this coefficient of similarity for binary variables ranges from 0 (no similarity) to 1 (full similarity). The index is estimated based on the formula s/(u1+u2+s), where s represents the number of shared domains assessed by two measures, and u1 and u2 are the number of domains assessed respectively in the first and the second measure.²⁸ A rule of interpretation is that a Jaccard coefficient of 0.00–0.19 indicates very weak similarity, 0.20–0.39 weak similarity, 0.40–0.59 moderate similarity, 0.60–0.79 strong similarity and 0.80–1 very strong similarity.²⁸

Evaluation and comparison of the development, content validity and other psychometric properties of the measures of mental pain

We evaluated each measure with COSMIN (online supplemental material 5a and 5b).³³ This method applies a filter approach to evaluate successively: (1) the measure’s development, (2) its content validity and (3) its further psychometric properties, if and only if the content validity was rated ‘adequate’ or ‘very good’.³³

Information sources to evaluate the development and content validity

We used three sources to evaluate the development and content validity of the measures. First, we used all studies tagged ‘development and validation study’ during the data extraction process of the systematic review. Second, we screened all references of these studies to identify further references.^{32 33} Third, we emailed each corresponding author of each tool to request all relevant further development/validation materials (published and unpublished studies).^{32 33} Only the authors of the Three-Dimensional Psychological Pain Scale (TDPPS) (Li et al) failed to respond. The final sample of studies is listed in online supplemental material 6.

Evaluation of the measures

Three investigators with different backgrounds and expertise (CC, AC and YM) independently rated development, content validity and, if content validity was adequate, all other measurement properties. Disagreements, if any, were resolved by consensus with experts. The investigators reported no conflict of interest with any of these measures.

First, the investigators evaluated the 35 features of each measure’s development, classified in two categories: (1) 'design' (eg, the underlying conceptual framework, description of the target population and appropriateness of the qualitative data collection used to identify relevant items) and (2) ‘cognitive interview study or pilot test’ (eg, was a cognitive interview performed and were patients asked about the comprehensiveness of the measure?). Details of the 35 features can be found in box 1 of the COSMIN methodology for content validity user manual, available on their website and reproduced in online supplemental material 5a. Each of the 35 features is to be rated ‘very good’, ‘adequate’, ‘doubtful’, ‘inadequate’ or ‘not applicable’. Following the COSMIN recommendation that ‘poor methodological aspects of a study cannot be compensated by [different] good aspects’, the overall rating of each primary category is the lowest score of the features. However, COSMIN differentiates ‘fatal flaws’, which lead automatically to the rating of the entire category as ‘inadequate’, from other flaws that are rated adequate or doubtful, to lower their impact in global scoring.³³ For instance, not using an appropriate qualitative data collection method to identify relevant items for a new PROM leads to the rating of the overarching ‘design’ category as ‘inadequate’.

Second, we evaluated the content validity of each measure of mental pain. COSMIN defines three critical aspects of content validity: relevance (relevance of the items and response options for the target population, in the context of use), comprehensiveness (no key concepts missing) and comprehensibility (are the instructions, items and response options appropriately worded and understood by the target population). This definition is operationalised in 31 features, classified in 5 main categories: (1) ‘asking patients about the relevance of PROM items’, (2) ‘asking patients about comprehensiveness’, (3) ‘asking patients about comprehensibility’, (4) ‘asking professionals about relevance’ and (5) ‘asking professionals about comprehensiveness’. Details of the 31 features can be found in box 2 of the ‘COSMIN methodology for content validity’ user manual (online supplemental material 5b). These features were clarified by sample questions applied to each relevant item, such as ‘Was an appropriate method used for assessing the relevance/comprehensiveness/comprehensibility of the PROM?’ ‘Were skilled moderators used?’ ‘Was each item tested in an adequate number of patients?’ ‘Were professionals from all relevant disciplines included?’ Each of the 31 domains can be rated ‘very good’, ‘adequate’, ‘doubtful’, ‘inadequate’ or not applicable. Like the assessment of development, the overall rating is based on the lowest score of the features.

Third, we evaluated the other psychometric properties (structural validity, internal consistency, cross-cultural validity, measurement invariance, reliability, measurement error, criterion validity, hypothesis testing for construct validity and responsiveness) if and only if the content validity was rated above the level of ‘inadequate’ in the previous step.³³

Findings

Identification of the measures of mental pain and description

Data extraction on 8 February 2022 retrieved 2658 publications, with 299 full texts searched resulting in the final inclusion of 167 unique studies (flow chart in online supplemental material 7). Among them, we identified 107 observational studies, 38 development and validation papers, 11 reviews, 1 meta-analysis and 10 intervention studies (3 of them randomised controlled trials). Among the 153 primary research projects involving participants, most took place in North America (n=49) and Europe (n=44). Studies including clinical populations (n=75) mostly involved people with mood disorders (n=51). Overall, 97 studies investigated the relation between mental pain and suicidal ideation and behaviour. See online supplemental material 8 for a more detailed description of the included studies.

We identified 10 different standardised measures of mental pain (table 1). All of these measures are self-rated scales published between 1999 and 2021 (cited here from the oldest to the newest): the Psychological Pain Assessment Scale (PPAS), Psychache Scale (PAS), Orbach and Mikulincer Mental Pain Scale (OMMPS), Tolerance of Mental Pain Scale (TMPS), Physical and Psychological Pain Visual Analogic scale (PPP-VAS), Mee-Bunney Psychological Assessment Pain Scale (MBPPS), Three-Dimensional Psychological Pain Scale (TDPPS, Mental Pain Questionnaire (MPQ), Unbearable Psychache Scale (UP3) and Psychache Pain Scale (PPS) (table 1). The OMMPS has an original version composed of 44 items, and four shorter versions (41, 40, 31, 8 items), resulting from attempts to validate the scale in different samples (see references in online supplemental material 6). The original TMPS comprised 20 items, but a 10-item version was validated in three different samples (the USA, Turkey, Italy) (online supplemental material 6). The PPS has an original unpublished version of 20 items and a published version of 12 items (online supplemental material 6).

Table 2 presents the underlying definitions of mental pain as reported by the authors of each measure. Five measures used Shneidman’s definition. Seven measures were initially developed to explore the relations between mental pain and suicide (PPAS, PAS, OMMP, TMPS, PPP-VAS, MPQ, PPS), and three were developed to predict and prevent suicide in routine clinical practice (MBPPS, TDPPS, UP3) (table 1).

Table 2.

Definitions of mental pain underlying each measure of mental pain

	Definitions of mental pain
The Psychological Pain Assessment Scale Shneidman 1999	Definition given in the scale’s instructions: “Psychological pain is the same as somatic or physical pain, It is how you feel as a person; how you feel in your mind or heart. It refers to how much you hurt as a human being. It is mental suffering; inner torment. It is called psychache. Psychache refers to hurt or misery. It is the pain of shame, or guilt, or grief, or humiliation, or hopelessness, or loneliness, or sadness, or anguish. It is how you feel inside. It is an ache in the mind”. The author’s definition: ‘Psychological pain is the introspective experience of negative emotions such as anger, despair, fear, grief, shame, guilt, hopelessness, loneliness and loss—psychache’.
The Psychache Scale Holden 2001	Definition given in the scale’s instructions: “A hurting feeling inside, often described as pain you feel in your heart or mind. It indicates how much you hurt emotionally or mentally”. The authors had no specific definition and thus relied on Shneidman’s: 'Shneidman (1993) has asserted that the cause of suicide is psychache, that is, pain associated with psychache. This psychological pain is an aversive state that encompasses shame, guilt, humiliation, loneliness, fear, angst, dread, anguish, etc. When psychache surpasses an individual’s threshold for tolerance, it will cause suicidal behaviour’.
The Orbach and Mikulincer Mental Pain Scale Orbach and Mikulincer 2003	No definition given in the scale’s instructions. The author’s definition: 'Mental pain is defined as a wide range of subjective experiences characterised as an awareness of negative changes in the self and in its functions accompanied by negative feelings. Different factors reflect a distinct negative modification in self-image, self-definition, and self-functioning, tapping a different experiential aspect of these changes'.
The Tolerance of Mental Pain Scale Orbach 2004 (unpublished) Meerwijk 2019 (10-item version)	No definition given in the scale’s instructions. No specification of what is mental pain but a definition of the tolerance of mental pain. 'Tolerance for psychological pain refers to the ability to endure psychological pain'.
The Physical and Psychological Pain Scale Visual Analogue Scale Olié 2009	No definition given in the scale’s instructions: 'Instructions are purposely limited to simple sentences and no explicit definition is given'. The authors had no specific definition and thus relied on the definition of Shneidman: ‘Psychological pain or “psychache” is a central feature of mental disorders. It is defined as “the introspective experience of negative emotions such as dread, despair, fear, greed, shame, guilt, frustrated love, loneliness and loss”’ (Shneidman 1996).
The Mee-Bunney Psychological Assessment Pain Scale Mee and Bunney 2011	Definition given in the scale’s instructions: 'Intense psychological (mental pain) is a feeling which is experienced as unbearable torment. It can be experienced during a psychiatric disorder or a tragic loss such as the death of a child'. The authors' definition: 'In this study, intense “unbearable” psychological (mental) pain is defined as an emotionally based extremely aversive feeling which can be experienced as torment. It can be associated with a psychiatric disorder or with a severe emotional trauma such as the death of a child’.
The Three-dimensional Psychological Pain scale Li 2014	No definition given in the scale’s instruction. The authors had no specific definition and thus relied on the definition of Shneidman: 'Psychological pain, or psychache, has been defined as the “introspective experience of negative emotions such as dread, despair, fear, grief, shame, guilt, frustrated love, loneliness and loss” (Shneidman 1996), and is a common theme in suicide notes'.
The Mental Pain Questionnaire Fava 2016	Definition in the scale’s instruction: Mental or psychological pain is an experience that is part of life. It is different from physical pain. Definition of the authors: ‘Mental pain may be worse than most forms of physical pain, because it is not localised and often has no apparent reason. Grief provides an example of the sense of emptiness, loss of meaning and suffering that mental pain entails’.
The Unbearable Psychache Scale Pachkowski 2019	No definition given in the scale’s instructions. No specific definition of the authors, hence they relied on the definition of Shneidman: ‘First proposed by Shneidman, psychache refers to “the hurt, anguish, soreness, aching, psychological pain in the psyche, the mind”’ (Shneidman 1993).
The Psychic Pain Scale Lewis 2020	No definition given in the scale’s instructions. Definition of the authors: ‘Nearly all definitions of psychological pain focus on the central experience of overwhelming negative affect, and the devastating combination of experiencing this emotional state as both unbearable and impossible to resolve. Definitions of psychological pain in a general sense have typically included negative affect labels such as shame, sadness, and despair, and terms that allude to a sense of compromised mental integrity and psychological boundedness, such as disintegration, emotional flooding, and loss of control’.

We extracted the definition of mental pain proposed (1) in the instructions of each of the 10 scales if any and (2) provided in the paper cited in the left column.

The measure cited most often in the studies included in our review is the PAS (85 of 167 identified studies), followed by the OMMPS (44 studies) and the PPP-VAS (21 studies) (table 1).

Content overlap and similarity of the measures

Semantic analysis of the 10 measures identified 7 different words to designate the pain: pain, hurt, ache, suffering, torment, misery and anguish. Four measures used only the word ‘pain’ (OMMPS, PPP-VAS, MPQ, UP3) in framing the items; four used two different words (TMPS, MBPPS, TDPPS, PPS); and two used three or more different words (PPAS, PAS). Across the measures, the pain was described as ‘psychological’, ‘emotional‘ or felt ‘inside’, ‘in the mind’, ‘in the heart’, ‘in the whole body’ or ‘everywhere’. The qualification of ‘psychological’ was used most frequently—in five measures (PPAS, PAS, PPP-VAS, MBPPS, TDPPS); ‘mind’ was used in four (PPAS, PAS, TMPS, TDPPS), ‘inside’ in four (PPAS, PAS, TMPS, PPS), ‘heart’ in three (PPAS, TDPPS, MPQ) and finally ‘emotional’ (TDPPS), ‘everywhere’ (MPQ) and ‘whole body’ (OMMPS) were used in only one measure each. Two tools specified the pain’s localisation ‘in the mind’, as ‘psychological’ or ‘emotional’ only (PPP-VAS and MBPPS), whereas three measures only mentioned locations that were not explicitly mental, such as ‘inside’, ‘everywhere’ or in the ‘heart’ or ‘whole body’ (OMMPS, MPQ, PPS). The UP3 mentions no location for the pain.

The 10 measures together feature 112 items. The qualitative content analysis of all 112 retrieved 35 domains. For clarity, we present them in five broad categories: pain characterisation (10 domains, eg, ‘intensity’ and ‘frequency’), associated emotions (13 domains, eg, ‘emptiness’ and ‘sadness’), its functional consequences (7 domains, eg, ‘cognitive impairment’ and ‘suicidality’), destruction of the self (4 domains, eg, ‘self-estrangement’ and ‘feeling destroyed’) and a measure of physical pain (one domain: ‘current physical pain’).

Figure 1 presents the content overlap between all measures. No domain was common to all 10 measures, and 42.9% (15/35 domains) appeared in only one or two measures. ‘Pain intensity’ was the domain of mental pain appearing in the most measures (7/10), followed by ‘feeling destroyed’, ‘pain tolerance’, ‘suicidality’, ‘helplessness’, ‘hopelessness’ and the qualification of ‘worse than physical pain’. While some measures focused on only a few domains, such as the UP3 and PPP-VAS, which capture 3 and 4 domains, respectively, others capture substantially more, such as the PPAS (17 domains) and OMMPS (18 domains).

Our analysis of similarity resulted in 45 individual Jaccard Similarity Index scores (one for each pair of measures). All individual scores were 0.5 (PPP-VAS and MBPPS) or lower, that is, had at best; ‘moderate’ similarity. Indeed, 57.7% of the Jaccard scores were below 0.19, that is, had ‘very weak’ similarity (table 3).

Table 3.

Jaccard indices for quantifying the similarity of the 10 measures of mental pain

	PPAS	PAS	UP3	OMMPS	PPS	PPP-VAS	MBPPS	TDPPS	MPQ	TMPS
PPAS	1
PAS	0.18	1
UP3	0	0.22	1
OMMPS	0.35	0.13	0.05	1
PPS	0.30	0.22	0.15	0.35	1
PPP-VAS	0.11	0.08	0	0	0.06	1
MBPPS	0.16	0.27	0.17	0	0.13	0.50	1
TDPPS	0.22	0.43	0.18	0.12	0.27	0.15	0.33	1
MPQ	0.24	0.29	0.10	0.17	0.16	0.18	0.17	0.25	1
TMPS	0.19	0.33	0.06	0.32	0.28	0.12	0.25	0.24	0.14	1
Mean similarity	0.19	0.24	0.10	0.17	0.21	0.13	0.22	0.24	0.19	0.21

This table shows the Jaccard Index of pairwise overlap for each pair of scales, estimated with the Jaccard Similarity Index. Scores range from 0 (no overlap) to 1 (complete overlap). Interpretation: 0.00–0.19: very weak; 0.20–0.39: weak; 0.40–0.59: moderate; 0.60–0.79: strong; 0.80–1: very strong.

For example, the overlap between the MBPPS and the PPP-VAS is 0.5, which indicates moderate overlap.

MBPPS, Mee-Bunney Psychological Assessment Pain Scale; MPQ, Mental Pain Questionnaire; OMMPS, Orbach and Mikulincer Mental Pain Scale; PAS, Psychache Scale; PPAS, Psychological Pain Assessment Scale; PPP-VAS, Physical and Psychological Pain Visual Analogue Scale; PPS, Psychache Pain Scale; TDPPS, Three-Dimensional Psychological Pain Scale; TMPS, Tolerance of Mental Pain Scale; UP3, Unbearable Psychache Scale.

In four cases, the Jaccard Index was 0, indicating that not a single item overlapped across two measures. Assessing the average similarity of one measure with all the others, we found the highest average similarity coefficient was 0.24, for both PAS and TDPPS. That is, the highest average similarity of mental pain measures with all other measures was ‘weak’.

Evaluation and comparison of the development and content validity of the measures of mental pain

Following the COSMIN standards, we used the 43 development and validation studies listed in online supplemental material 6 to rate the quality of the measure’s development. It was rated ‘inadequate’ (PPAS, PAS, PPS, UP3, MPQ, TDPPS, PPP-VAS, OMMPS) or ‘doubtful’ (MBPPS, TMPS), mainly because of the paucity of data about the methods used to generate items (table 4). There was no published report dedicated to the development of any of these scales. Only for the OMMPS and the MBPPS were there a few lines in a validation paper about the measure’s development (qualitative analysis of interviews or notes of patients), but not enough information for a rigorous assessment.

Table 4.

Assessment and comparison of the development of the 10 measures of mental pain

Measures	Design of the measures							Cognitive interview (CI) study*				Total: design + CI study
	General design requirements					Concept elicitation†	Total design	General design requirements	Comprehensibility	Comprehensiveness	Total CI study
	Clear construct	Clear origin of construct	Clear target population for which the measure was developed	Clear context of use	PROM developed in sample representing the target population‡			CI study performed in sample representing the target population
Psychological Pain Assessment scale	V	V	A	A	I (no such study reported)	0	I	0	0	0	0	I
Psychache Scale	V	V	A	A	I (no such study reported)	0	I	0§	0	0	0	I
Unbearable Psychache Scale (3 items of the PAS)	V	V	A	V	I (no such study reported)	0	I	0	0	0	0	I
Orbach and Mikulincer Mental Pain Scale (44 items)	NA	V	A	A	A	D (no comprehensive report of the method used for concept elicitation)	D	0	0	0	0	I
Psychache Pain Scale (12 items)	A	V	A	A	I (no such study reported)	0	I	0	0	0	0	I
Physical and Psychological Pain Visual Analogue Scale	V	V	D (unclear)	A	I (no such study reported)	0	I	0	0	0	0	I
Mee-Bunney Psychological Assessment Pain Scale	V	V	V	V	V	D (no comprehensive report of the method used for concept elicitation)	D	V	D (no comprehensive report of the method used for concept elicitation)	D (no comprehensive report of the method used for concept elicitation)	D (no comprehensive report of the method used for concept elicitation)	D
Three-Dimensional Psychological Pain Scale	A	A	D (unclear)	V	I (no such study reported)	0	I	0	0	0	0	I
Mental Pain Questionnaire	D (unclear)	D (unclear)	D (unclear)	D (unclear)	I (no such study reported)	0	I	0	0	0	0	I
Tolerance of Mental Pain Scale	A	V	D (unclear)	D (unclear)	I (no such study reported)	0	I	V	D (no comprehensive report of the method used for concept elicitation)	D (no comprehensive report of the method used for concept elicitation)	D (no comprehensive report of the method used for concept elicitation)	D

This table is based on the COSMIN methodology for assessing the content validity of PROMs, user manual V.1.0 (https://www.cosmin.nl/wp-content/uploads/PROM-Development-ratings-for-COSMIN-website-v1.pdf).

For the detailed 35 items of the COSMIN checklist, please see online supplemental material 5a.

*0 indicates that a CI study (or part of it) was not performed. Following the COSMIN guidelines, if a cognitive interview study or any other kind of pilot test was not performed, the rest of the box can be skipped and the total quality of the PROM development study will be rated as inadequate. See COSMIN user manual p. 28.

†0 indicates that the measure was not developed in a sample representing the target population, the concept elicitation was therefore not rated as recommended by COSMIN pp. 19–20.

‡COSMIN considers input from members of the target population as essential in the development of a PROM. COSMIN recommends the use of a qualitative study that include a diversity of patients to cover the breadth of the construct of interest. If such study is not performed, this item should be rated ‘inadequate’. See COSMIN user manual p. 20.

§The Spanish version of the scale performed a pilot study to evaluate the comprehensibility of the PAS (Ordonez 2019). The Spanish version is rated ‘doubtful’ for the cognitive interview section because of the paucity of methodological information available.

A, adequate; COSMIN, Consensus-based Standards for the selection of health Measurement Instruments; D, doubtful; I, inadequate; NA, not applicable; PROM, patient-reported outcome measure; V, very good.

In addition, we found no study evaluating the content validity, as defined by COSMIN, that is, no study involved patients and healthcare professionals in evaluating the relevance, comprehensiveness and comprehensibility of the measures. Accordingly, we rated the content validity of all measures as ‘insufficient’ or ‘doubtful’ (online supplemental material 9).

Because the content validity of all measures was inadequate, we did not evaluate further psychometric properties. The COSMIN filter approach is, after all, based on the principle that a measure can be precise, reliable and consistent, yet fail to capture the intended construct.³³

Conclusions and clinical implications

Our systematic review identified 10 measures of mental pain developed over the past 20 years, mainly for the purpose of predicting suicide and used mostly in the field of clinical psychology. Some scales, such as the OMMPS, have shorter versions that we did not review independently.³⁷ The 10 measures identified vary substantially in their theoretical frameworks, definitions of mental pain, semantics and content (poor content overlap and weak to very weak similarity). This heterogeneity might well be a limitation in comparing or combining the results of studies using these measures, for example, for systematic reviews or meta-analyses.³⁰

We found no comprehensive report of the development of any of these 10 measures, nor any comprehensive assessment of their content validity. Therefore, as required by our study protocol that planned assessment according to COSMIN, the design of the development of these measures was rated ‘inadequate‘ or ‘doubtful’, while the content validity was rated as ‘inadequate’. The articles reporting on the scale validation tested these measures instead for several other psychometric properties.

To our knowledge, this is the first review to comprehensively identify measures of mental pain. It is also the first study to analyse in detail their content and their similarity with both qualitative and quantitative approaches. Finally, it is the first study to evaluate and compare their development and content validity. We hope our work lays the foundations for similar papers about other constructs, to enhance the level of evidence of validity in psychiatry and psychology.

One limitation of our study may be our choice of search terms. Although we included ‘psychache, psychic pain’, ‘psychological pain’ and ‘mental pain’, we omitted the term ‘emotional pain’, which is at times used in the more biological (and less clinical) literature. The reason for not including the term is our focus on clinical psychology and psychiatry.

Furthermore, we chose not to evaluate further psychometric or clinimetric properties to comply with the COSMIN guidelines, in accordance with our protocol.^{33 38} However, these properties, including dimensionality and internal consistency, may provide further information about mental pain to inform scale construction and iteration.^{38 39}

This study leads to three conclusions. First, based on the COSMIN guidelines, we cannot currently recommend one scale over any other for research or clinical purposes, mainly because of the lack of available data to evaluate the tools’ development and content validity. This development must be reported in detail and their content validity must be investigated. Second, too few measures were developed with the input of persons with lived experience of mental pain. This absence is especially notable given that all the measures are patient-reported outcomes.³³ Hence, there is a need to collect more empirical data about the subjective experience of mental pain to (1) evaluate the content validity of these tools, (2) better define the construct and (3) develop relevant, comprehensive and comprehensible items if the content validity proved insufficient.³³ Third, learning about constructs such as mental pain requires iterative exchange between theory and measurement.^{19 40} The existing measures can therefore be understood as important first efforts to gain an understanding of mental pain. These efforts must now inform our conceptualisation of mental pain and thereby pave the way towards improving it.^{19 41}

In conclusion, because mental pain is an outcome that matters greatly to patients, considerably more attention and efforts are called for to assess it properly. One critical next step is therefore to develop a patient-reported routine outcome measure for mental pain for both research and clinical purposes.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplementary information.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

Not applicable.