Skip to main content
NCBI home page
Clinical Neuropsychiatry logoLink to Clinical Neuropsychiatry
. 2022 Aug;19(4):197–205. doi: 10.36131/cnfioritieditore20220402

Assessment of Unresolved/Disorganized State of Mind in Relation to Attachment: A ROC Curve Study Using the Adult Attachment Interview and the Measure of Parental Style

Claudio Imperatori 1,✉, Mauro Adenzato 2,✉, Luigia Palmiero 1, Benedetto Farina 1, Rita B Ardito 2
PMCID: PMC9442855  PMID: 36101645

Abstract

Objective

The main objective of the present study was to use the Receiver Operating Characteristic (ROC) curve analysis to identify cut-off points for a self-report measure assessing parental style, i.e., the Measure of Parental Style (MOPS), that are able to discriminate individuals with disorganized internal working models (IWMs) of attachment with adequate accuracy, in terms of sensitivity and specificity. Establishing cut-off points for the MOPS could provide clinicians and researchers with a valuable tool to investigate the role of disorganized IWMs as a link between parental styles and mental health.

Method

A sample of 90 university students (mean age = 21.21 ± 2.05, females = 66) was enrolled in the study. We used the Adult Attachment Interview (AAI) to assess disorganized IWM and the MOPS to assess parental styles. Subsequently, we used ROC curve analysis to pursue the objective of the study.

Results

The ROC curve analysis showed that the MOPS total score (i.e., the combination of maternal and paternal dimensions) was able to discriminate individuals with disorganized IWMs from individuals with organized IWMs (AUC= 0.77). Specifically, a score ≥ 25 (Youden index= 0.497) categorized individuals with a sensitivity of 0.69 (69% of participants with disorganized IWMs were correctly identified) and a specificity of 0.81 (19% of participants were incorrectly identified as having disorganized IWMs).

Conclusions

Although the AAI has demonstrated high psychometric properties for assessing attachment representations in adulthood, its use is difficult when studies with large samples are to be conducted. As an alternative to the AAI, the MOPS can be used in studies with large populations, but no cut-off has yet been proposed. Here, we have identified cut-off points for the MOPS that are capable of detecting disorganized IWMs of attachment with adequate accuracy, and we suggest that this self-report is a useful brief instrument for detecting disorganized IWMs when time constraints prevent the use of the AAI (e.g., in studies with large samples or epidemiological studies).

Keywords: adult attachment interview, dysfunctional parenting, disorganized internal working models, measure of parental style, receiver operating characteristic (ROC) curve

Introduction

Many areas of research in disciplines such as psychology, psychotherapy, and psychiatry focus their interest on the role of parenting in mental health and personality development (e.g., Adenzato et al., 2006; Ardito et al., 2004; Bowlby, 1969; Collishaw et al., 2007; Hesse & Main, 2006). This central role is justified, inter alia, by the longitudinal data available to date on the relationship between parental style and short-, medium-, and long-term cognitive and emotional developmental outcomes (e.g., Carlo et al., 2011; Morrell & Murray, 2003; Pettit et al., 1997; Raby et al., 2015; Sulik et al., 2015; van IJzendoorn & Bakermans-Kranenburg, 2019; Whittle et al., 2014).

Several instruments have been proposed to measure parental style, and one of the most commonly used is the Parental Bonding Instrument (PBI) proposed by Parker et al. (1979). Compared to other self-report instruments (e.g., the Adverse Childhood Experiences International Questionnaire; World Health Organization, 4-5 May 2011) that can assess different types of potentially traumatic childhood events (e.g., physical, sexual, emotional abuse/neglect, peer violence, etc.), the PBI is a self-report questionnaire that focuses on two basic parenting dimensions: Care and Over-protection. More specifically, it measures the types of behaviors the individual remembers experiencing from his or her parents during the first 16 years of life and consists of 25 items: twelve items relate to the degree of empathy and caring (care scale) and 13 to the degree of control and protection (over-protection scale). The PBI has demonstrated satisfactory psychometric properties (Parker, 1989) and has been used in numerous studies to demonstrate the relationship between variables assessed with the instrument and various psychiatric disorders, particularly anxiety and depression (Kidd et al., 2022; Kullberg et al., 2020). Nevertheless, the PBI has been criticized (Harris & Brown, 1996) for not capturing experiences of abuse. To overcome this limitation, Parker and colleagues (1997) proposed a refined version of the PBI in 1997, the Measure of Parental Style (MOPS).

The MOPS is a complemented version of the PBI that captures experiences of abuse and of parental separation and loss. These experiences are thought to strongly predispose individuals to psychological distress and psychopathology (Bowlby, 1969). It consists of 30 items that separately assess maternal (15 items) and paternal (15 items) parental styles during the first 16 years of life. The items are grouped into three dimensions for each parent: indifference (e.g., “Ignored me”, “Left me on my own a lot”, “Was uninterested in me”), over-control (e.g., “Over-protective of me”, “Over-controlling of me”, “Critical of me”), and abuse (e.g., “Verbally abusive of me”, “Physically violent or abusive of me”, “Made me feel in danger”). The indifference and over-control MOPS scales correspond to the care and over-protection PBI dimensions, while the newly added scale MOPS abuse has sufficient independence to be considered separately (Parker, et al., 1997). Various studies have shown that the MOPS is a reliable and valid tool for assessing parental style (Parker et al., 1997; Picardi et al., 2013; Rumpold et al., 2002).

As originally proposed by Parker et al. (1997), the MOPS can serve as a measure of the likelihood of exposure to dysfunctional parenting. The importance of assessing dysfunctional parenting is evidenced by the fact that dysfunctional parenting, such as very low care, emotional abuse, and high over-control, has been reported to be a factor of psychopathological vulnerability, affecting development at both neurobiological and psychological levels (e.g., Adenzato et al., 2019; Farina et al., 2021; Measelle et al., 2017; Poletti et al., 2022; Teicher et al., 2016). Various forms of dysfunctional parenting have also been shown to be the strongest predictor of symptom severity, negative prognosis, and worst treatment outcome in several psychiatric disorders, and particularly in affective disorders (e.g., Lima et al., 2010; Lippard & Nemeroff, 2020; McCrory et al., 2017).

Many authors have also emphasized the role of dysfunctional parenting in the etiology of disorganized attachment. For example, highly insensitive care by an attachment figure (Main & Solomon, 1990), helpless-fearful profiles of parenting (Lyons-Ruth & Spielman, 2004), and frightening and threatening parental behaviors (Hesse & Main, 2006) have been linked to the development of disorganized attachment. Moreover, empirical studies showed that parents of children with disorganized attachment were often abusive and neglecting and that severe forms of disorganized attachment can constitute a type of childhood trauma that, along with other forms of childhood adverse experiences, is a risk factor for psychopathology during development and adulthood (Amos et al., 2011; Farina et al., 2019; Liotti, 2004; Velotti et al., 2022).

The processes underlying the development of disorganized attachment have been described by attachment theorists (e.g., Bowlby, 1969; Lyons-Ruth & Jacobvitz, 2016; Main & Solomon, 1986). From the perspective of attachment theory, each child builds interpersonal cognitive/affective schemas or internal working models (IWMs) as a result of ongoing interactions with caregivers. IWMs are mental representations of an individual’s attachment experiences in childhood that are formed from repeated interactions between the child and caregiver and typically remain stable into adulthood (Bowlby, 1988; Bretherton, 1991; Grossmann & Grossmann, 1991; Waters et al., 2000).

Individuals can have organized or disorganized IWMs of attachment. Individuals who have developed organized IWMs show coherent strategies for requesting care and comfort (Ainsworth, 1982; Bowlby, 1988; Main & Solomon, 1990) and coherent states of mind when recalling attachment in adulthood (Hesse, 2008). Conversely, individuals who have developed disorganized states of mind exhibit IWMs characterized by anxious, incoherent, and inconsistent expectations towards attachment figures, concurrent or sequential inconsistent behavioral strategies, and disoriented and/ or disorganized memories of trauma or loss (Hesse, 2008).

Given the role that disorganized IWMs of attachment play in predisposing individuals to significant vulnerability to many mental disorders, it is crucial to have reliable tools for its detection. Several studies (Bakermans-Kranenburg & Van IJzendoorn, 1993, 2009; Ravitz et al., 2010; Sagi et al., 1994) have shown that the Adult Attachment Interview (AAI; George et al., 1996) is an instrument with high reliability and validity to assess adult attachment and the corresponding organized or disorganized IWMs. The AAI is a semistructured clinical interview that focuses on early attachment experiences and their effects (see Method section for more details). This interview classifies individuals’ “state of mind” in relation to their childhood attachment experiences and provides an understanding of the processes that characterize the development of attachment experiences from the first months of life into adulthood.

Although the AAI has repeatedly shown high psychometric properties for assessing attachment in adulthood (e.g., Ravitz, et al., 2010), it is time-consuming because it must be administered, transcribed, and scored by trained professionals. Consequently, although it likely remains the best tool to use in clinical and research settings, its use is often difficult or infeasible when studies with large samples are to be conducted. In these cases, the use of self-report may be an appropriate choice. Unfortunately, although there are several self-report measures of adult attachment in the literature, such as the Adult Attachment Styles (AAS; Hazan & Shaver, 1987), the Adult Attachment Questionnaire (AAQ; Simpson et al., 1996), and the Experiences in Close Relationships-Revised (ECR-R; Fraley & Shaver, 2000), none of these measures is able to assess the construct of disorganization, so they are limited to the secure, anxious, and avoidant styles. The only exception is the 9-item Adult Disorganized Attachment (ADA; Paetzold et al., 2015), which was developed in the field of social psychology in the context of studying romantic relationships and is thus outside the context of the relationship between the development of disorganized IWMs and parental styles examined here.

Therefore, considering the association between dysfunctional parenting and disorganized attachment (Manassis et al., 1999), the main aim of the present study was to add to the existing literature a tool capable of detecting disorganized IWMs of attachment with adequate accuracy in terms of both sensitivity and specificity. To this end, we use the AAI and Receiver Operating Characteristic (ROC) curve analysis to determine appropriate cut-off points for MOPS. Determining these cut-off points could provide clinicians and researchers with a valuable tool to investigate the role of disorganized IWMs as a link between parental styles and mental health.

Material and Methods

Participants

This research is part of a larger prospective study examining the effects of dysfunctional parenting at psychopathological, cognitive, and neurophysiological levels (Adenzato et al., 2019; Farina et al., 2021; Massullo et al., 2022).

For the present study, an a priori power analysis was performed using MedCalc statistical software, version 19.2.6 (MedCalc Software Ltd, 2020). It showed that at a probability level of 0.05, a sample size of 72 (24 positive cases and 48 negative cases) was required to provide an adequate statistical power (1– β= 80%) to detect a satisfactory (i.e., ≥ 0.70) area under the curve (AUC) with a null hypothesis value of 0.50. Thus, a convenience sample of 90 university students were enrolled for the present study. Exclusion criteria assessed using a dichotomous checklist, were: diagnosis/ history of major neuropsychiatric disorders; inability to understand Italian; and refusal to give written informed consent.

Given the low prevalence (i.e., approximately 17%) of unresolved/disorganized attachment in student samples (Bakermans-Kranenburg & van IJzendoorn, 2009) and the time-consuming nature of administering, transcribing, and coding the AAI (Thomson & Jaque, 2017), 296 participants were pre-screened for parenting quality using the PBI quadrants (Parker, et al., 1979), which are known to be associated with attachment styles (Manassis, et al., 1999; Wilhelm et al., 2016). More specifically, of the 296 participants1, a random sample of 41 participants (i.e., almost twice as many as the 24 positive cases identified by the power analysis) who fell into the third quadrant (i.e., poor care and over-controlling parenting) and/or the fourth quadrant (i.e., neglecting bonding) with both parents (or at least with one parent in the absence of one parent) was included. A random sample of 49 participants who did not fall into these quadrants with both parents (or at least with one parent in the absence of one parent) was also included.

All participants voluntarily participated in the study, were informed of the general purpose of the study, and received neither compensation nor academic credit. The study was approved by the European University’s ethics review board according to the standards of the Declaration of Helsinki.

Self-report Assessment

After providing the written informed consent, all participants were administered the MOPS (Parker, et al., 1997) and a checklist assessing socio-demographic variables (i.e., sex, age, and education level). Participants were tested individually in a quiet room by a trained psychologist who informed them of the purpose of the study and explained how to complete the questionnaire.

The MOPS (Parker et al., 1997) is a 30-item self-report measure that separately assesses maternal (15 items) and paternal (15 items) parental styles during the first 16 years of life. It is the revised version of the Parental Bonding Instrument (Parker et al., 1979), in which items are rated on a 4-point Likert scale (from "not true at all" to "extremely true" i.e., from 0 to 3) and grouped into three dimensions for each parent, as confirmed by principal component analysis (Parker et al., 1997): indifference, over-control, and abuse. Higher scores indicate higher levels of self-reported dysfunctional parenting. In the present study, the Italian version of the instrument was used (Picardi et al., 2013), and the Cronbach's alpha in the present sample was ≥ 0.74 for the MOPS sub-scales.

Adult Attachment Interview

On a different day from the self-report assessment, participants underwent the AAI (George et al., 1996). The interviews were conducted, transcribed, and coded by two clinical psychologists (R.B.A. and L.P., who are certified as reliable AAI coders), using the coding system of Main and colleagues (Main et al., 2003). Participants were classified into one of five categories for overall state of mind with respect to attachment: 1) Secure/Autonomous (F); 2) Dismissing (Ds); 3) Preoccupied (E); 4) Unresolved/Disorganized (U/d); 5) Cannot classify (CC). In contrast to F, Ds, and E (i.e., organized IWMs), individuals classified as U/d show signs of disorientation and disorganization in monitoring reasoning or discourse during discussions of potentially traumatic events such as loss or abuse. Individuals classified as CC instead show a global disruption of attachment strategy, with oscillations between opposite and contradictory mental states (Ds, E) or low coherence, indicating a general inability to adopt an organized stance. Categories U/d and CC are considered disorganized classifications (i.e., disorganized IWMs).

According to previous reports (Barone & Carone, 2020; Massullo et al., 2022), 20% of the AAIs (N= 18) were double-coded. Inter-rater agreement for the two-way analysis (disorganized vs. organized) was 100%, kappa= 1.00; inter-rater agreement for the four-way analysis (F, Ds, E, U/d or CC) was 94%, kappa= .82. All coders were blind to all other measures and participant information.

Statistical analyses

All analyses were performed with SPSS 18.0 Statistical Package for the Social Sciences (IBM, Armonk, NY, USA). In the present study, the Receiver Operating Characteristic (ROC) test procedures (Ruopp et al., 2008) were used to assess the performance of MOPS in categorizing organized (i.e., F, Ds, E) and disorganized (i.e., U/d and CC) individuals according to the AAI classification.

A ROC curve is a two dimensional depiction of test performance (Fawcett, 2006) and the area under the ROC curve (i.e., the probability that a randomly sampled respondent will be correctly assigned to the appropriate group) is considered the key outcome variable (Centor & Schwartz, 1985). The AUC reflects the overall accuracy of the instrument in categorizing individuals with values between 0.70-0.80, 0.800.90, and ≥ 0.90, considered adequate, excellent, and outstanding, respectively (Mandrekar, 2010; Zhang et al., 2020). In the present study the Youden Index (J; Youden, 1950) was also computed in order to detected the optimum cut-off points of MOPS dimensions (Ruopp et al., 2008). This index identifies the thresholds that maximize both sensitivity (i.e., the proportion of subjects who have the target condition and provide positive test results) and specificity (i.e., the proportion of subjects who do not have the target condition and provide negative test results). The non-parametric distribution assumption was chosen.

Differences between the organized and disorganized groups were also examined by chi-square (χ2) and Kolmogorov-Smirnov tests for dichotomous and dimensional variables, respectively. The use of nonparametric tests was chosen because none of the examined variables were normally distributed (Shapiro– Wilk test, p< 0.05). To avoid family-wise Type I error, a formal Bonferroni correction was applied by dividing the limit of significance by the number of comparisons (i.e., 12). Therefore, the threshold level for significance was p= 0.05/12 = 0.004.

Results

Fifty-three (58.9%) and thirty-seven (41.1%) participants were classified as organized and disorganized, respectively, according to the AAI coding system. The differences between the groups are shown in table 1. Compared with organized participants, disorganized individuals had higher scores on all MOPS dimensions. However, after a formal Bonferroni correction, only the MOPS total score, the maternal abuse sub-scale, and the maternal and paternal total score remained significantly different between the two groups.

Table 1.

Demographic and clinical data of the participants

Variables Total (N = 90) Organized (N = 53) Disorganized (N = 37) Test p
Females – N (%) 66 (73.3) 41(77.4 %) 25 (67.6 %) χ2= 1.07 0.301
Age – M ± SD 21.21 ± 2.05 20.51 ± 1.50 22.22 ± 2.31 Z= 1.42 0.036
Educational Level (years) –M ± SD 14.96 ± 1.48 14.74 ± 1.51 15.27 ± 1.41 Z= 0.98 0.297
MOPS total score –M (SD) 20.83 ± 18.66 13.81 ± 13.88 31.46 ± 20.09 Z= 2.28 < 0.001
Maternal indifference –M (SD) 2.74 ± 4.02 1.36 ± 2.27 4.73 ± 5.06 Z= 1.74 0.005
Maternal over-control –M (SD) 4.74 ± 3.32 3.92 ± 2.95 5.92 ± 3.50 Z= 1.54 0.017
Maternal abuse –M (SD) 2.54 ± 3.65 1.26 ± 2.05 4.38 ± 4.57 Z= 2.07 < 0.001
Maternal sub-scale total score –M (SD) 10.03 ± 9.59 6.55 ± 6.26 15.03 ± 11.29 Z= 2.22 < 0.001
Paternal indifference –M (SD) 4.14 ± 5.11 2.81 ± 4.36 6.14 ± 5.56 Z= 1.58 0.013
Paternal over-control –M (SD) 3.64 ± 3.10 2.81 ± 2.86 4.89 ± 3.08 Z= 1.42 0.035
Paternal abuse –M (SD) 3.05 ± 4.25 1.64 ± 2.81 5.17 ± 5.15 Z= 1.63 0.010
Paternal sub-scale total score –M (SD) 10.82 ± 11.09 7.26 ± 9.22 16.20 ± 11.62 Z= 2.11 < 0.001
Adult Attachment Interview classification
Secure/Autonomous –N (%) 21 (23.3) 21 (39.6) - - -
Dismissing –N (%) 22 (24.4) 22 (41.5) - - -
Preoccupied –N (%) 10 (11.1) 10 (18.9) - - -
Unresolved/disorganized –N (%) 34 (37.8) - 34 (91.9) - -
Cannot classify –N (%) 3 (3.3) - 3 (8.1) - -
Open in a new tab

Abbreviation: MOPS= Measure of Parental Style

Note: bold indicates significant differences between groups after a formal Bonferroni correction (i.e., p= 0.004).

Discriminant validity of MOPS scores

Detailed statistics on the ROC curves are shown in table 2.

Table 2.

Detailed ROC curves statistics

Test Variable(s) AUC Standard error p Confidence Interval
Lower Upper J index – MOPS cut-off (Sensitivity; 1-Specificity)
Maternal sub-scale total 0.74 0.06 < 0.001 0.63 0.85

0.476 – 9.50

(0.70; 0.23)
Maternal indifference 0.73 0.05 < 0.001 0.63 0.84

0.373 – 3.50

(0.49; 0.11)
Maternal over-control 0.67 0.06 0.006 0.55 0.79

0.330 – 5.50

(0.60; 0.26)
Maternal abuse 0.73 0.06 < 0.001 0.62 0.84

0.444 – 2.50

(0.60; 0.15)
Paternal sub-scale total 0.75 0.05 < 0.001 0.65 0.86

0.460 – 5.50

(0.80; 0.34)
Paternal indifference 0.70 0.06 0.001 0.59 0.82

0.345 – 3.50

(0.57; 0.23)
Paternal over-control 0.70 0.06 0.001 0.59 0.81

0.310 – 1.50

(0.86; 0.55)
Paternal abuse 0.70 0.06 0.001 0.59 0.82

0.354 – 3.50

(0.54; 0.19)
MOPS total score 0.77 0.05 < 0.001 0.67 0.87

0.497 – 24.5

(0.69;0.19)
Open in a new tab

Abbreviation: AUC= Area under the curve; MOPS= Measure of Parental Style; J= Youden index

The ROC curve analysis showed that the MOPS total score (i.e., the combination of maternal and paternal dimensions) was able to categorize (AUC= 0.77) individuals with disorganized IWMs (N= 352) from individuals with organized IWMs (N= 53) (figure 1). Specifically, a score ≥ 25 (Youden index= 0.497) categorized individuals with a sensitivity of 0.69 (69% of all participants with disorganized IWMs were correctly identified) and a specificity of 0.81 (19% of participants were incorrectly identified as having disorganized IWMs).

Figure 1.

Figure 1.

Open in a new tab

ROC curve graph for the ability of the MOPS total score (i.e., the sum of the maternal and paternal dimensions) to discriminate between individuals with organized and disorganized IWMs of attachment

When considering the MOPS maternal dimension, a ROC curve procedure (figure 2A) showed that all MOPS maternal sub-scales (AUC ≥ 0.73), with the exception of maternal over-control (AUC= 0.670), were able to classify disorganized individuals (N= 37). Nevertheless, better statistical values in terms of sensitivity and specificity were obtained for the total score of the maternal sub-scales. Specifically, a score ≥ 10 on the maternal dimension (Youden index= 0.476) categorized individuals with a sensitivity of 0.70 (70% of all disorganized individuals were correctly identified) and a specificity of 0.77 (23% of participants were incorrectly identified as disorganized individuals).

Figure 2.

Figure 2.

Open in a new tab

ROC curve graph for the ability of the MOPS maternal (panel A) and paternal (panel B) dimension to discriminate between individuals with organized and disorganized IWMs of attachment

Finally, considering the MOPS paternal dimension, a ROC curve procedure (figure 2B) showed that all MOPS paternal sub-scales (AUC ≥ 0.70) could classify disorganized individuals (N= 351). Similar to the maternal dimension, better statistical values in terms of sensitivity and specificity were obtained for the total score of the paternal sub-scales. Specifically, a score ≥ 6 on the paternal dimension (Youden index= 0.460) categorized individuals with a sensitivity of 0.80 (80% of all disorganized individuals were correctly identified) and a specificity of 0.66 (34% of participants were incorrectly identified as disorganized).

Discussion

The main aim of the present study was to determine cut-off points for MOPS capable of detecting disorganized IWMs of attachment. Our results showed that all MOPS sub-scales, with the exception of maternal over-control (AUC= 0.670), were adequate in discriminating (i.e., AUC ≥ 0.70) between individuals with disorganized IWMs and individuals with organized IWMs. However, better statistical values in terms of sensitivity and specificity were obtained for the MOPS total score (i.e., the sum of the maternal and paternal dimensions), which was able to discriminate (AUC= 0.77) between individuals with disorganized IWMs and individuals with organized IWMs with a sensitivity of 0.69 and a specificity of 0.81.

As mentioned previously, the forms of dysfunctional parenting measured by the MOPS significantly increase the likelihood of developing disorganized attachment and predispose individuals to significant vulnerability to various mental disorders (Liotti, 2004). Therefore, the detection of disorganized IWMs of attachment in adults is of great importance to both clinicians and researchers. Indeed, disorganized IWMs of attachment are considered a vulnerability factor for psychopathology in general and specifically for affective and trauma-related disorders (Farina & Meares, 2022; Liotti, 2004, 2009; van Hoof et al., 2019). In samples of families at high risk for psychopathology, the percentage of disorganized attachment can be as high as 80% (Lyons-Ruth & Jacobvitz, 2008). Individuals with a history of disorganized attachment are overrepresented among patients with mental disorders, and regardless of diagnosis, they exhibit relational problems, emotional and behavioral dysregulations, and mentalization difficulties, which complicates their treatment (Farina et al., 2019; Fonagy et al., 1991; Liotti, 2017; Meares, 2012). Regardless of diagnosis, individuals with elements of disorganized attachment tend to have specific therapeutic difficulties caused by the activation of disorganized IWMs in the therapeutic relationship (Farina et al., 2019).

It is very important for researchers to have an agile tool for assessing disorganized IWMs, as it may allow the study of the role that disorganized IWMs play in populations with different mental disorders. In longitudinal or prevention studies, it may also allow the investigation of disorganized IWMs as a risk factor. There is little doubt that the AAI is the best instrument available in the literature for these purposes. However, the procedure of this highly accurate interview is time-consuming and difficult when examining large samples (and indeed, studies conducted with the AAI rarely have sample sizes greater than 100). As an alternative to the AAI, the MOPS can be used in the study of large populations, but no cut-off has ever been proposed that can distinguish between individuals at higher psychopathological risk due to disorganized IWMs and individuals at lower risk due to organized IWMs. In this study, we identified cut-off points for MOPS that are able to detect disorganized IWMs of attachment with adequate accuracy in terms of sensitivity and specificity. Specifically, we have shown that a cut-off score ≥ 25 to the total score of the MOPS provide clinicians and researchers with a rapid but reliable index to investigate the role of disorganized IWMs as a link between parental styles and mental health.

It is important to underline that the correspondence between the AAI classification of the 90 participants involved in the present study and the overall accuracy of MOPS in categorizing individuals (expressed as AUC in the ROC curve analysis) was adequate but not excellent. In our opinion, this finding primarily reflects the fact that even if dysfunctional parenting (assessed by MOPS) may constitute the basis for the development of disorganized attachment (assessed by the AAI), this causal relationship is not linear. Indeed, it has been shown that variables other than dysfunctional parenting should be considered when examining disorganized attachment. In particular, Wazana et al. (2015) examined the combined effects of constitutional risk, molecular genetics, and environmental influences on the development of attachment disorganization, and demonstrated a complex interplay between variables such as birth weight, the dopamine receptor D4 gene, and early maternal care. Therefore, when using MOPS as a proxy for assessing disorganized attachment, researchers should consider that although dysfunctional parenting may play an important role in the etiology of disorganized attachment, other variables may contribute to the process.

The contributions of the study should be considered in light of some limitations. First, we did not examine the psychometric properties of the MOPS in patients with mental disorders, so our interpretations are specific to non-clinical individuals. In addition, there was a high proportion of female participants and young adults in our sample, which limits the generalizability of these results to other populations. Therefore, further studies should be conducted to investigate the performance of MOPS in detecting disorganized IWMs in different groups, especially in clinical samples.

Despite these limitations, to our knowledge, this is the first study ever that provides cut-off scores for the MOPS and examines the accuracy of MOPS in detecting disorganized IWMs. Taken together, our results suggest that the MOPS, particularly the MOPS total score (i.e., the sum of the maternal and paternal dimensions), is a useful brief self-report instrument for detecting disorganized IWMs when time constraints prevent the use of the AAI (e.g., studies with large samples or epidemiologic studies).

Acknowledgements

MA and RBA were supported by University of Turin grants (“Ricerca scientifica finanziata dall’Università”).

Footnotes

1

According to the PBI quadrants, the following prevalences of parental style were found in the pre-screened participants (N= 296): 41.2% (N= 122) maternal optimal bonding, 26.7% (N= 79) maternal affectionate constraint, 17.6% (N= 52) maternal affectionless control, 13.9% (N= 49) maternal neglectful parenting, 37.8% (N= 112) paternal optimal bonding, 25% (N= 74) paternal affectionate constraint, 10.8% (N= 74) paternal affectionless control, 24% (N= 71) paternal neglectful parenting. Two and seven participants were unable to complete the maternal and paternal dimensions of the PBI, respectively.

2

Two participants were not able to complete paternal dimension of the MOPS (i.e., early death).

References

  1. Adenzato, M., Ardito, R. B., & Izard, E. (2006). Impact of maternal directiveness and overprotectiveness on the personality development of a sample of individuals with acquired blindness. Social Behavior and Personality, 34(1), 17-26. doi: 10.2224/sbp.2006.34.1.17 [DOI] [Google Scholar]
  2. Adenzato, M., Imperatori, C., Ardito, R. B., Valenti, E. M., Marca, G. D., D'Ari, S., . . .Farina B., (2019). Activating attachment memories affects default mode network in a non-clinical sample with perceived dysfunctional parenting: An EEG functional connectivity study. Behavioural Brain Research, 372, 112059. doi: 10.1016/j.bbr.2019.112059 [DOI] [PubMed] [Google Scholar]
  3. Ainsworth, M. D. S. (1982). Attachment: Retrospect and prospect. In Parkes C. M. & Stevenson-Hinde J. (Eds.), The Place of Attachment in Human Behavior (pp. 3-30). New York: Basic Books. [Google Scholar]
  4. Amos, J., Furber, G., & Segal, L. (2011). Understanding maltreating mothers: a synthesis of relational trauma, attachment disorganization, structural dissociation of the personality, and experiential avoidance. Journal of Trauma & Dissociation, 12(5), 495-509. doi: 10.1080/15299732.2011.593259 [DOI] [PubMed] [Google Scholar]
  5. Ardito, R. B., Adenzato, M., Dell'Osbel, G., Izard, E., & Veglia, F. (2004). Attachment representations in adults with congenital blindness: association with maternal interactive behaviors during childhood. Psychological Reports, 95(1), 263-274. doi: 10.2466/pr0.95.1.263-274 [DOI] [PubMed] [Google Scholar]
  6. Bakermans-Kranenburg, M. J., & Van IJzendoorn, M. H. (1993). A psychometric study of the Adult Attachment Interview: Reliability and discriminant validity. Developmental Psychology, 29(5), 870-879. doi: 10.1037/00121649.29.5.870 [DOI] [Google Scholar]
  7. Bakermans-Kranenburg, M. J., & Van IJzendoorn, M. H. (2009). The first 10,000 Adult Attachment Interviews: distributions of adult attachment representations in clinical and non-clinical groups. Attachment & Human Development, 11(3), 223-263. doi: 10.1080/14616730902814762 [DOI] [PubMed] [Google Scholar]
  8. Barone, L., & Carone, N. (2020). Childhood abuse and neglect experiences, Hostile-Helpless attachment, and reflective functioning in mentally ill filicidal mothers. Attachment & Human Development, 23(6), 1-24. doi: 10.1080/14616734.2020.1738510 [DOI] [PubMed] [Google Scholar]
  9. Bowlby, J. (1969). Attachment and loss, Vol. 1. Attachment. London: Hogarth Press. [Google Scholar]
  10. Bowlby, J. (1988). A secure base. New York: Basic Books. [Google Scholar]
  11. Bretherton, I. (1991). Pouring new wine into old bottles: the social self as internal working model. In Gunnar M. & Sroufe L. A. (Eds.), Minnesota Symposia in Child Psychology: Self Processes in Development (Vol. 23, pp. 1-41). Hillsdale, NJ: Lawrence Erlbaum Associates. [Google Scholar]
  12. Carlo, G., Mestre, M. V., Samper, P., Tur, A., & Armenta, B. (2011). The longitudinal relations among dimensions of parenting styles, sympathy, prosocial moral reasoning, and prosocial behaviors. International Journal of Behavioral Development, 35(2), 116-124. doi: 10.1177/0165025410375921 [DOI] [Google Scholar]
  13. Centor, R. M., & Schwartz, J. S. (1985). An evaluation of methods for estimating the area under the receiver operating characteristic (ROC) curve. Medical Decision Making, 5(2), 149-156. doi: 10.1177/0272989X8500500204 [DOI] [PubMed] [Google Scholar]
  14. Collishaw, S., Pickles, A., Messer, J., Rutter, M., Shearer, C., & Maughan, B. (2007). Resilience to adult psychopathology following childhood maltreatment: evidence from a community sample. Child Abuse & Neglect, 31(3), 211-229. doi: 10.1016/j.chiabu.2007.02.004 [DOI] [PubMed] [Google Scholar]
  15. Farina, B., Imperatori, C., Adenzato, M., & Ardito, R. B. (2021). Perceived parental over-protection in non clinical young adults is associated with affective vulnerability: A cross-sectional study. Journal of Affective Disorders, 292, 496-499. doi: 10.1016/j.jad.2021.05.071 [DOI] [PubMed] [Google Scholar]
  16. Farina, B., Liotti, M., & Imperatori, C. (2019). The Role of Attachment Trauma and Disintegrative Pathogenic Processes in the Traumatic-Dissociative Dimension. Frontiers in Psychology, 10, 933. doi: 10.3389/fpsyg.2019.00933 [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Farina, B., & Meares, R. (2022). The Traumatic Disintegration Dimension. In Dorahy M. J., Gold S. N. & O'Neil J. A. (Eds.), Dissociation and the dissociative disorder: past, present, future. New York: Routledge. [Google Scholar]
  18. Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters 27. doi: 10.1016/j.patrec.2005.10.010 [DOI] [Google Scholar]
  19. Fonagy, P., Steele, M., Steele, H., Moran, G. S., & Higgitt, A. C. (1991). The capacity for understanding mental states: The reflective self in parent and child and its significance for security of attachment. Infant Mental Health Journal, 12(3), 201-218. doi: 10.1002/1097-0355(199123)12:3<201::AID-IMHJ2280120307>3.0.CO;2-7 [DOI] [Google Scholar]
  20. Fraley, R. C., & Shaver, P. R. (2000). Adult romantic attachment: Theoretical developments, emerging controversies, and unanswered questions. Review of General Psychology, 4(2), 132-154. doi: 10.1037/10892680.4.2.132 [DOI] [Google Scholar]
  21. George, C., Kaplan, N., & Main, M. (1996). Adult Attachment Interview Protocol, (3rd ed). University of California at Berkekey.
  22. Grossmann, K. E., & Grossmann, K. (1991). Attachment quality as an organizer of emotional and behavioral responses in a longitudinal perspective. In Parkes C. M., Stevenson-Hinde J. & Marris P. (Eds.), Attachment across the life cycle (pp. 93-114). London: Tavistock/Routledge. [Google Scholar]
  23. Harris, T. O., & Brown, G. W. (1996). Social causes of depression. Current Opinion in Psychiatry, 9(1), 3-10. [Google Scholar]
  24. Hazan, C., & Shaver, P. (1987). Romantic love conceptualized as an attachment process. Journal of Personality and Social Psychology, 52(3), 511-524. doi: 10.1037//00223514.52.3.511 [DOI] [PubMed] [Google Scholar]
  25. Hesse, E. (2008). The Adult Attachment Interview: Protocol, method of analysis, and empirical studies. In Cassidy J. & Shaver P. R. (Eds.), Handbook of attachment: Theory, research, and clinical applications (pp. 552–598). New York: The Guilford Press. [Google Scholar]
  26. Hesse, E., & Main, M. (2006). Frightened, threatening, and dissociative parental behavior in low-risk samples: description, discussion, and interpretations. Development and Psychopathology, 18(2), 309-343. doi: 10.1017/S0954579406060172 [DOI] [PubMed] [Google Scholar]
  27. Kidd, K. N., Prasad, D., Cunningham, J. E. A., de Azevedo Cardoso, T., & Frey, B. N. (2022). The relationship between parental bonding and mood, anxiety and related disorders in adulthood: A systematic review and meta-analysis. Journal of Afective Disorders, 307, 221-236. doi: 10.1016/j.jad.2022.03.069 [DOI] [PubMed] [Google Scholar]
  28. Kullberg, M. L., Maciejewski, D., van Schie, C. C., Penninx, B., & Elzinga, B. M. (2020). Parental bonding: Psychometric properties and association with lifetime depression and anxiety disorders. Psychological Assessment, 32(8), 780795. doi: 10.1037/pas0000864 [DOI] [PubMed] [Google Scholar]
  29. Lima, A. R., Mello, M. F., & Mari Jde, J. (2010). The role of early parental bonding in the development of psychiatric symptoms in adulthood. Current Opinion in Psychiatry, 23(4), 383-387. doi: 10.1097/yco.0b013e32833a51ce [DOI] [PubMed] [Google Scholar]
  30. Liotti, G. (2004). Trauma, dissociation and disorganized attachment: three strands of a single braid. Psychotherapy Theory Research Practice Training, 41(4), 472-486. doi: 10.1037/0033-3204.41.4.472 [DOI] [Google Scholar]
  31. Liotti, G. (2009). Attachment and dissociation. In Dell P. F. & O’Neil J. A. (Eds.), Dissociation and the Dissociative Disorders (pp. 53-65). New York: Routledge. [Google Scholar]
  32. Liotti, G. (2017). Conflicts between motivational systems related to attachment trauma: Key to understanding the intra-family relationship between abused children and their abusers. Journal of Trauma & Dissociation, 18(3), 304318. doi: 10.1080/15299732.2017.1295392 [DOI] [PubMed] [Google Scholar]
  33. Lippard, E. T. C., & Nemeroff, C. B. (2020). The Devastating Clinical Consequences of Child Abuse and Neglect: Increased Disease Vulnerability and Poor Treatment Response in Mood Disorders. American Journal of Psychiatry, 177(1), 20-36. doi: 10.1176/appi.ajp.2019.19010020 [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Lyons-Ruth, K., & Jacobvitz, D. (2008). Attachment disorganization: genetic factors, parenting contexts, and developmental transformation from infancy to adulthood. In Cassidy J. & Shaver P. R. (Eds.), Handbook of attachment: Theory, research, and clinical applications (pp. 666-697). New York: The Guilford Press. [Google Scholar]
  35. Lyons-Ruth, K., & Jacobvitz, D. (2016). Attachment disorganization from infancy to adulthood. In Cassidy J. & Shaver P. (Eds.), Handbook of attachment: Theory, research, and clinical applications (3rd ed., pp. 667-695). New York: Guilford Press. [Google Scholar]
  36. Lyons-Ruth, K., & Spielman, E. (2004). Disorganized Infant Attachment Strategies and Helpless-Fearful Profiles of Parenting: Integrating Attachment Research with Clinical Intervention. Infant Mental Health Journal, 25(4), 318335. doi: 10.1002/imhj.20008 [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Main, M., Goldwyn, R., & Hesse, E. (2003). Adult attachment classification system. Version 7.2. University of California at Berkekey.
  38. Main, M., & Solomon, J. (1986). Discovery of a new, insecure-disorganized/disoriented attachment pattern. In Yogman M. & Brazelton T. B. (Eds.), Affective development in infancy (pp. 95-124). Norwood, NJ: Ablex. [Google Scholar]
  39. Main, M., & Solomon, J. (1990). Procedures for identifying infants as disorganized/disoriented during the Ainsworth Strange Situation. In Greenberg M. T., Cicchetti D. & Cummings E. M. (Eds.), Attachment in the preschool years: Theory, research, and intervention (pp. 121-160). Chicago: University of Chicago Press. [Google Scholar]
  40. Manassis, K., Owens, M., Adam, K. S., West, M., & Sheldon-Keller, A. E. (1999). Assessing attachment: convergent validity of the adult attachment interview and the parental bonding instrument. Australian and New Zealand Journal of Psychiatry, 33(4), 559-567. doi: 10.1080/j.1440-1614.1999.00560.x [DOI] [PubMed] [Google Scholar]
  41. Mandrekar, J. N. (2010). Receiver operating characteristic curve in diagnostic test assessment. Journal of Thoracic Oncology, 5(9), 1315-1316. doi: 10.1097/JTO.0b013e3181ec173d [DOI] [PubMed] [Google Scholar]
  42. Massullo, C., Imperatori, C., De Vico Fallani, F., Ardito, R. B., Adenzato, M., Palmiero, L., . . . Farina B. (2022). Decreased brain network global efficiency after attachment memories retrieval in individuals with unresolved/ disorganized attachment-related state of mind. Scientific Reports, 12(1), 4725. doi: 10.1038/s41598-022-08685-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. McCrory, E. J., Gerin, M. I., & Viding, E. (2017). Annual Research Review: Childhood maltreatment, latent vulnerability and the shift to preventative psychiatry – the contribution of functional brain imaging. Journal of Child Psychology and Psychiatry, 58(4), 338-357. doi: 10.1111/jcpp.12713 [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Meares, R. (2012). The dissociation model of borderline personality disorder. New York: Norton Professional Books. [Google Scholar]
  45. Measelle, J. R., David, J., & Ablow, J. C. (2017). Increased levels of inflammation among infants with disorganized histories of attachment. Behavioural Brain Research, 325(Pt B), 260-267. doi: 10.1016/j.bbr.2016.12.001 [DOI] [PubMed] [Google Scholar]
  46. MedCalc Software Ltd, O., Belgium. (2020). https://www.medcalc.org/.
  47. Morrell, J., & Murray, L. (2003). Parenting and the development of conduct disorder and hyperactive symptoms in childhood: a prospective longitudinal study from 2 months to 8 years. Journal of Child Psychology and Psychiatry, 44(4), 489-508. doi: 10.1111/1469-7610.t01-1-00139 [DOI] [PubMed] [Google Scholar]
  48. Paetzold, R. L., Rholes, W. S., & Kohn, J. L. (2015). Disorganized attachment in adulthood: Theory, measurement, and implications for romantic relationships. Review of General Psychology, 19(2), 146-156. doi: 10.1037/gpr0000042 [DOI] [Google Scholar]
  49. Parker, G. (1989). The Parental Bonding Instrument: psychometric properties reviewed. Psychiatric Developments, 7(4), 317-335. [PubMed] [Google Scholar]
  50. Parker, G., Roussos, J., Hadzi-Pavlovic, D., Mitchell, P., Wilhelm, K., & Austin, M. P. (1997). The development of a refined measure of dysfunctional parenting and assessment of its relevance in patients with affective disorders. Psychological Medicine, 27(5), 1193-1203. doi: 10.1017/s003329179700545x [DOI] [PubMed] [Google Scholar]
  51. Parker, G., Tupling, H., & Brown, L. B. (1979). A parental bonding instrument. British Journal of Medical Psychology, 52(1), 1-10. doi: 10.1111/j.2044-8341.1979.tb02487.x [DOI] [Google Scholar]
  52. Pettit, G. S., Bates, J. E., & Dodge, K. A. (1997). Supportive parenting, Ecological Context, and Children's Adjustment: A seven-Year Longitudinal Study. Child Development, 68(5), 908-923. doi: 10.1111/j.1467-8624.1997.tb01970.x [DOI] [PubMed] [Google Scholar]
  53. Picardi, A., Tarsitani, L., Toni, A., Maraone, A., Roselli, V., Fabi, E., . . . Biondi M., (2013). Validity and reliability of the Italian version of the Measure of Parental Style (MOPS). Journal of Psychopathology, 19, 54-59. [Google Scholar]
  54. Poletti, M., Gebhardt, E., & Raballo, A. (2022). Childhood Maltreatment and the Subjective Roots of Mental Health Suffering. Clinical neuropsychiatry, 19(1), 5-7. doi: 10.36131/cnfioritieditore20220102 [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Raby, K. L., Lawler, J. M., Shlafer, R. J., Hesemeyer, P. S., Collins, W. A., & Sroufe, L. A. (2015). The interpersonal antecedents of supportive parenting: a prospective, longitudinal study from infancy to adulthood. Developmental Psychology, 51(1), 115-123. doi: 10.1037/a0038336 [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Ravitz, P., Maunder, R., Hunter, J., Sthankiya, B., & Lancee, W. (2010). Adult attachment measures: a 25-year review. Journal of Psychosomatic Research, 69(4), 419-432. doi: 10.1016/j.jpsychores.2009.08.006 [DOI] [PubMed] [Google Scholar]
  57. Rumpold, G., Doering, S., Höfer, S., & Schüßler, G. (2002). A German version of the Measure of Parental Style (MOPS). Zeitschrift fur Psychosomatische Medizin und Psychotherapie, 48(1), 59-74. doi: 10.13109/zptm.2002.48.1.59 [DOI] [PubMed] [Google Scholar]
  58. Ruopp, M. D., Perkins, N. J., Whitcomb, B. W., & Schisterman, E. F. (2008). Youden Index and optimal cut-point estimated from observations affected by a lower limit of detection. Biometrical Journal, 50(3), 419-430. doi: 10.1002/bimj.200710415 [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Sagi, A., van IJzendoorn, M. H., Scharf, M., Koren-Karie, N., Joels, T., & Mayseless, O. (1994). Stability and discriminant validity of the Adult Attachment Interview: A psychometric study in young Israeli adults. Developmental Psychology, 30(5), 771-777. doi: 10.1037/0012-1649.30.5.771 [DOI] [Google Scholar]
  60. Simpson, J. A., Rholes, W. S., & Phillips, D. (1996). Conflict in close relationships: an attachment perspective. Journal of Personality and Social Psychology, 71(5), 899-914. doi: 10.1037//0022-3514.71.5.899 [DOI] [PubMed] [Google Scholar]
  61. Sulik, M. J., Blair, C., Mills-Koonce, R., Berry, D., & Greenberg, M. (2015). Early Parenting and the Development of Externalizing Behavior Problems: Longitudinal Mediation Through Children's Executive Function. Child Development, 86(5), 1588-1603. doi: 10.1111/cdev.12386 [DOI] [PMC free article] [PubMed] [Google Scholar]
  62. Teicher, M. H., Samson, J. A., Anderson, C. M., & Ohashi, K. (2016). The efects of childhood maltreatment on brain structure, function and connectivity. Nature Reviews Neurosciences, 17(10), 652-666. doi: 10.1038/nrn.2016.111 [DOI] [PubMed] [Google Scholar]
  63. Thomson, P., & Jaque, S. V. (2017). Adverse childhood experiences (ACE) and adult attachment interview (AAI) in a non-clinical population. Child Abuse & Neglect, 70, 255-263. doi: 10.1016/j.chiabu.2017.06.001 [DOI] [PubMed] [Google Scholar]
  64. van Hoof, M. J., Riem, M. M. E., Garrett, A. S., van der Wee, N. J. A., van, I. M. H., & Vermeiren, R. (2019). Unresolved-disorganized attachment adjusted for a general psychopathology factor associated with atypical amygdala resting-state functional connectivity. European Journal of Psychotraumatology, 10(1), 1583525. doi: 10.1080/20008198.2019.1583525 [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. van IJzendoorn, M. H., & Bakermans-Kranenburg, M. J. (2019). Bridges across the intergenerational transmission of attachment gap. Current Opinion in Psychology, 25, 3136. doi: 10.1016/j.copsyc.2018.02.014 [DOI] [PubMed] [Google Scholar]
  66. Velotti, P., Rogier, G., Beomonte Zobel, S., Chirumbolo, A., & Zavattini, G. C. (2022). The Relation of Anxiety and Avoidance Dimensions of Attachment to Intimate Partner Violence: A Meta-Analysis About Perpetrators. Trauma Violence Abuse, 23(1), 196-212. doi: 10.1177/1524838020933864 [DOI] [PubMed] [Google Scholar]
  67. Waters, E., Hamilton, C. E., & Weinfield, N. S. (2000). The stability of attachment security from infancy to adolescence and early adulthood: general introduction. Child Development, 71(3), 678-683. doi: 10.1111/14678624.00175 [DOI] [PubMed] [Google Scholar]
  68. Wazana, A., Moss, E., Jolicoeur-Martineau, A., Graffi, J., Tsabari, G., Lecompte, V., . . . Meaney M. J., (2015). The interplay of birth weight, dopamine receptor D4 gene (DRD4), and early maternal care in the prediction of disorganized attachment at 36 months of age. Development and Psychopathology, 27(4 Pt 1), 1145-1161. doi: 10.1017/S0954579415000735 [DOI] [PMC free article] [PubMed] [Google Scholar]
  69. Whittle, S., Simmons, J. G., Dennison, M., Vijayakumar, N., Schwartz, O., Yap, M. B., . . . Allen N. B., (2014). Positive parenting predicts the development of adolescent brain structure: a longitudinal study. Developmental Cognitive Neuroscience, 8, 7-17. doi: 10.1016/j.dcn.2013.10.006 [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Wilhelm, K., Gillis, I., & Parker, G. (2016). Parental Bonding and Adult Attachment Style: The Relationship between Four Category Models. International Journal of Women’s Health and Wellness(2), 016. doi: 10.23937/24741353/1510016 [DOI] [Google Scholar]
  71. World Health Organization. (4-5 May 2011). Adverse Childhood Experiences International Questionnaire: piloty study review and finalization meeting. WHO Headquarters, Geneva Meeting Report.
  72. Youden, W. J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32-35. doi: 10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3 [DOI] [PubMed] [Google Scholar]
  73. Zhang, X., Tan, S. S., Bilajac, L., Alhambra-Borras, T., Garces-Ferrer, J., Verma, A., . . . Raat H., (2020). Reliability and Validity of the Tilburg Frailty Indicator in 5 European Countries. Journal of the American Medical Directors Association, 21(6), 772-779 e776. doi: 10.1016/j.jamda.2020.03.019 [DOI] [PubMed] [Google Scholar]

Articles from Clinical Neuropsychiatry are provided here courtesy of Giovanni Fioriti Editore

RESOURCES