‘We're not going to start lifting stones now…’: Stakeholder perspectives on the role of psychometric methods in outcome measurement

Abstract

Objectives

Psychological outcome measures guide research and clinical decision‐making, yet many widely used tools were developed with limited psychometric rigour. Although advanced methods (e.g., item response theory, structural equation modelling) are now widely available, their added value in applied research remains uncertain and applied researcher perspective regarding such are unexplored. We aimed to address this gap in knowledge by examining key stakeholder perspective.

Method

To explore how these methods are perceived, we conducted semi‐structured interviews with 21 stakeholders spanning psychometrics, clinical practice, applied research, statistics and academia. Data were analysed using reflexive thematic analysis.

Results

Analysis identified three overarching themes: (1) growing recognition that heterogeneity in latent traits challenges assumptions underlying many measures, (2) the enduring use of entrenched but psychometrically weak tools and (3) nuanced views on when advanced methods meaningfully influence research findings. Participants acknowledged gaps in psychometric literacy and emphasised the need for more training and collaboration with psychometricians.

Conclusion

These findings highlighted persistent limitations in measurement practice, clarify contexts where psychometric methods add genuine value, and point to opportunities for strengthening outcome measurement in psychology and psychiatry research.

Practitioner points.

• Many researchers may have insufficient knowledge of psychometrics to critically appraise the outcome measures they use to collect research data.

• Commonly used outcome measures may not be fit for purpose. This appears to be recognised in the research community, but these measures remain in use for reasons outside of scientific rigour.

• Psychometrics were seen to matter most to the development and validation of accurate outcome measures.

• Researchers should pursue appropriate training in psychometric methods, as well as collaboration with psychometricians, to ensure ‘good measurement’ in research projects.

INTRODUCTION

Do psychometrics matter? Existing research

Quantitative measurement in psychology research, as well as social and health science research more broadly, is inextricably linked with the use of composite multi‐item outcome measures. These measures can be used to assess a host of different outcomes, such as attitudes, mood states, health behaviours, and symptom burden (Chan, 2014; Embretson & Reise, 2000). Psychometric methods are typically used to develop outcome measures and evaluate their validity. Latent trait approaches consider underlying characteristics like depression to be unobservable and take the position that these characteristics can only be understood by examining groups of observable symptoms. Latent trait approaches include factor analysis (FA) or item response theory (IRT), and are used to compile sets of items (e.g., symptoms) that represent manifest expressions of underlying latent traits (e.g., depression). Some argue the best way to alleviate the symptoms is to address the underlying latent trait, with the aim of subsequently reducing symptom severity (De Champlain, 2010; Reise & Waller, 2009). In contrast to latent trait methods, network analysis (NA) eschews the requirement to specify a causal latent trait and instead plots a complex system of inter‐connected, mutually‐causal symptoms (Borsboom, 2017; Borsboom & Cramer, 2013).

These psychometric methods can also be used to potentially improve underperforming measures by identifying and removing redundant items, thereby reducing measurement error (Akdemir et al., 2001; Broen et al., 2015). Furthermore, they can be used to calculate weighted scores. These are severity scores that are recalculated based on the importance of each symptom in the respective psychometric statistical model, with more important symptoms having their scores increased and less important symptoms having their scores decreased. This may increase the accuracy of outcome measurements when compared to unweighted sum scores (Gorter et al., 2015, 2016).

Although multi‐item outcome measures tend to predominate in psychology research, there is some contention regarding their validity. Psychometric assessment of some of the most commonly used outcome measures in antidepressant clinical trials has produced mixed results (Bringmann et al., 2015; Byrne, Doyle, et al., 2024; Carmody et al., 2006). For example, the Hamilton Rating Scale for Depression (HRSD; Hamilton, 1960) – which has long been considered the ‘gold standard’ depression measure – demonstrates poor item validity and is structurally and factorially inconsistent (Bagby et al., 2004; Byrne, Doyle, et al., 2024). Attempts to utilise weighted fembret scores have also been called into question, with Kline (2005) proposing that the correlation between psychometrically refined weighted scores and traditional unweighted sum scores tends to be so close to 1.0 that the psychometric refinements are not worth attempting. However, this position is challenged by the findings of Gorter et al. (2015, 2016), who used IRT methods to analyse randomised controlled trial and simulated data. They demonstrated that sum scores could be biased by approximately one standard deviation, and that weighted scores derived from IRT models were better indicators of actual outcomes. Other researchers (e.g., Harrison, Plessen, et al., 2023; Harrison, Hossain, et al., 2023) have also examined the implications of using weighted scores, noting that findings that were statistically significant when using sum scores became non‐significant when analysing weighted scores.

We recently tried to address the question of whether psychometric approaches might influence trial effects using individual participant data from multiple randomised trials of antidepressants (N = 6962). We aimed to examine the implications of using advanced psychometric methods to reanalyse and potentially augment existing depression measures, and to determine whether this would inform any statistically or clinically important differences in depression trial outcomes (Byrne, Boland, et al., 2025; Byrne, Doyle, et al., 2024; Byrne, Doyle, et al., 2025; Byrne, Ghoshal, et al., 2024; Doyle et al., 2023). We examined the psychometric properties of two of the most commonly used depression outcome measures: the 17‐item HRSD‐17 (Hamilton, 1960) and the 10‐item Montgomery‐Asberg Depression Rating Scale (MADRS; Montgomery & Åsberg, 1979). Briefly, psychometric analyses for both scales indicated that models differed by method used. Of the 17 items in the HRSD, 4 (FA), 10 (NA) and 13 (IRT) items were retained, while of the 10 MADRS items, 7 (FA), 8 (NA) and all 10 (IRT) were retained. Both measures were limited by sex non‐invariance, meaning male and female outcomes may not be meaningfully comparable when these scales are used (Byrne, Boland, et al., 2025; Byrne, Doyle, et al., 2025).

We compared FA, IRT and NA weighted scores to unweighted sum scores and abbreviated scale sum scores (i.e., sum of retained items only). While original trial effects (HRSD: d = .067–.077; MADRS: d = .071–.081) were smaller than would be expected for antidepressant RCT outcomes, as evidenced by a large‐scale meta‐analysis (Cipriani et al., 2018), there was no evidence of increased effect sizes for IRT or NA approaches, although FA methods informed a ~10%–15% increase in effects when compared to original trial outcomes. However, the weighted scores derived from each method offered no additional utility beyond the removal of redundant items. These findings bring into question whether the application of such approaches could have any tangible effect on research outcomes, as asserted by Kline (2005).

The present study

Our previous findings raise important concerns regarding the potential benefits of advanced psychometric methods for ‘good’ measurement in psychology research. Psychometricians have advocated for the continued assessment of existing measures (Doyle et al., 2023). However, we found serious limitations with some of the most commonly used scales (Byrne, Boland, et al., 2025; Byrne, Doyle, et al., 2025), which have also previously been recognised (Bagby et al., 2004; O'Brien & Glaudin, 1988), and yet they still predominate in contemporary research. Furthermore, the implications of psychometrics for tangible research outcomes remains relatively unclear. In the present study, we used our quantitative research findings to frame a discourse with key psychometrician, clinician and academic stakeholders to explore these issues and garner informed perspectives on the place of psychometrics in psychology research.

The aims of this study were to examine the perspectives of key stakeholders regarding the results found in the quantitative phase of our research (Byrne, Boland, et al., 2025; Byrne, Doyle, et al., 2025) and explore stakeholder perspectives regarding the relationship between psychometrics and psychology and psychiatry research, more broadly.

METHOD

Research design

An experiential approach within a constructivist epistemology was adopted for this study to pursue an understanding of meaning and sense‐making regarding our quantitative findings, as well as perceptions and experiences of using psychometrics and underlying methods in participants' respective fields. A predominantly inductive approach informed data generation using semi‐structured interviews and analysis using reflexive thematic analysis (RTA; Braun & Clarke, 2019, 2021a, 2021b; 2022). Deduction was used to ensure that the data were generated and analysed in a manner that reflected the underlying theory of the psychometric methods that were to be discussed, and that findings appropriately addressed the study aims (Byrne, 2022). Although interviews reflected a broad discourse regarding psychometric methods, discussions were constructed around considerations for our previous qualitative analyses addressing the impact of psychometric methods on effect sizes. As such, it should be considered that participant perceptions were framed and nuanced by effect size considerations. Our analysis is reported in line with Reflexive Thematic Analysis Reporting Guidelines (RTARG) (Braun & Clarke, 2024). Ethical approval for this study was awarded by the RCSI University of Medicine and Health Science Ethics Committee (No: 212560819).

Participants

Purposive sampling was initially used to recruit participants, with snowball sampling subsequently used whereby participants were asked to raise awareness of the study with anyone in their extended network whom they believed may be interested in participating. Inclusion criteria were candidates who have experience developing, evaluating or using multi‐item outcome measures, who have experience of applied research within psychometrics, statistics, psychology or psychiatry (clinical or observational). The recruitment process lasted 4 months, with interviews being completed within the final 2 months of this period. A total of 69 candidates were approached, with 21 agreeing to participate. A broad range of perspectives was represented, with participants predominantly located in the Republic of Ireland and the UK, and some international representation also included (see Table 1).

TABLE 1.

Sample characteristics.

Pseudonym	Position/discipline	Location
P01	Professor, Psychology	N. Ireland
P02	Professor, Psychology	N. Ireland
P03	Professor, Psychology	Ireland
P04	Professor, Psychiatry	UK
P05	Lecturer, Statistics	Ireland
P06	Lecturer/Researcher, Psychiatry	UK
P07	Professor, Psychology	Netherlands
P08	Lecturer, Psychology	Ireland
P09	Lecturer, Applied Psychology	Ireland
P10	Professor, Psychology/Statistics	UK
P11	Professor, Statistics	USA
P12	PhD, Digital Mental Health	UK
P13	Lecturer, Psychology	Ireland
P14	Lecturer, Psychology	Ireland
P15	Lecturer, Psychology	UK
P16	Assistant Professor, Psychology	Ireland
P17	Professor, Psychiatry	Ireland
P18	Professor, Psychometrics	Ireland
P19	Professor, Behaviour Change	Netherlands
P20	Professor/Dean, Research	USA
P21	Lecturer, Psychology	Ireland

Data generation

Data were generated using semi‐structured interviews. After a brief introduction, participants were presented with the quantitative findings of our project, which examined the impact of psychometric methods on clinical trial effect size outcomes (Byrne, Boland, et al., 2025; Byrne, Doyle, et al., 2025). This included an outline of methods used, an overview of the psychometric performance of the outcome measures, and a comparison of original versus psychometrically‐informed outcomes. Participants were then asked about their thoughts and perspectives regarding these findings and the place of psychometric methods in psychology and psychiatry research. An interview guide (Supplementary Item 1) consisting of 14 questions was used to guide discussions. Participants were initially asked about their background and familiarity with psychometrics, as well as their use of such methods in research contexts. These discussions informed the line of questioning during the interviews. For example, participants who reported being very familiar with psychometrics were engaged with discussion of advanced methods and particular functions within respective methods that might have influenced resulting models or weighted scores. Participants who considered themselves less familiar with the intricacies of these methods often pursued discussions of the tangible implications of the findings for applied research.

All interviews were conducted online using either Microsoft Teams or Zoom, at the preference of the participant. The duration of interviews ranged from 56 min to 112 min (mean duration: 99 min). Each participant was provided with a participant information leaflet (Supplementary Item 2), which outlined the study aims and the nature of their participation, as well as their rights to access and/or withdraw any and all of their data. When interviews were complete, participants were reminded of their rights, offered the opportunity to review verbatim transcriptions of their interview (once transcribed), and were asked to complete a consent form.

Analysis

Data were analysed using the six‐phase process of reflexive thematic analysis (Braun & Clarke, 2019, 2021a, 2021b; 2022). RTA was chosen to facilitate our interpretations of a diverse applied researcher sample with varying degrees of knowledge of psychometrics. This began with data immersion, which involved developing a high level of familiarity with the data, reflecting on initial thoughts and sense‐making and noting and contextualising initial data trends (Braun & Clarke, 2019). A reflexive journal was maintained to document this process and monitor for potential sources of researcher bias (Braun & Clarke, 2024). Coding was performed iteratively, with codes being edited, amended, introduced or removed in subsequent iterations as interpretations of what was meaningful to participants evolved. Coding was both semantic and latent, which meant data could be dual‐coded according to the expressed meaning communicated by participants, and also according to the subtext we may have interpreted from discussions (Byrne, 2022). Initial themes were developed by reviewing and collating codes into broader topics (Braun et al., 2019). These were then interrogated for their meaning in relation to the research questions and organised into themes. The themes and constituent codes/topics were reviewed and amended as appropriate to ensure they were appropriate representations of the researchers' interpretation of what was presented as most meaningful by participants (Byrne, 2022). Finalised themes were framed in relation to a central organising concept (which was embedded in each theme name), and presented using an analytical reporting convention (Braun & Clarke, 2013, 2024).

Researcher reflexivity

The authors have a keen interest in psychometric methods and their implications for applied research. As such, we endeavoured to remain cognisant of our own potential preferences and biases and how these could inform our interpretations of the data. For example, we were aware that more frequent use of (or a preference to use) particular psychometric methods could inadvertently lead to reporting accounts of these methods more favourably. To address these concerns, the research team met regularly to discuss analyses and the evolving interpretation of the data. Evolving interpretations of the data (and instances where threads of analysis were unfruitful) were documented in a spreadsheet and discussed regularly. In addition, a multi‐disciplinary steering committee, comprised of statisticians, psychometricians, academics and public/patient involvement representation, was also consulted to ensure our interpretations of the data were defensible from different stakeholder perspectives. This reflexively addresses concerns of methodolatry, which is a topic discussed in these results.

RESULTS AND DISCUSSION

Analysis of the data resulted in the interpretation of three themes. The first, ‘Heterogeneity of Latent Traits’, addresses the perceived plurality of psychological constructs, such as depression, as evidenced by the lack of model invariance presented to participants. The second, ‘A Legacy of Poor Measurement’, describes factors that participants believed inform sub‐optimal research practices. This consists of two subthemes: ‘Basic Knowledge of Psychological Measurement’, which outlines potential shortcomings in researcher skillsets, and ‘Poor Practice in Psychological Research’, which highlights sub‐standard research practices. Finally, theme three, ‘When Psychometrics Matter’, provides accounts of when participants thought that psychometrics may be most useful in applied research, namely the identification of valid measurement items noted in the psychometric findings.

Heterogeneity of latent traits

Discussions the place of psychometrics in psychology research often began with considerations for the different constructs we try to measure. In this regard, participants critiqued often held perceptions that constructs like depression are unitary and homogeneous. Participants pushed back against this idea and argued that such constructs are in fact heterogeneous: ‘…the fact that we call it depression, doesn't suddenly make it one thing’ [p7]. This perspective reflects historical critiques of homogeneous considerations of psychological phenomena. For example, Goldberg (2011) argued the folly of International Classification of Diseases (ICD) and Diagnostic and Statistical Manual of Mental Disorders (DSM) diagnostic criteria, which latently (or otherwise) reify interpretations of depression as a ‘homogeneous, unitary entity’. Goldberg offered the example that patients experiencing different symptomologies (e.g., patient 1 experiences lassitude, hypersomnia and weight gain, while patient 2 experiences agitation, insomnia and weight loss) can have identical depression scores, which he argued is highly problematic for research design and can obscure important differences in depressive symptomology. This issue was further elucidated when another participant spoke of one study that explored the way in which different depression measures can consist of different symptoms:

I mean, there's a guy, Eiko Fried; he did a very nice study. He looked at 54 [sic] different measures of depression, you know, and found massive heterogeneity in terms of the item content. He did the same with anxiety and found massive amounts of heterogeneity across anxiety. But, what they did find; the only homogeneity was across the anxiety and depression measures! [P2]

Participant 2 referenced Fried (2017), which assessed the overlap of 52 depression symptoms from seven commonly used depression measures. This study found very low symptom overlap across measures, noting that 40% of symptoms examined were unique to a single measure. The author contended the heterogeneity of the depressive syndrome, arguing that findings obtained using one measure may not be replicable using another (Fried, 2017). The lack of overlap in depression measure symptomology was also identified in a large systematic review of 450 depression trials, which found poor convergence of symptoms with 80 pre‐specified depression domains. This was exemplified by the HRSD, which only addressed a maximum of 59% of these depression domains (Veal et al., 2024).

The work of Fried (2017) and Veal et al. (2024) highlights the difficulty of attempting to reify singular interpretations or understandings of psychological phenomena that are often very difficult to measure (Jahn & Dunne, 1997; Lind, 2019). This was well exemplified by one participant, who discussed the tendency to conflate different constructs due to poorly conceived or operationalised latent variables; ‘…this jingle/jangle fallacy. We have a lot of different measurements that measure the same thing and so on, with different definitions and the other way round’ [P19]. The jingle/jangle fallacy is the tendency for researchers to unduly assume different measures assess the same construct because they have similar names/labels, or different constructs because they have different names/labels (Cornelius et al., 2023; Flake & Fried, 2020). This issue is demonstrated in Fried's (2017) work in particular, as not only were depression measures found to be heterogeneous (the jingle fallacy), but homogeneity was observed between depression and anxiety measures (the jangle fallacy). Participants also reflected on the influence of these types of issues in their own research: ‘…people who are presenting with only depression or only anxiety are, kind of, psychological unicorns.’ [P2]. Participant 2 went on to suggest that their encounter with the jangle fallacy may have arisen because of how poorly delineated and susceptible to overlap constructs like depression and anxiety often are, and recognised this as a significant barrier to good measurement (Eysenck & Fajkowska, 2018; Lowe et al., 2008).

Some participants suggested that psychology research has long been married to notions of objectivity inherited from attempts to emulate natural sciences. The usefulness of this ethos was often directly challenged: ‘I think the challenge to it is a language and a philosophy challenge and how we break that metaphor of natural science diseases’ [P16]. While typically framed as informing good measurement, it has recently been suggested that natural science perspectives may often be adopted to recast theory‐avoidance as the pursuit of objectivity. In this regard, researchers could feign objectivity to justify otherwise questionable methodologies in order to avoid the difficult task of providing theoretically‐informed explanations of complex psychological processes (Gigerenzer, 2024). This too was discussed in detail and widely critiqued: ‘It might give us the superficial idea that we can safely make in person, but instead what we're doing is just hiding a bit of the world that is there’ [P7]. Participants demonstrated a willingness to move away from natural science perspectives and towards embracing the ‘ugly truth’ of psychological phenomena:

The truth is the mess! The truth is actually this horrible thing that gives terrible results and is total chaos. And, you can't make sense of it…but they're the truth. [P5]

In this regard, participants contended that there is ‘no a‐priori reason to assume that the constructs that we invent are the same for everyone’ [P7], and it was considered that there is little agreement as to what might constitute depression, or any such psychological construct. Interestingly, contemporary arguments suggest this may actually be preferable to pursuing consensus (Hyman, 2010; Jones & Robinaugh, 2021). Participants widely agreed with this position, arguing the erroneous nature of efforts to achieve agreement on singular representations of such constructs: ‘consensus efforts don't get to the truth. You just get the consensus’ [P7]. This point was further contextualised when participant 7 suggested that we may be on the right track by embracing the ‘known’ heterogeneity of psychological constructs, but that there is still more work to be done to identify and address what is ‘unknown’: ‘I think having heterogeneity is necessary to actually start understanding depression. We just need to eliminate this hidden heterogeneity’ [P7].

A legacy of poor measurement

‘I don't concern myself with why a centimetre is a centimetre…’: Basic knowledge of psychological measurement

Participants generally considered that psychometric methods were wide open to interpretation and that there is no single correct way to conduct such analyses. This sentiment was widely shared and accompanied a belief that, contrary to natural science, social science measurement, and psychometric analysis in particular, is an inherently subjective, highly specialised and difficult undertaking: ‘People just don't understand it’ [P18]; ‘…it's not for the faint of heart at all, you know’ [P15]; ‘In general speaking, psychometrics is a tricky thing because there's just…There are [sic] just no standardised way to it’ [P12]. As such, participants believed that there is currently a high degree of reticence among applied researchers to adopt psychometric methods within their research:

I would say there would be great reluctance to that […] You can imagine the principal saying ‘no, we definitely don't want to do that! That's the last thing we want to do!’ You know, ‘we're not going to start lifting stones now because we might find something unpleasant underneath!’ [P2]

This suggests further evidence of theory‐avoidance in psychology research (Gigerenzer, 2024), and also points towards what participants believed was a relative devaluing of psychometric methods: ‘I also have to say that a lot of people don't care. They see, like, this is some kind of extra thing ‐ you kind of jump through the hoop kind of thing…’ [P19]. This was said to be wholly detrimental to research quality and was argued to be evident in the current state‐of‐the‐art of psychological inquiry: ‘Unfortunately, I think – as a discipline – we have been poor about that. And, I find it remarkable that we're head‐scratching about replication crisis and so on’ [P2]. Relating the way in which applied researchers may often cede from interrogating the psychometric properties of the measures they use to the documented replication crisis in psychology is an astute observation, as this crisis has been partially attributed to the quality of the measures used by researchers (Slaney & Garcia, 2015; Uher, 2022, 2023). In this regard, a critical perspective of psychometric measures was said to be fundamentally important to good measurement: ‘…we need to think more and deeper and do more research about the instruments we're using, definitely’ [P17].

However, participants also suggested that many applied researchers simply do not have the requisite skillset to critically appraise the measures that are commonly used to collect data: ‘…most of us, the vast, vast majority of us don't have enough knowledge of the intricacy of scale construction and evaluation, and really testing it to make good decisions’ [P21]. There was an evident awareness that applied researchers should perhaps have some degree of knowledge of these methods to address related issues in their own research: ‘…occasionally questions will arise on which we need, you know, deeper understanding’ [P20]. This is a compelling argument that reflects previous assertions as to the need for researchers to upskill in this regard. For example, Vetter and Cubbin (2019) argued that the selection of an appropriate measure is fundamentally important to the accurate measurement of participant outcomes, and that a key step in selecting an appropriate measure is determining its usefulness. They advocated for applied researchers trusting that existing measures were developed to acceptable psychometric standards, but that researchers should interrogate their validity and practical utility in the context of their own research.

Many participants traced an underdeveloped psychometrics skillset back to training and education issues, particularly at third‐level. It was often highlighted that it is not ‘explicitly detailed what level of psychometric knowledge an undergraduate psychology student is expected to have by the end of their degree’ [P14]. This is an issue that appears evident at all strata of third‐level education. For example, a curriculum review of graduate psychology and education programs in the U.S. found that psychometrics were critical to students' professional readiness and subsequent career success, but that theoretical knowledge development in this regard was often devalued in relation to more general technical/statistical skills (Randall et al., 2021). Participants believed psychometric methods should be incorporated at all strata of third‐level education: ‘…incorporating that into the college curricula would be a really good move’ [P6], although they were keenly aware of the difficulty in doing so: ‘it's really hard to get my colleagues to change this, you see. Everybody is comfortable teaching what they learned during their own education’ [P7]. This was framed as ‘an entropy within the system that just restricts change’ [P5] and informed a cyclical trend whereby students would be taught statistics and research methods in a particular manner and would then go on to apply this in their research career, and ultimately teach it to the next generation of students. Ultimately, there was an apparent sense of resignation to the perceived negative opinions of psychometrics in academia:

I don't even bother anymore…I wish that I had support internally that would allow me to require it or at least to recommend it more strongly. [P15]

While a manifest appreciation for the importance of psychometrics in ‘good measurement’ was initially proposed by participants, their accounts suggest that this may disappear somewhat when it comes to the tangible practice of conducting applied research. This was well illustrated earlier, when participant 2 mentioned ‘something unpleasant’ that might be difficult to explain or otherwise jeopardise the study outcomes and highlights the relative lack of value for psychometric methods in the context of applied research.

‘Dogma and rituals have replaced thinking’: Poor practice in psychology research

Participants highlighted the way in which applied researchers seemingly forego critical reflection on different important aspects of methodology that would otherwise inform appropriate collection, interpretation and analyses of data. Adherence to ritualistically learned methods was said to be firmly entrenched among psychology researchers, resulting in formulaic and credulous approaches to conducting research:

…you know, there's a template people have in their head, ‘ok, you need an outcome measure. Ok, if it's self‐report, what's the reliability?’ And validity, you know, they don't even ask that. ‘Is there good evidence for reliability?’, and if you can say yes, then you're in. [P1]

It was considered common that researchers would typically have a particular preferred way to conduct research, from which they rarely deviate. This is a central tenet of methodolatry, which is considered a significant issue in psychology research (Chamberlain, 2000; Chirkov, 2015; Rey, 2019; Wertz, 1999). A prominent characteristic of methodolatry is an over‐emphasis on how research is done at the expense of understanding the implications of what is done. This can lead to the absence or avoidance of theoretically‐informed critical perspectives of research data, and the implications for outcomes therein (Chamberlain, 2000). Participant 1 exemplified this in relation to a critical psychometric perspective by highlighting the formulaic nature of satisfying certain criteria when selecting an outcome measure. It is noteworthy that validity is singled out as the problematic criterion. As participant 2 highlighted earlier, assessing the validity of psychometric measures could bring to light ‘something unpleasant’ that could be very difficult to address. In fact, it was a widely held opinion among participants that there would indeed be something unpleasant to be found, as most were highly critical of the validity of the measures used in contemporary psychology research:

I think, my view – and it might be something of an extremist view – I think most measurements in psychology are terrible. Em…they lack any kind of validity. [P3]

Psychology textbooks often state that ‘psychologists do not simply assume that their measures work’ and that ‘if research demonstrates that a measure does not work, then it is no longer used’ (e.g., Chiang et al., 2015). However, this position was refuted by some participants, who believed serious issues of measurement validity are often simply overlooked: ‘we pretend that isn't the case. And, we say ‘these measurement problems don't really exist’, and we interpret our findings as if these problems don't exist’ [P3]. The continued use of sub‐optimal and sometimes invalid measures is demonstrable with the example of the HRSD, which has prevailed as the gold standard of depression measurement despite well‐documented evidence of serious validity and structural issues (Bagby et al., 2004; Byrne, Doyle, et al., 2024; Steinmeyer & Moller, 1992). Despite recognising its shortcomings, some participants acknowledged that they are still using the HRSD: ‘It's still very prevalent and I'm still using it myself, you know, in research’ [P6].

Several explanations were put forward as to why measures that are known to be sub‐optimal may remain in use. One frequently proposed reason was habit or tradition: ‘There's a lot of tradition, because we've always done it this way’ [P20]; ‘I think a lot of people, when they're doing research, they just do what the person before them did’ [P5]. This points towards an inherited legacy of suboptimal practice, as outlined earlier, and could equally be informed by a lack of knowledge of such measures or a preference for a particular way of conducting research (Chamberlain, 2000). It was also suggested that it is simply more convenient to adopt the most commonly used measures: ‘…they just look, “okay it's published there. It's validated. Let's go‘” [P17]. Alternatively, some participants proposed that researchers might chose outcome measures to allow easier comparisons of findings with existing literature: ‘there's also the, kind of, wanting your findings to be comparable with what's in the published literature’ [P3]. Over the course of these discussions, participants spoke openly of the difficulty in correcting these behaviours. Some suggested that applied researchers relied on statisticians and psychometricians to highlight and correct bad measurement, but were sceptical of their ability to do so:

…many of us in clinical trials, but in other areas of research, are operating on faith that another area, another group in our discipline, you know, statisticians, psychometricians, are going to do that work and flag it to us, except then we don't read that work [P21]

Crucially, this participant highlighted that, although it is tacitly expected that these groups will help guide best research practice, it is apparent that applied researchers may not engage with requisite research output that is being published by psychometricians. This has previously been argued to be a broad reality, as it has been argued that outputs from psychometric research are very often too technically advanced for applied researchers to understand, and this may discourage others from engaging with this important work (Sijtsma, 2012). Some participants proposed means by which psychometrics could be further imbued in psychology research. One suggestion was to create infrastructure that better facilitates the exchange of knowledge between psychometricians and applied researchers:

Wouldn't it be great if we had some kind of online resource where you can search for measures and tools and instruments but then side‐by‐side rate them according to inter‐rater reliability, consistency over time and you know all the other measures that you like to see and there's a little summary to say ‘beware, this is not gender invariant’…that kind of thing. [P6]

However, more concerted arguments were made in favour of simply increasing collaboration: ‘…it needs to be collaborative, and you need to have different people with different expertise […] include some expertise on psychometrics’ [P17]. While this was said to be a more pragmatic option, it was recognised that it may be impractical in many research projects. Ultimately, participants were eager to bridge the known gap between psychometrics and applied research (Wijsen et al., 2022), but recognised that there is much work to be done to achieve broader recognition and implementation of psychometrics in psychology research.

When psychometrics matter

When discussing the potential that the application of psychometric methods could inform more accurate trial and research outcomes, most participants were reluctant to comment on the implications of our quantitative research findings. While it was typically considered that more research would be needed to support the findings of a single study, participants' primary concern was the initial small effect size: ‘…the effect sizes were so small, to look at a difference between tiny effect sizes is sort of like you're looking for a needle in a haystack’ [P8]. Here, it could be construed that participants were exercising reasonable caution about potentially over‐interpreting findings involving secondary analyses of small effect sizes. Careful consideration of these accounts brought to light a connotation that ‘effects were small so comparisons were meaningless’. Dismissal of small effects without critical appraisal of such has been argued to be a common phenomenon in psychological research (Cortina & Landis, 2009), However, Cohen (1988) indicated that the small/medium/large effect criteria he put forward were somewhat arbitrary and that interpretation of effect sizes should be contextualised within the relative domain of inquiry. Indeed, effect sizes needed to be examined to the third decimal place to note a 10%–15% increase in effects from the original trial outcomes when applying FA methods (Byrne, Boland, et al., 2025; Byrne, Doyle, et al., 2025). It is therefore necessary to conduct further research to better answer the question as to whether psychometrics matter to effect size outcomes.

An area where some participants offered more definitive perspectives on where psychometrics matter was scale validity, offering critical appraisals of the psychometric performance of the measures that were used in our research. The performance of the scales was often contextualised in relation to the degree of redundancy or non‐performing items found during psychometric modelling. In this regard, it was considered that the MADRS outperformed the HRSD: ‘I think I would go for the MADRS rather than Hamilton just because Hamilton is redundant’ [P6]. Here, some participants highlighted the way in which psychometrics matter when designing practical measurement tools, linking the abbreviation of scales to reduced burden and increased compliance: ‘I think it's great when you can have an abbreviated version that works well and can reduce burden’ [P1]; ‘Having shorter versions really helps with the compliance’ [P4]. In this regard, psychometrics were considered to have demonstrated clear utility in identifying shorter, more practical measures. Conversely, some participants thought that abbreviation would threaten measurement validity: ‘I think there's a real possibility of increasing measurement error with shorter scales’ [P1].

Interestingly, research has identified adverse implications of using measures that are not valid, and how a lack of critical appraisal of psychometric properties can lead to the misinterpretation of model performance. For example, Waldman et al. (2023) demonstrated that it is possible for a factor with a large number of moderately‐performing items to produce similar statistics to a factor consisting of a smaller number of high‐performing items. This contradicts the concerns of some participants, highlighted above, that abbreviation of measures could be detrimental to validity. On the contrary, when done correctly, abbreviation could potentially increase model fit, validity and theory alignment, which is arguably the case with our MADRS model (Byrne, Doyle, et al., 2025). The scale‐abbreviation argument of practicality versus validity demonstrates the different perspectives that can be adopted when critically appraising the utility of psychometric methods and how they matter.

The widely held position was that psychometrics are fundamentally important to ensuring valid and accurate measurement: ‘…If you're not getting your scales right nothing really matters'’ [P9]. Participant 9 exemplified the view of many participants that good research is rooted in good measures, and that research outcomes could be jeopardised by sub‐optimal performance of the test instrument used to collect data. This reflects historical contentions that the ability to detect changes and generalise assertions regarding unobservable latent constructs is fundamentally dependent on the validity of the instruments used to measure them (Bagozzi, 1984).

Perhaps unsurprisingly, psychometricians have long asserted that continuous validation of outcome measures is necessary (Doyle et al., 2023), and it appears that this sentiment is also shared among applied researchers: ‘I think all the scales should be reviewed all the time.’ [P5]; ‘for me, good science is about consistently checking these kinds of issues’ [P3]; ‘I think it's essential. Absolutely important and ideally would be done systematically’ [P13]. On these bases, it is arguable that continuous critical appraisal of measures of latent psychological constructs should not be the sole remit of psychometricians, but also of theoreticians and applied researchers who use them. Moreover, participants considered that psychometrics were critically important to the development of valid measures, and that psychometrics matter most to ‘good measurement’.

CONCLUSION AND RECOMMENDATIONS

This study presents some concerns regarding the state‐of‐the‐art of psychological measurement. Our findings elicit broader discussions regarding the constructs we measure and how we try to measure them. Specifically, participants recognised the difficulty in trying to measure dynamic and complex psychological phenomena, such as depression, which have been shown to manifest as a plurality of different symptomologies (Fried, 2017; Veal et al., 2024). This was strongly reflected in the reluctance of participants to comment on the outcomes of our effect size analyses due to the sub‐optimal performance of the measures used and the small size of found effects. A significant dissonance was evident with regard to what participants believed to be standard practice and what they believed was good practice in psychology research. Shared accounts reflected known concerns of methodolatry in psychology research (Chamberlain, 2000; Chirkov, 2015; Rey, 2019; Wertz, 1999), and this was traced back to identified gaps in training and education. Participants argued that researchers generally have a low level of psychometric knowledge, or are reluctant to use knowledge they may have to ask ‘tough questions’ of their data. This was said to inform poor measurement practices and the predominance of sub‐optimal psychometric outcome measures (Bagby et al., 2004; Byrne, Doyle, et al., 2024; Steinmeyer & Moller, 1992). Overall, participants' accounts presented a concerning narrative of measurement in psychology research.

Addressing these issues is arguably imperative to improving current research practice, and psychometrics were seen as fundamentally important in this endeavour. As such, a number of recommendations can be made that would aim to further imbue psychometrics in psychology research:

First, all applied researchers should possess a minimum requisite knowledge of psychometrics and related methods, such that they can critically interrogate the validity and suitability of any given outcome measure for use in their research, with a focus on theory being ultimately vital for validity. For example, this could include an understanding of the way in which dimensionality can be determined, as well as basic knowledge of fit indices and the implications if these are not met. Education regarding these types of psychometric indices should be delivered with consideration for valid theory, such as recognition of item content validity in respect of diagnostic criteria.

Second, as highlighted by participants, perceived shortcomings in research practice appear to be systemic and can be traced back to gaps in university‐level education. Therefore, it is recommended that training and education in psychometric methods should feature more strongly in university‐level psychology programs. As minimum requirements, we suggest that undergraduate students should have an awareness of psychometric methods, while postgraduate students should be able to conduct basic psychometric analyses. As postgraduate researchers have the potential to contribute to cumulative science, they should meet the criteria of the first recommendation and be able to independently determine the validity of the outcome measure used in their research project.

Finally, participants highlighted the known gap between psychometrics and applied psychology research. We advocate for increased collaboration between psychologists and psychometricians on applied research projects, where possible. In addition, it may also be conducive to pursue a centralised repository reporting basic psychometric properties of existing outcome measures in user‐friendly terms to facilitate researchers in selecting appropriate measures for their research.

AUTHOR CONTRIBUTIONS

David Byrne: Writing – original draft; writing – review and editing; methodology; data curation; formal analysis; investigation. Fiona Boland: Writing – review and editing; validation; methodology; funding acquisition; investigation; supervision; project administration. Susan Brannick: Writing – review and editing; investigation. Robert M. Carney: Writing – review and editing; investigation. Pim Cuijpers: Writing – review and editing; investigation. Alexandra L. Dima: Writing – review and editing; investigation. Kenneth E. Freedland: Writing – review and editing; investigation. Suzanne Guerin: Writing – review and editing; investigation. David Hevey: Writing – review and editing; investigation. Bishember Kathuria: Writing – review and editing; investigation. Emma Wallace: Writing – review and editing; investigation. Frank Doyle: Funding acquisition; investigation; writing – review and editing; validation; methodology; supervision; project administration.

CONFLICT OF INTEREST STATEMENT

None.

ETHICS STATEMENT

Ethical approval for this study was awarded by the RCSI University of Medicine and Health Science Ethics Committee (No: 212560819).

Supporting information

Data S1.

Data S2.

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available from the corresponding author upon reasonable request.