The Making of Psychological Methods

Abstract

Psychological Methods celebrated its 20-year anniversary recently, having published its first quarterly issue in March 1996. It seemed time to provide a brief overview of the history, the highlights over the years, and the current state of the journal, along with tips for submissions. The article is organized to discuss: (1) the background and development of the journal, (2) the top articles, authors and topics over the years, (3) an overview of the journal today, and (4) a summary of the features of successful articles that usually entail rigorous and novel methodology described in clear and understandable writing and that can be applied in meaningful and relevant areas of psychological research.

Brief Background on the Development of the Psychological Methods Journal

What went into the making of the Psychological Methods journal? As far back as 1978, the Publications and Communications (P&C) Board of the American Psychological Association (APA) began discussions on the feasibility of starting an APA statistical-methodological journal. For approximately 35 years prior to this, articles on statistics and research methodology were published as part of the mission of Psychological Bulletin. The main concern about considering a new journal was that it would need to avoid being highly technical and instead strive to complement Psychological Bulletin and Psychological Review in publishing highly relevant and widely applicable articles. A number of others, in addition to those on the P&C Board, were involved with early discussions including Division 5 leaders, editors and other individuals involved with other APA journals, particularly Psychological Bulletin and Psychological Assessment, and other quantitative researchers of note in the field. There were many conversations about the benefit of adding another APA journal, particularly as it wasn’t clear to some that it would reach the very wide audience of APA members and psychologists in general.

In 1979, the P&C Board appointed a committee to consider the viability of a new journal on quantitative methods. The committee included such luminaries in the field as Duncan Luce, who served as chair, along with Darrell Bock, Anita DeVivo, William Estes, Bert Green, Richard Herrnstein, Lloyd Humphreys, Lyle Jones, and David Zeaman. Due to the lack of a clear consensus at the time, it was decided instead to establish a Quantitative Methods section as part of Psychological Bulletin. That way, there would be a specific outlet for methodological papers while still reaching the broad readership of Psychological Bulletin, one of the hallmark journals of APA.

A full 15 years went by before there was enough assurance and momentum to make specific plans to launch a separate journal. A Division 5 journal committee (i.e., Leona Aiken: Chair, with James Butcher, Linda Collins, Roger Kirk, and Stephen West) and other advocates such as William C. Howell who was the Director of the APA Science Directorate, along with Carol Dwyer and others, helped to reinforce the need for a new quantitative journal. Uppermost in the plans was the development of a mission statement that would delineate the focus and scope of the journal so as to appeal to expert methodologists and applied researchers in psychology and related fields, with the firm caveat to avoid having the journal be overly technical.

As chair of the P&C in 1993, Donald J. Foss chaired the search committee to identify and hire the first editor for Psychological Methods, a journal that not everyone was convinced was needed. A number of individuals were nominated and considered with one of them rising to the top of the list. In early 1994, Mark Appelbaum was selected to serve as the founding editor and the journal’s birth began. Appelbaum was encouraged to appoint a Consulting Editorial Board of individuals with a broad base of strengths in the content, methods, teaching, research, applications, and theory related to quantitative methods for psychological researchers. He then began requesting submissions in order to publish the first volume in March of 1996 (Appelbaum & Sandler, 1995).

The Top Articles, Authors and Topics in the First Two Decades

First Issue

From the start, Psychological Methods has found a place in the literature, publishing on a variety of methodological topics of interest to a range of researchers. Table 1 lists the seven articles that were published in the first issue in March 1996, along with the authors and the current citation counts. Note that the counts may differ from those from other sources although the procedure for and source of the citation counts (i.e., APA PsycNET website on January 20, 2017) was consistent across the tables and text in this article. The average citation count was 332.86 over the seven articles, with much of the count carried by two articles (i.e., Curran, West & Finch, 1996; McGraw & Wong, 1996). The topics address issues that are still of interest today, including intraclass correlation coefficients (McGraw & Wong, 1996), structural equation modeling (Curran, West & Finch, 1996), measurement (Li, Rosenthal, & Rubin, 1996; Steinberg & Thissen, 1996), meta-analysis (Shadish, 1996), and statistical inference (Bakeman, Robinson, & Quera, 1996; Bushman & Wang, 1996).

Table 1.

Articles Published in the First issue of Psychological Methods, March 1996, Vol 1, Issue 1, and their Citations and Authors

Citations	Authors	Article Title
1098	McGraw, K. O. & Wong, S. P.	Forming inferences about some intraclass correlation coefficients.
962	Curran, P. J., West, S. G. & Finch, J. F.	The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis.
95	Shadish, W. R.	Meta-analysis and the exploration of causal mediating processes: A primer of examples, methods, and issues.
76	Steinberg, L. & Thissen, D.	Uses of item response theory and the testlet concept in the measurement of psychopathology.
56	Li, H., Rosenthal, R., & Rubin, D. B.	Reliability of measurement in psychology: From Spearman-Brown to maximal reliability.
28	Bakeman, R., Robinson, B. F. & Quera, V.	Testing sequential association: Estimating exact p values using sampled permutations.
15	Bushman, B. J. & Wang, M. C.	A procedure for combining sample standardized mean differences and vote counts to estimate the population standardized mean difference in fixed event models.

Note: Citation counts were obtained from the APA PsycNET website on January 20, 2017. It is recognized that these may differ from those from other sources although the procedure for and source of the citation counts was consistent across the tables and text in this article.

Most Cited Articles

Across the first two decades of Psychological Methods, there have been seven articles that had extraordinary impact, each being cited at least 1000 times. Table 2 lists the number of citations, authors, article title and year for these most cited papers from 1996 to 2015. Two of the articles are concerned with mediation (i.e., MackKinnon, Lockwood, Hoffman, West & Sheets, 2002; Shrout & Bolger, 2002). Two are on a topic related to structural equation modeling (Hu & Bentler, 1998; MacCallum, Browne & Sugawara, 1996) or factor analysis (Fabrigar, Wegener, MacCallum & Strahan, 1999). Another article concerns missing data (Schafer & Graham, 2002), and one of the articles, focusing on the intraclass correlation coefficient, was also published in the first issue (see Table 1: McGraw & Wong, 1996). If there is a common theme among these highly cited articles it is that they are very accessible and instructive, offering comparisons, perspective, recommendations, guidelines, and/or evaluation.

Table 2.

Most Cited Articles Published in Psychological Methods from 1996 through 2015

Citations	Authors	Article Title (Publication Year)
3618	MacKinnon, Lockwood, Hoffman, West, & Sheets	A comparison of methods to test mediation and other intervening variable effects. (2002)
3428	Schafer & Graham	Missing data: Our view of the state of the art. (2002)
3099	Shrout & Bolger	Mediation in experimental and nonexperimental studies: New procedures and recommendations. (2002)
2554	Hu & Bentler	Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. (1998)
1867	MacCallum, Browne, & Sugawara	Power analysis and determination of sample size for covariance structure modeling. (1996)
1856	Fabrigar, Wegener, MacCallum, & Strahan	Evaluating the use of exploratory factor analysis in psychological research. (1999)
1098	McGraw & Wong	Forming inferences about some intraclass correlation coefficients. (1996)

Note: Citation counts were obtained from the APA PsycNET website on January 20, 2017. It is recognized that these may differ from those from other sources although the procedure for and source of the citation counts was consistent across the tables and text in this article.

Most Cited Authors

In addition to evaluating the most cited articles, it is worthwhile to investigate which authors were cited the most over the first 20 years of Psychological Methods. Table 3 provides a list of the top 15 authors who reached citation counts of over 1000 for one to 13 articles published from 1996 through 2015. MacCallum tops the list, accruing the most citations (i.e., 6003) across a set of nine articles published during the first two decades of Psychological Methods. Two other authors, Curran and Preacher, also stand out with highly prolific records (i.e., 12, and 13 articles, respectively) that earned high citation counts (i.e., 2767 and 2630, respectively) during this 20-year period. Two of the authors, Hu and Fabrigar, garnered high citations (i.e., 2554, and 1856, respectively) with just one published article with very high citation counts.

Table 3.

Most Cited Authors Published in Psychological Methods (PM) from 1996 through 2015

Citations	Author	Total PM Articles
6003	MacCallum, R. C.	9
4115	Schafer, J. L.	4
3702	MacKinnon, D. P.	3
3305	Shrout, P. E.	4
2767	Curran, P. J.	12
2630	Preacher, K. J.	13
2554	Hu, L.	1
1856	Fabrigar, L. R.	1
1255	Muthén, B. O.	9
1201	Nagin, D. S.	3
1112	McGraw, K. O.	2
1110	Edwards, J. R.	2
1094	Bauer, D. J.	10
1040	Enders, C. K.	5
1035	Hedges, L. V.	3

Note: Citation counts were obtained from the APA PsycNET website on January 20, 2017. It is recognized that these may differ from those from other sources although the procedure for and source of the citation counts was consistent across the tables and text in this article.

Similar to what held with the articles that were the most cited (in Table 2), these highly cited authors tended to write in a very clear and understandable manner on topics of great interest to a broad range of researchers. The nature of the topics that occurred the most is presented next.

Most Frequent Topics

It is informative to consider what kind of topics were discussed in articles published in the first 20 years of Psychological Methods. Table 4 lists the top 11 index terms, along with the frequency of endorsement for articles published in this journal from 1996 through 2015. The most recurring topics included statistical analysis, models and structural equation modeling, which reinforces the claim by Rodgers (2010) that we are amidst a statistical revolution that values and emphasizes modeling methods. Among other topics that emerged were statistical estimation, effect size, meta-analysis and statistical power, all of which focus on the movement toward making more informed statistical inferences rather than with using just traditional null hypothesis testing (e.g., Cumming & Calin-Jageman, 2017; Harlow, Mulaik & Steiger, 2016).

Table 4.

Most Frequent Index Terms in Psychological Methods from 1996 through 2015

Frequency	Index Terms
184	Statistical Analysis
126	Models
102	Structural Equation Modeling
79	Statistical Estimation
63	Meta-Analysis
60	Effect Size (Statistical)
58	Factor Analysis
56	Experimental Design
54	Statistical Power
51	Statistical Data
50	Methodology

Note: Frequencies were obtained from the APA PsycNET website on January 20, 2017.

Psychological Methods Today

Psychological Methods currently has one editor (i.e., Lisa Harlow) and eight associate editors (AEs) (i.e., Jaime DeCoster, Herbert Hoijtink, Jee-Seon Kim, Siwei Liu, Keith Markus, Fred Oswald, Lijuan Wang, and Hao Wu). Three associate editors who worked for several years recently but who have stepped down within the last year include Sy-Miin Chow, Ken Kelley, and Ke-Hai Yuan. There are also 48 individuals serving on the current Consulting Editorial (CE) board. On the full editorial board of 57 individuals, including 1 editor, 8 AEs, and 48 CEs, 42% are women and 30% are ethnic minorities.

Over the years, there have been 117 to 299 submissions per year, with an average of about 250 in recent years. After a careful reading of all manuscripts, approximately 50% are rejected without external review, usually due to one of three reasons, (1) the paper is too technical, (2) the paper is too content based, or (3) the paper does not make enough of a contribution or is not adequate for publication. This process helps authors to improve or redirect their manuscripts to a more relevant journal, and saves time for reviewers and AEs who spend a great deal of effort reviewing the remaining 50% of submissions. About 35 manuscripts a year are published, with an overall rejection rate of about 80–85% that has remained fairly constant since 1996, and with a current impact factor of 5.000 and a 5-year impact factor of 9.464.

Since September 2014, Psychological Methods has had an ongoing General Call for Tutorials with 50 tutorial submissions so far, 11 of which have been accepted. Beginning in March 2016, at least one tutorial has been published as the first article in each issue of the journal, although it is not always explicitly listed as such in the actual table of contents. Tutorials have always been part of the mission although methodological researchers sometimes have mixed views on the value of submitting them. One of the goals of the special call is to shift the thinking that these kinds of papers are “just tutorials” to an appreciation that tutorials are illuminating and instructive articles on cogent areas of methodology and reach a wide readership.

Mission

Throughout its history, the mission of Psychological Methods has stayed consistent with the early goals stated in the first editorial by Appelbaum and Sandler (1996). The journal encourages articles that are of interest to a wide range of researchers and that highlight a broad spectrum of topics, such as: methodological innovations, quantitative and qualitative methods, measurement, research design, and clear and overarching tutorials.

Submission Guidelines

The Psychological Methods webpage provides a clear description of the mission and input for submitting manuscripts. Some informal general guidelines for submitting a paper to Psychological Methods are offered here, realizing that successful papers will vary while tending to adhere to many or all of the following.

1. Provide a well-written and understandable description of a methodological focus and how other psychology researchers could use this methodology.

2. Avoid an overly technical focus and provide any needed equations with clear input on all of the terms and uses, understandable to an applied methodological readership.

3. Provide context on why the proposed procedure is needed compared to existing methods and how several areas of psychological research could benefit from this methodology.

4. Demonstrate the methodology with a simulation and/or real-data examples that consider a number of relevant and justified conditions.

5. Describe specific steps for those who wish to apply the methodology to their own research, and whether there are specific computational procedures that could be shared.

6. Be clear about the conditions in which the proposed procedure or methodology would offer useful results, and provide several limitations to its use.

7. Discuss the implications of the findings and what these demonstrate with respect to the proposed methodology, and why this would appeal to a wide audience of researchers.

Conclusion

Throughout its history, Psychological Methods has offered a forum for presenting innovative methods that would be of interest to a broad array of researchers. The individuals that initially saw a role for this journal and believed in its place in the world of publishing deserve appreciation and gratitude for their vision and persistence. In approximately 600 manuscripts published since the first issue in 1996, there have been numerous high-impact articles such that, if an h-index could be allotted to our first two decades of Psychological Methods, it would be 103, with 103 articles reaching at least 103 citations or more (Note: A table of these 103 articles and their respective citations is available from the author). The journal of Psychological Methods is still in the making, owing to a diverse and talented cadre of leaders, authors, researchers, and other readers over its two-plus decades of existence. All have contributed much ongoing interest and support in keeping Psychological Methods as a major applied methodological outlet in psychology and related fields, with much promise of continuing its place into the future.