From the Editors: External Peer Review at HSR

In 2003, we initiated a series of editorials entitled “From the Editors” that were designed to make our policies and procedures as transparent as possible. To date, we have written about our special issues and special sections, our transformation to an electronic peer review process, and the rates of and time-to-make different types of editorial decisions.1 The perspective on our peer review process taken to date has emphasized information most important to our authors. In this editorial we examine our process from the perspective of the reviewer and address four areas: (1) the HSR process for selecting external reviewers, (2) what editors want to know from reviewers, (3) HSR's experience in using reviews in our new internet-based system, and (4) examples of how reviews can impact the content of manuscripts.

External peer review is integral to the process of evaluating the scientific design and evidence-base and of ensuring the integrity of our field. It helps ensure that published manuscripts have undergone careful and open examination of the theoretical background and models used; that the methods and data sources and findings have withstood critical scrutiny; that the conclusions and implications have been evaluated regarding their importance and validity; and that the manuscript has been judged for its overall worthiness of publication. Key to this effort is a small army of potential reviewers (HSR has about 2500 in its database) and members of the Editorial Board who agree to bear an unusually large load as well as other duties. The official payments for reviewing consist of an annual “thank you” and “thanks to anonymous reviewers” from some grateful authors. The principal reward, however, is simply knowing they have contributed to the process to ensure quality of research. Other incentives include advising some colleagues [although anonymously], the potential to see cutting-edge work before its publication, and receiving feedback from the review process itself. We are indebted to all our reviewers for their time and effort, and this editorial is dedicated to them. And, while we hope that experienced reviewers will find this editorial edifying, it is particularly designed for young investigators and scientists-in-training. We invite you all to join our army.2

Our Process for Selecting External Reviewers

In order to maximize the contributions from external reviewers, either of the two co-Editors-in-Chief (EIC) first screens all manuscripts to determine their potential suitability for HSR, both in terms of content and quality. The EIC also judges whether any disclosed conflicts of interest or prior dissemination preclude publication in HSR. Overall, the EICs reject about 30% of HSR manuscripts without external review at this step.

If manuscripts pass this initial screening, the EIC then assigns the manuscript to one of the six Senior Associate Editors (SAE), on the basis of their expertise and overall workload. At the same time, the EIC suggests several external reviewers and may comment on the types of expertise that may be appropriate to draw upon. We intend to select reviewers who provide a balance of the areas of expertise most pertinent for reviewing the manuscript, and so the EIC may create two lists to draw upon.

The SAE, after reviewing the manuscript and the EIC's suggestions about reviewers, makes the actual selection of reviewers, often using reviewers who are not on the original EIC list. The list may include up to 10 or more potential reviewers with appropriate expertise, because the SAE needs to invite reviewers until two agree to review. Occasionally, a potential reviewer will provide an alternative name if he/she cannot review.

There are several criteria and sources the editorial staff (i.e., the EICs and SAEs) draw upon in selecting potential external reviewers for a manuscript, including: (1) the names submitted by the authors as potential reviewers or reviewers to avoid [the latter is always respected; we don't necessarily use the former, but always find it helpful]; (2) authors of selected references used in the paper; (3) an examination of names we have in our reviewer database or who are otherwise known to us; and (4) a keyword search using our automated system. MEDLINE searches are used as a tool to help ensure we do not include reviewers who have co-authored recent papers with the authors or are at the same institution. MEDLINE can also identify authors of papers on similar topics and help us examine the suitability of the publications of potential reviewers.

How Our Electronic Peer Review System Works for Reviewing

We started using an internet-based peer review administration system in June of 2003. This system allows us to track key steps, including submission of manuscripts and their revisions as well as e-mail communications about the manuscript, reviews, and recommended decisions at various levels.

Focusing on the electronic process as it relates to reviewers, this system permits the editorial staff to look at the brief biographical information and key words entered by each reviewer; also recorded for each person is the number of manuscripts currently under review, the reviewer's average time to return reviews, whether they are on our Editorial Board, and whether they have recorded a “vacation message” requesting that they not be selected for a given period of time. The EICs or SAEs can add comments too; here, we typically remark when the reviews were unusually excellent or problematic.

The editorial staff at all times knows the identity of the reviewers and authors; otherwise, our system is officially double-blinded, that is, the reviewers and authors are unknown to each other. In practice, of course, either direct reference by authors to their own work or other hints may give names away to knowledgeable reviewers or authors, but we try to maintain anonymity of reviewers “forever” and of authors until publication.

The system permits various levels of searches by keywords to try to find a match between reviewers and the content of the manuscript. This is why it is important for potential reviewers to enter keywords that describe their areas of expertise and interests as a reviewer. These help us identify manuscripts most appropriate for them. Because we keep track of the number of reviews in process and completed, adding keywords does not result in too many requests, but it does increase the chances the request will be on target.

Once the SAE selects a potential reviewer, the system sends an invitation letter that includes the abstract of the article and notes the expected timeframe for returning a review. Occasionally the first two reviewers asked will agree to provide reviews. When this is not the case, additional potential reviewers are asked, following the prioritized list the SAE and EIC have developed. As one might expect, this sometimes gets a bit complicated when several different types of expertise or viewpoints are needed for a balanced review.

After a reviewer agrees to review, reviewers are given an access code that allows them to see the manuscript file “for reviewers”, which consists of anonymous PDF files of the manuscript and any attachments intended for reviewers. When they are ready to file their review, they use this same system to answer a “scorecard” of questions about the manuscript's quality and importance, suitability for a full research article or research brief, and appropriateness for a special commentary. Reviewers are asked to state their recommendation for publication (reject, minor or major revisions, accept) for the editorial staff only and indicate their willingness to re-review this manuscript if it is revised. Then they can attach or create separate comments intended only for the EIC and SAE and comments for the authors. Only the last type of information is shared with authors.

The system tracks the elapsed time between each expected step in the process as well (such as the number of days after an invitation to review is sent and a reviewer agrees) and, if expected actions have not been completed, sends e-mail reminders to either the Managing Editor for her action, or to the reviewers.

When both reviews have been returned, the SAE examines the reviewers' information and the manuscript again and makes a recommendation to the EIC. The EIC examines the reviews, the reviewers' scorecard information, the manuscript, and comments from the SAE and then makes a final judgment. In cases where the EIC disagrees with the SAE's recommendation, they consult until consensus is reached.

Once the EIC's decision is sent to the authors, the system helps send a thank you to reviewers, which includes the decision reached by the EIC and copies of all anonymous review comments to the authors. For particularly outstanding reviews, we send a separate “gold star thank you” to the reviewer, with the expectation that reviewers may wish to forward our commendation of their exemplary help to others as appropriate.

What We Want to Know from Reviewers

In short, of course, we want a careful reading of the manuscript and as unbiased an assessment of the quality of the manuscript as possible, as well as suggestions as to how it can be improved.

There are several aspects of reviews that are most useful [or not]. Here is a list of things to do:

In the comments for the authors:

• Please separate general comments relating to the overall approach taken by the authors, such as the appropriateness of the methods used relative to the question at hand, from those that relate to specific paragraphs or pages.

• If possible, distinguish major points from minor ones. The former are issues that are likely to be critical to the judgment of acceptability of the manuscript. The latter are things that should (e.g., a stilted writing style) or must (e.g., an inaccurate reference) be corrected, but do not jeopardize the validity of the manuscript.

• It is helpful to break each point into a separate paragraph so authors can refer to them in a response letter.

• It is also important, for both the editorial staff and the authors, that reviewers offer enough comments to the authors to adequately support their recommended decision. Sometimes reviewers describe “fatal flaws” in their comments to the editorial staff that are not mentioned in the comments to authors. Needless to say, authors are then “surprised” by our decision, given the apparently favorable external reviews, even when we address additional concerns in our letter. It is far better for an author to be pleasantly surprised by a set of detailed critiques that are accompanied by request for a revision than to be befuddled by a rejection with only minor comments in the reviews.

In comments for the editorial staff:

• Reviewers should tell us whether they believe they have not been able to adequately review aspects of the manuscript and suggest what other expertise would be necessary to cover areas overlooked by them.

• If there are appendices “for reviewers only” or suggested by the authors as intended for publishing, we would like your opinion about whether they warrant publication on Blackwell Synergy, a website for viewing HSR's published articles electronically. While we can include some appendices in this electronic system, overuse of this option is not helpful, and we would appreciate your assessment about whether publication of the appendix in HSR merits such important and publicly available access.

• If there are concerns about human subjects issues, conflicts of interests, or prior dissemination of which reviewers are aware, we want to be informed. We will then be alerted to the need to make a more careful determination. We will work with you to handle these issues in a manner that appropriately protects your confidentiality.

• As noted above, comments to the editors are not shared with authors. If you think you have found “fatal flaws”, those concerns should also be detailed in the comments to the authors. However, if you have serious concerns that you may not wish to state, or state so strongly, to authors, please convey such comments to the editorial staff.

• Please help us understand the extent to which your concerns are evidence-based or hunches. For example, you may have detected so many examples of inattention to detail in the manuscript that you feel uncomfortable the authors will be able to carry out the complicated changes you feel are necessary before its suitability for publication.

• If you believe that the manuscript is better suited for other journals, please be as specific as possible as to why, and if possible, suggest other journals or types of audiences.

• Try to separate your assessments of technical quality and importance. If you believe that the manuscript is “okay” on technical grounds, but does not deserve space in our journal, please separate these two aspects in a recommendation because sometimes the manuscripts are for special sections or special issues where use of this space does not detract from regular manuscripts.

• Try also to separate your assessment of the manuscript's technical quality from its ideological or research perspective. For example, you might feel that a manuscript has done a good job but is addressing the wrong question or overlooks certain key issues or implications. If you wish to challenge the author(s) on those points in your comments, that is appropriate.

• You may also suggest that a commentary providing a contrary viewpoint—or emphasizing the policy importance—would be desirable for this manuscript if accepted. We appreciate suggestions of possible authors for such commentaries and indicating your own willingness to do so is welcome. We do not always depend on reviewers to decide whether to solicit a commentary but greatly appreciate reviewers' opinions.

• If you believe you know who authored the paper, it is appropriate to let us know. Our official process tries to blind the authors to the reviewers and vice versa, but it is not always effective. For example, we do not require that authors not refer to their own work during the peer review process, and there are other reasons that reviewers can recognize who has authored a paper. Simply knowing who an author is does not jeopardize the usefulness of your review. If you believe you know the authors and this knowledge makes you concerned that your critique is likely to be biased, please let us know and do not review the manuscript. If your critique is not biased by probably knowing who has authored the paper, please proceed with the review.

In this next list, here are a few points to avoid in your reviews:

In comments for the authors:

• Don't state your recommended decision in your comments to authors.

• Don't request that the authors add references to your own work and otherwise “toot your own horn.”

• Don't write only positive comments or minor criticisms in your notes to authors and then refer to “damning” fatal flaws only in your comments to the editors.

• Don't be personal or sarcastic or mean-spirited. As a rule of thumb, always approach a review as though you are advising a valued colleague who knows who you are. We do not edit your comments before sending them to the authors, but we may remark on and apologize for their inappropriate tone in our cover letter to the authors and may share them with the full editorial staff. If you find yourself reacting so strongly to a manuscript that you feel you cannot be civil, it is best to decline to review it.

• Don't assume you know exactly who the authors are and therefore try to address your comments accordingly. Many reviewers who think they know who have authored the paper are wrong—or at least do not realize that the lead author is a new investigator.

In comments for the editorial staff:

• One of the most difficult types of review situations is when an author takes too long to respond; HSR has had some authors take well over a year after being invited to revise and resubmit. Understandably, you probably cannot remember reviewing the original manuscript. In such cases, please be sure to evaluate anew whether it meets quality and contribution standards. We warn authors that delaying re-submission puts their manuscript at risk of being seen as no longer relevant or up-to-date.

HSR's Experience in Using Reviews and Their Impact on Decisions

The experience reported here is from our electronic peer review system, covering manuscripts submitted between June 10, 2003 and March 2, 2004. The assessment of the manuscript's status was made during the week of July 2, 2004, that is, a minimum of 4 and a maximum of 13 months after the manuscript was first submitted.

There were 498 manuscripts submitted during that time, representing 257 unique submissions after discounting revisions. The peer review process was completed to at least the first round of decisions for all but two of these manuscripts; (subsequent rounds are appropriate if the EIC decision was to revise and resubmit). After removing the two incomplete submissions and the 5 commentaries and 57 manuscripts submitted for special issues, there were 193 unique submissions for publication in regular issues of HSR during this period, for which there was at least one round of EIC decisions completed.

Of these unique submissions, 29.5% were rejected by the EIC without requesting external review. Of greatest interest here is what happened to those 136 submissions that underwent an initial external review. For each of these manuscripts, 2 external reviewers agreed to review, but three did not return their reviews after repeated requests and one reviewer made comments, but did not recommend a decision—leaving us with 268 external reviews with reviewer recommendations for a decision, during this period.

While this snapshot of our experience does not permit detailed assessments, we feel that it illustrates how the contributions of external reviewers fit into our peer review process. Specifically, we report on the extent of agreement of reviewers with each other for a given manuscript, the overall agreement between the reviewer's recommended decisions and the EIC's decision, and what happened during subsequent revisions.

Full agreement on the details of the review of a manuscript is not expected for a variety of reasons—most notably because we usually try to maximize the expertise of the reviewers so as to review all of the clinical, managerial, policy, and scientific issues raised. In fact, the richness of reviewers' comments to the authors and editors is crucial to the judgment of the editors for making the final decision on publication and is particularly important to direct authors to consider when revising their manuscripts. However, in this analysis, we focus on the overall judgments about the suitability for publication in HSR.

We take the perspective of a reviewer receiving the editor's decision and perhaps wondering how that outcome occurred, given his/her recommendation. Table 1 is a snapshot at Round One for the 132 manuscripts that received two external reviews.

Table 1.

Agreement among 2 External Peer Reviewers and the EIC (Analyzed at the Manuscript level)

EIC's Decision at Round One:	Both Reviewers Agreed with EIC's Decision	Reviewers Agreed with Each Other, but not with the EIC's Decision	≥1 Reviewers Agreed with the EIC's Decision or were Less Lenient
Reject n=50	18%	4%	84%
Major revision n=56	26%	6%	95%
Minor revision n=26	42%	16%	100%
Accept n=0	–	–	–

The strongest type of agreement is when both reviewers agree with each other and with the EIC's decision. For example, in 18% of the cases in which the EIC's decision was to reject, both reviewers had recommended rejection. In general, the more “lenient” (i.e., favorable to the authors) the EIC's decision, the more likely this decision reflected the consensus of the two external peer reviewers; however, for all types of EIC decisions, a 3-way agreement occurred among fewer than 50% of manuscripts.

The next column of Table 1 shows the percent of manuscripts where both reviewers agreed but the EIC made a different decision. For example, in 16% of the manuscripts in which both reviewers thought only minor revisions were needed, the EIC decided to request a major revision or rejected the manuscript.

The “flip” perspective is to examine how often the reviewers disagreed with each other. Calculating the complement of the first columns together reveals the incidence of disagreement, which was 78% of the time for rejected manuscripts, 68% of the time for major revisions and 42%, for minor revisions. Though not discernible from this table, the two reviewers disagreed for 72% of the manuscripts overall.

The last column in Table 1 shows the percent of manuscripts within each type of decision when at least one reviewer had recommended what the EIC decided or recommended a less lenient decision. Here we see that for the overwhelming majority of EIC decisions, at least one reviewer either agreed with the EIC or recommended a less lenient decision. The complement percentage for this column (i.e., 16%, 5% and 0% respectively) represents the percentage of manuscripts for which both reviewers had recommended a more lenient decision than what the EIC conveyed. [No manuscript was accepted at Round One and, although not displayed here, no manuscript ever received a recommendation of “accept” from both reviewers.]

In Table 2, we examine the set of all 268 reviewers' recommendations compared to all 136 EIC decisions, without linking them by manuscript. At Round One, all manuscripts received external review. For reasons explained above, 4 had only 1 reviewer's recommendation; these 4 manuscripts and reviewers' recommendations are included in the analysis in Table 2 but not in Table 1.

Table 2.

Overall Decisions Recommended by External Reviewers and EIC

TYPES OF DECISIONS:	Recommendations of External Reviewers	EIC's Decision	% Reviewed Externally (noted by EIC decision)
AT ROUND 1 (N)	268	136	136
Accept	4.1%	-0-	100%
Minor revision	29.5%	19.1%	100%
Major revision	37.3%	41.9%	100%
Reject	29.1%	39.0%	100%
AT ROUND 2 (N)	35	38	22 manu. with ≥1 review
Accept	37.1%	50.0%	26%
Minor revision	34.3%	10.5%	75%
Major revision	14.3%	26.3%	80%
Reject	14.3%	13.2%	80%

Overall, recommendations by external reviewers at Round One tended to be slightly more lenient than the EIC's decisions. In particular, the EICs tended to reject a higher percent of manuscripts (∼40%) compared to the overall percent of reviewers who recommended rejection (∼30%).

At Round Two of the review process, only those manuscripts that received an EIC decision for major or minor revision at Round One were germane for further consideration. Among these, not all had completed the review process at the time of our examination of our experience—either because the authors had not yet resubmitted their manuscript or the peer review process was incomplete for resubmitted versions, leaving only 38 manuscripts with an EIC decision finished at Round Two; of these 38, half had been requested to make major, and half to make minor revisions in Round One.

Sixteen (42%) were not sent for re-review by external reviewers, primarily because: (1) the EIC and SAE upon reviewing the responses from the authors and the revised manuscript felt that a decision could be made without additional external review or (2) one or both reviewers said they would not be willing to re-review the particular manuscript. Not surprisingly, the SAE and EIC were more likely to send for re-review those which had been requested to make major revisions at Round One (13 of 19) than if they had been minor at Round One (9 of 19).

As displayed in the last column of Table 2, few manuscripts that the EIC accepted at Round Two had been sent for re-review. However, about three-fourths of manuscripts where the EIC decided to reject or request a revision again at Round Two were re-reviewed by one or both external reviewers. This pattern is part of the reason that the external reviewers at Round Two appeared to be less lenient than the EICs—apparently because they were requested to re-review the more problematic manuscripts where the EIC and SAE wanted input about whether the responses to the reviews were satisfactory.

While not displayed in these tables, it is interesting but not too surprising to note that reviewers often “disagree” with themselves between rounds—sometimes even changing from recommending “reject” in Round One to “accept” in Round Two, based on the revisions and responses.

We identified several manuscripts where—on the basis of the recommendations of reviewers and the decision conveyed—there seemed to be a mismatch of what external reviewers advised and the EIC decided. Looking into the reviewers' comments, either to the editors or to the authors, and other answers about the originality of the work or its importance for policy or suitability for our audience, usually revealed a rich fount of information that explained why the manuscript was not publishable in HSR—or—why it was “worth” trying to encourage the authors to fix major problems. Indeed, we turn next to a tiny sample of these much richer sources for advice and insights from reviewers to the authors and editors.

How Reviews Can Impact the Content of Manuscripts

We present 3 examples excerpted from actual HSR reviews in the interest of illustrating how and whether external reviews can help improve the field's science and reports of specific studies in particular. In Example A, the reviewer made thoughtful and extensive remarks, but these mostly reflected alternative perspectives that other researchers have taken. While useful as a commentary, it was less helpful as a critique of the approach actually used in the manuscript. In Example B, the critiques were seen as demeaning and constructive responses perceived as somewhat “hopeless” by the authors. In Example C, we briefly present the case that we are featuring on our webpage, http://www.HSR.org, to illustrate how constructive comments from a reviewer helped improve the manuscript that was ultimately published in HSR.

A. Example of a lengthy review that is a commentary rather than a critique

The full review was 14 pages, doubled-spaced. While admirable as a potential commentary and contribution to a cross-disciplinary debate on methods and models, it was difficult for the authors to respond to since it was framed as a critique of the field rather than of the paper. An excerpt:

“This paper describes an econometric shell game attempting to whisk from view the finding that the correlation between having private medical insurance and being healthy is largely or entirely spurious. The analyses use various stratagems to hide that apparently unwelcome observation. At heart, they come down to treating the positive effects of good jobs on health as necessarily resulting from the medical insurance typically provided by those jobs. The results show that IF the health benefits of good jobs were entirely due to medical insurance, THEN medical insurance would have a more positive and statistically significant estimated effect on health. The problem is that there is no reason to believe the IF and lots of reasons to doubt it, as detailed below. … The introduction claims that previous studies … do not address “endogeneity.” The introduction uses various implicit definitions of the term, but by any of them the previous studies do address endogeneity. …‘Endogeneity’ refers to the potential problems that other social, behavioral, and health sciences call ‘confounding’ and ‘suppression.’”

B. Example of using a caustic tone in a review

These comments by Reviewer A are from a second round review, and were in response to the authors' response to Reviewer A from Round One. While the authors attempted to be responsive and ignore the tone at several steps during the entire review process for this manuscript, they raised concerns to us about the apparent put-down and double-standard used by the reviewer:

“This response doesn't make sense at all. … This isn't something that can be argued away in such a cavalier manner. … To publish in a prestigious journal like HSR it is important to conduct rigorous analyses. If the study team does not have the expertise to perform the proper analyses, then they need to find it or be prepared for peer review that points out the problem and the potential implications in terms of decisions about the acceptability of the manuscript for publication.”

in a later point in their response to the comments on Round One, the authors disagreed with a strongly worded assertion from Reviewer A and referred to their experience from interviews with subjects. Reviewer A in turn retorted,

“My comments were based on published literature. One's personal undocumented and unreviewed experiences carry little weight in this sort of argument. I could say that in my personal experience the world is flat, but that doesn't make it so.”

C. Example of a peer review that improves manuscripts

With the permission of the authors and both reviewers and their agreement to remove anonymity, we are creating an extended example of a manuscript that was improved by external review. The authors are R. Tamara Konetzka, Deokhee Yi, Edward C. Norton, and Kerry E. Kilpatrick and their manuscript, “Effects of Medicare Payment Changes on Nursing Home Staffing and Deficiencies,” was published in the June 2004 issue of HSR.

The authors have agreed to let us place a full copy of their manuscripts as submitted in Rounds One and Two, along with their responses to the reviewers. Reviewer A [Orna Intrator] and Reviewer B [Charlene Harrington] have each agreed to let us place their reviews and comments on the webpage too. Here is an excerpt from Reviewer A's initial comments to the authors:

• “This is an excellent paper, very well developed and concentrated on a very timely topic, the reimbursement for skilled nursing care by Medicare under prospective payment system (PPS). Some basic theoretical problems remain. However, I believe that the authors will be able to adequately address them.

• 1. The use of percent Medicare residents is not conceptualized or motivated enough. It appears to be in the “right direction”, but lacks clarity. In particular, percent Medicare measures the intersection of two concepts: firstly capacity: a facility builds its staff, knowledge base, certification, and environment to be able to handle a specific volume of Medicare residents. Secondly, market share: a facility is able to attract a certain amount of fee-for-service Medicare residents and MCO skilled residents. Thus, the observed percent Medicare residents in a facility is a “realized solution” of both concepts. It would change with market demand, policy environment, and competition. If the only change was policy environment (with the introduction of PPS), then the measure at baseline, pre-PPS, may capture facilities' capacity to care for SNF residents. This appears to be the argument driving this analysis. Yet, it is not presented clearly, nor is it motivated enough. I wonder whether the within facility variability in this rate is an important control. I also wonder whether the proportion of certified beds provides another angle at this measure. I would like to see the authors provide a more theoretically based justification for the use of this measure, and its anticipated effect. Potentially this is a misspecification problem, which would only require acknowledgement in the limitation section of the discussion. The discussion of the endogeneity of this measure is also not entirely clear. Finally, an explanation of the main hypothesis tested (the interaction of PPS and % Medicare) needs to be explained. The issue of % Medicare is at the heart of this paper, therefore it may deserve a section to itself. ….

• 3. It is important to note what other pre/post differences there were, and how that may have reflected on the effect of the introduction of PPS. On page 12 you state that the difference-in-difference approach takes into account the general staffing effect within a state … however, if that changes, how does the method allow you to differentiate between the effect of changes in availability of staffing and the effect of introduction of PPS. A sensitivity analysis could possibly be done to determine what magnitude of effect, and what correlation with PPS, would have been necessary to alter the effect of PPS to make it void.

• There are also some relatively more minor issues:

• 1. It is important to explain what a difference in difference model is as the audience is health services researchers and not economists. …

• 3. Please make sure to use SNF to describe the level of care and not nursing facilities. From the definitions in the State Book (Harrington, C., Swan, J. H., Wellin, V., Clemena, W., Bedney, B., & Carillo, H. (1999). 1998 State Data Book on Long Term Care Program and Market Characteristics. San Francisco, CA: Department of Social and Behavioral Sciences, University of California.): A nursing home facility (NF) is a state licensed facility providing skilled nursing and/or intermediate care services to individual residents on a 24 hour basis. This category was created by OBRA l987 Nursing Home Reform legislation. SNF: Skilled nursing facility. Under the implementation of OBRA in 1990, these facilities became 'nursing facilities' or ‘NFs’. This designation is retained by some states to characterize the level of care needs of residents rather than the classification of the facility. …

• 5. Background is confusing and not well organized. It should also be shortened.

• a. Why does length of stay relate to quality of SNF care? You argue on page 6 that hospital PPS and SNF PPS are different. I agree, but your argument does not work well with the importance of LOS to quality. This should be a separate paragraph as well.

• b. The beginning of the paragraph, regarding the results of Norton and colleagues is not clear. Again, too much economic jargon; what is elasticity of .16 and .20??

• [Continued … ]”

In the future, with the permission of all participants, we plan to include additional examples on our webpage along with comments and responses from reviewers and authors and editors, with the hope that these will become useful teaching tools for the field.

And a final note, lest potential authors or reviewers feel discouraged by our making the process and experience at HSR more “transparent” and thereby potentially exposing some of the disagreement and imperfections in peer review as well as its strengths, we offer two points of encouragement: (1) The EICs and SAEs pay very careful attention to external reviewers, but their comments and recommendations are taken as advice, not votes. Notably, for the manuscripts in these examples, all were accepted or are in final stages of the peer review process. (2) The manuscript awarded as HSR's Eisenberg Article-of-the-Year for 2004, that is, was selected for its overall excellence in an area of health services research supported by the late John M, Eisenberg—underwent three rounds of submission for peer review at HSR before being accepted (Miranda et al. 2003). Its principal author, Jeanne Miranda, in accepting the Eisenberg award at our Board dinner, remarked with delight that, during the peer review process, she had not been sure that the reviewers and HSR would ever finally accept her work, let alone award it a prize for excellence. But that is precisely the challenge that peer review, done well, is supposed to issue: to push authors, not only to improve their work, but also to achieve excellence.