This special issue of Health Services Research is intended to provide the reader with technical documentation of the development and testing of the CAHPS Hospital Survey and on the decisions that shaped the final product. In addition, it gives readers a sense of the steps involved and the various factors to be considered in developing a survey for widespread use.
CAHPS surveys are used by health care decision makers to help them make informed choices of health care providers. For example, the CAHPS Health Plan Survey is used by a wide variety of organizations, including the National Committee for Quality Assurance, Centers for Medicare and Medicaid Services, state Medicaid programs, Department of Defense, purchasing coalitions, and health plans for a wide variety of purposes, including informing consumers, accrediting health plans, assuring accountability, monitoring performance, and identifying quality improvement targets.
Because CAHPS data are used to inform various health care decisions described above, the development of the CAHPS surveys must incorporate input from a wide range of stakeholders to ensure that the final product will meet the needs of those decision makers. In addition, rigorous scientific methods need to be applied in the development and evaluation of CAHPS data to ensure that the products are credible and useful.
The first article in the special issue by Elizabeth Goldstein and colleagues provides an overview of the process to solicit input on the content of the questionnaire and the methods for sampling, data collection, and analysis. Input from stakeholders was garnered through a variety of venues, including a literature review, one-on-one meetings, focus groups with target consumers, and issuance of Federal Register Notices. Obtaining this information was critical for the CAHPS Hospital Survey because the hospital industry already has a number of surveys to measure patients' level of satisfaction and their assessment of quality of care. The information allowed the team to develop a CAHPS Hospital Survey that could be integrated with the existing surveys conducted by hospitals, while providing sufficient standardization to ensure valid comparisons among hospitals.
The survey development process began with a review of the scientific literature on hospital patient surveys, described in the second article in the supplement by Nick Castle and colleagues. Because a number of surveys are already in use and most are proprietary, the literature review was supplemented with a call for measures from survey vendors and others.
Following the development of an early draft but prior to field testing, the team conducted a series of one-on-one interviews with recently hospitalized patients to assess how well draft items measure what the team had intended and to direct revisions to the instrument. This important step is described in the Roger Levine et al. article on cognitive interviewing.
The questionnaire was then field tested in a pilot survey conducted in three states (Arizona, Maryland, and New York). The bulk of the articles in this special issue present analyses of the pilot survey data and the findings that provided evidence for subsequent revisions of the questionnaire. These analyses assess the impact of factors that may systematically reduce the accuracy of reports about hospital care and increase bias between hospital comparisons. By taking these factors into account, it is possible to obtain a more accurate picture of what patients report about the care they receive and how they assess the quality of that care. The three-state pilot study afforded the opportunity to study the effects of these factors and to revise the survey to reduce potential bias.
The CAHPS team conducted research on the types of information that consumers would like to see in such a report and also on how best to report the results so that they would be used by those making decisions about from which hospital to receive care. The article by Shoshanna Sofaer et al. presents these analyses.
A primary purpose of the three-state pilot survey was to evaluate the reliability and validity of the instrument and to assess its ability to discriminate among hospitals. The article by San Keller et al. describes how the various forms of reliability and validity were assessed and how these findings led to the refinement or elimination of survey items and to an understanding of the survey's underlying constructs.
The CAHPS Hospital Survey is designed to collect data that will enable consumers to compare patients' perspectives on the quality of care delivered by different hospitals. The article by James O'Malley et al. describes the methods and results of a hospital level analysis of the covariation of survey items.
It is increasingly difficult for surveys to achieve a representative set of responses from those sampled. Marc Elliott and his colleagues discuss the impact of both unit (i.e., the person sampled) and item nonresponse on the CAHPS Hospital Survey in their article.
Successful implementation of the CAHPS Hospital Survey may require integration with surveys currently carried out by hospitals, while providing sufficient standardization to assure valid comparisons among hospitals. Almost all hospitals use either mail or telephone administration for their patient surveys. Research comparing results obtained from mail and telephone administration of surveys often finds that telephone respondents provide a more favorable assessment of health care. The article by Han de Vries et al. reports the results of an analysis of data from the three-state pilot survey that examine the possible effects of telephone versus mail mode of administration in the CAHPS Hospital Survey. In the national implementation of the CAHPS Hospital Survey, the Centers for Medicare and Medicaid Services (CMS) will permit the use of mail, telephone, a combination of mail and telephone, or a form of interactive voice response (IVR) known as active IVR, initiated by a telephone interviewer. In addition to other small-scale studies of the CAHPS Hospital Survey at other test sites, CMS is planning a large-scale field test to assess the effect of mode on the survey results.
With the growing number of Spanish-speaking residents in the United States, it is increasingly important for surveys to obtain a representative response from this segment of the population. Therefore, the CAHPS Hospital Survey was developed and tested in both English and Spanish. The Spanish language version was developed using a rigorous translation protocol. However, the team also sought to assess the psychometric equivalence of the instrument in the two languages. Their findings are reported in the article by Margarita Hurtado et al.
Another influence on survey results are systematic differences in reports and ratings of care by different types of respondents. For example, CAHPS survey research shows that those who self report poorer health, tend to rate their care negatively. Because the primary goal of the CAHPS Hospital Survey is to provide data for between-hospital comparisons, there is reason to consider whether the survey results might be influenced by such factors as self-reported health status, and if so, to adjust for differences in patient mix when making comparisons among hospitals. The analyses of the effect of patient mix on the CAHPS Hospital Survey are reported in James O'Malley et al.
The CAHPS Hospital Survey was developed by the Agency for Healthcare Research and Quality, with funding from and in collaboration with CMS. The scientific analyses that supported the development of the CAHPS Hospital Survey were conducted by grantee teams from the American Institutes for Research, Harvard Medical School, and RAND.
Hopefully, this special issue will provide the reader with the information needed to evaluate the scientific soundness of the CAHPS Hospital Survey and give insight into the process followed to develop the survey.