Development of and Field-Test Results for the CAHPS PCMH Survey

Abstract

Objective

To develop and evaluate survey questions that assess processes of care relevant to Patient-Centered Medical Homes (PCMHs).

Research Design

We convened expert panels, reviewed evidence on effective care practices and existing surveys, elicited broad public input, and conducted cognitive interviews and a field test to develop items relevant to PCMHs that could be added to the CAHPS^® Clinician & Group (CG-CAHPS) 1.0 Survey. Surveys were tested using a two-contact mail protocol in 10 adult and 33 pediatric practices (both private and community health centers) in Massachusetts. A total of 4,875 completed surveys were received (overall response rate of 25%).

Analyses

We calculated the rate of valid responses for each item. We conducted exploratory factor analyses and estimated item-to-total correlations, individual and site level reliability, and correlations among proposed multi-item composites.

Results

Ten items in four new domains (Comprehensiveness, Information, Self-Management Support, and Shared Decision-Making) and four items in two existing domains (Access and Coordination of Care) were selected to be supplemental items to be used in conjunction with the adult CG-CAHPS 1.0 survey. For the child version, four items in each of two new domains (Information and Self-Management Support) and five items in existing domains (Access, Comprehensiveness-Prevention, Coordination of Care) were selected.

Conclusions

This study provides support for the reliability and validity of new items to supplement the CG-CAHPS 1.0 survey to assess aspects of primary care that are important attributes of Patient-Centered Medical Homes.

Introduction

There is growing interest in organizing and providing medical care in a way that is consistent with the principles of the Patient-Centered Medical Home (PCMH).¹ Key features of the PCMH include an orientation toward each patient’s needs, culture, values, and preferences; comprehensive, team-based care; coordination of care across all elements of the complex health care system; access to care; and a systems-based approach to quality and safety.² In many states, medical practices are eligible for financial incentives for adopting the PCMH model and the federal government is participating in multi-payer demonstrations and supporting efforts to deploy the model in community health centers.³ The National Committee for Quality Assurance’s (NCQA’s) Patient-Centered Medical Home recognition program is the most widely used method for qualifying practices for rewards in multi-payer demonstrations.⁴ NCQA recently released updated standards for recognition of PCMHs.⁵

Several of the important aspects of PCMH care (e.g., whole person orientation, comprehensiveness) can best be assessed by asking patients about their care experiences and many have argued that patients’ reports about their experiences should be an integral part of assessments of the care provided by PCMHs.^6,7 Currently, no patient survey assesses all of the key functions of PCMHs. The goal of this study was to develop survey questions to assess patient experiences that reflect key elements of the PCMH care model.

METHODS

Survey Development

NCQA and the Consumer Assessment of Healthcare Providers and Systems (CAHPS^®) Consortium originally initiated independent efforts to develop patient survey questions for assessing PCMHs. After soliciting public input on priorities for topics that could be used to assess PCMH care and reviewing six existing surveys with NCQA’s stakeholder and survey expert panels, NCQA decided to use the CAHPS Clinician & Group 1.0 Survey as the foundation for a survey to assess PCMHs because it was already widely used, assesses several key functions of PCMHs, was the only patient experiences survey endorsed by the National Quality Forum, and is part of a family of survey instruments with comparable questions.

The CAHPS Consortium conducted focus groups to confirm the PCMH domains of interest, to learn more about what is important to patients receiving ambulatory care, and to identify the wording that patients use to talk about their experiences. The focus groups were conducted in both English and Spanish, with adult patients and parents of pediatric patients. We included both patients in medical home practices and those in primary care practices that are not categorized as medical homes. Patients with chronic conditions were represented in all focus groups. One group was composed entirely of parents of children with special health care needs. Four focus groups, with a total of 30 participants, were conducted in Boston, Seattle, Philadelphia, and a rural town in Minnesota.

Candidate PCMH items were cognitively tested with adult patients and parents of pediatric patients to ensure that they were consistently understood as intended and that survey respondents could report their experiences using the response options offered. There were two iterative rounds of testing in both English (total n=15) and Spanish (total n=14). In addition to the sites where the focus groups were held, cognitive interviews were also conducted in Florida, Georgia and Maryland.

Based on the results of the focus groups, cognitive interviews, and stakeholder input, we developed a draft questionnaire that included 115 items about: access, communication, coordination of care, comprehensiveness, self-management support, shared decision-making, office staff, as well as a global rating of the provider question and items related to eligibility and patient characteristics. We used items from the CAHPS Clinician & Group (CG-CAHPS) 1.0 Survey^8,9 and other existing surveys as well as new items developed using CAHPS principles¹⁰ addressing the same content.

Field Test

Sites

Our field test was conducted in collaboration with the Massachusetts Health Quality Partners (MHQP), a not-for-profit coalition of physicians, hospitals, health plans, purchasers, consumers, academics, and government agencies. MHQP recruited practices from health care networks, including a collaborative of safety net providers. Forty-three practices, including 10 sites serving adults and 33 serving children in the Boston area were recruited for the field test.

Sample

The sample comprised adult (age 18 or older) and pediatric (under age 18) primary care patients who had visited a study practice between July 16, 2009 and July 15, 2010. A parent or guardian was asked to complete the survey for eligible children. Patients were considered primary care patients if they received care from a physician, physician assistant, or nurse practitioner in Internal Medicine, Family Medicine, General Medicine, or Pediatrics.

Patients were assigned to sites using visit, enrollment, and site data provided by sites. All patients with an eligible visit to a site were equally likely to be sampled regardless of the number of visits, type of visit, or number of providers seen. Once the samples were drawn, patients were assigned to the provider with whom they had the most recent visit. No more than one patient per home address was sampled. Sampling started at sites with the smallest available sample frames to minimize the potential impact of cross-site de-duplication on achieving the desired sample sizes. Sample sizes were selected that were expected to yield approximately 200 completed questionnaires per site. At practices with a majority of patients covered by Medicaid or other non-commercial payer (excluding Medicare), we sampled approximately 833 patients per site. Because we anticipated a higher response rate at practices with a majority of patients covered by commercial insurers or Medicare, we sampled approximately 667 patients per site in those practices. Twenty-three of the pediatric sites were supported by different funding than the rest of the sites and for internal reasons their sample sizes varied as a function of number of clinicians at the practice.

Procedures

Survey Protocol

The initial questionnaire packets were mailed in November 2010 and replacement packets to non-responders mailed in January 2011. Two mailed contacts is comparable to the data collection protocol for the CAHPS Hospital and Medicare surveys.^11,12 No incentive was offered for survey completion.

Analyses

We counted as incomplete any questionnaire that had responses for less than half the items that all respondents were eligible to answer. We used the American Association for Public Opinion Research formula RR1¹³ for calculating response rates. We used forward cleaning to eliminate responses to items where skip instructions were not correctly followed and calculated the rate of missing data and distribution of valid responses for each item.

To identify groupings of items for multi-item composites, we conducted exploratory principle factor analyses. To identify the number of underlying dimensions, we examined multiple criteria including the eigenvalues of factors and the loadings of items on factors. After determining the number of dimensions, we performed an oblique (Promax) factor rotation. We imputed missing data in factor analyses using SAS PROC MI (SAS Version 9.2) and obtained maximum likelihood estimates of the covariance matrix using expectation-maximization (EM) algorithm (SAS PROC MI, SAS version 9.2).¹⁴ Once we made decisions about candidate composites, we calculated scores using proportional scoring and the summated rating method, i.e., we calculated the mean of the responses to each item, after transforming each response to a 0 to 100 scale. We estimated item-scale correlations and the internal consistency reliability (Cronbach’s alpha) of multi-item scales or composites.¹⁵ Since the new PCMH items were to be used with the CG-CAHPS 1.0 survey, we analyzed the responses to all questions together. After reviewing factor analysis results, we tested the individual and practice level reliabilities of different combinations of items.

For each item and composite we estimated practice-level reliability and the number of respondents required to achieve a reliability of 0.70.¹⁶ We examined practice-level reliability using the following formula:

$$\text{Reliability}=(\mathrm{F}-1)/\mathrm{F}$$

Where F is the F-test of the variation among sites.

For the self-management questions, we examined the performance of survey items and potential composites for patients with and without a chronic condition. Presence of a chronic condition was determined using responses to a series of questions from the CG-CAHPS 1.0 Survey that asked whether an adult patient received care for a condition or problem that had lasted for at least 3 months or used prescription medicine to treat a condition or problem that had lasted for at least 3 months. The pediatric chronic condition identifiers require care for a physical, developmental, behavioral or emotional condition or medication to treat it for at least 12 months.

RESULTS

Response rates

We received 4,875 completed questionnaires, for a response rate of 25% for adults and 26% for children (Table 1). Older patients and patients with more visits and chronic conditions were significantly more likely to respond to the adult survey, as were patients of physicians (as opposed to physician assistants or nurse practitioners) (P < 0.001). Medicaid patients were less likely to respond than patients with commercial insurance or Medicare (p < 0.0001). There were no significant differences in the gender of responders and non-responders (p = 0.66). For the pediatric survey, response rates were significantly lower for children with Medicaid coverage, patients of a nurse practitioner, and children between the ages of 2 and 9 (p < 0.001). Below we describe the final determinations that were made for new PCMH items in the context of the CAHPS Clinician & Group 1.0 Survey for both adults (Table 2) and children (Table 3). Items that were omitted from the CAHPS PCMH survey are included in Appendix A.

Table 1.

Characteristics of Adult Survey Respondents, Child Survey Respondents, and Children who were the patients asked about on the Child Surveys

		Respondents for Adult Survey		Respondents for Child Survey		Children asked about
		N	Percent	N	Percent	N	Percent
Age	18–34 years	210	12%	364	12%
	35–44 years	204	11%	950	30%
	45–55 years	376	21%	1152	37%
	55–64 years	382	21%	464	15%
	>=65 years	559	31%	52	2%
	Missing	59	3.%	147	5%	114	4%
	<1 years					128	4%
	1–5 years					1004	32%
	6–10 years					760	24%
	11–15 years					791	25%
	16– 17 years					332	11%
Gender	Male	805	45%	300	10%	1495	48%
	Female	928	52%	2769	89%	1533	49%
	Missing	57	3%	60	2%	101	3%
Education	< High School	132	7%	43	1%
	High School Graduate	333	19%	185	6%
	Some College	431	24%	467	15%
	College Graduate	825	46%	2368	76%
	Missing	69	4%	66	2%
Race/Ethnicity	White	1393	78%			2428	78%
	Black	159	9%			118	4%
	Hispanic	83	5%			205	7%
	Asian	53	3%			232	7%
	Other	31	2%			54	2%
	Unknown	71	4%			92	3%
Health	Excellent	275	15%			1930	62%
	Very Good	559	31%			889	28%
	Good	607	34%			203	7%
	Fair	221	12%			28	1%
	Poor	33	2%			1	0%
	Missing	95	5%			78	3%
Presence of Chronic Condition		1324	74%			212	7%
N of Visits	One	377	21%			709	23%
	Two	430	24%			801	26%
	3–4	612	34%			951	30%
	>=5	257	14%			429	14%
	Missing	114	6%			239	8%

Table 2.

Results for the Adult Version of the CAHPS Clinician and Group Core Survey and New PCMH Items/Composites

	Core or PCMH1	Response Set2	Cronbach’s Alpha for Composite			Composite		Practice Level Reliability
				Correlation with Total	Cronbach’s Alpha with Variable Deleted	Mean	SD
Access			0.74			74.39	23.11	0.87
Got appointment for urgent care as soon as needed	Core	N-A		0.61	0.67			0.70
Got appointment for check-up or routine care as soon as needed	Core	N-A		0.52	0.69			0.71
Got answer to medical question the same day you phoned	Core	N-A		0.61	0.66			0.51
Got answer to medical question as soon as you needed when phoned after hours	Core	N-A		0.53	0.69			0.39
Saw provider within 15 minutes of appointment time	Core	N-A		0.32	0.77			0.93
Items not scored in composite
Got needed care on evenings, weekends, or holidays	PCMH	N-A						0.73
Days you had to wait for an appointment for urgent care	PCMH	0–7 Days						0.60
Information about After Hours Care (not scored as a composite)
Got information about what to do if you needed care on evenings, weekends, or holidays	PCMH	Y-N						0.75
Received reminders between visits	PCMH	Y-N						0.92
Communication			0.91			90.21	17.4	0.62
Provider explained things in a way that was easy to understand	Core	N-A		0.72	0.89			0.77
Provider listened carefully	Core	N-A		0.78	0.89			0.02
Provider gave easy to understand instructions about taking care of health problems or concerns	Core	N-A		0.78	0.89			0.26
Provider seemed to know important information about your medical history	Core	N-A		0.69	0.90			0.64
Provider respected what you had to say	Core	N-A		0.78	0.89			0.10
Provider spent enough time with you	Core	N-A		0.74	0.89			0.75
Coordination of Care (not scored as a composite)
Provider’s office followed up to give you results of blood test, x-ray, or other test	Core	N-A						0.87
Provider seemed informed and up-to-date about care you got from specialists	PCMH	N-A						0.09
Talked with you about your prescriptions	PCMH	N-A						0.71
Comprehensiveness-Behavioral/Whole Person Orientation			0.68			62.2	38.53	0.89
Talked about personal or family problem/alcohol or drug use	PCMH	Y-N		0.45	0.64			0.87
Talked about worry and stress in your life	PCMH	Y-N		0.55	0.51			0.81
Talked about feeling sad or depressed	PCMH	Y-N		0.48	0.60			0.82
Self-Management Support			0.62			39.17	39.31	0.83
Work with you to set specific goals for your health	PCMH	Y-N		0.52	.			0.86
Ask if there are things make it hard to take care of your health	PCMH	Y-N		0.52	.			0.73
Shared Decision-Making			0.61			78.84	25.03	0.61
Provider talked about reasons to take a medicine	PCMH	Not-A lot		0.48	0.44			0.24
Provider talked about reasons not to take a medicine	PCMH	Not-A lot		0.54	0.42			0.51
Provider asked what you thought was best for you regarding medicine	PCMH	Y-N		0.44	0.60			0.68
Office Staff			0.85			83.61	22.46	0.91
Office staff at this office were as helpful as you though they should be	Core	N-A		0.74	.			0.90
Office staff at this office treated you with courtesy and respect	Core	N-A		0.74	.			0.88

¹this column indicates whether the item is part of the CG-CAHPS core survey or an item newly developed for the PCMH survey.

²

this column indicates the response sets used for the item.

• N-A= Never, Sometimes, Usually, Always
• Y-N= Yes, No
• Not-A lot = Not at all, A little, Some, A lot

Table 3.

Results for the Child Version of the CAHPS Clinician and Group Core Survey and New PCMH Items/Composites

	Core or PCMH1	Response Set2	Cronbach’s Alpha for Composite			Composite		Practice Level Reliability
				Correlation with Total	Cronbach’s Alpha with Variable Deleted	Mean	SD
Access			0.67			81.56	18.45	0.88
Got appointment for urgent care as soon as needed	Core	N-A		0.48	0.61			0.76
Got appointment for check-up or routine care as soon as needed	Core	N-A		0.44	0.61			0.78
Got answer to medical question the same day you phoned provider’s office	Core	N-A		0.50	0.60			0.88
Got answer to medical question as soon as you needed when phoned provider’s office after hours	Core	N-A		0.49	0.59			0.72
Saw provider within 15 minutes of appointment time	Core	N-A		0.34	0.70			0.95
Items not scored in composite
Got needed care on evenings, weekends, or holidays	PCMH	N-A						0.74
Days you had to wait for an appointment for urgent care	PCMH	0–7 Days						0.89
Information about After Hours Care (not scored as a composite)
Got information about what to do if you needed care on evenings, weekends, or holidays	PCMH	Y-N						0.77
Received reminders between visits	PCMH	Y-N						0.91
Communication			0.88			93.91	12.61	0.76
Provider explained things in a way that was easy to understand	Core	N-A		0.71	0.86			0.80
Provider listened carefully	Core	N-A		0.74	0.85			0.41
Provider gave easy to understand instructions about taking care of health problems or concerns	Core	N-A		0.71	0.86			0.49
Provider seemed to know important information about your medical history	Core	N-A		0.64	0.87			0.75
Provider respected what you had to say	Core	N-A		0.71	0.86			0.49
Provider spent enough time with you	Core	N-A		0.68	0.86			0.72
Stand-alone items related to Communication
Provider explained things in a way that was easy for child to understand	Core	N-A						0.55
Provider listened carefully to child	Core	N-A						0.64
Provider gave you enough information about what was discussed with child when you were not there	Core	Y-N						0.58
Provider gave you enough information about needed follow-up for child’s care	Core	Y-N						0.11
Coordination of Care (not scored as a composite)
Provider’s office followed up to give you results of blood test, x-ray, or other test	Core	N-A						0.54
Provider seemed informed and up-to-date about care you got from specialists	PCMH	N-A						0.52
Talked with you about your prescriptions	PCMH	N-A						0.55
Comprehensiveness-Development			0.73			70.46	32.93	0.85
Talked about child’s learning ability	core	Y-N		0.51	0.69			0.69
Talked about behaviors that are normal for child at this age	core	Y-N		0.61	0.62			0.85
Talked about how your child’s body is growing	core	Y-N		0.44	0.72			0.81
Talked about child’s moods and emotions	core	Y-N		0.56	0.65			0.74
Comprehensiveness-Prevention			0.78			61.34	30.23	0.91
Talked about things to do to keep child from getting injured	core	Y-N		0.58	0.73			0.90
Given information about keeping child from getting injured	core	Y-N		0.55	0.74			0.92
Talked about how much time child spends in front of a computer/TV	PCMH	Y-N		0.52	0.74			0.88
Talked about food your child eats	core	Y-N		0.44	0.76			0.90
Talked about exercise your child gets	core	Y-N		0.47	0.76			0.74
Talked about how child gets along with others	core	Y-N		0.55	0.74			0.84
Talked about if there were problems in household that might affect child	core	Y-N		0.40	0.77			0.80
Self-Management Support			0.57			71.89	35.46	0.69
Work with you to set specific goals for your health	PCMH	Y-N		0.50	.			0.63
Ask you if there are things that make it hard for you to take care of your health	PCMH	Y-N		0.50	.			0.61
Office Staff			0.88			84.39	22.88	0.91
Office Staff at this office were as helpful as you though they should be	Core	N-A		0.79	.			0.91
Office Staffs at this office treated you with courtesy and respect	Core	N-A		0.79	.			0.89

¹this column indicates whether the item is part of the CG-CAHPS core survey or an item newly developed for the PCMH survey.

²

this column indicates the response sets used for the item.

• N-A= Never, Sometimes, Usually, Always
• Y-N= Yes, No
• Not-A lot = Not at all, A little, Some, A lot

Access

The CG-CAHPS 1.0 Survey includes 5 questions about access to care. Two additional PCMH access items were selected because of their salience in the PCMH model. One item addressing after hours care (“Got needed care on evenings, weekend or holidays”) had good practice level reliability (0.73 for adult survey and 0.74 for pediatric survey) and correlated highly with the existing C&G Access items (r= 0.46 for adult survey and r= 0.50 for pediatric survey). The second item, about the number of days it took to get an appointment for urgent care, had lower practice-level reliability in the adult survey (0.60 versus 0.89 for child survey). These two items were retained but not included in the access composite. It was decided that these items did not improve the composite enough to justify including them in it. In addition, the days to urgent appointments question had a different response scale. Only a small number of respondents reported they had sought advice by email. The 3-item composite from the CAHPS Health Information Technology (Health IT) survey is recommended as a supplemental set of items where access to and use of e-mail for advice are more salient.¹⁷

Information about After Hours Care

Two questions about a practice’s efforts to provide information about after hours care and reminders between visits were selected. These items were initially chosen to assess other domains (Access and Communication respectively). However, the factor analyses showed that the items did not group well with the original domains. The question about information related to after-hours care had a loading of −0.07 for the adult survey and −0.18 for the pediatric survey on the Access factor. The question about reminders between visits loaded 0.01 for the adult survey and 0.14 for the pediatric survey on the Communication factor. Also, they did not form a cohesive separate scale (Cronbach’s alpha = 0.33 for adults and 0.20 for child). Their content is particularly germane to the PCMH, however, and they demonstrated good practice-level reliability (0.75 and 0.92).

Communication

The CG-CAHPS Survey includes 6 items about communication in the adult survey and 10 items in the child survey. We tested 6 new PCMH items related to communication; all of these loaded strongly on the same factor as the core CAHPS communications items. In addition, new items developed to address “whole person orientation” also correlated strongly with the communication items (e.g., “Provider knew you as a person” correlated 0.65 with the CAHPS Core Communication composite). The site-level reliability for a composite based on the combined set of items was higher than the CAHPS core composite for both the adult and pediatric survey, but the practice-level reliability of the core items in this field test was much lower than what has been observed in other settings. For example, previous analyses of the CG-CAHPS 1.0 Survey showed a reliability for the composite of 0.71, compared to 0.62 in this report, although the sample size in that study was much larger.⁹ Furthermore, the internal consistency reliability of the expanded set of communication items was lower than the core communication items for both the adult and pediatric survey. The internal consistency reliability estimate was 0.90 and 0.86 for the adult and pediatric core items, respectively, but only 0.88 and 0.86 for the composites that also included the additional PCMH items. Given concerns about survey length, we decided to retain the core communication composite without any additional items.

Care Coordination

The CG-CAHPS 1.0 Survey includes one item in the care coordination domain – “Provider’s office followed up to give you results of blood test, x-ray or other test.” Of the seven other care coordination items tested, two were retained: “Provider seemed informed and up-to-date about care you got from specialists” and “Provider talked with you about your prescriptions.” These items were included because they addressed different aspects of coordination relevant to a PCMH practice. The psychometric results suggest that they do not form an internally consistent scale (Cronbach’s alpha = 0.49 for adult and 0.50 for child). In addition, the item about coordination with specialty care had low practice level reliability (0.09 in the adult survey).

Stakeholders placed a high priority on care coordination as an essential component of PCMH practices. The question about coordination with specialists (“whether the provider was informed and up-to-date on care received from specialists”) is especially important because of frequent reports of poor communication between primary care and specialty providers. This question has performed well in other surveys (e.g., the CAHPS Health Plan survey), so we decided to retain it after making question wording and format revisions. In the field test version, this question about was placed after the access and communication sections. A final round of cognitive interviews conducted during the field test showed that some respondents were confused about whether the “provider” was the focal provider in the first part of the survey or the specialist. Based on these cognitive interviews, we moved this question to immediately follow the rating of the provider and changed the wording to clarify the referent: “In the last 12 months, how often did the provider named in Question 1 seem informed and up-to-date about the care you got from specialists?” In addition, introductory text was added prior to subsequent questions that asked about the referent provider: “Please answer these questions about the provider named in Question 1 of this survey.”

Comprehensiveness and Whole Person Orientation

We originally identified a set of items to assess “whole person orientation” but analyses indicated that several of those items assessed aspects of communication already adequately captured in that communication composite. Other items were more correlated with items that assessed comprehensiveness, so we developed a short composite of items assessing behavioral and/or emotional health needs for adults. This 3-item composite had good practice-level reliability (0.89) and internal consistency reliability (alpha = 0.68) and it addresses an important but often overlooked domain of care.

The pediatric CG-CAHPS 1.0 Survey already includes two composites addressing development and prevention. For the Child CAHPS PCMH, items related to development and prevention performed better than the behavioral health domains (Table 3). A new question about “computer and television screen time” was added to the existing pediatric CAHPS preventive care items.

Self-Management Support

We developed new PCMH items to address self-management support for general health needs as well as for chronic conditions. Factor analysis showed that almost all items related to self-management support loaded heavily on a single factor. For the adult survey, all items had a loading of 0.47 or higher except for two questions about setting goals and getting support of making changes to habits or lifestyle, which were not retained (loadings of 0.39 and 0.03). For the pediatric survey, all loadings were at least 0.46.

These items were administered to all respondents in the field test, but we were interested in whether some items would work better with a targeted sample of patients with chronic conditions. For the adult survey, the self-management support items performed better with patients with self-reported chronic conditions, and some of the candidate items are less appropriate for a non-chronic condition population. We considered a composite targeted at individuals with chronic conditions but this would have required additional screening questions and a complex skip pattern. We chose instead to include items that were relevant to help maintain good health as well as improve health in the presence of chronic conditions. This composite includes two items: “Work with you to set specific goals for your health” and “Ask you if there are things that make it hard for you to take care of your health.” The composite had better psychometric properties in the adult survey than in the child version (alpha = 0.62 and practice level reliability = 0.83 for the Adult versus 0.57 and 0.69 for the Child version) but there was strong stakeholder support for including this composite in both surveys.

Shared Decision-Making

We tested a series of three items about decision-making for two different kinds of decisions: stopping or starting a medication and having a “surgery or procedure.” Just under half of adult respondents were eligible to answer the questions about medications; about one quarter of respondents screened into the questions about surgery or a procedure. Neither set of items had a site-level reliability of 0.70 or greater. However, because this topic is of critical importance to consumers, we included the items on medications decisions in the PCMH item set for adults and recommend that the responses to these items be combined to calculate a composite score. Because of the small proportion of children for whom this series of questions was relevant (about 25%) and the large number of items related to prevention and development, we did not recommend this composite as a supplement for the child survey.

Office Staff

This domain was not prioritized by our advisory groups as particularly important for the PCMH model, but the Office Staff composite is part of the CG-CAHPS 1.0 Survey. The composite had good reliability (alpha = 0.85, practice-level reliability = 0.91).

DISCUSSION

Using broad stakeholder input and both qualitative and quantitative testing, we developed a set of items that can be used to evaluate attributes of primary care that are particularly relevant for PCMHs. The CG-CAHPS 1.0 Survey with the new PCMH items assesses access, coordination, shared decision-making, self-management support, comprehensiveness and information-sharing.

For the adult survey, to assess Access, two new items were developed that can be used as separate items (i.e., not part of the Access composite). Two new items that are not part of any composite were developed to access information provided by the practice. Two new questions about Coordination of Care can be used as separate items (i.e., not part of a composite) with the CG-CAHPS 1.0 coordination question. A new three item composite about Comprehensiveness-Behavioral Health/Whole Person Orientation, a two item composite about Self-Management Support, and a three item composite to assess Shared Decision-Making were developed. Comparable items were developed for the pediatric survey, although some of the comprehensiveness items were not new. Subsequent field tests will assess modifications to existing care coordination items and possible new questions. We also will continue to assess different ways of assessing shared decision making.

While the CG-CAHPS 1.0 survey supplemented with these new PCMH questions is particularly germane to evaluating performance of practices that have adopted the PCMH model, the items are relevant for all primary care settings. Recommendations for survey sampling and data collection, and quality improvement efforts are available on the AHRQ website (https://www.cahps.ahrq.gov/Surveys-Guidance/Item-Sets/PCMH.aspx).

Input from diverse stakeholders shaped the new CAHPS PCMH content: a public comment process led by NCQA identified communication, shared decision-making, care coordination and self-management support among the top priorities for evaluating PCMH practices. Adding the recommended new PCMH items to the core CG-CAHPS Survey increases the survey length from 34 to 52 for the Adult survey (55 to 66 items for the Child survey). Previous research by the CAHPS Consortium suggests that the survey length should not affect survey response rates).¹⁸

Overall, the CG-CAHPS PCMH items and composites have sound psychometric properties. However, some items were included because of the salience to the PCMH model. For example, we chose to include the Shared Decision-Making composite in the Adult survey despite internal consistency and practice-level reliability less than 0.70. We anticipate that as practices focus more on involving patients in decision-making (one of the tenets of the PCMH model), more patients will report that they discussed starting or stopping a medication with their providers (thus increasing the eligible denominator sizes for the measure) and these items will better discriminate among practices that do and do not offer discussion of the benefits and drawbacks of medications. The CAHPS Consortium is continuing to do research on the PCMH survey and will re-evaluate the items as more information becomes available.

Patient survey data now are considered an integral part of the evaluation of health plans and hospitals; they should also be included in the evaluation of care at the provider and practice level. While the CAHPS PCMH survey was developed to evaluate care provided by PCMHs, the content is equally relevant for Accountable Care Organizations and health plans.

This study had several potential limitations. The responses rates were lower than seen in some other surveys, although they are similar to response rates in some implementations of CAHPS surveys. Fielding during the winter holidays was necessary to stay on schedule and may have contributed to lower response rates. Also, limited resources and time available for the field test precluded the use of telephone follow-up which typically boosts response rates, particularly in sites serving large numbers of Medicaid patients.^18–20 Nevertheless, a low response rate, while potentially biasing the distribution of responses, should not have a major effect on analyses of associations, the primary focus of these analyses.

Another potential limitation is that the field test was only conducted in the Boston area. Thus, the results may not generalize to other areas. However, we think that although the distribution of responses and practice level reliability may be different in other areas, the associations among items are less likely to vary across geographic regions.

Patient-centeredness is a critical domain of health care quality. In spite of its potential limitations, this study provides support for the reliability and validity of the CAHPS PCMH survey. This survey can be used to evaluate important aspects of primary care that are consistent with the PCMH model of care as well as with other efforts to reform the health care delivery system.

Appendix A. Items Tested but Not Included in CG-CAHPS Patient Centered Medical Home Survey

Access

Days you had to wait for an appointment for urgent care

Visited emergency room on evenings, weekends, or holidays because could not get needed care from provider

Got answer to medical question as soon as you needed when e-mailed provider’s office

Got information about office hours, appointments, and prescription refills

Kept informed about wait time for appointment

Communication

Asked how you would like to receive information

Received reminders between visits

Provider used medical words you did not understand

Provider interrupted you

Provider answered all your questions

Provider gave you as much information as you wanted about health problems or concerns

Coordination of Care

Provider had your medical records

Got test results as soon as you needed

Got help managing care from different providers

Offered after visit notes

Got help contacting child’s school or daycare

Comprehensiveness

Usual place for urgent care

Usual place for routine care

Usual place for care for on-going problem

Got needed help for personal or family problem/alcohol or drug use

Whole Person Orientation

Provider seemed to understand what was important to you about your health

Provider knew you as a person

Rating of provider’s knowledge of your medical history

Rating of provider’s knowledge of your responsibilities at home/work/school

Rating of provider’s knowledge of you as a person

Self-Management Support

Talked about things you could do to change your habits or lifestyle

Talked about what was available in community to change habits or lifestyle

Got help making changes in your habits or lifestyle

Talked about health monitoring

Talked about health goals

Offered help for things that make it hard to manage your health

Got instructions about how to manage your health

Talked about what was available in community to manage your health

Got help to get community services to manage your health

Helped you learn skills to manage your health

Shared Decision Making

Provider talked about reasons to have surgery or procedure

Provider talked about reasons not to have surgery or procedure

Provider asked what you thought was best for you regarding surgery or procedure