Abstract
Background:
Over the last decade, international surveys of patients and clinicians have been used to compare health care across countries. Findings from these surveys have been extensively used to create aggregate scores and rankings.
Objective:
To assess the concordance of survey responses provided by patients and clinicians.
Methods:
Analysis of 16 pairs of questions that focused on coordination, organizational factors, and patient-centered competencies from the Commonwealth Fund International Health Policy Survey of older adults (2014) and of primary care physicians (2015). Concordance was assessed by comparing absolute rates and relative rankings.
Results:
In absolute terms, patients and clinicians gave differing responses for questions about coordination of care (patients were more positive) and provision of after-hours care (patients were less positive). In relative terms, country rankings were positively correlated for 5 of 16 question pairs (Spearman ρ > .6 and P < .05).
Conclusion:
Patterns of concordance between patient and clinician perspectives provides information to guide the use of survey data in performance assessment. However, this study highlights the need to assess the complementarity and substitutive nature of patients’ and clinicians’ perspectives before combining them to create aggregate assessments of performance.
Keywords: health-care performance, concordance, international surveys, rankings
Background
Health-care systems are complex and multifaceted. This complexity means that the measurement of health-care performance should incorporate a wide array of information. At the same time, quality in health-care is a contested construct, shaped by expectations, experiences, context, and priorities of different stakeholders and so meaningful assessment should also encapsulate different perspectives (1,2). Considering the extent of concordance in perspectives from different stakeholder groups may reveal elements of performance that are unwaveringly viewed in the same way (either positively or negatively), both within and across health-care systems. It can also reveal aspects of performance that are seen very differently from a patient and from a provider perspective—with this difference in perspective replicated in all health-care systems. Finally, it can reveal elements of performance that are shaped by differences in health system context, resulting in little concordance both between stakeholders and across jurisdictions.
Patient surveys are a common feature in most developed health-care systems and international surveys occupy an important space in performance assessment efforts and policy discussions in many countries (3 –5). Patients provide a unique view of health care as they are often the only constant presence across episodes of care, treatments, and clinicians and are best placed to observe the extent to which care is patient centered and integrated (6). Increasingly, provider or clinician perspectives are also used in quality assessment and performance reporting (7 –9). Clinicians are well placed to report on issues such as coordination and integration of care through technology, communication, and teamwork across teams (10 –12). Clinicians are also able to assess the culture of safety in their organization; to describe how they work and the tools they have available to enable them to deliver care to patients; and to reflect on coordination with, and information flow from, other clinicians.
The use of multisource feedback is increasingly used to provide information for revalidation and other performance applications in health care (13). Both patients and clinicians are seen as key informants, and both groups serve as an input to policy making (14,15). This is especially important in the context of wide recognition that care should be patient-centered and also promote good experience in delivering health care for clinicians (16,17). Patient and clinician surveys have been used to assess health care performance in terms of the provision of accessible and appropriate care and with regard to the outcomes of care, across and within countries, combining their perspectives using qualitative or quantitative aggregation techniques (3 –5,18). However, studies that compare perspectives have shown that physicians are sometimes limited in their ability to accurately self-assess (19). This means that a range of perspectives are often required to fully explore issues of performance.
While many assessments use scoring and ranking approaches to compare performance and create aggregate measure, there are concerns about the validity of many scoring and ranking methods. The Commonwealth Fund’s work is an example where patient and provider surveys are used to create aggregate ranking scores to compare countries (3 –5).
The idea of “concordance” of patient and clinician perspectives has been considered in alternative ways in research. Some studies have assessed concordance with regard to demographic characteristics of patients and clinicians such as gender, language, or ethnicity (20). Others have explored the alignment of patient and clinician perspectives with regard to shared decision-making (21), attitudes about respect (22), and medication adherence (23). Here, we are interested in concordance as the level of agreement between patients and clinicians at a geographic level with regard to satisfaction with or experience of care (24).
The extent to which the views of patients and clinicians agree, or are concordant, remains to be assessed. This study aims to assess how patient-reported and clinician-reported measures on similar questions provide concordant assessment of performance within 11 countries with developed health care systems across Europe, North America, and Australasia. In addition, it aims to assess the impact of using patient-reported or clinician-reported measures to compare countries with regard to performance in health care. Furthermore, this article proposes an approach to investigate concordance between patient and clinician perspectives regarding particular aspects of care, such as coordination or availability of urgent appointments, for measures reported across countries. This approach may inform both measurement efforts and the interpretation of measures combining patient and clinician perspectives.
Methods
This study is exploratory in nature and uses international surveys of patients and primary care clinicians to assess concordance of assessment. We consider survey concordance in 2 ways: absolute and relative. Absolute concordance refers to the similarity—within a jurisdiction, in the percentage of patients and of providers who selected a particular response option (usually the most positive response option). For example, within a jurisdiction, if the percentage of patients who said their regular doctor spends enough time with them is similar to the percentage of providers who said they were satisfied with the amount of time they have to spend per patient—that is deemed as high absolute concordance.
Relative concordance places the comparison between patient and provider responses in a broader context. It considers whether the extent of concordance is consistent across jurisdictions. For example, looking across all of the participating jurisdictions in the survey—if both patients and providers from a particular jurisdiction gave the highest proportion of positive responses (ie, the jurisdiction was ranked 1 in both surveys), that would be deemed to high relative concordance (regardless of whether the absolute % responses were similar or highly disparate in that jurisdiction).
Data Source and Variables
We used the Commonwealth Fund International Health Policy Surveys of older adults (2014) and primary care physicians (2015). Surveys were conducted by telephone in 11 countries—Australia, Canada, France, Germany, the Netherlands, New Zealand, Norway, Sweden, Switzerland, the United Kingdom, and the United States. The number of older adult respondents ranged across countries from 928 to 7206, and the number of primary care clinicians’ respondents ranged from 502 to 2905 (see Appendix A). Analyses were weighted to reflect the distribution of adults aged 55 years and older and of practicing primary care physicians in each country (11,12).
To identify question pairs that addressed the same aspects of care, 2 researchers independently matched survey questions across the patient and provider surveys. Disagreements in matching were resolved by discussion within the broader research group. In total, 16 pairs of questions were identified (although there were slight differences in phrasing for some question pairs), covering topics such as availability and timeliness of care, coordination of care, and overall views on the health system (Appendix B). A performance dimension, such as accessibility, appropriateness, efficiency or sustainability, was assigned to each question pair based on existing performance reports (5,17). Matching response categories for each question were selected for analysis. In the case of one question regarding overall health system views, both the most and least positive categories were used in this analysis. Appendix C provides descriptive statistics for each pair of questions by country.
Assessment of Concordance
The differences between patient and clinician responses were calculated to provide an estimate of the absolute concordance. Differences were calculated both within countries and based on the average value across countries. The relative concordance between country rankings was assessed for each pair of measures using Spearman rank-order correlation test. Results with P < .05 results were noted as being concordant. Based on the correlation coefficient, ranks were considered to be moderately concordant (>0.4) or highly concordant (>0.6) (21). The averages of each of the concordance measures (absolute and rank differences) were used to summarize results across countries. Within each country, the average concordance values across the 16 pairs of questions were calculated for each of the measures. Finally, to explore possible relationships between the differences in perspectives and the sample information, a rank correlation between the concordance measures with both sample size and response rates was calculated.
Results
Descriptive results and measures of concordance between levels and rankings for each of the 16 pairs of questions are summarized in Table 1. The absolute differences provide an indication of whether patients or clinicians tended to be more positive. Patients responded more positively than clinicians to questions regarding coordination. For example, a minority of patients said their records were unavailable or they had care coordination issues, whereas a majority of clinicians said these issues had occurred in the preceding month. Patients were more likely to say their regular place of care seemed informed about the hospital care they received, while clinicians were less positive about receiving notification from hospitals. In contrast, clinicians responded more positively than patients about the availability of after-hours care. Differences in average patient and clinician responses were small (<10 percentage points) for measures about overall system views, discussing end-of-life care, provision of written plans for chronic conditions, and the time spent in consultation with clinicians.
Table 1.
Descriptive Statistics Including Percentage, Average Across Countries by Patient or Provider Perspective, and Absolute and Relative Concordance Measures Between Perspectives by Question Pairs.a
| Performance Dimension | Survey of Adults Aged 55 Years and Over (2014) | Survey of Primary Care Physicians (2015) | Concordance | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Questions | Average of Countries (%) | Range | Questions | Average of Countries (%) | Range | Difference in Averages (Percentage Points) | Average Rank Difference | Spearman Rank-Order Test | ||
| Coefficient | P Value | |||||||||
| Accessibility | Health-care professional makes contact for chronic condition | 22 | 36.9 | Have staff who contact patient to monitor chronic condition | 34 | 37.9 | 12 | 1.6 | 0.83 | .00b |
| Accessibility | Availability of same or next day appointments | 66 | 39.3 | Almost all patients able to get same or next day appointments | 47 | 44.4 | 19 | 1.6 | 0.72 | .03b |
| Accessibility | Waited 2 months or longer for specialist appointment | 15 | 25 | Patients experience long waits for specialist appointments | 47 | 61.5 | 32 | 1.8 | 0.7 | .00b |
| Accessibility | Skipped care due to cost | 9 | 20.4 | Patient had difficulty paying for medical expenses | 24 | 58.2 | 15 | 1.8 | 0.68 | .01b |
| Accessibility | Very easy to get after-hours primary care | 24 | 32.7 | Practice has after-hours arrangements for patients | 75 | 54.6 | 51 | 1.5 | 0.61 | .01b |
| Appropriateness | Discussed with family, friend, health-care professional about treatment | 44 | 58.9 | Had conversations about treatment wishes with older/sicker patients | 43 | 45.4 | 1 | 2.6 | 0.51 | .08 |
| Sustainability | Health system is working well, only minor changes | 46 | 40.6 | Health system is working well, only minor changes | 39 | 51.2 | 7 | 2.6 | 0.51 | .13 |
| Appropriateness | Received written plan for management of chronic condition | 35 | 37.2 | Patients with conditions given written plan to manage care | 30 | 41.5 | 5 | 2.6 | 0.47 | .83 |
| Appropriateness | Medical staff seemed informed about care in hospital | 86 | 25.6 | Received notification about patient’s care in hospital | 34 | 62.0 | 52 | 2.9 | 0.45 | .35 |
| Appropriateness | Received a list of medications | 58 | 49.4 | Practice can generate list of patient’s medications | 72 | 60.3 | 14 | 2.2 | 0.45 | .75 |
| Sustainability | Health system needs a complete rebuild | 11 | 29.7 | Health system needs a complete rebuild | 6 | 13.6 | 5 | 2.7 | 0.21 | .31 |
| Efficiency | A test repeated because results unavailable | 7 | 11.0 | A patient’s test was repeated because results unavailable | 30 | 30.4 | 23 | 3.6 | 0.10 | .11 |
| Accessibility | GP always spent enough time | 65 | 36.4 | Satisfied with time you have to spend per patient | 57 | 48.5 | 8 | 3.6 | 0.08 | .51 |
| Appropriateness | Experienced care coordination problem | 20 | 25.7 | Patients had care coordination problems | 51 | 50.3 | 31 | 4 | –0.07 | .08 |
| Appropriateness | Medical staff seemed informed about care in hospital | 86 | 25.6 | Received notification about patient’ care in ED | 33 | 61.7 | 53 | 2.9 | 0.32 | .13 |
| Efficiency | Test results or medical records not available at the time of visit | 9 | 13.2 | A patient’s medical record or relevant clinical information not available at the time of visit | 65 | 23.6 | 56 | 4.2 | –0.29 | .8 |
Abbreviations: ED, emergency department; GP, general practitioner.
aThe dark shading indicates the lowest levels of concordance, lighter shading denotes a moderate level of concordance, and no shading indicates a high level of concordance or a significant concordance of rankings (P < .05). For absolute concordance and rank difference measures, shading is based on tertiles. For the rank-order coefficient, the most concordant measure pairs have coefficient >0.6 and P < .05, the moderately concordant pairs are those where 0.4 < coefficient < 0.6, and anything between (−0.4 and 0.4) is noted as not concordant in this analysis.
b P < .05.
The Spearman rank correlation analysis shows a strong concordance (coefficient >.6 and significant P < .05) between rankings based on patients and clinician responses for 5 questions. For these questions, all of which were related to the dimension of accessibility, the average difference in rankings was less than 2 places. However, when looked at in terms of the percentage differences for all 5 questions, there was moderate and low agreement with differences of over 10 percentage points. For the only question that was the same in both surveys, there was strong concordance based on the levels and moderate concordance based on the rankings in terms of positive patient and clinician views seeing health system as working well (Figure 1).
Figure 1.
Example of moderate relative concordance and high absolute concordance: overall health system views.
According to the relative concordance using ranking, there was low concordance for 6 of 16 questions, covering 3 performance dimensions (efficiency, accessibility, and appropriateness). Within this subset of 4 questions, the average difference in rank was approximately 3 or more places between perspectives. The least concordant question pair addressed patients’ medical records, tests, or clinical information being unavailable at the time of their visit, with an average rank difference between perspectives of 4 places (Figure 2).
Figure 2.
Example of low relative and low absolute concordance: medical records or tests not available.
How Do Perspectives Vary by Country?
The differences between patient and clinician perspectives are summarized across the 16 question pairs in Table 2. In terms of absolute differences between patient and clinician perspectives, the Netherlands had the smallest average (absolute) difference of <20 percentage points and Norway had the largest average difference of 30 percentage points. Within countries, the average difference in rankings between questions ranged from 1.9 in Sweden to 3.3 places in the United Kingdom. Countries with larger sample sizes, such as Sweden and Canada, tended to have smaller differences in ranking and smaller correction factors. There were strong negative Spearman rank correlation coefficients between the number of respondents in the patient survey and the rank differences (ρ = −.7).
Table 2.
Average Absolute Difference and Rank Difference Across 16 Pairs of Questions, by Country.a
| Average Difference Between Patient and Provider Responses (Percentage Points) | Average Rank Difference | |
|---|---|---|
| Sweden | 29 | 1.9 |
| United States | 24 | 2.1 |
| Canada | 29 | 2.3 |
| Australia | 29 | 2.6 |
| Switzerland | 26 | 2.6 |
| France | 28 | 2.7 |
| New Zealand | 25 | 2.8 |
| Germany | 29 | 2.9 |
| Netherlands | 19 | 2.9 |
| Norway | 30 | 3.1 |
| United Kingdom | 22 | 3.3 |
| Spearman rank-order coefficient between concordance measures and survey respondent/response rates | ||
| 2014 respondents | 0.33 | −0.70 |
| 2014 response rate | 0.02 | −0.28 |
| 2015 respondents | 0.16 | −0.48 |
| 2015 response rate | −0.05 | 0.04 |
a For full survey details on the number of respondents and response rate, see Appendix A.
Discussion
This study examined the concordance of measures from patient and clinician perspectives used in international surveys and compared a range of methods to assess concordance. Overall, concordance was stronger for certain aspects of health care, such as access. In addition, results for concordance in ranking differ from those for concordance in absolute levels, and in many cases, the magnitude of differences is important.
Our academic and gray literature review did not find any studies that looked at approaches to interpret concordance of measures from patient and clinician perspectives used in international performance comparisons. However, there are examples of geographic correlation in satisfaction of patients and clinicians in the United States and the United Kingdom (25). Further, one country-level study in Ukraine did consider population and physician views about several measures of quality of health care (26). It reported that both groups felt the health system should be reformed, care for common chronic diseases was poor, and affordability was a problem. However, their perspectives differed on quality of care where household respondents saw it as a priority and physicians did not, and results were consistent with our findings of moderate concordance with regard to affordability of care and overall views about performance. As far as we are aware, ours is the first study that has looked at concordance of performance rankings and perspectives from 2 international surveys.
Complementary or Substitutive Perspectives?
Our findings may help to better understand the value of measuring aspects of care from different perspectives, the need to consider both rankings and absolute levels, and the fact that the extent to which perspectives are concordant may be sensitive to context.
Our study identified strong concordance for some pairs of performance measures, regardless of the method used to assess concordance. Questions where perspectives aligned well or moderately well in terms of both levels and rankings suggest performance measurement based on either perspective would have similar conclusions and could be substitutive. In other words, we may be able to aggregate the patient and provider perspectives to compare performance despite differences in expectations between patients and providers, or we may not need to ask both patients and providers about this aspect of care.
For other pairs of performance measures, the assessment of concordance was sensitive to the method of concordance assessment used. For some questions (eg, on after-hours care), there was little difference in rankings while the concordance based on absolute levels was low. For others, absolute concordance was good but rankings varied much more (eg, time spent in consultations), especially when the actual range of results was small between countries. These may point to interesting divergence of perspectives within countries and aspects where assessment may benefit from both perspectives, with patients and clinicians providing complementary assessments of performance.
In situation of strong concordance, the 2 perspectives can be combined. However, we could potentially rely on the use of only one of the perspectives to understand performance. In case of low concordance, the aggregation of perspective risks drawing an inaccurate conclusion and the measurement of both perspectives provide complementary understanding.
Why Might Patients’ and Clinicians’ Responses Vary
From a purely applied perspective, both patients and clinicians provide valuable insights on their own. From a comparative perspective, however, additional insights may come when we look at the concordance of these patient-based and clinician-based assessments across health-care systems. Varying expectations, differences in actual exposures to aspects of health-care delivery, and temporality of experiences of health care and its reporting through surveys may be factors that affect the levels of concordance across patients’ and clinicians’ assessments. Patients’ responses may not show the same time lag between “real experience” and responses at the time of surveys. For example, clinicians are experiencing health care every day, while most patients only experience health care on an infrequent basis. Therefore, changes in performance may actually be picked up more quickly by clinicians compared to most patients. Interestingly, our assessment showed that despite moderate to high concordance in terms of overall views, the United Kingdom ranked second in terms of whether the system was working well based on the perspective of adults aged 55 years and over, while it ranked ninth from a clinician perspective on the same question 1 year later. This could show how expectations toward health care could be influenced at a different pace depending on the group providing their views about health care. Our results suggest that caution is required in the construction of composite measures and aggregation of measures—particularly those that combine different stakeholder perspectives.
This analysis also points to areas where survey data should be used with caution, or performance measures selected more intentionally, from only one perspective. Patients or clinicians may be better placed to report on some of the measures. For instance, patients can report whether they were treated with respect, and doctors can report whether they generally received appropriate information from a colleague following their patients’ referral. Where there is a lack of concordance across methods of assessment, such as the case for reporting of the availability of medical records at the time of patient visits, the conclusions of performance assessment depend on the perspective used. For example, clinicians see many patients and have more chance to encounter issues with availability of records, whereas a patient may see only 1 clinician during a time frame, and lack of availability of records may be a rarer occurrence from their perspective. This could also explain differences with regard to the assessment of coordination of care, since providers and patients will have a very different actual experience of the flow of services. Interestingly, access seems less associated with discordance. Ultimately, patients are the witness of their entire experience with various providers, while providers have an experience shaped by the totality of their roster of patients. Moves toward patient-centered care and a stronger focus on key aspects that relate to coordination of care, communication, and team-based approaches may result in an increased concordance of experience of care between providers and patients.
Limitations
There are several limitations that need to be considered. First, we acknowledge the challenges in comparing the average levels of responses from 2 groups of respondents, using different survey questions with varying response formats (eg, satisfaction vs reporting of experience), over 2 time periods, based on unlinked data sets. However, while not identical, the paired questions ask about the same constructs or elements of care. Assessing the same pair across jurisdictions allows us to see the relationship in terms of patient experience and provider satisfaction. We acknowledge that we have not established whether the questions used in this study actually measure the same constructs, for all raters, as no testing of measurement invariance of multisource feedback has been performed.
Correlational analysis based on 11 countries and the use of country averages and arbitrary cutoffs are also limitations. Further, despite being used repeatedly in several international surveys, the survey measures have not been thoroughly validated. Despite these methodological issues, some clear and consistent themes emerged by triangulating perspectives and methods (absolute and relative rankings).
Survey methods may also affect differences in rankings between patient and clinician perspectives within countries. Countries with larger numbers of respondents such as Sweden and Canada have smaller differences between rankings. This may be due in part to the smaller margins of error or less variability in estimates from larger samples.
Conclusion
Comparing patient and clinician views across jurisdictions provides new insight into assessing performance using survey data. Patterns of concordance between patient and clinician perspectives provides information to guide the use of survey data in performance assessment. However, this study highlights the need to assess the complementarity and substitutive nature of patients’ and clinicians’ perspectives before combining them to create aggregate assessments of performance.
This was a simple, mainly descriptive analysis with potential for greater development. Conceptually, it considers survey data in 4 “planes”—patient perspectives, clinician perspectives, jurisdictional context, and time. Given that heterogeneity—lack of concordance is hard to interpret, however, high concordance may provide more guidance. High concordance could be the reflection of fundamental features of health care that are viewed in the same way by patients and providers and across different contexts and jurisdictions. Future research could more purposefully assess the concordance of perspectives between patients and providers using a prospective approach.
These results reinforce conclusions drawn from the use of many survey measures in international comparisons and show a robustness of survey data as complementary information in performance assessment alongside more established administrative and emerging patient outcome measures. The strong concordance in rankings for accessibility measures indicates that from either perspective, country comparisons of performance would result in similar conclusions about relative performance in access to care. However, there is more work to be done to develop survey questions into performance measures. Measures that are not concordant may point to lessons to be learned about the sensitivity of rankings to the survey or perspective. Patterns across countries showed potential differences that may result from the different timing of surveys in the context of change at the system level. To understand differences in perspectives at the country level, results need to be analyzed separately by measure and with a more in-depth assessment of the health-care policy environment in the specific country.
Acknowledgments
The authors would like to acknowledge the support of the staff of the Bureau of Health Information.
Author Biographies
Jean-Frederic Levesque MD, PhD is Chief Executive of the Agency for Clinical Innovation in New South Wales and Conjoint Professor at the Centre for Primary Health Care and Equity at the University of New South Wales. His career spans both academic and decision-making roles that relates to the measurement and improvement of health care performance in Australia, Canada and India.
Lisa Corscadden is a PhD candidate at James Cook University exploring experiences of disparities in access to healthcare for people with mental health conditions in Australia. She is a lead of performance reports at the Bureau of Health Information.
Anushree Dave has a Masters degree in Bioethics / Medical ethics and is currently a social innovation program manager at Epicentre at McGill University. As part of her studies, she worked on a placement at the Bureau of Health Information.
Kim Sutherland MSc, MBA, PhD is a Specialist Advisor at the NSW Agency for Clinical Innovation. Her career has spanned a range of research, quality improvement, performance measurement and public reporting roles in academic and health system settings in Australia, Europe and North America.
Appendix A. Sample Size and Response Rate by Countries, 2014 and 2015 International Health Policy Surveys
| 2014 Survey of Adults 55+ | 2015 Survey of Primary Care Physicians | |||
|---|---|---|---|---|
| Number of Respondents | Response Rate | Number of Respondents | Response Rate | |
| Australia | 3310 | 31% | 747 | 25% |
| Canada | 5269 | 28% | 2284 | 32% |
| France | 1500 | 29% | 502 | 8% |
| Germany | 928 | 26% | 559 | 19% |
| Netherlands | 1000 | 25% | 618 | 41% |
| New Zealand | 750 | 27% | 503 | 28% |
| Norway | 1000 | 16% | 864 | 44% |
| Sweden | 7206 | 23% | 2905 | 47% |
| Switzerland | 1812 | 60% | 1065 | 39% |
| United Kingdom | 1000 | 23% | 1001 | 39% |
| United States | 1755 | 24% | 1001 | 31% |
Appendix B. Pairs of Measures Selected for Analysis from the International Health Policy Survey of Older Adults (2014) and of Primary Care Providers (2015)
Appendix B.
| Dimension | Older Adults Survey Questions (2014) | Primary Care Physician Survey Questions (2015) |
|---|---|---|
| Accessibility | Last time you were sick or needed medical attention, how quickly could you get an appointment to see a doctor or a nurse? (Same or next day) | What proportion of your patients who request a same- or next-day appointment can get one? (Almost all [more than 80%]) |
| After you were advised to see or decided to see a specialist, how long did you have to wait for an appointment? (2 months or longer) | How often do you think your patients experience the following? Experience long waiting times to see a specialist. (Often) | |
| How easy or difficult is it to get medical care in the evenings, on weekends, or holidays without going to the hospital emergency department? (Very easy) | Does your practice have an arrangement where patients can see a doctor or nurse if needed when the practice is closed without going to the hospital ED? (Yes) | |
| When you need care or treatment, how often does your regular doctor or medical staff you see spend enough time with you? (Always) | Please indicate how satisfied you are with the following aspects of your medical practice. The time you have to spend per patient. (Satisfied) | |
| During the past 12 months, was there a time when you skipped care (treatment, visit or prescription) due to cost? (Yes to any of the 3). | How often do you think your patients experience the following? Have difficulty paying for medications that they have to pay for themselves or other out-of-pocket costs. (Often) | |
| Between doctor visits, is there a health-care professional who contacts you to see how things are going with your condition? (Yes) | Do you and/or other personnel who work with you provide care in any of the following ways? Contact patients between visits to monitor their condition. (Yes, frequently) | |
| Appropriateness | Any coordination problem (test results not available, conflicting information, or unnecessary test). (Yes to any of the 3) | During the past month, did the following occur with any of your patients? A patient experienced problems because care was not well coordinated across multiple sites or providers. (Yes) |
| In the event you become very ill or injured and you cannot make decisions for yourself, have you had a discussion with family, friend, or a health professional about what treatment you want or do not want? (Yes) | Do you have conversations with older or sicker patients about the health-care treatment they want or do not want in the event they become very ill, injured, or cannot make decisions for themselves? (Yes, routinely) | |
| In the past 12 months, has a health-care professional given you a written list of all your prescribed medications? (Yes) | Can your practice generate a list of all medications taken by an individual patient (including those that may be prescribed by other doctors? (Yes) | |
| During the past year…has any health-care professional you see for your condition given you a written plan to help you manage your own care? (Yes) | Are your patients with chronic conditions given written instructions about how to manage their own care at home…? (Yes, routinely) | |
| After you left the hospital, did the doctors or staff at the place where you usually get medical care seem informed and up-to-date about the care you received in the hospital? (Yes) | When your patients go to the ED, how often do you receive? Notification your patient has been seen in the ED. (Always) | |
| When your patients go to the hospital, how often do you receive? Notification your patient has been seen. (Always) | ||
| Efficiency | Now thinking about the past 2 years, when receiving care for a medical problem, was there ever a time when test results or medical records were not available at the time of your scheduled medical care appointment? (Yes) | During the past month, did the following occur with any of your patients? A patient’s medical record or other relevant clinical information was not available at the time of the patient’s scheduled visit. (Yes) |
| Was there ever a time in the past 2 years when doctors ordered a medical test that you felt was unnecessary because it had already been done? (Yes) | During the past month, did the following occur with any of your patients? Tests or procedures had to be repeated because results were unavailable. (Yes) | |
| Sustainability | Which of the following statements comes closest to expressing your overall view of the health-care system in this country? (On the whole, the system works pretty well, and only minor changes are necessary to make it work better) | Which of the following statements comes closest to expressing your overall view of the health-care system in this country? (On the whole, the system works pretty well, and only minor changes are necessary to make it work better) |
| Which of the following statements comes closest to expressing your overall view of the health-care system in this country? (Our health-care system has so much wrong with it that we need to completely rebuild it) | Which of the following statements comes closest to expressing your overall view of the health-care system in this country? (Our health-care system has so much wrong with it that we need to completely rebuild it) |
Abbreviations: ED, emergency department; GP, general physician.
Appendix C. Percentage of Adults Aged 65+ (2014) and Providers (2015)Who Provided the Selected Response Option, by Country
| Patient Experience | Australia | Canada | France | Germany | Netherlands | New Zealand | Norway | Sweden | Switzerland | United Kingdom | United States | GP Experience | Australia | Canada | France | Germany | Netherlands | New Zealand | Norway | Sweden | Switzerland | United Kingdom | United States |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Medical record not available at the time of visit | 6 | 13 | 3 | 9 | 8 | 7 | 7 | 9 | 7 | 8 | 16 | A patient’s medical record not available at the time of visit | 74 | 61 | 58 | 54 | 59 | 77 | 76 | 62 | 59 | 69 | 62 |
| Experienced care coordination problem | 20 | 25 | 6 | 30 | 14 | 16 | 20 | 20 | 23 | 18 | 32 | Patients had care coordination problems | 64 | 51 | 29 | 57 | 47 | 64 | 40 | 54 | 33 | 79 | 46 |
| GP always spent enough time | 67 | 60 | 47 | 72 | 83 | 74 | 57 | 50 | 80 | 60 | 65 | Satisfied with time you have to spend per patient | 75 | 67 | 65 | 55 | 45 | 59 | 67 | 41 | 68 | 26 | 55 |
| A test repeated because results unavailable | 8 | 7 | 2 | 11 | 4 | 4 | 5 | 4 | 12 | 8 | 13 | A patient’s test was repeated because results unavailable | 39 | 28 | 26 | 26 | 22 | 28 | 34 | 35 | 18 | 48 | 28 |
| Health system needs a complete rebuild | 6 | 10 | 3 | 12 | 20 | 6 | 7 | 11 | 5 | 5 | 32 | Health system needs a complete rebuild | 2 | 3 | 13 | 12 | 2 | 1 | 1 | 11 | 2 | 6 | 14 |
| Medical staff seemed informed about care in hospital | 87 | 85 | 86 | 93 | 93 | 87 | 88 | 68 | 89 | 82 | 88 | Received notification about patient’s care in ED | 18 | 33 | 22 | 20 | 68 | 57 | 32 | 7 | 31 | 49 | 32 |
| Medical staff seemed informed about care in hospital | 87 | 85 | 86 | 93 | 93 | 87 | 88 | 68 | 89 | 82 | 88 | Received notification about patient’s care in hospital | 18 | 29 | 33 | 28 | 70 | 49 | 38 | 8 | 29 | 37 | 31 |
| Received a list of medications | 63 | 67 | 28 | 47 | 66 | 66 | 44 | 77 | 38 | 75 | 65 | Practice can generate list of patient’s medications | 78 | 56 | 35 | 69 | 93 | 86 | 82 | 70 | 48 | 96 | 74 |
| Received written plan for management of chronic condition | 40 | 37 | 42 | 25 | 22 | 45 | 23 | 24 | 22 | 59 | 50 | Patients with conditions given written plan to manage care | 40 | 18 | 20 | 33 | 41 | 28 | 14 | 10 | 25 | 52 | 46 |
| Discussed treatment wishes at end of life | 53 | 61 | 14 | 65 | 43 | 40 | 21 | 31 | 53 | 35 | 73 | Had conversations about treatment wishes with older/sicker patients | 40 | 44 | 36 | 50 | 59 | 34 | 22 | 24 | 48 | 67 | 48 |
| Health system is working well, only minor changes | 51 | 35 | 41 | 46 | 44 | 49 | 55 | 44 | 62 | 56 | 22 | Health system is working well, only minor changes | 48 | 36 | 29 | 27 | 51 | 57 | 68 | 19 | 54 | 22 | 16 |
| Very easy to get after-hours primary care | 20 | 13 | 13 | 28 | 39 | 39 | 23 | 6 | 28 | 35 | 21 | Practice has after-hours arrangements for patients | 78 | 48 | 73 | 87 | 95 | 92 | 81 | 75 | 69 | 89 | 40 |
| Skipped care due to cost | 11 | 12 | 4 | 7 | 8 | 12 | 5 | 4 | 9 | 5 | 25 | Patient had difficulty paying for medical expenses | 25 | 31 | 17 | 13 | 53 | 30 | 3 | 6 | 9 | 12 | 61 |
| Waited 2 months or longer for specialist appointment | 16 | 28 | 13 | 17 | 11 | 21 | 25 | 20 | 3 | 14 | 3 | Patients experience long waits for specialist appointments | 57 | 70 | 65 | 62 | 11 | 66 | 48 | 56 | 9 | 41 | 34 |
| Availability of same or next day appointments | 67 | 44 | 83 | 81 | 76 | 81 | 53 | 54 | 68 | 63 | 54 | Almost all patients able to get same or next day appointments | 46 | 28 | 61 | 65 | 52 | 56 | 40 | 20 | 54 | 38 | 53 |
| Health-care professional makes contact to check on chronic condition | 22 | 15 | 22 | 12 | 21 | 30 | 13 | 17 | 10 | 47 | 330 | Have staff who contact patient to monitor chronic condition | 27 | 28 | 31 | 38 | 39 | 49 | 16 | 35 | 19 | 54 | 39 |
Abbreviations: ED, emergency department; GP, general physician.
Authors’ Note: This study is a secondary analysis of anonymous surveys and therefore did go through an ethical review process. The study was conducted as part of the work of the Bureau of Health Information and the Agency for Clinical Innovation and respected the ethical conduct policies of these public organizations.
Declaration of Conflicting Interests: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) received no financial support for the research, authorship, and/or publication of this article.
ORCID iD: Jean-Frederic Levesque, MD, PhD 
https://orcid.org/0000-0002-5418-8593
References
- 1. Leatherman S, Sutherland K. Designing national quality reforms: a framework for action. Int J Qual Health Care. 2007;19(6):334–40. Epub October 18, 2007. [DOI] [PubMed] [Google Scholar]
- 2. Papanicolas I, Cylus J, Smith PC. An analysis of survey data from eleven countries finds that ‘satisfaction’ with health system performance means many things. Health Aff (Millwood). 2013;32(4):734–42. doi:10.1377/hlthaff.2012.1338. [DOI] [PubMed] [Google Scholar]
- 3. Davis K, Schoen C, Schoenbaum SC, Doty MM, Holmgren AL, Shea K, et al. Mirror on the Wall: An International Update on the Comparative Performance of American Health Care. New York, NY: The Commonwealth Fund; 2007. [Google Scholar]
- 4. Davis K, Schoen C, Stremikis K. Mirror, Mirror 2017: International Comparison Reflects Flaws and Opportunities for Better U.S. Health Care. New York, NY: The Commonwealth Fund; 2017. [Google Scholar]
- 5. Davis K, Stremikis K, Squires D, Schoen C. Mirror Mirror on the Wall: How the Performance of the US Health Care System Compares Internationally. New York, NY: The Commonwealth Fund; 2014. [Google Scholar]
- 6. Browne K, Roseman D, Shaller D, Edgman-Levitan S. Analysis & commentary measuring patient experience as a strategy for improving primary care. Health Aff (Millwood). 2010;29(5):921–5. doi:10.1377/hlthaff.2010.0238. [DOI] [PubMed] [Google Scholar]
- 7. Bodenheimer T, Sinsky C. From triple to quadruple aim: care of the patient requires care of the provider. Ann Fam Med. 2014;12(6):573–6. doi:10.1370/afm.1713. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Braithwaite J, Hibbert P, Blakely B, Plumb J, Hannaford N, Long JC, et al. Health system frameworks and performance indicators in eight countries: a comparative international analysis. SAGE Open Med. 2017;5:2050312116686516 doi:10.1177/2050312116686516. eCollection 2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Carinci F, Van Gool K, Mainz J, Veillard J, Pichora EC, Januel JM, et al. Towards actionable international comparisons of health system performance: expert revision of the OECD framework and quality indicators. Int J Qual Health Care. 2015;27(2):137–46. doi:10.1093/intqhc/mzv004. Epub March 10, 2015. [DOI] [PubMed] [Google Scholar]
- 10. Cooley WC, McAllister JW. Building medical homes: improvement strategies in primary care for children with special health care needs. Pediatrics. 2004;113(suppl 5):1499–506. [PubMed] [Google Scholar]
- 11. Osborn R, Moulds D, Schneider EC, Doty MM, Squires D, Sarnak DO. Primary care physicians in ten countries report challenges caring for patients with complex health needs. Health Aff (Millwood). 2015;34(12):2104–12. doi:10.1377/hlthaff.2015.1018. [DOI] [PubMed] [Google Scholar]
- 12. Osborn R, Moulds D, Squires D, Doty MM, Anderson C. International survey of older adults finds shortcomings in access, coordination, and patient-centered care. Health Aff (Millwood). 2014;33(12):2247–55. doi:10.1377/hlthaff.2014.0947. Epub November 19, 2014. [DOI] [PubMed] [Google Scholar]
- 13. Campbell JL, Roberts M, Wright C, Hill J, Greco M, Taylor M, et al. Factors associated with variability in the assessment of UK doctors’ professionalism: analysis of survey results. BMJ 2011;343:d6212 doi:10.1136/bmj.d6212. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Confederation NHS. Feeling Better? Improving Patient Experience in Hospital. London, England: NHS Confederation; 2010. [Google Scholar]
- 15. DeCourcy A, West E, Barron D. The national adult inpatient survey conducted in the english national health service from 2002 to 2009: how have the data been used and what do we know as a result? BMC Health Serv Res. 2012;12:71 doi:10.1186/1472-6963-12-71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Berwick DM, Nolan TW, Whittington J. The triple aim: care, health, and cost. Health Aff (Millwood). 2008;27(3):759–69. doi:10.1377/hlthaff.27.3.759. [DOI] [PubMed] [Google Scholar]
- 17. OECD. Health at a Glance 2015. Paris, France: OECD Indicators; 2015. [Google Scholar]
- 18. Bureau of Health Information. Healthcare in Focus 2015. How Does NSW Compare. 2016. Retrieved from Sydney Available from: http://www.bhi.nsw.gov.au/BHI_reports/healthcare_in_focus. Retrieved on February 6, 2019.
- 19. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296(9):1094–102. doi:10.1001/jama.296.9.1094. [DOI] [PubMed] [Google Scholar]
- 20. Street RL, O’Malley KJ, Cooper LA, Haidet P. Understanding concordance in patient-physician relationships: personal and ethnic dimensions of shared identity. Ann Fam Med. 2008;6(3):198–205. doi:10.1370/afm.821. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21. Coran JJ, Koropeckyj-Cox T, Arnold CL. Are physicians and patients in agreement? Exploring dyadic concordance. Health Educ Behav. 2013;40(5):603–11. doi:10.1177/1090198112473102. Epub January 23, 2013. [DOI] [PubMed] [Google Scholar]
- 22. Beach MC, Roter DL, Wang NY, Duggan PS, Cooper LA. Are physicians’ attitudes of respect accurately perceived by patients and associated with more positive communication behaviors? Patient Educ Couns. 2006;62(3):347–54. Epub July 21, 2006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Chakrabarti S. What’s in a name? Compliance, adherence and concordance in chronic psychiatric disorders. World J Psychiatry. 2014;4(2):30–36. doi:10.5498/wjp.v4.i2.30. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24. DeVoe J, Fryer GE, Jr, Straub A, McCann J, Fairbrother G. Congruent satisfaction: is there geographic correlation between patient and physician satisfaction? Med Care. 2007;45(1):88–94. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Aiken LH, Sloane DM, Ball J, Bruyneel L, Rafferty AM, Griffiths P. Patient satisfaction with hospital care and nurses in England: an observational study. BMJ Open. 2018;8(1):e019189 doi:10.1136/bmjopen-2017-019189. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Luck J, Peabody JW, DeMaria LM, Alvarado CS, Menon R. Patient and provider perspectives on quality and health system effectiveness in a transition economy: evidence from Ukraine. Soc Sci Med. 2014;114:57–65. doi:10.1016/j.socscimed.2014.05.034. Epub May 21, 2014. [DOI] [PubMed] [Google Scholar]