Skip to main content
PMC home page
Bulletin of the World Health Organization logoLink to Bulletin of the World Health Organization
. 2024 Jun 4;102(7):509–520. doi: 10.2471/BLT.24.290519

Patient satisfaction and value based purchasing in hospitals, Odisha, India

Satisfaction des patients et achats basés sur la valeur dans les hôpitaux d’Odisha, Inde

Satisfacción de los pacientes y compras basadas en el valor en hospitales de Odisha (India)

رضا المرضى والشراء على أساس القيمة في المستشفيات، أوديشا، الهند

印度奥里萨邦医院的患者满意度情况和基于价值的购买项目

Удовлетворенность пациентов и закупки на основе ценностей в больницах, Одиша, Индия

Liana Woskie a,✉, Anuska Kalita b, Bijetri Bose b, Arpita Chakraborty c, Kirti Gupta c, Winnie Yip b
PMCID: PMC11197647  PMID: 38933484

Abstract

Objective

To examine how a general inpatient satisfaction survey functions as a hospital performance measure.

Methods

We conducted a mixed-methods pilot study of the Hospital Consumer Assessment of Health Providers and Systems survey in Odisha, India. We divided the study into three steps: cognitive testing of the survey, item testing with exploratory factor analysis and content validity indexing. Cognitive testing involved 50 participants discussing their interpretation of survey items. The survey was then administered to 507 inpatients across five public hospitals in Odisha, followed by exploratory factor analysis. Finally, we interviewed 15 individuals to evaluate the content validity of the survey items.

Findings

Cognitive testing revealed that six out of 18 survey questions were not consistently understood within the Odisha inpatient setting, highlighting issues around responsibilities for care. Exploratory factor analysis identified a six-factor structure explaining 66.7% of the variance. Regression models showed that interpersonal care from doctors and nurses had the strongest association with overall satisfaction. An assessment of differential item functioning revealed that patients with a socially marginalized caste reported higher disrespectful care, though this did not translate into differences in reported satisfaction. Content validity indexing suggested that discordance between experiences of disrespectful care and satisfaction ratings might be due to low patient expectations.

Conclusion

Using satisfaction ratings without nuanced approaches in value-based purchasing programmes may mask poor-quality interpersonal services, particularly for historically marginalized patients. Surveys should be designed to accurately capture true levels of dissatisfaction, ensuring that patient concerns are not hidden.

Introduction

In 2018, the Indian government launched the world’s largest health insurance scheme, Pradhan Mantri Jan Arogya Yojana.1 The scheme aims to cover secondary and tertiary care for 500 million newly insured citizens, corresponding to 40% of the country’s most vulnerable population.2–4 The government has focused on the quality of care covered through the scheme, including patient satisfaction as a key quality metric in several accountability programmes.5,6 A proposed nationwide programme would formally tie hospital performance to payment with up to 15% of reimbursement depending on the quality of services delivered.7 Satisfaction is the programme’s primary proposed measure of patient-centred care, similar to many value-based purchasing programmes in high-income countries that incentivize high-quality care by linking hospital payments to performance.8 Hence, poor performance on patient satisfaction measures may represent a substantial financial risk for hospitals.

The Ministry of Health and Family Welfare of India has long prioritized measuring patients’ satisfaction with secondary and tertiary care. For example, Mera Aspataal (My Hospital) is a health ministry digital platform used to capture patient feedback on services received from both public and private health facilities.9 To develop this platform, the health ministry used a review of validated patient surveys.6 Mera Aspataal data have informed three policy efforts: a public reporting programme, the national hospital accreditation programme, and a results-based incentives effort focused on hospital cleanliness and physical infrastructure.6 Alternate sources of information, such as insurance claims data, on the quality of health services delivered in inpatient settings across India are scarce.10,11 However, the use of patient satisfaction measures within payment programmes has been controversial8and there are debates on how best to interpret and value satisfaction ratings.12,13 Implicit in any survey-based measure is the assumption that tools are consistently understood by the patient and that variation represents the underlying construct being assessed, as opposed to differences in how people understand or interpret a concept or tool.14 Critics argue that due to information asymmetry, some patients may rate the superficial aspects of the visit (for example, an imposing lobby) rather than the technical or interpersonal quality of care provided by health workers.15 This issue may be particularly relevant as low- and middle-income countries improve access to hospital-based care, and newly insured patients may use secondary and tertiary services for the first time.2,16 While the health ministry already prioritizes patient satisfaction, we lack an in-depth understanding of how patients understand and value aspects of the care interaction, and how those understandings inform satisfaction reporting in the context of a value-based purchasing programme.7

To better understand how satisfaction ratings function within an Indian inpatient setting, we conducted a pilot study using a comprehensive survey tool that assesses both patients’ experiences with a given clinical interaction and their overall satisfaction rating. Considering the proposed value-based purchasing programme, we posed the following research questions: what aspects of patient experience do patients value when rating their satisfaction with care? Does the tool function similarly across different patient types? What factors might drive differences in reporting and to what extent might they be systematic?

Methods

We conducted a mixed-methods assessment of a comprehensive patient experience survey tool, focusing on how patients report overall satisfaction with general inpatient care.7 We employed methods similar to those used in the development of the tool (Table 1).17 We divided the study into three steps: cognitive testing of the survey; item testing and exploratory factor analysis; and content validity indexing. We built on prior work on patient satisfaction in Indian clinical settings.5,18 We used the Hospital Consumer Assessment of Health Providers and Systems survey, due to its use in the nationwide value-based purchasing programme in the United States of America19 and its relevance to India’s proposed programme.7 The survey includes a direct overall measure of patient satisfaction and has been tested in nine countries worldwide.20–24 In India, the tool and its derivatives have been used to assess hospital quality and inform digital health platforms.6 The survey includes questions assessing aspects of the patients’ experience across six domains: interpersonal care from nurses; interpersonal care from doctors; the hospital environment; general experience; after-discharge care; and understanding of care.25 These patient experience questions employ a four-point Likert scale, and additional questions collect demographic information, such as age and gender.

Table 1. Methods used to pre-test and pilot the Hospital Consumer Assessment of Health Providers and Systems survey, Odisha, India, 2020.

Stepa Purpose Process No. of participantsb
1. Cognitive testing To refine translation of survey tool.
To ensure variation in responses do not reflect differences in understanding of a given question, we aimed to identify how individuals interpret each survey item and how their cognitive processing relates to the construct intended by the researcher and original survey instrument		 Focus groups discuss all survey items to assess if framing is logical and answerable, if response options are adequate, etc. We paired each item with structured verbal probes to elicit participants’ cognitive processes and assess their understanding and interpretation of each survey item 50
2. Item testing and exploratory factor analysis Quantitively assess how survey items relate and if exposure to quality of care informs our overall variable of interest: patient satisfaction Hospital-based exit interviews with eligible patients; responses anonymized and analysed using an exploratory factor analysis and series of ordinary least squares models with overall satisfaction posed as a dependent variable, controlling for patient complexity and interview characteristics, for example privacy and enumerator ID 507
3. Content validity indexing Assess to which extent the tool items represent facets of the construct patient experience, that is, do the survey items represent what is important to patient-centredness in Odisha, India One hour-long individual interviews, conducted in non-clinical settings with five patients, five health workers and five health-system experts.
For each survey item, each interviewee rates the relevance to patients’ satisfaction and relevance given hospital environment, using a four-point Likert scale.
Subsequently, interviewees describe the reasons for their ratings		 15
Open in a new tab

a Steps were conducted consecutively.

b Participants partook only in one step, that is each group was distinct.

Step 1

To ensure that observed variation reflects real differences and is not the result of heterogeneity in how the questions are interpreted,14 we used cognitive testing.17,26–29 In this assessment, respondents discussed what each survey item meant to them with the goal of exploring the processes by which respondents answer survey questions. We followed the protocol developed for the Hospital Consumer Assessment of Health Providers and Systems survey.17 Participants included 50 convenience-sampled Odia-speaking individuals, 27 women and 23 men (gender was self-reported). We conducted the cognitive testing in Bhubaneswar, India, with all assessments in Odia, and clarifying discussions in Odia, Hindi and English. During a day-long session, participants reviewed each survey question in full, working in focus groups of 7 to 12 individuals to discuss their understanding of each question. We reimbursed the individuals for their participation. We used scripted probes to elicit additional insights into cognitive processes and conceptual equivalence in processing survey items.30 We used deductive qualitative analysis to categorize identified issue types.

Step 2

We administered the Odia-translated Hospital Consumer Assessment of Health Providers and Systems survey to patients at the time of discharge who had been hospitalized for at least 24 hours. We sampled five public hospitals across Odisha from purposively selected districts. Districts were first grouped according to administrative units, then selected to represent the diversity of the state in terms of tribal population, urbanization, coastal and mining areas, which are believed to influence health, health-care utilization and health-related expenditure. For each hospital, we surveyed approximately 100 patients (20 female obstetrics inpatients, 40 general female and male inpatients each) with an average survey duration of 35 minutes. When the number of patients being discharged exceeded the number of patients the enumerators were able to survey, we used a stratified random sampling strategy with a list frame approach to reduce bias. We set the target sample to 500 respondents, which exceeds recommendations for quantitative validation involving patients (250–350 patients)31 and meets the threshold of very good for factor analysis.32

With the resulting survey data, we conducted an exploratory factor analysis using principal-component factors (assuming no unique factors), and calculated the average of all correlations between each item and the total score (Cronbach's α). Additionally, we ran three models examining the relationship between individual survey items and overall patient satisfaction. Model I is an unadjusted bivariate ordinary least squares regression where overall satisfaction is the dependent variable, and each patient experience survey item is treated as a separate independent variable. Model II adds the patient’s age and gender, as well as variables relevant to clinical complexity: if the patient was admitted through the emergency department; the patient’s self-reported rating of health; length of stay; and facility type. Model III adds variables relevant to the interview: interviewer ID and an enumerator rating of interview privacy. Finally, we assessed differential item functioning by disaggregating results by caste, assessing differences in means with a two-sample t-test, and producing a Spearman’s rank correlation coefficient for each subgroup to assess the strength of the relationship between exposure to disrespectful care and odds of reporting dissatisfaction. Dissatisfaction is shown as an unweighted proportion, with the four most negative response options (of 10) combined to generate one negative rating.

Step 3

To assess the degree to which questionnaire items constitute an adequate operational definition of our construct of interest,33 that is, patients’ overall satisfaction, we used item-level content validity indexing.21 We interviewed 15 individuals, purposively sampled across three categories – patients, health workers and experts. Patients were people familiar with public hospital care in Odisha and included hospital patients on the day of discharge; health workers were currently providing clinical care in Odisha; and experts were researchers experienced in collecting patient data from inpatient settings in Odisha. Each interview was in-person and lasted approximately one hour. The interviews involved providing verbal instructions on how to use the Likert scale (1: not relevant; 2: somewhat relevant; 3: relevant; and 4: highly relevant) to evaluate the relevance of survey items, followed by questions to explain why they did, or did not, think the item was relevant. Two separate scores were captured: (i) the item’s relevance to patient satisfaction; and (ii) the item’s relevance given the clinical setting. By allowing interviewees to provide two distinct scores, we were able to address concerns regarding care expectations identified during cognitive testing. This approach helped us better distinguish whether low ratings were due to concerns with the item’s relevance to patient satisfaction, or other factors, such as feasibility and structural constraints in the study setting.

Disaggregating expectations

Finally, to outline policy-relevant implications of this work, we used Thompson and Sunol’s framework to organize sources of variation into four categories: ideal expectations, predicted expectations, normative expectations and patient expression.34

Ethical considerations

Institutional Review Board approval was provided through Harvard TH Chan School of Public Health, Boston, United States of America (IRB18–1675); Research and Ethics Committee of the Directorate of Health Services, Government of Odisha ID: 60/PMU/187/17; and Sigma, registered with the Division of Assurance and Quality Improvement of the Office for Human Research Protections, USA (IRB00009900). All participants gave informed consent to participate in the study before taking part.

Results

Participants in the cognitive testing surfaced several fundamental concerns. They flagged six out of 18 questions as having relevance issues to the Odisha inpatient setting. These issues centred around responsibility for care. For example, families, not health workers, may be responsible for cleanliness. Furthermore, participants thought that doctors were responsible for communicating clinical information, but did not think they were responsible for explaining the information. These concerns informed conversations about which tasks were the responsibilities of health-care professionals (Table 2).

Table 2. Cognitive testing issues identified in items in the Hospital Consumer Assessment of Health Providers and Systems survey, Odisha, India, 2020.

Survey domain and item Full item text Cognitive testing issue
Brief description Typea
Interpersonal care from nurses
Courtesy and respect During this hospital stay, how often did nurses treat you with courtesy and respect? No issues raised NA
Listen carefully During this hospital stay, how often did nurses listen carefully to you? Listening carefully may not be seen as distinct from being treated with respect Construct
Explain During this hospital stay, how often did nurses explain things in a way you could understand? Patient must define “how often,” as the concept often lacks a point of reference Construct
Interpersonal care from doctors
Courtesy and respect During this hospital stay, how often did doctors treat you with courtesy and respect? No issues raised NA
Listen carefully During this hospital stay, how often did doctors listen carefully to you? Doctors are often not responsible for listening to patients Relevance
Explain During this hospital stay, how often did doctors explain things in a way you could understand? Doctors are often not responsible for explaining care to patients Relevance
Hospital environment
Room clean During this hospital stay, how often were your room or ward and bathroom kept clean? Families, not providers, are often responsible for cleanliness Relevance
Quiet During this hospital stay, how often was the area around your room/ward quiet at night? Lack of clarity on the concept quiet. In open hospital wards, it may not be possible to maintain quiet Construct and relevance
General experience
Bathroom help How often did you get help in getting to the bathroom or in using a bedpan as soon as you wanted? Families, not providers, are often responsible for bedpans Relevance
Talk about pain During this hospital stay, how often did hospital staff talk with you about how much pain you had? Patient must define “how often,” as the concept often lacks a point of reference Construct
Talk about pain treatment During this hospital stay, how often did hospital staff talk with you about how to treat your pain? Patient must define “how often,” as the concept often lacks a point of reference Construct
Explain medication purpose Before giving you any new medicine, how often did hospital staff tell you what the medicine was for? Lack of clarity on what constitutes new medicine. External purchase of medication most common and doctors rarely provides the medicine Information and relevance
Explain side-effects of medication Before giving you any new medicine, how often did hospital staff describe possible side-effects in a way you could understand? Lack of clarity on what constitutes new medicine. External purchase of medication most common and doctors rarely provides the medicine Information and relevance
After discharge
Assessment of post-discharge During this hospital stay, did doctors, nurses or other hospital staff talk with you about whether you would have the help you needed when you left the hospital? Understood as: when you go home will you get the help that you need Construct
Receipt of discharge guidance During this hospital stay, did you get information in writing about what symptoms or health problems to look out for after you left the hospital? Written guidance may be irrelevant if patients are illiterate Relevance
Understanding of care
Taking preferences seriously During this hospital stay, staff took my preferences and those of my family or caregiver into account in deciding what my health care needs would be when I left. The doctors may not concern themselves with care after discharge, as it is not within the scope of the doctor’s professional role Relevance
Understand responsibilities When I left the hospital, I had a good understanding of the things I was responsible for in managing my health. Lack of clarity on what the patient is told versus what the patient understands Construct
Understand purpose of medications When I left the hospital, I clearly understood the purpose for taking each of my medications? No issues raised NA
Open in a new tab

NA: not applicable

a Construct issues were raised when the item was understood differently than its intended construct. Information issues were raised when there was unclear or inadequate information for a patient to answer the question reliably. Relevance issues were when there was something about the question that raised concern, e.g. relevance in the Odisha inpatient setting.

In step 2, enumerators surveyed 507 patients. Educational backgrounds varied, with most male inpatients having completed a primary or middle school education (77/193), while most female inpatients had no formal schooling (62/209). The majority identified as Hindu (494/507) and most spoke Odia (442/507) as their primary language (Table 3).

Table 3. Characteristics of public hospital-based exit interviewees, Odisha, India, 2020.

Characteristic No. of respondents (%)a
Male inpatients
(n = 193)		 Female inpatients
(n = 209)		 Inpatients of obstetrics–gynaecology departments
(n = 105)
Age in years, mean (SD) 47.2 (17.6) 45.2 (17.4) 25.5 (5.3)
Highest educational attainment
Illiterate 13 (6.7) 32 (15.3) 0 (0.0)
No formal schooling 32 (16.6) 62 (29.7) 11 (10.5)
Under primary 11 (5.7) 22 (10.5) 13 (12.4)
Primary 39 (20.2) 21 (10.1) 15 (14.3)
Upper primary and middle 38 (19.7) 24 (11.5) 18 (17.1)
Secondary 29 (15.0) 25 (12.0) 23 (21.9)
Higher secondary 19 (9.8) 13 (6.2) 21 (20.0)
Graduate 7 (3.6) 7 (3.4) 4 (3.8)
Caste
Scheduled tribe 34 (17.6) 40 (19.1) 28 (26.7)
Scheduled caste 23 (11.9) 36 (17.2) 25 (23.8)
Otherwise backward class 74 (38.3) 64 (30.6) 22 (20.9)
Generalb 61 (31.6) 67 (32.1) 29 (27.6)
Religion
Hindu 189 (97.9) 205 (98.1) 100 (95.2)
Muslim 4 (2.1) 4 (1.9) 1 (1.0)
Christian 0 (0.0) 0 (0.0) 4 (3.8)
Primary languagec
Odia 171 (88.6) 193 (92.3) 78 (74.3)
Hindi 4 (2.1) 4 (1.9) 1 (1.0)
Telugu 0 (0.0) 2 (1.0) 3 (2.9)
Tribal dialect 16 (8.3) 9 (4.3) 21 (20.0)
Open in a new tab

SD: standard deviation.

a Values are no. (%) if not otherwise given.

b No historically marginalized caste designation.

c Languages spoken by less than 1% of respondents not included, hence the sum does not equal 100%.

Note: we limited the sampling to public hospitals which are slated to be incorporated within the proposed value-based purchasing programme.

The exploratory factor analysis yielded six eigenvalues greater than 1, indicating a six-factor structure. These results explained 66.7% of the variance within the model. All Cronbach’s α values exceeded the threshold of 0.7. Uniqueness at the item-level, variance not shared with other variables, ranged from 17.1% (understand responsibilities) to 55.6% (doctors listen carefully). Regression models revealed that the hospital environment category had the weakest association with overall satisfaction (Model III coefficient: 0.23), whereas interpersonal care from doctors and nurses had the strongest association (Model III coefficients: 0.76 and 0.70, respectively; Table 4).

Table 4. Results of exploratory factor analysis and of overall satisfaction models, Odisha, India, 2020.

Category and experience item Mean item value
(SE)		 Exploratory factor analysis and item-level testing
 Coefficient, by level
  Model Ib
 Model IIc
 Model IIId
Item uniqueness Cronbach’s αa   Item Category Item Category Item Category
Interpersonal care from nurses (λ: 3.5)e
Courtesy and respect 3.4 (0.034) 0.221 0.785   0.65*** 0.76*** 0.58*** 0.69*** 0.59*** 0.70***
Listen carefully 3.4 (0.032) 0.218 0.781   0.79*** 0.74*** 0.75***
Explain 3.3 (0.036) 0.371 0.780   0.81*** 0.75*** 0.77***
Interpersonal care from doctors (λ: 1.9)e
Courtesy and respect 3.5 (0.031) 0.359 0.785   0.91*** 0.82*** 0.84*** 0.74*** 0.86*** 0.76***
Listen carefully 3.3 (0.033) 0.556 0.779   0.82*** 0.73*** 0.75***
Explain 3.3 (0.033) 0.319 0.785   0.72*** 0.65*** 0.66***
Hospital environment (λ: 1.7)e
Room clean 2.9 (0.040) 0.293 0.798   0.47*** 0.33*** 0.39*** 0.25** 0.38*** 0.23**
Quiet 2.5 (0.044) 0.287 0.807   0.18*** 0.10 0.08
General experience (λ: 1.3)e
Talk about pain 2.6 (0.056) 0.445 0.790   0.90*** 0.68*** 0.81*** 0.60*** 0.87*** 0.62***
Talk about pain treatment 2.9 (0.036) 0.310 0.786   0.64*** 0.56*** 0.57***
Explain medication purpose 2.8 (0.055) 0.330 0.802   0.50*** 0.42*** 0.42***
After discharge (λ: 1.3)e
Assessment of post-discharge 0.9 (0.022) 0.345 0.811   0.26* 0.54** 0.09 0.34** 0.09 0.33**
Receipt discharge guidance 1.6 (0.017) 0.542 0.801   0.81*** 0.59*** 0.57***
Understanding of care (λ: 1.1)e
Taking preferences seriously 3.6 (0.024) 0.300 0.804   0.69*** 0.69*** 0.55*** 0.58*** 0.54*** 0.57***
Understand responsibilities 3.6 (0.023) 0.171 0.801   0.72*** 0.59*** 0.58***
Understand purpose of medications 3.6 (0.026) 0.277 0.804   0.67*** 0.59*** 0.58***
Open in a new tab

SE: standard error; *P ≤ 0.05; **P ≤ 0.01; ***P ≤ 0.001.

a A typical exclusion threshold for α coefficient is 0.70. The higher the α coefficient, the more the items have shared covariance and may measure the same underlying concept. Highly correlated items will also produce a high coefficient and can therefore be interpreted as a sign of redundancy. As we did not conduct the analysis to shorten the Hospital Consumer Assessment of Health Providers and Systems survey, we retain all items regardless of performance.

b Model I represents the unadjusted results of a bivariate ordinary least square regression where overall satisfaction is the dependent variable and each row represents a different patient experience item posed to patient.

c Adjusted for patient age, gender and clinical complexity.

d Adjusted for Model II factors plus interview characteristics.

e Eigenvalues (λ) shown for retained factors. Corresponding item categories are discrete and align with factor loadings most relevant to defining each factor’s dimensionality.

Note: we excluded two items (bathroom help and explanation of medicine side-effects) from this table because fewer than 50 respondents needed support with the bathroom or were prescribed medicines.

Disaggregating results by patient characteristics, we identified differential functioning of survey items based on caste. Patients who identified as part of a scheduled caste, otherwise backward class or scheduled tribe were significantly more likely to report receiving disrespectful care compared to patients with no marginalized class designation (P-value: > 0.05; Fig. 1; Table 5). In contrast, there was no statistical difference in reporting dissatisfaction between the groups. Only patients who identified as part of an otherwise backward class had a significant correlation between exposure to disrespectful care and reporting dissatisfaction (ρ: 0.19; P-value: 0.02). Moreover, all values fall well below the 15% satisfaction threshold set within the proposed value-based purchasing programme, meaning the difference in exposure to disrespectful care by caste would not translate to a difference in hospital payment.

Fig. 1.

Share of patients reporting receipt of disrespectful treatment and share reporting overall dissatisfaction with care, by caste, Odisha, India, 2020

Notes: the proposed value-based purchasing programme in India sets an initial threshold of 85% satisfaction (15% dissatisfaction). We combined the four most negative response options (of 10) to generate a combined negative rating. We used this interpretation of dissatisfaction because the satisfaction ratings in India’s proposed value-based purchasing programme will be evaluated using a 5-point Likert scale of which the two least favourable responses will be combined to a negative rating. Difference is assessed with a two-sided t-test comparing to the base group, individuals with no historically marginalized designation.

Fig. 1

Open in a new tab

Table 5. Share of patients reporting receipt of disrespectful treatment and share reporting overall dissatisfaction with care, by caste, Odisha, India, 2020.

Caste group Reporting disrespectful treatment Reporting dissatisfaction Spearman’s ρa (P)
Generalb (n = 157) 0.34 (< 0.01)
% of respondents (no.) 32.5 (51) 3.2 (5)
Scheduled caste (n = 84) 0.14 (0.19)
% of respondents (no.) 47.6 (40) 3.6 (3)
Difference from general group, % points (P) 15.1 (< 0.01) 0.4
Otherwise backward class (n = 160) 0.19 (0.02)
% of respondents (no.) 44.4 (71) 1.3 (2)
Difference from general group, % points (P) 11.9 (0.01) −1.9
Scheduled tribe (n = 102) 0.17 (0.09)
% of respondents (no.) 61.8 (63) 2.0 (2)
Difference from general group, % points (P) 29.3 (< 0.01) −1.2
Open in a new tab

a Spearman’s ρ assessing the relationship between reporting disrespectful treatment and reporting dissatisfaction.

b The general group refers to individuals with no historically marginalized class designation.

Finally, our content validity indexing results suggest that reporting discordance (that is, experiencing disrespectful care but not reporting dissatisfaction) may be due to low expectations rather than a difference in what patients value. When participants were asked about item relevance, hospital environment relevance scored lower (Fig. 2) than relevance to patients’ satisfaction in 13 of 18 questions. These results align with cognitive testing results; for example, participants valued doctors listening carefully, but did not expect this to occur in practice because they did not believe it was a physician’s responsibility within the Odisha inpatient setting.

Fig. 2.

Mean content validity indexing scores assessing items’ relevance to patient satisfaction and hospital environment, Odisha, India, 2020

Notes: we interviewed five patients, five health workers and five health-system researchers, who rated the relevance of survey item to either patient satisfaction or the hospital environment, including feasibility or likelihood of an event occurring in the inpatient setting. Rating scale for each individual question was 1: not relevant; 2: somewhat relevant; 3: relevant; 4: highly relevant.

Fig. 2

Open in a new tab

Interviews revealed that understandings of clinical responsibilities and corresponding expectations informed patients’ overall ratings. For example, a patient participant stated:

“I do feel the doctors were disrespectful, but they are the boss and this is how it is, no? So I think disrespect is important to me and my family, but if this is the same treatment I got last time, why complain? This is why my [satisfaction] score is still high.”

These pilot study findings raise concerns regarding the use of an overall satisfaction rating within provider payment programmes and how we interpret traditional quantitative approaches to validation, which may assume low item functioning means low importance to the patient or satisfaction. Potential sources of variation in patient satisfaction ratings and considerations for value-based purchasing policies are presented in Table 6. These sources suggest a need to consider predicted expectations in addition to other sources of variation.

Table 6. Sources of variation in patient satisfaction ratings and considerations for value-based purchasing policies, Odisha, India, 2020.

Source of variation Descriptiona Policy considerations for value-based purchasing
Values Ideal expectations are similar to aspirations, desires or preferred outcomes; what a person ultimately values, that is, in a situation without limitation Values can, and likely do, vary between patients and contexts; expectations represent an anticipated source of variation, allowing satisfaction ratings to reflect a diverse range of patient values
Expectations Predicted expectations are realistic, practical or anticipated outcomes that result from personal experiences, reported experiences of others and sources of knowledge such as the media Addressing variation that results from differences in predicted expectations may include the following:
- Collecting basic demographic information about patients that are potentially associated with historical marginalization, for example, religious identity, caste and educational attainment. These data can be used to better understand hospitals’ baseline population as well as augment clinically-focused risk adjustment, which is often used within value-based purchasing programmes and focuses on case mix, i.e. morbidity type and severity
Normative expectations are based on what should or ought to happen, often based on a mutually agreed upon threshold for what constitutes patient-centred care (similar to human rights standards) Addressing variation that results from differences in normative expectations may include the following:
- Pair subjective satisfaction ratings with more objective assessments of what a patient is experiencing during a given clinical interaction (that align with normative guidance) and look for discordance in patient ratings, that is, when patients give positive ratings to potentially inadequate careb
- Due to low and variable thresholds for reporting dissatisfaction when exposed to low quality care, do not use a satisfaction rating to trigger sub-items, which are sometimes only posed to dissatisfied patients
Expression Expression is how patients convey or report their satisfaction with care to others, which may differ for patients regardless of ideal, predicted, or normative expectations of care and inform reporting bias,c that is, how satisfaction is expressed may differ among patients with a similar level of true satisfaction Addressing variation that results from differences in expression may include the following:
- Consider the addition of variables within surveys used for value-based purchasing that may inform reporting bias. For example, interview privacy and interviewer ID. Consider these factors when analysing data to address underreporting, which may be more prevalent for marginalized patients.
- If resources allow, follow up with a random subset of interviewed patients to assess if there is a variation in responses once they left the hospital
Open in a new tab

a Adapted from Thomson & Sunol, 1995.34

b For example, being yelled at by a provider is generally seen as unacceptable by both national and international standards. It is important to understand if patients consistently give positive feedback to such care, as this helps ensure that these forms of poor-quality care are challenged, particularly among marginalized patients.

c Thomson & Sunol34 include a related concept, which they call “unformed expectations,” which is when individuals are unable to articulate their expectations because they do not have expectations, have difficulty expressing their expectations or do not wish to reveal their expectations due to fear, anxiety or conforming to social norms.

Discussion

In this pilot study, we find aspects of the care interaction beyond the physical environment, such as the quality of interpersonal care, had a strong relationship with overall satisfaction. However, these results raise concerns for the use of satisfaction ratings within a nationwide performance policy. Observed differences in care ratings may not reflect true differences in patients’ satisfaction, which may vary between sociocultural groups. These findings are timely as the Indian government considers using satisfaction ratings to hold hospitals accountable to patients.

Satisfaction ratings, as a single metric, are appealing in that they theoretically capture a wide range of underlying preferences. Conversely, absent of clinical expertise, patients may place undue value on more superficial aspects of the care interaction – aspects more subject to manipulation to improve ratings.35 Contrary to this concern, we found the physical environment had a weak relationship with satisfaction. Patients did appear to value interpersonal aspects of care, for example, being listened to carefully and having care explained adequately. Even when examining questions that did not perform well in the factor analysis or regression models, such as receipt of post-discharge guidance, content validity indexing suggested this guidance was valued, but participants did not anticipate it to occur in practice. Traditionally, in tool validation studies, low item performance in quantitative approaches indicates that the item is not an important driver of patient satisfaction. As a result, the item may be excluded. However, our results indicate that low coefficients may result from low predicted expectations rather than low ideal expectations.

The proposed value-based purchasing programme sets an 85% satisfaction rating threshold, with facilities scoring below facing reduced health insurance scheme reimbursement.7 In our study, despite a high proportion of respondents reporting disrespectful care, reimbursement would not be affected since dissatisfaction ratings fell well below 15%. As such, the currently designed programme may not adequately surface low-quality interpersonal care provided to marginalized patients. This type of variation in reporting, which results from differences in predicted expectations, is problematic particularly if certain patients or groups of patients have been systematically subjected to lower quality of care than others. Different thresholds for reporting satisfaction raise concern for the use of overall ratings within value-based purchasing.36 Many public reporting and payment programmes treat satisfaction as a stand-alone measure, which is both a feasible and simple approach, particularly if variation results from differences in ideal expectations. However, this approach may fail to surface low-quality interpersonal care experienced by individuals unlikely to report overall dissatisfaction – either due to low predicted expectations or issues of expression. Scheduled tribe patients, for example, may have lower expectations of the system due to experiences of disrespect. Furthermore, patients with higher education may have unreasonable predicted expectations of the health system and/or a lower threshold for the expression of dissatification.37 Researchers developing the World Health Surveys coined the term universally legitimate expectations, which refers to a normative set of expectations.37 Accordingly, we provide actionable considerations for improving satisfaction ratings within value-based purchasing programmes (Table 6).

This work extends the existing literature assessing patient experience and satisfaction in Indian clinical settings.5,38,39 We build on this work by focusing on general inpatient care, instead of specific conditions or specialties, and consider policy applications given the proposed value-based purchasing programme. While some studies have used the Hospital Consumer Assessment of Health Providers and Systems tool in India as an outcome measure,40 we were unable to find any documentation of formal adaptation or pre-testing processes that might be useful in informing the tool’s use in payment policies. Our work also extends the patient vignette literature, which aims to understand differences in how individuals judge care for a fixed clinical example.41,42 This literature exposes differences in ratings based on patient characteristics, but cannot disentangle why ratings differ. By using a formative mixed-methods approach, we were able to assess patients’ values and expectations.

This study has several limitations. First, the sample size is small and we lacked a reliable sampling frame. For example, due to the small sample, we were unable to examine how patient characteristics interact with one another. However, the results and concerns raised should inform larger studies. Second, we conducted this pilot study in a rural state with a large tribal population, which may pose challenges to generalizing these findings. However, researchers have estimated that the largest increases in hospital utilization will likely occur in states like Odisha, and we lack research on survey tools that assess health system performance in the state.43 Third, the study was run as a hospital exit interview as opposed to a non-hospital-based setting, which is considered best practice in mitigating reporting bias.44–46 For example, the likelihood of reporting disrespectful or abusive delivery of care in the United Republic of Tanzania increased nearly 10 percentage points in a post-discharge survey compared to an exit interview.47 However, almost half of the women in our study had at most a primary school education, which made the enumerators administer the tool verbally. In addition, only 82.1% (416/507) of patients could provide a phone number and for 70.0% (291/416) of them, the phone belonged to a family member or neighbour. These findings reaffirmed the reliance on exit interviews as the most practical method. The limitation of using an exit interview tool motivated us to adjust for interview characteristics in one of our regression models. Finally, the sample sizes for the cognitive testing and content validity indexing are small and not necessarily representative of the final populations that would be surveyed. In our study, the sample sizes exceeded those published in the pre-testing of the Hospital Consumer Assessment of Health Providers and Systems tool in 2005 (cognitive testing: 41 versus 50 participants; and content validity indexing: 12 versus 15 participants).

In conclusion, increased access to health care does not always guarantee better health outcomes,48 potentially due to low-quality services.49 Therefore, improving the quality of care is crucial, but measuring it can be challenging. Patient-reported measures offer a promising opportunity for assessment. However, without a nuanced approach to identify sources of systematic reporting error, using satisfaction ratings within value-based purchasing programmes may obscure poor-quality interpersonal care for marginalized patient populations.

Acknowledgements

We thank Mahrokh Irani, Ashish K Jha, the team at Oxford Policy Management, New Delhi, India, the Government of Odisha, India, and the study participants. LW is also affiliated with the Brown School of Public Health and AK with the Lancet Citizens' Commission on Reimagining India's Health System, New Delhi, India.

Funding:

The funding for this study was provided in part by the Bill and Melinda Gates Foundation. The data collection was supported by the Tata Trusts. LW reports additional funding from the HF Guggenheim Foundation and Horowitz Foundation for doctoral thesis support, of which this work was a part.

Competing interests:

None declared.

References

  • 1.About Pradhan Mantri Jan Arogya Yojana (PM-JAY) [internet]. New Delhi: National Health Authority of India; 2023. Available from: https://nha.gov.in/PM-JAY [cited 2023 Nov 30].
  • 2.Kastor A, Mohanty SK. Disease and age pattern of hospitalisation and associated costs in India: 1995-2014. BMJ Open. 2018. Jan 24;8(1):e016990. 10.1136/bmjopen-2017-016990 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Chatterjee P. India launches Ayushman Bharat’s secondary care component. Lancet. 2018. Sep 22;392(10152):997. 10.1016/S0140-6736(18)32284-0 [DOI] [PubMed] [Google Scholar]
  • 4.Jana A, Basu R. Examining the changing health care seeking behavior in the era of health sector reforms in India: evidences from the National Sample Surveys 2004 & 2014. Glob Health Res Policy. 2017. Mar 6;2(1):6. 10.1186/s41256-017-0026-y [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Chahal H, Mehta S. Modeling patient satisfaction construct in the Indian health care context. Int J Pharm Healthc Mark. 2013. Mar;7(1):75–92. 10.1108/17506121311315445 [DOI] [Google Scholar]
  • 6.SAATHII. Mera Aspataal: an initiative to capture patient feedback and improve quality of services. New Delhi: Ministry of Health and Family Welfare; 2019. p. 35. [Google Scholar]
  • 7.Volume-based to value-based care: ensuring better health outcomes and quality healthcare under AB PM-JAY. New Delhi: National Health Authority of India; 2022. Available from: https://abdm.gov.in:8081/uploads/VBHC_Policy_Document_For_Upload_a20f871a55.pdf [cited 2023 Nov 30].
  • 8.Tsai TC, Orav EJ, Jha AK. Patient satisfaction and quality of surgical care in US hospitals. Ann Surg. 2015. Jan;261(1):2–8. 10.1097/SLA.0000000000000765 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Mera Aspataal. Share your experience to improve hospitals [internet]. New Delhi: Ministry of Health and Family Welfare; 2024. Available from: https://meraaspataal.nhp.gov.in/ [cited 2024 May 26].
  • 10.Sarwal R, Kalal S, Iyer V. Best practices in the performance of district hospitals. New Delhi: NITI Aayog; 2021. 10.31219/osf.io/y79bg [DOI] [Google Scholar]
  • 11.Mohanan M, Hay K, Mor N. Quality of health care in India: challenges, priorities, and the road ahead. Health Aff (Millwood). 2016. Oct 1;35(10):1753–8. 10.1377/hlthaff.2016.0676 [DOI] [PubMed] [Google Scholar]
  • 12.Richman BD, Schulman KA. Are patient satisfaction instruments harming both patients and physicians? JAMA. 2022. Dec 13;328(22):2209–10. 10.1001/jama.2022.21677 [DOI] [PubMed] [Google Scholar]
  • 13.Kravitz R. Patient satisfaction with health care: critical outcome or trivial pursuit? J Gen Intern Med. 1998. Apr;13(4):280–2. 10.1046/j.1525-1497.1998.00084.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Collins D. Pretesting survey instruments: an overview of cognitive methods. Qual Life Res. 2003. May;12(3):229–38. 10.1023/A:1023254226592 [DOI] [PubMed] [Google Scholar]
  • 15.Schoenfelder T, Klewer J, Kugler J. Determinants of patient satisfaction: a study among 39 hospitals in an in-patient setting in Germany. Int J Qual Health Care. 2011. Oct;23(5):503–9. 10.1093/intqhc/mzr038 [DOI] [PubMed] [Google Scholar]
  • 16.Joe W, Perkins JM, Kumar S, Rajpal S, Subramanian SV. Institutional delivery in India, 2004-14: unravelling the equity-enhancing contributions of the public sector. Health Policy Plan. 2018. Jun 1;33(5):645–53. 10.1093/heapol/czy029 [DOI] [PubMed] [Google Scholar]
  • 17.Levine RE, Fowler FJ Jr, Brown JA. Role of cognitive testing in the development of the CAHPS Hospital Survey. Health Serv Res. 2005. Dec;40(6 Pt 2):2037–56. 10.1111/j.1475-6773.2005.00472.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Rao KD, Peters DH, Bandeen-Roche K. Towards patient-centered health services in India–a scale to measure patient perceptions of quality. Int J Qual Health Care. 2006. Dec;18(6):414–21. 10.1093/intqhc/mzl049 [DOI] [PubMed] [Google Scholar]
  • 19.The hospital value-based purchasing (VBP) program [internet]. Baltimore: Centers for Medicare & Medicaid Services; 2024. Available from: https://www.cms.gov/medicare/quality/value-based-programs/hospital-purchasing [cited 2024 Mar 16].
  • 20.Alanazi MR, Alamry A, Al-Surimi K. Validation and adaptation of the hospital consumer assessment of healthcare providers and systems in Arabic context: evidence from Saudi Arabia. J Infect Public Health. 2017. Nov-Dec;10(6):861–5. 10.1016/j.jiph.2017.02.002 [DOI] [PubMed] [Google Scholar]
  • 21.Squires A, Bruyneel L, Aiken LH, Van den Heede K, Brzostek T, Busse R, et al. Cross-cultural evaluation of the relevance of the HCAHPS survey in five European countries. Int J Qual Health Care. 2012. Oct;24(5):470–5. 10.1093/intqhc/mzs040 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Delnoij DMJ, ten Asbroek G, Arah OA, de Koning JS, Stam P, Poll A, et al. Made in the USA: the import of American consumer assessment of health plan surveys (CAHPS) into the Dutch social insurance system. Eur J Public Health. 2006. Dec;16(6):652–9. 10.1093/eurpub/ckl023 [DOI] [PubMed] [Google Scholar]
  • 23.Islam S, Muhamad N. Patient-centered communication: an extension of the HCAHPS survey. Benchmarking: An International Journal. 2021;28(6):2047–74. 10.1108/BIJ-07-2020-0384 [DOI] [Google Scholar]
  • 24.Aoki T, Yamamoto Y, Nakata T. Translation, adaptation and validation of the hospital consumer assessment of healthcare providers and systems (HCAHPS) for use in Japan: a multicentre cross-sectional study. BMJ Open. 2020. Nov 19;10(11):e040240. 10.1136/bmjopen-2020-040240 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.HCAHPS: patients’ perspectives of care survey. Public reporting [internet]. Baltimore: Centers for Medicare and Medicaid Services; 2021. Available from: https://www.cms.gov/medicare/quality/initiatives/hospital-quality-initiative/hcahps-patients-perspectives-care-survey [cited 2021 Feb 14].
  • 26.Erkut S. Developing multiple language versions of instruments for intercultural research. Child Dev Perspect. 2010. Apr 1;4(1):19–24. 10.1111/j.1750-8606.2009.00111.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Johnson TP. Methods and frameworks for crosscultural measurement. Med Care. 2006. Nov;44(11 Suppl 3):S17–20. 10.1097/01.mlr.0000245424.16482.f1 [DOI] [PubMed] [Google Scholar]
  • 28.Tran T, Nguyen T, Chan K. Developing cross-cultural measurement. Volume 1. Oxford: Oxford University Press; 2018. [Google Scholar]
  • 29.Weeks A, Swerissen H, Belfrage J. Issues, challenges, and solutions in translating study instruments. Eval Rev. 2007. Apr;31(2):153–65. 10.1177/0193841X06294184 [DOI] [PubMed] [Google Scholar]
  • 30.Westbrook KW, Babakus E, Grant CC. Measuring patient-perceived hospital service quality: validity and managerial usefulness of HCAHPS scales. Health Mark Q. 2014;31(2):97–114. 10.1080/07359683.2014.907114 [DOI] [PubMed] [Google Scholar]
  • 31.White M. Sample size in quantitative instrument validation studies: a systematic review of articles published in Scopus, 2021. Heliyon. 2022. Dec 12;8(12):e12223. 10.1016/j.heliyon.2022.e12223 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32.Comrey AL, Lee HB. A first course in factor analysis (2nd ed.). Hillsdale: Lawrence Erlbaum Associates, Inc; 1992. [Google Scholar]
  • 33.Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 2007. Aug;30(4):459–67. 10.1002/nur.20199 [DOI] [PubMed] [Google Scholar]
  • 34.Thompson AG, Suñol R. Expectations as determinants of patient satisfaction: concepts, theory and evidence. Int J Qual Health Care. 1995. Jun;7(2):127–41. 10.1093/intqhc/7.2.127 [DOI] [PubMed] [Google Scholar]
  • 35.Press I, Fullam F. Patient satisfaction in pay for performance programs. Qual Manag Health Care. 2011. Apr-Jun;20(2):110–5. 10.1097/QMH.0b013e318213aed0 [DOI] [PubMed] [Google Scholar]
  • 36.Jha AK, Zaslavsky AM. Quality reporting that addresses disparities in health care. JAMA. 2014. Jul 16;312(3):225–6. 10.1001/jama.2014.7204 [DOI] [PubMed] [Google Scholar]
  • 37.De Silva A. A framework for measuring responsiveness. GPE Discussion Paper Series: No. 32. Berlin: ResearchGate GmbH; 2015. Available from: https://www.researchgate.net/publication/265280238_A_FRAMEWORK_FOR_MEASURING_RESPONSIVENESS [cited 2021 Mar 8].
  • 38.Agarwal A, Garg S, Pareek U. A study assessing patient satisfaction in a tertiary care hospital in India: the changing healthcare scenario. J Commun Dis. 2009. Jun;41(2):109–12. [PubMed] [Google Scholar]
  • 39.Goel S, Sharma D, Singh A. Development and validation of a patient satisfaction questionnaire for outpatients attending health centres in North Indian cities. J Health Serv Res Policy. 2014. Apr;19(2):85–93. 10.1177/1355819613508381 [DOI] [PubMed] [Google Scholar]
  • 40.Khetrapal S, Acharya A, Mills A. Assessment of the public-private-partnerships model of a national health insurance scheme in India. Soc Sci Med. 2019. Dec;243:112634. 10.1016/j.socscimed.2019.112634 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Valentine N, Verdes-Tennant E, Bonsel G. Health systems’ responsiveness and reporting behaviour: multilevel analysis of the influence of individual-level factors in 64 countries. Soc Sci Med. 2015. Aug;138:152–60. 10.1016/j.socscimed.2015.04.022 [DOI] [PubMed] [Google Scholar]
  • 42.Rice N, Robone S, Smith PC. Vignettes and health systems responsiveness in cross-country comparative analyses. J R Stat Soc Ser A Stat Soc. 2012. Apr;175(2):337–69. 10.1111/j.1467-985X.2011.01021.x [DOI] [Google Scholar]
  • 43.Yip W, Kalita A, Bose B, Cooper J, Haakenstad A, Hsiao W, et al. Comprehensive assessment of health system performance in Odisha, India. Health Syst Reform. 2022. Jan 1;8(1):2132366. 10.1080/23288604.2022.2132366 [DOI] [PubMed] [Google Scholar]
  • 44.Elliott MN, Zaslavsky AM, Goldstein E, Lehrman W, Hambarsoomians K, Beckett MK, et al. Effects of survey mode, patient mix, and nonresponse on CAHPS hospital survey scores. Health Serv Res. 2009. Apr;44(2 Pt 1):501–18. 10.1111/j.1475-6773.2008.00914.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 45.Leone T, Sochas L, Coast E. Depends who’s asking: interviewer effects in demographic and health surveys abortion data. Demography. 2021. Feb 1;58(1):31–50. 10.1215/00703370-8937468 [DOI] [PubMed] [Google Scholar]
  • 46.de Vries H, Elliott MN, Hepner KA, Keller SD, Hays RD. Equivalence of mail and telephone responses to the CAHPS Hospital Survey. Health Serv Res. 2005. Dec;40(6 Pt 2):2120–39. 10.1111/j.1475-6773.2005.00479.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 47.Kruk ME, Kujawski S, Mbaruku G, Ramsey K, Moyo W, Freedman LP. Disrespectful and abusive treatment during facility delivery in Tanzania: a facility and community survey. Health Policy Plan. 2018. Jan 1;33(1):e26–33. 10.1093/heapol/czu079 [DOI] [PubMed] [Google Scholar]
  • 48.Kruk ME, Gage AD, Arsenault C, Jordan K, Leslie HH, Roder-DeWan S, et al. The Lancet Global Health Commission High-quality health systems in the sustainable development goals era: time for a revolution. Lancet Glob Health. 2018. Nov;6(11):e1196–252. 10.1016/S2214-109X(18)30386-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 49.National Academies of Sciences, Engineering and Medicine. Crossing the global quality chasm: improving health care worldwide. Washington, DC: The National Academies Press; 2018. 10.17226/25152 [DOI] [PubMed] [Google Scholar]

Articles from Bulletin of the World Health Organization are provided here courtesy of World Health Organization

RESOURCES