Abstract
IMPORTANCE
In rural India, as in many developing countries, childhood mortality remains high and the quality of health care available is low. Improving care in such settings, where most health care practitioners do not have formal training, requires an assessment of the practitioners’ knowledge of appropriate care and the actual care delivered (the know-do gap).
OBJECTIVE
To assess the knowledge of local health care practitioners and the quality of care provided by them for childhood diarrhea and pneumonia in rural Bihar, India.
DESIGN, SETTING, AND PARTICIPANTS
We conducted an observational, cross-sectional study of the knowledge and practice of 340 health care practitioners concerning the diagnosis and treatment of childhood diarrhea and pneumonia in Bihar, India, from June 29 through September 8, 2012. We used data from vignette interviews and unannounced standardized patients (SPs).
MAIN OUTCOMES AND MEASURES
For SPs and vignettes, practitioner performance was measured using the numbers of key diagnostic questions asked and examinations conducted. The know-do gap was calculated by comparing fractions of practitioners asking key diagnostic questions on each method. Multivariable regressions examined the relation among diagnostic performance, prescription of potentially harmful treatments, and the practitioners’ characteristics. We also examined correct treatment recommended by practitioners with both methods.
RESULTS
Practitioners asked a mean of 2.9 diagnostic questions and suggested a mean of 0.3 examinations in the diarrhea vignette; mean numbers were 1.4 and 0.8, respectively, for the pneumonia vignette. Although oral rehydration salts, the correct treatment for diarrhea, are commonly available, only 3.5% of practitioners offered them in the diarrhea vignette. With SPs, no practitioner offered the correct treatment for diarrhea, and 13.0% of practitioners offered the correct treatment for pneumonia. Diarrhea treatment has a large know-do gap; practitioners asked diagnostic questions more frequently in vignettes than for SPs. Although only 20.9% of practitioners prescribed treatments that were potentially harmful in the diarrhea vignettes, 71.9% offered them to SPs (P < .001). Unqualified practitioners were more likely to prescribe potentially harmful treatments for diarrhea (adjusted odds ratio, 5.11 [95% CI, 1.24–21.13]). Higher knowledge scores were associated with better performance for treating diarrhea but not pneumonia.
CONCLUSIONS AND RELEVANCE
Practitioners performed poorly with vignettes and SPs, with large know-do gaps, especially for childhood diarrhea. Efforts to improve health care for major causes of childhood mortality should emphasize strategies that encourage pediatric health care practitioners to diagnose and manage these conditions correctly through better monitoring and incentives in addition to practitioner training initiatives.
Diarrhea and pneumonia remain leading drivers of mortality among children worldwide, causing 2 million deaths in 2011, including 24% of deaths among children aged 1 to 4 years.1,2 Reducing these deaths requires investments in improved prevention, accurate diagnosis, and appropriate treatment.3,4 Therefore, assessment of the capability of health care delivery systems is critical for the correct diagnosis and management of these conditions.1 We herein examine the knowledge of rural pediatric health care practitioners, many of whom have no formal training, and assess the quality of care they deliver for childhood diarrhea and pneumonia in Bihar, India.
For many of Bihar’s 100 million inhabitants, only low-quality pediatric health care is accessible, contributing to the highest infant mortality rate in India (55 per 1000 live births),5,6 preventable morbidity, and esclating antibiotic resistance.7 Documented deficiencies include low levels of knowledge and even lower levels of observed performance among these health care practitioners.8–11 Low quality of care can occur even when health care practitioners have appropriate knowledge because of the know-do gap, whereby practitioners’ actions diverge from what they know they should do.10,12,13 Understanding know-do gaps of health care practitioners is a critical step toward developing effective, practical strategies to improve delivery of necessary health care. For example, recent empirical evidence on pay-for-performance programs suggests the potential of incentive contracts to improve health outcomes, even in low-resource settings.14–18 With further knowledge concerning know-do gaps, appropriate training and monitoring programs to improve the quality of health care delivery can be developed, and effective regulations and incentives can then support such efforts.19
This study aimed to disentangle the low levels of health care practitioner knowledge from poor effort and delivery of care. We present, to our knowledge, one of the first estimates of know-do gaps in the context of health care systems in developing countries that uses a standardized patient (SP) method for rigorously measuring the performance of health care practitioners.13,20,21 We analyzed data from interviews with health care practitioners using vignettes to estimate competence in terms of knowledge as to what care these practitioners would provide for a hypothetical patient22 and compared the responses with data from SP-based assessments that accurately describe what practitioners did when they were presented with the same case. The know-do gap is the difference between the 2 measures.
Methods
Study Design
The study protocol of the Bihar Evaluation of Social Franchising and Telemedicine (BEST) project was approved by Duke University (approval No. 29755) and India’s Health Ministry Steering Committee (No.12/2008/30-HMSC/4). Three hundred forty health care practitioners provided oral informed consent. We assessed the quality of health care provided for childhood diarrhea and pneumonia using data collected from June 29 through September 8, 2012, as part of the BEST project evaluation of a large-scale telemedicine program.23 Among 360 clusters in the BEST project, 80 were selected at random for the study described herein. Each cluster consists of a group of geographically contiguous villages within 11 districts of Bihar. We generated a list of all health care practitioners visited in the past 6 months—regardless of medical qualifications—from interviews with 64 randomly selected households with children younger than 5 years. Although the study included the 5 most frequently visited primary health care practitioners in each cluster, our analytical sample is restricted to 340 practitioners for whom we had complete data from the vignettes and SPs. Using this restricted sample yields a consistent sample size for our analyses. All of our results remain consistent regardless of the sample used (justification and sample comparison are available in the eMethods and eTables 1 and 2 in the Supplement).
Data
The study used 3 data sources collected before implementation of the telemedicine intervention in the BEST project. Surveys captured information on the characteristics of the health care practitioners, including age, educational level, medical training, experience, and types of illnesses treated. Vignettes and SPs were used to measure the knowledge of the practitioners and the quality of care. The health care practitioners consented to vignette interviews and visits by unannounced SPs within 2 months; 178 practitioners were randomized to receive SPs presenting with childhood diarrhea, and 162 were randomized to receive SPs presenting with childhood pneumonia.
Measuring Knowledge Using Vignettes
The vignettes estimate the clinical knowledge of health care practitioners by presenting a hypothetical case in an interview setting administered by 2 interviewers.9,22,24 In the diarrhea vignette, health care practitioners are told that a father seeks treatment for his 2-year-old son who has had loose stools for 2 days. While interviewer 1 records questions asked by the health care practitioners, interviewer 2 reads scripted responses aloud. For the pneumonia vignette, the treating practitioner is informed that the child has had a fever and a cough for 5 days and appears to have trouble breathing (vignettes are available at http://cohesiveindia.org/publications-downloads.html).
Measuring Quality/Effort Using SPs
Although vignettes measure the knowledge of practitioners, they do not measure actual care delivered. Previous studies of know-do gaps have typically used methods whereby practitioners are observed by interviewers, but this method is vulnerable to Hawthorne effects.13,25 In addition, limitations from case mix and self-selection of patients make comparisons across practitioners difficult.9,12,25 The SP method is considered the criterion standard for practitioner performance measurement because it presents a well-defined incognito case in a clinically accurate and consistent manner to all practitioners.13,20,26
Following the methods of Das et al,13 we used a proxy SP case in which a father seeks treatment for his ill 2-year-old child; the child is not present for the interaction. This pattern of health care, in which a family member seeks care on behalf of the sick patient, is common in India27 and enables use of SP methods without putting a child at risk. The first case is a child with diarrhea (likely caused by rotavirus infection) but no clinical signs of dehydration for whom the only medical therapy indicated, aside from food-based fluids (eg, soup) or clean water, consists of oral rehydration salts (ORSs) to prevent dehydration.28 The second proxy case is a child with pneumonia (as defined by a cough and rapid breathing) who requires antibiotic treatment; given that the father also reports signs of respiratory distress, this child meets the criteria for severe pneumonia and needs an urgent referral for hospitalization.29 Immediately after the interaction, SPs were debriefed using exit interviews that recorded details of the interaction.
For diarrhea, correct treatment was defined to include ORSs with or without zinc supplements, with no prescription of unnecessary or potentially harmful drugs according to the 2005 World Health Organization guidelines.28 The 2013 World Health Organization recommendations30 include ORS and zinc supplementation. The correct treatment for severe pneumonia was defined to include appropriate antibiotics, absence of potentially harmful drugs, and referral to a hospital.29 Because our sample consisted exclusively of practitioners in outpatient settings, we do not have data on hospitalization and referrals and instead focus on the drugs prescribed. For the SPs and vignettes, performance was measured using the numbers of key diagnostic questions asked and examinations conducted. We used item response theory (IRT) to compute performance scores separately for SPs and vignettes following previously described methods.9,31 All analyses, including IRT scores, relied on a subset of 12 essential questions that help to diagnose the cause and severity of disease (eTable 3 in the Supplement). We refer to IRT-based performance for vignettes as the knowledge score and for SPs as the performance score.
Statistical Analysis
We tested for differences in characteristics between practitioners with and without medical qualifications using unpaired 2-tailed t tests and χ2 tests of proportions. We then compared practitioners’ knowledge for vignettes and performance with SPs to characterize know-do gaps as the fraction of practitioners who asked key diagnostic questions on each method.
We used regression analyses to examine associations among practitioners’ observable characteristics, performance, and the know-do gap. We estimated multivariable regressions in which the outcome is practitioner performance measured as the percentage of diagnostic questions asked for diarrhea and pneumonia. We checked the robustness of our findings using fractional logit models that account for the outcome having values ranging from 0 to 1. We conducted logistic regressions for the prescription of potentially harmful treatments (yes/no) for each case. All regressions control for age; medical qualification; practitioners’ work hours; patient volume; whether practitioners engage in public events, such as medical camps; whether the clinic is public or private; and clinic cleanliness as observed by the investigators. Analyses were adjusted for SEs for those practitioners who were sampled within clusters.
Results
Practitioners
Of the 340 practitioners included in this study, 80.0% had no formal medical degrees in allopathy, Ayurveda, homeopathy, or Unani medicine, which is consistent with other studies of health care in rural India32 (Table 1). Qualified practitioners had a higher caseload, worked longer hours, and were more likely to work in public facilities. We found no significant differences in mean years of experience between qualified and unqualified practitioners. Qualified practitioners are more likely to spend time on skill-oriented activities, such as consultation and laboratory-related duties, and less time selling drugs. Approximately 90% of all practitioners reported frequent prescription of allopathic treatments. Almost all practitioners reported treating diarrhea (98.5% of those with qualifications and 96.7% of those without), whereas fewer practitioners without formal qualifications reported treating pneumonia (81.6% vs 92.6%). However, even practitioners who reported that they do not treat patients with a stated condition actually do provide treatment because they may not always provide the correct diagnosis for these conditions.
Table 1.
Practitioner Characteristics According to Medical Qualification
| Characteristic | Data, Mean (SD)a | P Value | |
|---|---|---|---|
| No Medical Qualification (n = 272)b | Medical Qualification (n = 68)c | ||
| Age, y | 43.6 (11.2) | 45.5 (10.7) | .19 |
| Educational level beyond high school | 70.2 (45.8) | 100 (0.0) | <.001 |
| Ever used a computer | 11.4 (31.8) | 47.1 (50.3) | <.001 |
| Experience, y | 18.3 (10.7) | 18.0 (10.2) | .87 |
| Patient caseload, No./d | 17.2 (7.0) | 20.3 (10.6) | .02 |
| Time working, h/wk | 48.6 (17.6) | 57.5 (16.6) | <.001 |
| Run camps | 4.4 (20.6) | 17.6 (38.4) | <.001 |
| Work in public health facility | 0.4 (6.1) | 8.8 (28.6) | <.001 |
| Infrastructure indexd | −0.4 (0.9) | 1.2 (2.9) | <.001 |
| Consultation fee, Rs | 14.4 (17.4) | 45.6 (47.3) | <.001 |
| Tasks | |||
| Consult with patients | 99.6 (6.1) | 100 (0.0) | .62 |
| Administer treatment | 91.2 (28.4) | 69.1 (46.5) | <.001 |
| Sell drugs | 55.9 (49.7) | 26.5 (44.4) | <.001 |
| Laboratory-related duties | 4.0 (19.7) | 7.4 (26.3) | .25 |
| Administrative duties | 63.6 (48.2) | 63.2 (48.6) | .96 |
| Ownership | 72.8 (44.6) | 66.2 (47.7) | .28 |
| Type of medicine practiced | |||
| Allopathic | 90.8 (28.9) | 89.7 (30.6) | .78 |
| Homeopathic/ayurvedic | 34.9 (47.8) | 45.6 (50.2) | .10 |
| Type of diseases treated | |||
| Diarrhea | 96.7 (17.9) | 98.5 (12.1) | .42 |
| Pneumonia | 81.6 (38.8) | 92.6 (26.3) | .03 |
| IRT scores | |||
| Knowledge: combined | −1.37 (2.53) | −0.84 (2.15) | .07 |
| Knowledge: diarrhea | −0.93 (2.21) | −0.49 (1.82) | .15 |
| Knowledge: pneumonia | −1.81 (2.75) | −1.42 (2.53) | .43 |
| Performance: combined | −1.31 (2.58) | −1.58 (2.64) | .42 |
| Performance: diarrhea | −1.73 (2.68) | −1.20 (2.56) | .23 |
| Performance: pneumonia | −0.89 (2.42) | −2.19 (2.69) | .02 |
Abbreviation: IRT, item response theory.
Unless otherwise indicated, data are expressed as percentage of practitioners. Data are obtained from the practitioner questionnaire, standardized patient exit interview, and vignettes.
Includes all practitioners with null medical training or courses/degree related in some way to medicine, such as pharmacy.
Includes practitioners with a master’s or a bachelor’s degree (bachelor of medicine, bachelor of surgery; bachelor of ayurvedic medicine and surgery; and bachelor of homeopathic medicine and surgery) and diploma in ayurvedic medicine and some other doctor of medicine degrees.
Computed from principal components analysis using the following variables: electricity, power backup, number of consulting rooms, number of beds for day observation, provision of tests, provision of radiological examinations, and a computer system. The resulting index score is a standardized score with a mean of 0 and an SD of 1. Negative values represent practitioners with an infrastructure index below the mean, and positive values represent practitioners with an infrastructure index above the mean.
Vignettes
Practitioners demonstrated low levels of knowledge of key diagnostic questions and examinations (Table 2). The most commonly asked question for diarrhea concerned the nature of stools (60.6%); 46.2% asked about the frequency of stools. Only 32.9% asked questions that provide critical information about dehydration severity (eg, weakness, ability to take fluids, and urinary frequency). Most practitioners (86.8%) failed to ask about blood in stools to distinguish the simple case of viral diarrhea from possible dysentery. Among the diagnostic questions and examinations for diarrhea listed in Table 2, practitioners asked a mean of 2.9 questions and suggested a mean of 0.3 examinations.
Table 2.
Percentage of Practitioners Who Asked Key Diagnostic Questions and Performed Key Examinations, Diagnosis, and Treatment in Vignettes (N=340)
| No. (%) of Practitioners | ||
|---|---|---|
| Treating Diarrhea | Treating Pneumonia | |
| Key Diagnostic Questions and Examinations | ||
| Fever/child warm | 68 (20.0) | NA |
| Urine/color normal | 27 (7.9) | NA |
| Time of last urination | 24 (7.1) | NA |
| Nature of stool | 206 (60.6) | NA |
| Frequency of stool | 157 (46.2) | NA |
| Quantity of stool | 43 (12.6) | NA |
| Blood or mucus in the stool | 45 (13.2) | NA |
| Worms in stool | 21 (6.2) | NA |
| Foul-smelling stool | 40 (11.8) | NA |
| Stomachache | 75 (22.1) | NA |
| Weak now | 30 (8.8) | NA |
| Vomiting | 172 (50.6) | NA |
| Drinking a lot of water | 70 (20.6) | NA |
| Weight of child | 23 (6.8) | 18 (5.3) |
| Temperature | 26 (7.6) | NA |
| Mucous membranes checked for moistness | 14 (4.1) | NA |
| Skin color and turgor | 19 (5.6) | NA |
| Palpation of the abdomen | 22 (6.5) | NA |
| High fever | NA | 113 (33.2) |
| Cough continuous | NA | 60 (17.6) |
| Cough increases at night | NA | 30 (8.8) |
| Fever and cough started 5 d ago | NA | 34 (10.0) |
| Runny/blocked nose | NA | 29 (8.5) |
| Breathing rapidly | NA | 82 (24.1) |
| Nostrils appear to be flaring when breathing | NA | 34 (10.0) |
| Noticed skin between the ribs or the stomach moves inward when breathing | NA | 47 (13.8) |
| Any particular sounds that the child made since difficult breathing started | NA | 40 (11.8) |
| Breathlessness in the past | NA | 17 (5.0) |
| Respiration rate | NA | 16 (4.7) |
| Auscultation of chest and heart | NA | 37 (10.9) |
| Pulse rate | NA | 18 (5.3) |
| Temperature | NA | 62 (18.2) |
| Examination of chest | NA | 28 (8.2) |
| Chest radiograph | NA | 27 (7.9) |
| Total leukocyte count | NA | 30 (8.8) |
| WBC differential | NA | 36 (10.6) |
| Other Questions | ||
| Expressed not having given antibiotics when they did | 36 (12.6) | 3 (1.1) |
| Diagnosis | ||
| Gave any diagnosis | 337 (99.1) | 332 (97.6) |
| Correct diagnosis | 251 (73.8) | 196 (57.6) |
| Correct diagnosis, if anya | 251 (74.5) | 196 (59.0) |
| Treatment | ||
| Gave any treatment | 338 (99.4) | 324 (95.3) |
| Correct treatment | 12 (3.5) | 30 (8.8) |
| Correct treatment, if anyb | 12 (3.6) | 30 (9.3) |
Abbreviations: NA, not applicable; WBC, white blood cell count.
Indicates of the 337 practitioners who gave any diagnosis in the diarrhea vignette and the 332 practitioners who gave any diagnosis in the pneumonia vignette.
Indicates of the 338 practitioners who gave any treatment in the diarrhea vignette and the 324 practitioners who gave any treatment in the pneumonia vignette.
For pneumonia, 33.2% of practitioners asked questions about fever, but only 24.1% asked about rapid breathing, and 20.9% asked about visual signs of respiratory distress (drawing in of the chest and nasal flaring were present in this case). Although difficulty in breathing was voluntarily described in the presenting complaints, only 10.9% of practitioners said they would auscultate the child’s chest (Table 2). Practitioners asked a mean of 1.4 questions and performed a mean of 0.8 examinations for pneumonia.
Despite seeking little diagnostic information, almost all practitioners made a diagnosis and prescribed treatments. Among 99.1% of practitioners with a diagnosis for diarrhea, 74.5% were correct. This finding did not surprise us because the diagnosis in the local language is a term for loose stools. Although ORSs are a commonly available treatment, only 3.5% of practitioners offered the correct ORS treatment, whereas 20.9% prescribed unnecessary antibiotics, corticosteroids, and other potentially harmful drugs without ORSs or zinc. (The drug list is available at http://cohesiveindia.org/publications-downloads.html). Another 68.8% prescribed ORSs with other unnecessary treatments (Table 3). When asked at the end of the vignette if the drugs prescribed included antibiotics, 12.6% of practitioners reported not prescribing antibiotics when in fact they had.
Table 3.
Type of Treatment Provided for Diarrhea and Pneumonia in SP Interactions and Vignettesa
| Treatment Prescribed | No. (%) of Practitioners | |
|---|---|---|
| Vignette | SP | |
| For Diarrheab | ||
| ORS ± zinc | 8 (2.3) | 0 |
| ORS + Ayurveda/homeopathy/intravenous bottles/glucose | 4 (1.2) | 0 |
| ORS + antibiotics | 143 (42.1) | 18 (10.1) |
| ORS + antibiotics + otherc | 79 (23.2) | 12 (6.7) |
| ORS + others (no antibiotics)c | 12 (3.5) | 1 (0.6) |
| No ORS, no harmful drugs | 21 (6.2) | 6 (3.4) |
| No ORS + harmful drugs (antibiotics or others) | 71 (20.9) | 128 (71.9) |
| No treatment | 2 (0.6) | 13 (7.3) |
| For Pneumoniad | ||
| Antibiotics ± nonharmful drugs | 10 (2.9) | 4 (2.5) |
| Antibiotics + analgesics ± nonharmful drugs | 18 (5.3) | 19 (11.7) |
| Antibiotics + analgesics + unnecessary drugs ± nonharmful drugs | 106 (31.2) | 53 (32.7) |
| Antibiotics + unnecessary drugs ± nonharmful drugs | 150 (44.1) | 28 (17.3) |
| No antibiotics | 40 (11.8) | 24 (14.8) |
| No treatment | 16 (4.7) | 34 (21.0) |
Abbreviations: ORS, oral rehydration salts; SP, standardized patient.
Data were obtained from the SP and vignette exit interviews.
Includes 340 practitioners for the vignette and 178 for the SP.
May include analgesics, antiulcer medication, antiallergy medicine, corticosteroids, cardiac medication, or psychiatric/neural medicine.
Includes 340 practitioners for the vignette and 162 for the SP. All drug categories in addition to antibiotics may include nonharmful medication, such as vitamins.
For pneumonia, 59.0% of all practitioners made the correct diagnosis of pneumonia. Only 8.8% of practitioners prescribed the correct treatment. An additional 43.8% offered antibiotics with unnecessary, potentially harmful drugs, such as allergy medications and drugs for cardiac conditions. The relatively high share of practitioners prescribing antibiotics likely reflects the general overprescription of antibiotics irrespective of the patients’ conditions.33,34 The most commonly prescribed incorrect treatments included corticosteroids or vitamin syrups without any antibiotics.
Standardized Patients
Practitioner effort, measured in their interactions with SPs, was low (Table 4). Practitioners spent a mean (SD) of 1.6 (1.7) minutes with the SP with diarrhea and 2.9 (3.8) minutes with the SP with pneumonia (eFigure 1 in the Supplement shows the distribution of time). Practitioners asked a mean of 2.7 essential questions for diarrhea and 2.8 for pneumonia, covering less than 30% of the questions needed to diagnose the cause and severity of the disease. Despite cursory consultations, practitioners prescribed a mean of 1.8 medicines for diarrhea and 2.2 medicines for pneumonia.
Table 4.
Diagnostic Questions Asked and Diagnosis and Treatment Given in SP-Practitioner Interactionsa
| SP-Practitioner Interaction | SP With Diarrhea (n = 178) | SP With Pneumonia (n = 162) |
|---|---|---|
| Key Diagnostic Questions and Examinations | ||
| Age of child | 166 (93.3) | 156 (96.3) |
| Nature of stool | 81 (45.5) | NA |
| Frequency of stool | 53 (29.8) | NA |
| Quantity of stool | 21 (11.8) | NA |
| Questions about urination | 4 (2.2) | NA |
| Child is active/playful | 5 (2.8) | NA |
| Fever | 28 (15.7) | NA |
| Abdominal pain | 33 (18.5) | NA |
| Vomiting | 38 (21.3) | NA |
| What has the child eaten | 20 (11.2) | NA |
| Taking fluids | 23 (12.9) | NA |
| Fever | NA | 99 (61.1) |
| Breathing is rapid | NA | 44 (27.2) |
| Difficulty in breathing/nostrils flaring/skin between ribs moves inward/neck muscles are strained | NA | 40 (24.7) |
| Type of cough | NA | 56 (34.6) |
| Runny/blocked nose | NA | 23 (14.2) |
| Sounds while breathing | NA | 12 (7.4) |
| Child is weak | NA | 7 (4.3) |
| Breastfeeding/immunization history | NA | 11 (6.8) |
| Other Questions | ||
| Father was asked to bring the child to clinic | 42 (23.6) | 53 (32.7) |
| Counseling on hygiene, especially washing hands | 4 (2.2) | NA |
| Asked if other children have similar symptoms | NA | 3 (1.9) |
| Diagnosis | ||
| Gave any diagnosis | 11 (6.2) | 19 (11.7) |
| Correct diagnosis | 6 (3.4) | 13 (8.0) |
| Correct diagnosis, if anyb | 6 (54.5) | 13 (68.4) |
| Treatment | ||
| Gave any treatment | 165 (92.7) | 128 (79.0) |
| Correct treatment | 0 | 21 (13.0) |
| Correct treatment, if anyc | 0 | 21 (16.4) |
| Length of SP Interaction, Questions Asked, and Medicines Prescribed, Mean (SD) | ||
| Visit length, min | 1.6 (1.7) | 2.9 (3.8) |
| Total No. of 12 essential questions asked by practitioners | 2.7 (1.9) | 2.8 (1.6) |
| No. of medicines prescribed/dispensed | 1.8 (1.1) | 2.2 (1.4) |
Abbreviations: NA, not applicable; SP, standardized patient.
Unless otherwise indicated, data are expressed as number (percentage) of practitioners. Data were obtained from the SP exit interview.
Indicates of the 11 practitioners who gave any diagnosis in the diarrhea vignette and the 19 practitioners who gave any diagnosis in the pneumonia vignette.
Indicates of the 165 practitioners who gave any treatment in the diarrhea vignette and the 128 practitioners who gave any treatment in the pneumonia vignette.
Practitioners asked the SPs even fewer questions about the severity of the disease than they did with the vignettes (Table 4). Although 45.5% of practitioners asked the SP with diarrhea about the quality of stools, only 29.8% asked about frequency, 2.2% about urination, and 2.8% about the child’s activity level. Similarly, only 27.2% asked the SP with pneumonia about rapid breathing, and 24.7% asked about signs of respiratory distress. Other than age, the most commonly asked question was about fever (61.1%), which was one of the presenting complaints described by the SP.
Practitioners participating in the SP method frequently offered treatment despite not seeing the child. Although 23.6% asked the SP to return with the child for diarrhea, 92.7% prescribed treatment (for the SP with pneumonia, 32.7% and 79.0%, respectively). The proportion of practitioners who prescribed the correct treatment to the SP was far lower for diarrhea (0) compared with pneumonia (13.0%).
Know-Do Gap
We found a clear know-do gap for diarrhea: practitioners reported during the vignette that they would ask diagnostic questions far more often than they asked the SP (eFigure 2 in the Supplement). For example, although 46.2% of practitioners asked about the frequency of stools in the vignette, only 29.8% asked the SP. For pneumonia, practitioners’ generally poor performance was roughly comparable between the vignette and the SP.
Likewise, the know-do gap for treatments offered is larger for diarrhea than for pneumonia (Table 3). For the diarrhea vignette, 72.4% of practitioners reported that they would offer ORSs (often in combination with other drugs); only 17.4% actually offered this type of treatment to the SP. Another critical dimension of the large know-do gap for diarrhea is reflected in practitioners prescribing potentially harmful treatments to the SP and not in the vignette. Compared with 20.9% of practitioners who said they would prescribe only potentially harmful treatments without ORSs for the vignette, 71.9% offered such treatments to the SP (P < .001). For pneumonia, this gap is smaller. Although 11.8% of practitioners reported that they would not prescribe antibiotics (classified as the incorrect treatment), 14.8% did not prescribe antibiotics (P = .09) to the SP.
Practitioner Effort and Characteristics
Practitioners asked the SP only 24.1% and 30.7% of the essential questions for diarrhea and pneumonia, respectively (Table 4). Table 5 shows the associations between practitioner characteristics and the percentage of diagnostic questions asked. Practitioners in public facilities asked the SPs significantly fewer questions (16 percentage points fewer for diarrhea [P = .006] and 25 percentage points fewer for pneumonia [P < .001]), controlling for measures of patient volume, type of practice, knowledge IRT scores, the practitioners’ age group, and the practitioners’ experience. Those practitioners with more working hours per week asked fewer diagnostic questions for the SPs with diarrhea (0.2 percentage points [P = .004]) and pneumonia (0.3 percentage points [P = .02]). Medical qualifications were associated with 8.4 percentage points fewer (P = .03) diagnostic questions for pneumonia. Practitioners with higher knowledge scores about diarrhea also asked more diagnostic questions (P = .005). Unlike the SPs, for the vignettes, we found no significant or large associations between other practitioner characteristics and the number of questions asked (eTable 4 in the Supplement). Results were highly similar when we used fractional logit regressions (eTable 5 in the Supplement).
Table 5.
Regression Analysis of Practitioner Characteristics on Practitioner Performancea
| Characteristic | Percentage of Diagnostic Questions Asked (OLS) | Prescribed Harmful Treatment (Logistic) | ||||||
|---|---|---|---|---|---|---|---|---|
| Estimated Effect, % (95% CI)b | P Value | Estimated Effect, % (95% CI)b,c | P Value | OR (95% CI)b | P Value | OR (95% CI)b,c | P Value | |
| Diarrhea | ||||||||
| Age, yd | ||||||||
| 20–29 | 7.5 (−5.4 to 20.5) | .25 | 6.3 (−4.7 to 17.3) | .26 | 0.38 (0.05 to 2.61) | .32 | 0.42 (0.06 to 2.84) | .36 |
| 40–49 | −0.7 (−9.0 to 7.5) | .86 | 0.1 (−7.3 to 7.5) | .98 | 3.82 (0.64 to 22.81) | .14 | 3.63 (0.62 to 21.33) | .15 |
| 50–59 | −0.9 (−10.9 to 9.0) | .85 | −1.4 (−10.6 to 7.8) | .76 | 1.99 (0.17 to 23.65) | .59 | 2.10 (0.17 to 26.20) | .56 |
| ≥60 | 7.8 (−8.4 to 24.0) | .34 | 5.5 (−10.4 to 0.2) | .50 | 0.32 (0.02 to 4.91) | .41 | 0.41 (0.02 to 7.26) | .54 |
| Experience, y | −0.0 (−0.5 to 0.4) | .91 | 0.0 (−0.4 to 0.5) | .83 | 1.02 (0.91 to 1.14) | .76 | 1.02 (0.91 to 1.13) | .78 |
| Medical qualificationse | 4.0 (−3.0 to 11.1) | .26 | 6.0 (−1.6 to 13.6) | .12 | 3.92 (1.24 to 12.43) | .020 | 5.11 (1.24 to 21.13) | .02 |
| Work time, h/wk | NA | −0.2 (−0.4 to −0.1) | .004 | NA | 1.02 (0.98 to 1.05) | .33 | ||
| Mean caseload | NA | −0.2 (−0.4 to 0.1) | .15 | NA | 1.03 (0.98 to 1.08) | .25 | ||
| Runs camps | NA | 3.7 (−9.2 to 16.6) | .57 | NA | 1.58 (0.18 to 13.82) | .68 | ||
| Works in public facility | NA | −16.1 (−27.5 to −4.8) | .006 | NA | 1.09 (0.11 to 10.80) | .95 | ||
| Cleanliness | NA | −1.1 (−5.8 to 3.6) | .63 | NA | 1.43 (0.63 to 3.24) | .39 | ||
| Knowledge IRT scoref | NA | 3.3 (1.0 to 5.6) | .005 | NA | 1.11 (0.54 to 2.27) | .77 | ||
| R2 value | 0.059 | 0.153 | 0.166 | 0.192 | ||||
| Pneumonia | ||||||||
| Age, yd | ||||||||
| 20–29 | −11.3 (−20.2 to −2.3) | .01 | −11.0 (−20.8 to −1.2) | .03 | 1.10 (0.22 to 5.55) | .90 | 1.09 (0.17 to 7.02) | .93 |
| 40–49 | −0.3 (−8.0 to 7.3) | .93 | 0.3 (−7.6 to 8.2) | .94 | 1.56 (0.64 to 3.83) | .33 | 1.76 (0.70 to 4.39) | .23 |
| 50–59 | 3.5 (−7.1 to 14.0) | .52 | 4.0 (−7.9 to 15.9) | .51 | 1.37 (0.33 to 5.75) | .65 | 2.05 (0.48 to 8.79) | .34 |
| ≥60 | 10.3 (−2.6 to 23.3) | .12 | 7.3 (−8.2 to 22.7) | .35 | 0.58 (0.11 to 2.99) | .51 | 0.61 (0.10 to 3.75) | .60 |
| Experience, y | −0.4 (−0.9 to −0.0) | .046 | −0.4 (−0.9 to 0.0) | .08 | 1.02 (0.96 to 1.08) | .52 | 1.02 (0.96 to 1.08) | .61 |
| Medical qualificationse | −9.5 (−17.7 to −1.3) | .02 | −8.4 (−16.0 to −0.7) | .03 | 2.09 (0.90 to 4.83) | .09 | 2.19 (0.88 to 5.45) | .09 |
| Work time, h/wk | NA | −0.3 (−0.5 to −0.1) | .02 | NA | 0.99 (0.96 to 1.01) | .36 | ||
| Mean caseload | NA | −0.0 (−0.4 to 0.4) | .98 | NA | 1.01 (0.95 to 1.08) | .67 | ||
| Runs camps | NA | 1.0 (−9.1 to 11.1) | .85 | NA | 3.43 (0.17 to 67.71) | .42 | ||
| Works in public facility | NA | −24.9 (−34.8 to −15.0) | <.001 | NA | ||||
| Cleanliness | NA | 5.3 (0.3 to 10.2) | .04 | NA | 1.56 (0.79 to 3.10) | .21 | ||
| Knowledge IRT scoref | NA | 0.7 (−2.3 to 3.6) | .66 | NA | 1.04 (0.72 to 1.50) | .83 | ||
| R2 value | 0.111 | 0.211 | 0.109 | 0.140 | ||||
Abbreviations: IRT, item response theory; NA, not applicable; OLS, ordinary least squares; OR, odds ratio.
Data are obtained from the practitioner questionnaire, standardized patient exit interview, and vignettes for 340 practitioners.
Adjusted for the time and day of the week of the consultation and practitioners’ age group, years of experience, and qualifications.
Adjusted in addition for covariates listed below.
Practitioners aged 30 to 39 years are the reference group.
Described in Table 1.
Based on practitioners’ responses to 12 key questions listed in eTable 1 in the Supplement.
Being an unqualified practitioner predicted a significantly higher likelihood of prescribing potentially harmful treatments (Table 5) for diarrhea compared with qualified practitioners (adjusted odds ratio, 5.11 [95% CI, 1.24–21.13]). For pneumonia, the adjusted odds ratio was 2.19 (95% CI, 0.88–5.45). Results were generally similar when performance IRT score was used as the outcome (eTable 6 in the Supplement).
Discussion
In rural India, pediatric health care practitioners demonstrated low levels of knowledge during vignettes for childhood diarrhea and pneumonia. We also found a large know-do gap in the practitioners’ treatment of childhood diarrhea but not pneumonia. In these settings, medically qualified practitioners offered far fewer potentially harmful treatments compared with unqualified practitioners.
Although 72.4% of practitioners reported that they would prescribe ORS treatment in the diarrhea vignette, only 17.4% actually prescribed ORSs to the SP, with the remainder prescribing potentially harmful drugs instead. Understanding factors that lead to this large know-do gap is critical to reducing preventable deaths due to diarrhea. The know-do gap for diarrhea appears paradoxical because most practitioners in our sample are in private practice and thus should be able to benefit financially from better performance unless patients cannot differentiate practitioner quality (an asymmetric information problem).35 The severity of this asymmetry may be seen in the lack of robust correlation between practitioner characteristics and performance with SPs; practitioner characteristics explain only 12% to 20% of the variation in performance.
The smaller know-do gap for pneumonia is striking. Among practitioners in our sample, 83.8% claimed to treat pneumonia (97.1% for diarrhea), suggesting that they might consider pneumonia to be more critical than diarrhea and might exert higher effort. However, because the technically correct treatment includes a referral for hospitalization, which we did not observe, the true rates of correct treatment are likely lower than the 13.0% that we report. Mean consultation time for pneumonia was almost twice that for diarrhea, yet was still short. Although the know-do gap for pneumonia was small, the level of practitioner knowledge during vignettes was very low. Our findings of low levels of practitioner knowledge and effort and that public practitioners had worse performance with SPs are consistent with previous research.10,11,36,37
Although our SP methods represent substantial improvements over other methods for measuring the quality of health care practitioners deliver in developing countries, they have limitations. First, the study is restricted to cases for whom the interviewers did not face any risk. Second, we used proxy cases presented by a parent of a sick child, although this presentation is common in India. From previous studies, these limitations do not affect measurements of practitioner quality but do present challenges for implementing similar methods in other contexts. Furthermore, because the presentations of patients were standardized, we do not know how practitioners might perform given a different set of symptoms. Nonetheless, the cases we chose are simple, uncomplicated presentations of conditions seen commonly in primary care in rural areas, making the findings important on their own merit. Finally, the absence of data on referrals for hospitalization for pneumonia cases suggests that our estimated rate of correct treatment (13.0%) represents the upper bound.
Although medically qualified practitioners do less harm than their unqualified counterparts, most practitioners in rural areas are unqualified.32,38,39 Although practitioners asked few questions and spent very little time with the patient, almost all practitioners, including the unqualified ones, prescribed treatments, none of which were correct for diarrhea and only 13.0% of which were correct for pneumonia.
Conclusions
Our findings highlight the need to better understand why pediatric practitioners in developing countries fail to correctly diagnose and manage the 2 leading causes of childhood mortality. The know-do gap we document supports the argument that more training focused on increasing knowledge alone is insufficient. Understanding the incentives faced by practitioners as well as the potential role of patients who do not have adequate information about practitioner performance and quality are critical. How such information can be appropriately communicated and targeted to patients in developing countries to counteract the know-do gap remains a major challenge.
Supplementary Material
Acknowledgments
Funding/Support: This study was supported by the Bill and Melinda Gates Foundation.
Role of the Funder/Sponsor: The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Footnotes
Supplemental content at jamapediatrics.com
Author Contributions: Dr Mohanan and Ms Giardili had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Mohanan, Vera-Hernández, Das, Goldhaber-Fiebert, Rabin, Schwartz.
Acquisition, analysis, or interpretation of data: Mohanan, Vera-Hernández, Giardili, Goldhaber-Fiebert, Rabin, Raj, Schwartz, Seth.
Drafting of the manuscript: Mohanan, Vera-Hernández, Goldhaber-Fiebert, Schwartz, Seth. Critical revision of the manuscript for important intellectual content: Mohanan, Vera-Hernández, Das, Giardili, Goldhaber-Fiebert, Rabin, Raj, Schwartz.
Statistical analysis: Mohanan, Vera-Hernández, Giardili, Goldhaber-Fiebert.
Obtained funding: Mohanan, Vera-Hernández, Goldhaber-Fiebert.
Administrative, technical, or material support: Mohanan, Raj, Seth.
Study supervision: Mohanan, Vera-Hernández, Das, Raj.
Conflict of Interest Disclosures: None reported.
Additional Information: Data set and STATA Do files can be downloaded from http://cohesiveindia.org/publications-downloads.html.
Additional Contributions: Purshottam, MA, Charu Nanda, MA, Rajan Singh, MA, Simi Chaturvedi, MA, Geeta, MA, Devender, MCom, Zargham Sayed, MA, Krishan Yadav, BA, and Varun Kumar, BA, of the Institute of Socioeconomic Research for Development and Democracy (ISERDD) team and the standardized patients implemented the field work for this study. We thank interviewers from Sambodhi Research and Communications for conducting the practitioner surveys and vignette interviews. Joanna Maselko, ScD, Departments of Psychiatry and Behavioral Sciences and Global Health, Duke University, provided comments on the manuscript. Contributors from the ISERDD received compensation for their work.
References
- 1.Walker CLF, Rudan I, Liu L, et al. Global burden of childhood pneumonia and diarrhoea. Lancet. 2013;381(9875):1405–1416. doi: 10.1016/S0140-6736(13)60222-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Lozano R, Naghavi M, Foreman K, et al. Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet. 2012;380(9859):2095–2128. doi: 10.1016/S0140-6736(12)61728-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Chopra M, Mason E, Borrazzo J, et al. Ending of preventable deaths from pneumonia and diarrhoea: an achievable goal. Lancet. 2013;381(9876):1499–1506. doi: 10.1016/S0140-6736(13)60319-0. [DOI] [PubMed] [Google Scholar]
- 4.Bhutta ZA, Das JK, Walker N, et al. Lancet Diarrhoea and Pneumonia Interventions Study Group Interventions to address deaths from childhood pneumonia and diarrhoea equitably: what works and at what cost? Lancet. 2013;381(9875):1417–1429. doi: 10.1016/S0140-6736(13)60648-0. [DOI] [PubMed] [Google Scholar]
- 5.You D, New JR, Wardlaw T. Levels & Trends in Child Mortality: Estimates Developed by the UN Inter-agency Group for Child Mortality Estimation. New York, NY: United Nations Children’s Fund; 2012. [Google Scholar]
- 6.DES-Delhi. Delhi Statistical Handbook. Delhi, India: Directorate of Economics and Statistics, Government of National Capital Territory of Delhi; 2012. [Google Scholar]
- 7.Peters DH, Bloom G. Developing world: bring order to unregulated health markets. Nature. 2012;487(7406):163–165. doi: 10.1038/487163a. [DOI] [PubMed] [Google Scholar]
- 8.Chaudhury N, Hammer J, Kremer M, Muralidharan K, Rogers FH. Missing in action: teacher and health worker absence in developing countries. J Econ Perspect. 2006;20(1):91–116. doi: 10.1257/089533006776526058. [DOI] [PubMed] [Google Scholar]
- 9.Das J, Hammer J. Which doctor? combining vignettes and item response to measure clinical competence. J Dev Econ. 2005;78(2):348–383. [Google Scholar]
- 10.Das J, Hammer J. Money for nothing: the dire straits of medical practice in Delhi, India. J Dev Econ. 2007;83(1):1–36. [Google Scholar]
- 11.Das J, Hammer J, Leonard K. The quality of medical advice in low-income countries. J Econ Perspect. 2008;22(2):93–114. doi: 10.1257/jep.22.2.93. [DOI] [PubMed] [Google Scholar]
- 12.Leonard KL, Masatu MC. Professionalism and the know-do gap: exploring intrinsic motivation among health workers in Tanzania. Health Econ. 2010;19(12):1461–1477. doi: 10.1002/hec.1564. [DOI] [PubMed] [Google Scholar]
- 13.Das J, Holla A, Das V, Mohanan M, Tabak D, Chan B. In urban and rural India, a standardized patient study showed low levels of provider training and huge quality gaps. Health Aff (Millwood) 2012;31(12):2774–2784. doi: 10.1377/hlthaff.2011.1356. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Miller G, Babiarz KS. Pay-for-Performance Incentives in Low- and Middle-Income Country Health Programs. 2013. (National Bureau of Economic Research Working Paper Series). Paper 18932. [Google Scholar]
- 15.Singh P. Performance Pay and Information: Reducing Child Undernutrition in India. 2014. (SSRN Working Paper Series). [Google Scholar]
- 16.Miller G, Luo R, Zhang L, et al. Effectiveness of provider incentives for anaemia reduction in rural China: a cluster randomised trial. BMJ. 2012;345:e4809. doi: 10.1136/bmj.e4809. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Basinga P, Gertler PJ, Binagwaho A, Soucat AL, Sturdy J, Vermeersch CM. Effect on maternal and child health services in Rwanda of payment to primary health-care providers for performance: an impact evaluation. Lancet. 2011;377(9775):1421–1428. doi: 10.1016/S0140-6736(11)60177-3. [DOI] [PubMed] [Google Scholar]
- 18.Gertler P, Vermeesch C. Using Performance Incentives to Improve Health Outcomes. Washington, DC: 2012. (World Bank Working Paper Series). [Google Scholar]
- 19.Miller GE. The assessment of clinical skills/competence/performance. Acad Med. 1990;65(9 suppl):S63–S67. doi: 10.1097/00001888-199009000-00045. [DOI] [PubMed] [Google Scholar]
- 20.Rethans J-J, Gorter S, Bokken L, Morrison L. Unannounced standardised patients in real practice: a systematic literature review. Med Educ. 2007;41(6):537–549. doi: 10.1111/j.1365-2929.2006.02689.x. [DOI] [PubMed] [Google Scholar]
- 21.Glassman PA, Luck J, O’Gara EM, Peabody JW. Using standardized patients to measure quality: evidence from the literature and a prospective study. Jt Comm J Qual Improv. 2000;26(11):644–653. doi: 10.1016/s1070-3241(00)26055-0. [DOI] [PubMed] [Google Scholar]
- 22.Leonard KL, Masatu MC. The use of direct clinician observation and vignettes for health services quality evaluation in developing countries. Soc Sci Med. 2005;61(9):1944–1951. doi: 10.1016/j.socscimed.2005.03.043. [DOI] [PubMed] [Google Scholar]
- 23.Bihar Evaluation of Social Franchising and Telemedicine (BEST) 2011 https://clinicaltrials.gov/ct2/show/study/NCT01345695. Accessed August 13, 2013.
- 24.Peabody JW, Luck J, Glassman P, et al. Measuring the quality of physician practice by using clinical vignettes: a prospective validation study. Ann Intern Med. 2004;141(10):771–780. doi: 10.7326/0003-4819-141-10-200411160-00008. [DOI] [PubMed] [Google Scholar]
- 25.Leonard KL, Masatu MC. Using the Hawthorne effect to examine the gap between a doctor’s best possible practice and actual performance. J Dev Econ. 2010;93(2):226–234. [Google Scholar]
- 26.Rethans J-J, Sturmans F, Drop R, van der Vleuten C. Assessment of the performance of general practitioners by the use of standardized (simulated) patients. Br J Gen Pract. 1991;41(344):97–99. [PMC free article] [PubMed] [Google Scholar]
- 27.Johnston HB, Ved R, Lyall N, Agarwal K. Where do rural women obtain postabortion care? the case of Uttar Pradesh, India. Int Fam Plan Perspect. 2003;29(4):182–187. doi: 10.1363/ifpp.29.182.03. [DOI] [PubMed] [Google Scholar]
- 28.World Health Organization. Handbook: Integrated Management of Childhood Illness. Geneva, Switzerland: World Health Organization Dept of Child and Adolescent Health and Development; 2005. [Google Scholar]
- 29.Arora N, India Clinical Epidemiology Network (IndiaCLEN) Task Force on Pneumonia Rational use of antibiotics for pneumonia. Indian Pediatr. 2010;47(1):11–18. doi: 10.1007/s13312-010-0015-4. [DOI] [PubMed] [Google Scholar]
- 30.World Health Organization. WHO Pocket Book of Hospital Care for Children. 2nd. Geneva, Switzerland: WHO; 2013. [PubMed] [Google Scholar]
- 31.Leonard KL, Masatu MC, Vialou A. Getting doctors to do their best: the roles of ability and motivation in health care quality. J Hum Resour. 2007;42(3):682–700. [Google Scholar]
- 32.Banerjee A, Deaton A, Duflo E. Health care delivery in rural Rajasthan. Econ Polit Wkly. 2004;39(9):944–949. [Google Scholar]
- 33.Laxminarayan R, Duse A, Wattal C, et al. Antibiotic resistance: the need for global solutions. Lancet Infect Dis. 2013;13(12):1057–1098. doi: 10.1016/S1473-3099(13)70318-9. [DOI] [PubMed] [Google Scholar]
- 34.Kotwani A, Chaudhury RR, Holloway K. Antibiotic-prescribing practices of primary care prescribers for acute diarrhea in New Delhi, India. Value Health. 2012;15(1 suppl):S116–S119. doi: 10.1016/j.jval.2011.11.008. [DOI] [PubMed] [Google Scholar]
- 35.Arrow K. Uncertainty and the welfare economics of medical care. Am Econ Rev. 1963;53:941–973. [Google Scholar]
- 36.Das J. The quality of medical care in low-income countries: from providers to markets. PLoS Med. 2011;8(4):e1000432. doi: 10.1371/journal.pmed.1000432. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142(4):260–273. doi: 10.7326/0003-4819-142-4-200502150-00008. [DOI] [PubMed] [Google Scholar]
- 38.MAQARI Team. Mapping Medical Providers in Rural India: Four Key Trends. New Delhi, India: Centre for Policy Research; 2011. [Google Scholar]
- 39.Rao M, Rao KD, Kumar AKS, Chatterjee M, Sundararaman T. Human resources for health in India. Lancet. 2011;377(9765):587–598. doi: 10.1016/S0140-6736(10)61888-0. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.