The Accuracy of Data Collected by Surgical Residents

Abstract

BACKGROUND

Clinician records are the primary information source for assessing of the quality of facial injury care, billing, risk management, planning of health services, and health-system management and reporting. Inaccuracies obscure outcomes assessment and affect the planning of health services.

OBJECTIVES

We sought to determine the accuracy of the clinician collected data by comparing them to similar information elicited by professional interviewers.

METHODS

We abstracted admissions data from the medical records of 185 patients treated for orofacial injury between January 2005 and January 2007. Clinician data on sociodemographics and substance use was compared to similar information elicited by trained research staff as part of a prospective study.

RESULTS

The accuracy of the clinician data sets varied considerably depending on the variable. Concordance with the interviewer data sets was highest for age (paired t-test p=.09), sex (κ = 1) and ethnicity (κ = .84) but dropped off considerably for marital status (κ = .22) and alcohol (κ = .18) and drug use (κ = .16). The missing data per variable ranged from 4.5% (gender) to 46.9% (employment and education).

CONCLUSIONS

Although more research is needed to evaluate the cause of inaccuracies and the relative contributions of patient, provider, and system level effects, it appears that significant inaccuracies in administrative data are common. Interventions aimed at identifying the sources and correcting these errors are necessary.

Individuals with intentional orofacial injuries comprise a significant subset of the patients treated by the trauma services at public hospitals (Mathog et al. 2000¹; Hall & Ofodile, 1991², Leathers et al 1998³). Although surgical care has traditionally focused on the physical injury, increasing recognition of covert psychosocial antecedents and sequelae is generating a greater interest in a broader and more comprehensive management of the injured patients (Wong et al. 2007⁴; Glynn et al. 2007⁵; Shetty et al. 2003⁶; Shephard 1992⁷). However, the expansion of resource-intensive trauma care (Abubaker & Lynam, 1998)⁸ to include needed psychosocial screenings and interventions is complicated by the growing financial constraints on our trauma centers. Decreasing reimbursements for trauma care and a rising expectation of evidence-based practice generate urgency for surveillance systems for cataloguing care information, monitoring outcomes and evaluating clinical effectiveness of different management strategies. Beyond shaping the delivery of quality, cost-effective care to injured patients, such trauma registries are necessary for tracking injury patterns in specific communities, establishing program priorities and evaluating health expenditures (Shetty et al. 2007⁹).

The clinician-generated patient record is still the primary information source for assessing and tracing quality of care and outcomes, billing, risk management, planning of health services, health-system management and reporting (Peabody et al. 2004¹⁰). Registries linking trauma-related data across systems derive primarily from clinician collection of patient and injury-specific information. Implicit to the use of such surveillance systems is the assumption that the data collected fulfill certain criteria in terms of completeness, reliability, and validity. This is true of both prospective and retrospective audit. Correspondingly, the accuracy of the information collected by clinicians is essential. Inaccurate patient records can lead to treatment errors, inadequate patient services, ineffective outcome assessments, contradictory treatment recommendations for similar conditions, and a masking of the true costs of providing care. Consistent documentation is also necessary for coordinating trauma care between various specialists and comparing outcomes across multiple sites.

Unfortunately, there is reason to be concerned about the accuracy of clinician documentation. Recent studies have indicated that clinician-related factors such as inter-rater reliability, interview quality, and rater bias, can significantly impact signal detection and study outcome in clinical trials (Kobak et al. 2007¹¹). In a study of the accuracy of medical records for hip fracture patients, Fox et al. (1998)¹² found discrepancies between the hospital face sheet and the medical record and between the abstracted surgical procedure and radiographs, thus, questioning the soundness of study conclusions relying on hospital chart data or administrative healthcare databases that utilize the hospital face sheet data. Peabody et al (2004¹⁰) investigated the accuracy of primary and secondary diagnoses recorded in administrative data sets by utilizing standardized patients to complete unannounced visits across three sites. Subsequent analysis of the medical records from these visits revealed that the correct primary diagnosis was recorded for only 57% of the visits; 13% of errors were caused by physician diagnostic error; 8% to missing encounter forms; and 22% to incorrectly entered data.

Despite the large body of literature devoted to the epidemiology and management of facial injuries, it is surprising that little attention has been devoted to measuring the accuracy of the data on which these reports and treatment recommendations are based. An ongoing, prospective study of facial injury patients provided us the opportunity to verify the validity of using actual patient records as a data source. By comparing basic sociodemographic and administrative data collected by the clinicians to that elicited by trained research staff, we investigated the reliability of data collected by clinicians in naturalistic settings and clarified the sources of data inaccuracy and any loss of information.

Methods

Study Setting and Participants

This study was conducted at a major urban trauma center in Los Angeles serving a large socioeconomically disadvantaged population. Care for patients presenting with orofacial injury, particularly mandible fractures, is provided primarily by residents from the oral and maxillofacial surgery (OMS) service working under the supervision of attending surgeons. As part of the admissions work-up, the surgical residents elicit and record administrative and injury information utilizing standardized, trauma registry questionnaires incorporating check-list fields. All adult (18 years or older) patients who presented with intentional facial injury to the OMS service between January 2005 and January 2007 were approached about participation in a randomized, controlled trial of a culturally competent motivational intervention focusing on antecedent substance use behaviors. Patients were considered eligible for recruitment if, within the past 30 days, they had recently used alcohol or drugs, and had at least one fracture due to interpersonal violence and involving the jaws, orbit, nose or cheekbone, as determined by clinical history, examination and radiographic findings. Patients with severe gunshot injuries, or altered mental status attributable to head injuries, or who were incarcerated or in treatment for mental illnesses were excluded, as were patients who were unable or unwilling to participate in motivational interviewing or return for follow-up care. Of 212 patients eligible for participation and approached in the clinic, 13% (n = 27) declined to participate. Primary reasons for refusals were facial pain, too busy due to dealing with the combination of work and injury treatment, not wanting to travel the distance to thee clinic again, and not wanting to deal with anything more than the injury itself.

Study Protocol

Subsequent to the admissions work-up by the surgical residents, eligible and consenting patients were interviewed by trained bilingual (English-Spanish) interviewers using structured questionnaires and interviewing approaches approved by the local Institutional Review Boards. The research staff used face-to-face interviews to collect extensive baseline information on sociodemographic characteristics and various psychosocial measures. Interviews averaged one hour in length and were conducted in a private exam room or office. The staff were trained by the project director and routinely monitored to assure the consistency and quality of the interviewing process. Although both provider and interviewer questionnaires included questions related to patient sociodemographics and substance use behaviors, the form of the questions varied. Data from the completed interviews as well as data abstracted from medical charts was entered into a computer and compiled into a database.

Measures

Patient Sociodemographics

The sociodemographic information collected by the residents derived from the administrative data sets utilized by the hospital financial system. Research interviewers utilized a demographic questionnaire developed by the Integrated Substance Abuse Program (ISAP) at UCLA. Variables on which data was collected included patient gender, country and date of birth, ethnic background, first language, marital status, children raised by patient, children currently living with patient, family history of substance use problems, time in jail/prison, time in homeless or emergency shelter, occupation, employment status, hours of work, education and religion or legal problems. The ISAP measures are particularly salient because they correspond closely to standard sociodemographics information collected in other nationwide surveys, thus allowing comparison of data sets across sites. The administrative dataset collected by the residents included the information on injury type and severity that was needed to determine eligibility for the study and could only be supplied by examining clinicians. In addition, the residents were asked to record, on the same admission form they used to record injury type and severity, basic sociodemographic information about each patient at the time of admission. This included date of birth, ethnicity, gender, education level, employment status, and use of alcohol or drugs. The data collected by the interviewers included these basic questions, but also provided an opportunity to follow-up and explore behavioral and social issues in depth via standard scales and indices during the hour that interviewers spent with each patient. The research interviewers were both linguistically competent in both English and Spanish, and primarily Latino. Surgical residents tended to be of a different ethnic and cultural background than most patients and were less likely to speak Spanish.

Substance use

The residents elicited drug use behaviors by directly asking the subjects if they used drugs and the specifics of the use. Additionally, they screened for alcohol use and if the patients responded positively, administered the CAGE questionnaire (Ewing 1984¹³) to ascertain the degree of alcohol use. The interviewers obtained information on drug use utilizing the Texas Christian University Drug Screen (TCUDS: Knight et al., 2000¹⁴), a 12- item index of severity of drug abuse. Developed as a short instrument for use with low-income, ethnic minority men who have contact with the criminal justice system, the TCUDS was evaluated on 18,364 Texas Department of Criminal Justice inmates (Knight et al., 2002¹⁵). TCUDS has a test-retest reliability of .95 (Peters et al., 2000¹⁶) and an internal reliability of .89 (Knight et al., 2002¹⁵). For our cohort of patients with orofacial injury, TCUDS had an internal consistency with an alpha = .85. In addition to the TCUDS, those patients reporting use of illicit drugs also provided a complete history of each drug used, age when regular use began, most recent use, and use of the drug in past 30 days.

Data analysis

Because the reasons and process of collecting the patient information differed between the clinicians and the interviewers, the coding of the information differed in the two settings. In order to make useful comparisons, demographic summaries such as ethnicity, level of education, marital status, and employment were collapsed to produce similar response categories in the two data sources (See Table 1). Ethnicity categories such as “other” and “multi-ethnic” were excluded, as the inherent ambiguity of such categorizations does not lend to useful concordance analysis. All discrepancies between the two data sources were checked individually, and those determined to have resulted from data processing errors were corrected. Thus, the reported disagreements between clinician and interviewer-collected data reflect the actual data collection in the two settings.

Table 1.

Coding of fields

Interviewer collected	Coding used for analysis	Clinician collected
Ethnicity
White Non-Latino	White	White
Black/African-American	Black	Black
Mexican/Mexican-American	Latino	Latino
Other Latino	Latino	Latino
Asian-Pacific Islander	Asian-Pacific Islander	Asian-Pacific Islander
Native American/Alaskan	Native American	Native American
Other	Other	Other
Multi-Ethnic	Other	Other
Education
Less than HS	Less than HS	8th Grade
Less than HS	Less than HS	Some HS
HS/GED	HS or More	HS Grad
Trade or Technical	HS or More	Trade School
Some College	HS or More	Some College
2 yr College (AA)	HS or More	College Grad
4 yr College (BA, BS)	HS or More	College Grad
Masters	HS or More	College Grad
PhD or similar	HS or More	College Grad
Other	HS or More	College Grad
Employment
Currently Employed	Employed	Part Time
Currently Employed	Employed	Full Time
Not Currently Employed	Unemployed	Disabled
Not Currently Employed	Unemployed	Retired
Not Currently Employed	Unemployed	Homemaker
Not Currently Employed	Unemployed	Unemployed
Marital Status
Widowed	Single	Single
Separated	Single	Separated
Divorced	Single	Divorced
Never Married	Single	Windowed
Married	Married/Live With	Married
Living With	Married/Live With	Married

For purposes of analysis, the detailed data collected by the interviewers was considered the “criterion standard” or “correct” data. To understand the basis for discordant data, we examined those cases where the subject interview and the provider report disagreed and measured the percent of time the provider interview agreed and disagreed with the subject interview. A lower agreement implies that the surgical resident obtained a different history than the research interviewer. The concordance between the two data sources was summarized with Cohen’s Kappa (κ) statistic, a standard measure of inter-rater reliability measuring agreement while accounting for simple chance. The only continuous measure, patient age, was compared with a paired t-test. All statistical analysis was performed using commercial software (SPSS - version 10, SPSS, Inc., Chicago, IL).

Results

Of the 185 patients interviewed by the research staff, the residents filled out the admissions form for only 179 patients. Hence, the concordance analysis was limited to this study population of 179 subjects. The average age of the subjects was 31 years in both the clinician and interviewer collected data sets (p = .088), with slight differences attributable to the time lag between hospital admission and interview. The data collected by the residents showed varying degrees of completeness. Missing responses varied from as little as 4.5% (patient gender) to as much as 46.9% (employment and education). Table 2 summarizes the percent of missing data for each variable as well as the concordance of the data gathered by the clinician with the information elicited by the interviewers. Depending upon the variable, the concordance between the clinicians and the interviewers ranged between 16% and 100 %. In terms of the gender variable, the clinician-collected data classified 100% (155/155) of the men correctly, and 100% (16/16) of the women correctly. For ethnicity categories, the percentage of patients correctly identified by the residents ranged from 33% (1/3) for Native Americans to 96% (74/77) for Latino, yielding a kappa of .84. For the education variable, the clinicians correctly elicited that 61% (19/31) of patients had less than a high school education, while 67% (43/64) were correctly identified as having at least a high school education, with a kappa of .26. Employment status yielded a kappa of .40, with 68% (43/63) correctly identified as unemployed and 75% (24/32) identified as employed. The kappa for marital status was .22 with 90% of the respondents (71/79) correctly identified as single, and 29% (7/24) identified as married or living with a partner. The residents correctly identified alcohol use in 58% (71/122) of patients, while lack of alcohol use was identified in 74% (17/23), with a kappa of .18. Similarly, the residents correctly identified drug use in 35% (19/54) patients, while 82% (49/60) of patients were correctly classified as non-users. The kappa for drug use was .16. Generally, the residents were reliable recorders of patient age and gender, with ethnicity information agreeing moderately well with that elicited by the interviewers. Other data items relating to drug use/alcohol use were not reliably recorded; residents documented patient information on alcohol use in only 81% of the cohort and drug use in only 63.7%.

Table 2.

Concordance between clinicians and interviewers

Variable	No. Patients per Interviewers	No. Patients per Clinicians	% Agreementa	Kappaa	% Missing
Sex			100	1	4.5
Male	162	155
Female	17	16
Ethnicityb			90.2	0.839	10.1
White	19	30
Black	48	42
Latino	84	83
Asian/Pacific Islander	4	3
Native American	3	2
Education			65.3	0.265	46.9
< HS	70	40
≥HS	109	55
Employment			70.5	0.396	46.9
Employed	67	44
Unemployed	111	51
Marital Status			75.7	0.219	42.5
Single	142	88
Married/Live With	37	15
Alcohol use			60.7	0.177	19.0
Yes	152	77
No	27	68
Drug use			59.3	0.161	36.3
Yes	82	30
No	97	84

^aCalculated using only cases with entries from both data sources.

^bExcludes patients classified as “other” or “multi-ethnic”.

Discussion

Our study is one of the first to investigate the accuracy of patient data collected by surgical residents involved in care of patients with facial injury. Our findings manifest some of the limitations of clinical records as a tool for documenting patient information, supporting patient care, conducting research, and ultimately, framing secondary interventions and health policy decisions. We found that the patient information collected by the surgical residents underreported alcohol and substance using behaviors. Equally low was the correspondence of clinician-elicited information with interviewer data on sociodemographic variables including education, employment and marital status. Clinician-elicited data reflected interviewer data only moderately well for patient ethnicity. Furthermore, the low level of concordance was compounded by the significant proportion of missing data relative to basic administrative information such as gender, age and ethnicity. Our findings question the common presumption that administrative data (e.g., age, gender, dates of admission and discharge) contained within medical records are the more reliable and accurate components of the medical record (Iezzoni, et al., 1988¹⁷; Institute of Medicine, 1977¹⁸).

The incompleteness and low accuracy of the data collected by the clinicians in our study may be attributed to several factors. Although the constraints of an acute care setting can pose challenges to data collection, it is more likely that the missing data reflected attitudinal differences between the clinicians and interviewers. Restitution of the physical injury is the primary focus of the surgical residents; they may have varying degrees of interest, indeed ownership, of any research protocol or administrative data beyond that related to the physical injury. Correspondingly, the filling out of data forms and the collection of seemingly unrelated patient demographic and administrative information may be regarded as a comparatively unimportant or superfluous issue. Additional factors influencing documentation habits may include the resident’s workload, time of patient encounter, and prior training in documentation for research studies. The variable type may play a role in how consistently information is collected by surgical residents under various pressures to accomplish this quickly. As reflected by the low concordance levels associated with the ethnicity, marital status and employment variables, the greater the number of potential responses, the greater the probability of mismatch between the information collected by the clinician and interviewers.

Of particular interest was the very low concordance of the clinician-elicited information on substance use behaviors. In the alien environment of the trauma center, patient responses are likely to be clouded by their perspective of whether they are talking to a clinician or a staff member. Vulnerable patients presenting with intentional injury, particularly those worried about the criminal justice system, are less likely to provide spontaneous information about substance use behaviors. Clinicians may falsely assume that patients will report all of their substance use behaviors when asked and do not prompt for more detail. In contrast to the surgical residents, the interviewers were able to get more textured information on substance use behaviors from the same set of patients. For one, the interviewers were trained to conduct psychosocial type of interviewing. Furthermore, research questionnaires probing substance use typically begin with non-threatening, general questions to allow a patient to get into the rhythm of answering the questions and establishing a relationship with the interviewer, before more sensitive questions (e.g., alcohol and drug use) are asked. Therefore, the structured interviews allow an opportunity to establish a rapport and obtain information on substance using behaviors that are not always divulged in the brief encounter with the surgical resident. Ethnic and cultural congruency also play a particular role in extracting sensitive information related to risky behaviors. Survey researchers (Reese et al, 1986¹⁹) have clearly demonstrated that the ethnicity of the interviewer can exert a significant “interviewer effect” among ethnic minorities. This finding has particular relevance to the quality of data collected by residents at inner-city trauma centers inasmuch as the residents may lack the ethnic background or the linguistic competencies to elicit more sensitive information. Particularly in the case of trauma centers serving predominantly minority populations, it seems reasonable to consider whether the selection of surgical residents should be based, in part, on an assessment of the applicants’ linguistic and cultural competencies. Training incoming residents in culturally competent interviewing techniques could partly alleviate some of the limitations of acute trauma care which do not allow the time required for effective patient-clinician interactions and trust to develop.

Although these discrepancies in documentation are not surprising to those who work in the acute care setting, they do raise several concerns. Clinical records are implicitly considered accurate representations of the state of the patient and the events occurring during the process of care. Beyond serving as the basis for patient care and informed decision making, patient records assist evaluations of various treatment approaches and allocation of scarce healthcare resources. Lack of valid and accurate documentation complicates each of these necessary activities and has serious implications for providers, as well as hospital administrators and policymakers. The low documentation and poor accuracy in documenting alcohol and drug use has particular ramifications on the comprehensive care of the injured patient. A growing body of evidence (Madan, Yu, & Beech, 1999²⁰; Mathog et al., 2000¹; Murphy, Shetty, Resell, Zigler, & Yamashita, in press²¹; Soderstrom et al., 1992²²; Vinson et al., 2003²³) implicates substance use behaviors with intentional injury and recidivism. Alcohol and drug use not only increase vulnerability to injury, but also complicate trauma care and recovery. The clinical impression of substance use is a frequent reason for oral and maxillofacial surgeons to choose the more involved and less fallible rigid fixation techniques over the less expensive maxillomandibular fixation (Shetty et al. 2003²⁴). In a five-year follow-up of 501 survivors of violence related injury, Sims et al.²⁵ found that 62% abused alcohol or drugs; thus, the substance use problems continued long-term. Given the substantive linkages between substance use and injury and reinjury, numerous investigators and national agencies have argued for treating trauma center admissions as a secondary symptom of an underlying substance use problem and for developing secondary prevention strategies that focus on the causal substance use behaviors (Semltzer & Redeker, 1992²⁶; Zatzick et al., 2004²⁷). Integrating screening of causal substance use problems sets the stage for comprehensive care of the injured patient and minimizes the risk of recidivism due to continued risk taking behaviors. Absent accurate identification of underlying substance use behaviors by the treating oral and maxillofacial surgeon, patients would not receive appropriate referrals for treatment of these antecedent risk factors or help with attendant psychosocial problems (Shetty et al. 2003⁶).

So what can be done to enhance the reliability and completeness of data collected on patients with facial injuries? The challenge is to create questionnaires and data collection systems that are easy to use while at the same time collecting complete and appropriate set of data for subsequent audit and analysis. Variables should be incorporated only if the information produced is relevant to an important clinical or service issue data and are actually used to implement change. Efforts to improve document accuracy and thoroughness may occur at the individual or clinic level. Individual level interventions to standardize documentation could include computer assisted data collection methods that incorporate logic checks and force residents to collect a particular data set before they can proceed to the next set, enforcing more standardized documentation through use of checklists and defined taxonomies (e.g., UCLA MISS/FLOTID⁹) instead of free comments, and consolidation of the variable to a bare minimum (e. g., unemployed or employed instead of determining the various levels of employment). Easily used and validated instruments for screening alcohol (e.g., AUDIT²⁸, CAGE¹³ or RAPS4²⁹), drug use (e.g., ASI³⁰; NIDA Risk Behavior Assessment³¹; NHI^32,33) and psychological issues (e.g., BSI³⁴ CESD³⁵; SCID³⁶) should become part of the patient record and residents must be trained in the administration of these questionnaires. Clinic wide interventions can include periodic audits to determine whether documentation is worse with certain providers, so that targeted efforts can be implemented to improve documentation practices. Regular presentation of the administrative audits is useful to show the relationships between the data collection activities and tracking the quality of care. Allowing the clinicians to relate the isolated data bits to the larger clinical picture gives meaning to the tasks of data collection and will makes it easier for them to implement the changes necessary to improve data quality.

The deficiencies associated with the systematic collection of accurate administrative data in our cohort of facial injury patients are illustrative of the quality problems that plague our entire trauma care system. The collection of valid and reliable patient information is a necessary first step toward improvement in the care of the injured patient. Only by improving the accuracy and thoroughness of the medical record will we be able to identify the need for and provide the comprehensive treatment required by many of our injured patients, recognize true differences in outcomes and minimize the wasting of health care resources responding to variations introduced by inaccurate patient data. The growing emphasis on comprehensive care of the injured patient and evidence based practice coupled with the diminishing health care resources have greatly increased the stakes for improving the accuracy of documentation. Failure to address this fundamental aspect of trauma care has critical implications for trauma centers and health care providers, both financially and in terms of credibility with funding agencies.