Abstract
Functional health status is an important factor not only for determining overall health, but also for measuring risks of adverse events. Our hypothesis is that important functional status data is contained in clinical notes. We found that several categories of phrases related to functional status including diagnoses, activity and care assessments, physical exam, functional scores, assistive equipment, symptoms, and surgical history were important factors. Use of functional health status level terms from our chart review compared to National Surgical Quality Improvement Program determination had varying sensitivities for correct functional status category identification, with 96% for independent patients, 60% for partially dependent patients, and 44% for totally dependent patients. Inter-rater agreement assessing term relevance to functional health status was high at 91% (Kappa=0.74). Functional status-related terms in clinical notes show potential for use in future methodologies for automated detection of functional health status for quality improvement registries and other clinical assessments.
Introduction
Functional health status is often defined as one’s ability to perform daily activities required to meet basic needs, fulfill usual roles, and maintain their health and well-being1. It is increasingly recognized that a patient’s functional health status is important for determining overall general health and has been used as a factor to estimate preoperative risk of complications and adverse events2-5.Unfortunately, the measurement and documentation of functional health status is often not standardized particularly for front line clinical practice. Often, physicians and other clinicians use a combination of scoring systems, clinical judgement, and physical exam to determine a patient’s functional health status.
There are several tools created to determine a patient’s functional health status. These include tools like the Karnofsky Performance Scale, an observational method of functional health status determination using a 0-100 point scale6,7. Functional health status may also be determined through patient or caregiver-completed assessments such as the Patient-Reported Outcomes Measurement Information System (PROMIS)8. Alternatively, functional health status may be determined through calculations as a composite score. One example of this is the Frailty Index, which measures patient medical problems compared to an age appropriate list of medical problems9. Metabolic equivalents (METs) are also often used to measure functional capacity, a surrogate of functional health status10. There are also frameworks and guidelines created to standardize the determination of a patient’s functional health status, such as the World Health Organization’s International Classification of Functioning, Disability and Health11.
The American College of Surgeons National Surgical Quality Improvement Program (NSQIP) is a quality improvement registry which collects patient data to track post-operative outcomes and complications. This program offers high quality, risk adjusted data that is nationally validated. Data is collected manually by the surgical clinical reviewer (SCR), a trained nurse with specific training in the registry and its definitions12. Participation in this program has been shown to improve surgical outcomes and decrease post-operative morbidity12,13. Despite its usefulness, data collection remains labor intensive and participation is expensive14. With respect to patient functional health status, the NSQIP trained reviewers most often manually review charts to abstract a determination for the level of the patient’s functional health status placing patients into one of three categories (Table 1): independent, partially dependent, and totally dependent. An independent patient is defined as one who does not require assistance from another person for any activities of daily living, including one who functions independently with the use of prosthetics, equipment, and/or devices. A partially dependent patient requires some assistance from another person for activities of daily living regardless of use of prosthetics, equipment, and/or devices. Finally, a totally dependent patient requires total assistance for all activities of daily living. Functional health status in the NSQIP database is determined within 30 days prior to the operation and is highly correlated with post-operative outcomes15.
Table 1:
| Performance Score | Functional Status Performance |
|---|---|
| 100 | No complaints, no evidence of disease |
| 90 | Able to complete major activities; minor signs and symptoms of disease |
| 80 | Normal activity with effort; some signs and symptoms of disease |
| 70 | Care of self; unable to carry on normal activities or do active work |
| 60 | Requires occasional assistance; able to care for most of personal needs |
| 50 | Requires considerable assistance and frequent medical care |
| 40 | Disabled; requires special care and assistance |
| 30 | Severely disabled; hospital admission is indicated; death not imminent |
| 20 | Very sick; hospital admission necessary and active treatment necessary |
| 10 | Moribund; fatal processes progressing |
| 0 | Death |
| NSQIP functional status scale | |
| Independent | Does not require assistance from another person for any activities of daily |
| living, including one who functions independently with the use of | |
| prosthetics, equipment, and/or devices. | |
| Partially Dependent | Requires some assistance from another person for activities of daily living |
| regardless of use of prosthetics, equipment, and/or devices. | |
| Totally Dependent | Requires total assistance for all activities of daily living. |
By reviewing and categorizing terms and phrases in clinical notes associated with functional health status, it may be possible to improve automated abstraction efforts. Functional health status is often poorly defined, or difficult to define with individual patients between different providers and scoring systems are not uniformly used. As a first step toward understanding the value of the clinical notes for functional health status, we sought to develop a library of terms associated with functional status. More broadly, our hypothesis is that much of the data relating to functional health status would be found within free-text documentation in clinical notes.
Methods
Institutional Review Board approval was obtained for this study. The study was completed at the University of Minnesota Medical Center, an integrated health system in partnership with Fairview Health Services based in Minneapolis, serving the upper Midwest. We performed our initial search using the medical center’s Clinical Data Repository (CDR). Data in from the CDR is compiled from the electronic health records of more than two million patients from eight hospitals and forty clinics16. The CDR was queried for all patients included in the NSQIP database over three years (2013-2015). Of these patients, a stratified random convenience sample of 75 patients (twenty-five patients in each of the three functional health status categories as defined by NSQIP: “independent”, “partially dependent”, and “totally dependent”) were selected.
Physician reviewers (SS and EA) were blinded to the NSQIP functional health status determination for each patient. Reviewers examined all clinical notes and forms for the 30 days prior to the operative procedure for which functional health status was originally measured and determined. All phrases associated with functional health status were recorded. Details associated with the phrase, such as clinical note section, type of note, author credentials, and author specialty were also recorded.
After completing the chart review and recording all functional health status data, our reviewers assigned a NSQIP functional health status category and a Karnofsky Performance Score (Table 1) to each of the patient charts based only on functional health status terms recorded during the chart review. These scores were compared with the gold standard scores, which were the determinations previously made in the NSQIP registry by the SCR. To assess the inter-rater reliability of our functional health status determination of associated phrases, a subset of 8 overlapping patients (10.7%) was performed by both reviewers with percentage agreement and Kappa calculated. Statistical analyses were performed using R software (Vienna, Austria, 2017).
Results
A total of 75 patient charts from 2013-2015 were reviewed. In these charts, a total of 1,353 clinical notes were reviewed. Within these clinical notes, there were 1,328 phrases identified which were associated with the determination of a patient’s functional health status. There was a good variety of surgical specialties represented by the operations for which functional status was assessed in the NSQIP registry. Given the interest in functional health status for this study, specialties that operate on problems associated with low functional health status are more highly represented (i.e., neurosurgery, plastic surgery, colorectal surgery, urology, orthopedic surgery). Patient demographic and surgical information is summarized in Table 2.
Table 2:
Patient Demographics and Surgical Specialty
| Median Age (Range) | Male Patients n (%) | |
|---|---|---|
| All Patients | 51.5 (21-91) | 39 (52%) |
| NSQIP Functional Status | ||
| Independent | 48 (21-77) | 11(44%) |
| Partially Dependent | 59.8 (28-91) | 13 (52%) |
| Totally Dependent | 47.6 (22-75) | 15 (60%) |
| Surgical Specialty | Charts n (%) | |
| Urology | 13 (17.3%) | |
| Orthopedic Surgery | 11 (14.7%) | |
| Gynecologic Surgery | 10 (13.3%) | |
| Neurosurgery | 10 (13.3%) | |
| General Surgery | 8 (10.7%) | |
| Colorectal Surgery | 7 (9.3%) | |
| Plastic Surgery | 5 (6.7%) | |
| Vascular Surgery | 5 (6.7%) | |
| Bariatric Surgery | 3 (4%) | |
| Otolaryngology | 2 (2.7%) | |
| Thoracic Surgery | 2 (2.7%) | |
All of the 1,328 phrases were also annotated according to the type of clinical note in which they appeared as well as the clinical note section in which they appeared. These distinctions were evaluated separately. Breakdown of clinical note type and clinical note section can be found in Table 3. The “progress note” category within the clinical note type includes both daily progress notes written by physicians as well as progress and miscellaneous notes written by nursing and ancillary staff. In some cases, clinical note sections were not clearly delineated or present; these notes are included in the “not applicable” category. This study found that the majority of functional health status information was found in history & physical notes, anesthesia assessments, and office visits, which would be likely to have some component of assessment of patient functional status prior to an operation. The clinical note sections that featured the most functional health status data were history of present illness, assessment/plan, review of systems, and physical exam. All of these sections are likely to describe symptoms, major medical problems, and impairments that affect the patient.
Table 3:
Functional Health Status Phrases by Clinical Note Type, Note Section, and Author
| Clinical Note Type | Phrases n (%) |
|---|---|
| History & Physical | 440 (33.1%) |
| Anesthesia Pre-Operative Assessment | 338 (25.5%) |
| Office Visit | 237 (17.8%) |
| Progress Note | 160 (12.0%) |
| Consultation Note | 69 (5.2%) |
| Emergency Department Visit | 51 (3.8%) |
| Telephone Note | 23 (1.7%) |
| Operative Note | 8 (0.6%) |
| Discharge Summary | 2 (0.2%) |
| Clinical Note Section | |
| History of Present Illness | 327 (24.6%) |
| Not Applicable | 215 (16.2%) |
| Assessment/Plan | 199 (15.0%) |
| Review of Systems | 185 (13.9%) |
| Physical Exam | 156 (11.7%) |
| Past Medical History | 141 (10.6%) |
| Social History | 38 (2.9%) |
| Past Surgical History | 26(2.0%) |
| Chief Complaint | 18 (1.4%) |
| Form Elements | 14 (1.1%) |
| Operative Indications | 9 (0.7%) |
| Type of Author | |
| Staff Physician | 795 (59.9%) |
| Midlevel Provider | 182 (13.7%) |
| Resident or Fellow | 144 (10.8%) |
| Registered Nurse | 64 (4.8%) |
| Other | 30 (2.3%) |
| Physical Therapist | 28 (2.1%) |
| Wound Ostomy Continence Nurse | 26 (2.0%) |
| Occupational Therapist | 25 (1.9%) |
| Medical Assistant | 21 (1.6%) |
| Social Worker | 18 (1.4%) |
| Medical Student | 5 (0.4%) |
Functional health status-related phrases were recorded in the electronic medical record most frequently by physicians. An analysis of functional health status terms ordered by author role is shown in Table 3. The vast majority of functional health status phrases were recorded by providers (physicians, trainees, midlevel provider).
Author specialty was also recorded. Functional health status data was recorded most frequently by anesthesiologists (23.6% of phrases) and internists (19.7% of phrases). When combined, surgical specialties amounted to 21.7% of the phrases and medical specialties recorded 34.3% of the phrases. Nursing and ancillary staff accounted for 19.7% of functional health status phrases.
Phrases related to functional health status were categorized into seven major categories including: diagnosis, activity/care needs, physical exam elements, functional scores, assistive equipment, symptoms, and surgical history. The phrases are divided according to category and NSQIP functional status determination in Table 4.The amount and proportion of functional health status-related diagnoses increased with increasing level of dependence.
Table 4:
Categorized Phrases According to NSQIP Functional Category
| Phrase Category n (%) | Independent n (%) | Partial Dependence n (%) | Total Dependence n (%) | All Patients n (%) |
|---|---|---|---|---|
| Total Phrases | 209 | 607 | 512 | 1328 |
| Diagnosis | 23 (11.0%) | 194 (32.0%) | 255 (49.6%) | 472 (35.5%) |
| Activity/Care Needs | 61 (29.2%) | 161 (26.5%) | 77 (15.0%) | 297 (22.3%) |
| Physical Exam | 35 (16.7%) | 71 (11.7%) | 48 (9.3%) | 154 (11.6%) |
| Elements | ||||
| Functional Scores | 47 (22.5%) | 41 (6.8%) | 32 (6.2%) | 120 (9.0%) |
| Assistive Equipment | 7 (3.3%) | 65 (10.7%) | 38 (7.4%) | 110 (8.3%) |
| Symptoms | 23 (11.0%) | 45 (7.4%) | 27 (5.3%) | 95 (7.2%) |
| Surgical History | 13 (6.2%) | 30 (4.9%) | 37 (7.2%) | 80 (6.0%) |
Unique phrases and terms were isolated from all phrases that were recorded during the chart review process. There was a total of 47 unique diagnoses. Unique diagnoses are listed in Table 5. Some of the diagnoses had associated modifiers, which were usually markers of severity or location. For example, the diagnosis “multiple sclerosis” had modifiers “with worsening plaques”, “progressive”, and “unclear”. “Pressure ulcer” had modifiers: “sacral”, “gluteal”, “coccygeal”, and “Stage IV” found in the clinical notes.
Table 5:
Unique Diagnoses
| scoliosis | kyphosis | lumbar stenosis | subdural hemorrhage | chronic pain | neurogenic bladder | spinal cord injury | multiple sclerosis |
|---|---|---|---|---|---|---|---|
| menin-gioma | CNS lymphoma | hemiplegia | monoplegia | autism | learning disability | Mobius Syndrome | paraplegia |
| ulcer | neurogenic bowel | hip fracture | dementia | malnutrition | Spina Bifida | hydrocephalus | decubitus ulcer |
| paralysis | Polio | post-Polio syndromes | hyper-reflexia | spasticity | Lyme Disease | Chiari Malformation | weakness |
| spasmodic dysphonia | limb hypo-genesis | congenital deformity | radiculitis | Cauda Equina | stroke | critical limb ischemia | seizure |
| Alzheimer Disease | Parkinson Disease | mental retardation | Cerebral Palsy | cognitive defects | dysreflexia | developmental delay |
A large portion of patient activity level and care needs were unique; however, the themes of these phrases were similar. Usually, activities measured were similar but there were varying levels of dependence and assistance required between patients. Care needs varied slightly by level of care and nomenclature for facility type. Activity level was often judged based on several activities of daily living: ambulation/walking, eating/cooking, climbing stairs, transferring, toileting, bathing/hygiene, dressing, and exercise. General statements were sometimes made to summarize the patient’s level of activity, such as “not frail”, “good mobility”, or “low exercise capacity”. Care needs were relayed through assistive facility (nursing home, long term care facility, adult foster care, assisted living) or by the individuals helping with daily cares (husband caregiver, personal care assistant, home nurse, daily skilled nursing care).
Physical exam elements were classified into unique terms and phrases. There were 87 unique physical exam elements that could be further divided into three physical exam categories. There were 37 unique “motor/strength/sensation” items, 37 unique “general exam/appearance” items, and 13 unique “cognitive” items.
Functional status scoring systems were particularly useful for making a functional status determination, but were not present for all patients. These scoring systems included American Society of Anesthesiologists (ASA) Class, Karnofsky Functional Scale, Berg Balance Scale, METs estimation, the EQ-5D quality of life questionnaire, and the PROMIS questionnaire. METS and ASA were most prevalent in this chart review and frequently recorded by anesthesiologists, likely reflecting the peri-operative status of these patients.
Unique assistive equipment is recorded in Table 6. There were 31 unique equipment items/devices that were relevant to functional health status.
Table 6:
Unique Assistive Equipment
| spinal cord stimulator | indwelling Foley | wheelchair | motorized wheelchair | Baclofen pump | urinary catheters | home oxygen | lift chair |
|---|---|---|---|---|---|---|---|
| walker | bath bench | leg brace | foot brace | knee brace | torso brace | neck brace | crutches |
| scooter | intrathecal pump | home ramp | prosthesis | ostomy pouch | Jay cushion | Roho cushion | Hoyer lift |
| shower chair | cane | BiPAP | hospital bed | stretcher | grab bars | ventilator |
Twenty-eight unique symptoms were found in this study. Like diagnoses, modifiers of the symptoms typically highlighted symptom severity, frequency, and location. The unique symptoms related to functional health status determination are found in Table 7.
Table 7:
Unique Symptoms
| pain | weakness | shortness of breath | tingling | numbness | spasticity | fatigue | secretion problems |
|---|---|---|---|---|---|---|---|
| multiple falls | hematuria with cath | urinary incontinence | fecal incontinence | altered sensation | paresis | paresthesias | radiculopathy |
| worsening gait | swelling | urinary retention | neuropathy | worsening motor function | worsening neurologic status | seizures | unresponsive |
| combative behavior | constant movement | memory deficit | slurred speech |
Finally, there were terms related to surgical procedures that were helpful for functional status determinations. There were 22 unique phrases in the surgical history that helped with determination listed in Table 8.
Table 8:
Unique Surgical Terms
| epidural injection | below knee amputation | thoracic spine surgery | artificial urinary sphincter | suprapubic catheter placement | Mitrofanoff | ventriculo-peritoneal shunt | ventriculo-pleural shunt |
|---|---|---|---|---|---|---|---|
| neck fusion | above knee amputation | tracheostomy | colostomy | craniotomy | urinary diversion | nephrostomy tubes | percutaneous gastrostomy |
| ileal conduit | Monti | bladder augmentation | urostomy | disarticula-tion | gastro-jejunostomy |
Sensitivity was measured for comparing functional status designation using only functional status level phrases identified in the review and NSQIP surgical clinical reviewer determination. Sensitivity decreased as functional status complexity increased. A sensitivity of 96% was obtained for independent patient identification, 60% for identification of partially dependent patients, and 44% for totally dependent patients. Table 9 demonstrates the distribution of the designation of functional status based on the NSQIP SCR designation as the gold standard and functional status level phrases in separate human reviewer designation.
Table 9:
NSQIP Functional Status (Reviewer Designation versus Gold Standard)
p = 3.2201860600803914e-11
The gold standard functional health status designation was also compared against the Karnofsky Performance Scale as determined by independent review (Figure 1). Independent status had a maximum score of 100, median of 90, and minimum of 70 (interquartile range [IQR] 80-100). Partially dependent status had a maximum score of 100, median of 60, and a minimum of 40 (IQR 60-70). Totally dependent patients had a maximum score of 70, median of 50 and minimum of 30 (IQR 40-50).
Figure 1:
Karnofsky Score versus NSQIP Functional Status
Lastly, phrases were assessed for their relevance to functional health status by a second reviewer (EA). Ten percent of the phrases were used for assessment. Inter-rater reliability for determination if a phrase was important for functional health status was scored with an agreement of 90.7% and a Cohen’s kappa of 0.737.
Discussion
At present, functional health status data has not been well integrated into electronic medical records and most clinical workflows in practice. The combination of use of a multitude of functional health status assessments along with using clinical judgement and incomplete/inconsistent documentation make this integration challenging. The purpose of this study was to characterize signals for functional health status in clinical notes to attempt to organize and classify functional health status-related data. This study demonstrated that a variety of phrase categories can be helpful for determining functional health status including diagnoses related to functional health status, activity descriptions, home care needs, physical exam, functional scores, assistive equipment and medical devices, symptoms, and surgical procedures. Anesthesiologists and internal medicine physicians were the most frequent recorders of functional health status data in this study, likely because of the perioperative status of the patient.
There were a large number of unique terms for patient activity level and home care needs. The remaining categories had a relatively small number of unique phrases, but many modifiers related to severity, frequency, and location (particularly for diagnoses and symptoms).
We found that correctly identifying NSQIP functional health status category (gold-standard) with functional health status-related terms was more challenging in our chart review with patients of increasing functional health status complexity. As seen in Table 9, there was still correlation between outcomes, however, there was particularly increased variability in the designation for the “totally dependent” category. This could be because functional health status is inherently difficult to classify. Alternatively, it is often difficult to make a determination based on descriptions of a complex topic such as functional health status which can span many different observation categories, particularly when different providers’ descriptions may not align. There were two cases that were classified as “totally dependent” by the SCR, yet “independent” by human reviewer. Because of the wide discrepancy, these cases were re-reviewed and found to have conflicting data in clinical notes. In these patients, there were signals of total/partial dependency for functional health status in some notes, while other providers’ notes clearly stated that the patients were completely independent. This highlights the complexity of functional health status and limitations associated with designating functional health status retrospectively.
This study is limited in its retrospective nature. Also, there are likely additional factors that were involved in the determination of functional health status level by the physicians performing the assessment that were not entered in the electronic medical record. As such, determining functional status in a prospective manner in a standard and rigorous fashion is most ideal. Additionally, this is a single institution study with a study of relatively small sample size, and the description of functional health status may differ between institutions or even between surgical services. In our cohort, several author types (physical therapist, occupational therapists, and social workers) may have been particularly underrepresented. These author types wrote 1.2% of the total clinical notes reviewed, yet contributed to 5.3% of the functional health status-related phrases. It is likely that these authors have a larger impact on the functional health status descriptors than represented in this study. This study reviewed surgical patients, which may have contributed to the lack of documentation from these authors. Perhaps the reason for low percentages of these notes overall is that for cases where patients were totally independent or totally dependent (n=50), physical therapy, occupational therapy, and social work may have been minimally involved, since the supportive needs were either very low, or conversely, maximal and established. Finally, certain surgical services more involved in the care of lower functional status patients were likely over-represented.
Other methods have been used to attempt to characterize functional health status. Another retrospective review was performed which collected functional health status terms within the Veteran Affairs electronic medical record and patient-reported functional health status data on social media, concluding that standard terminologies such as Unified Medical Language System do not sufficiently cover functional health status information17. In a separate project, the same group used topic modeling as a method to extract relevant frailty information in clinical text18.A study by Ruggieri et al. found that functional health status terminologies centered around verb phrases, particularly descriptions of motion. A “frame-semantic” method was used for functional health status representation with a final goal of improving information abstraction and natural language applications of functional health status19.
Similar work characterizing clinical note data has been performed at our institution to develop an automated method of data abstraction for determination of surgical site infections and other complications20,21. Using keywords related to these complications improved the accuracy of the detection of these complications. We identified and categorized terms and phrases used in surgical site infection descriptions in a previous study22. Translating this method to this current project, the new terms identified in this study could be valuable for the automated detection of functional health status level. Additionally, these terms could potentially be incorporated into an automated approach to examine notes as they are being written to determine if they meet minimum documentation requirements for functional health status. We believe that expansion of our previous work in an analogous fashion can improve automated detection techniques for functional health status determinations.
Conclusion
Functional health status is a difficult clinical entity to quantify. Determination of level of functional status likely differs between providers, and while functional status scores are often helpful in this determination, they are not always documented. Factors in functional health status determination can be found in clinical notes through diagnoses, activity and home care descriptions, physical exam elements, functional scores, assistive equipment, symptoms, and surgical procedures. The phrases identified in this study could potentially be used to assist in automation of detection of functional health status level.
Acknowledgements
This research was supported by the University of Minnesota Academic Health Center Faculty Development Award, Agency for Healthcare Research and Quality (R01HS24532), National Institutes of Health (NIH) Clinical and Translational Science Award (CTSA) program (UL1TR000114), NIH/National Institute of General Medical Sciences (NIGMS) (R01GM120079), Fairview Health Services, and University of Minnesota Physicians.
References
- 1.Wilson IB, Cleary PD. Linking clinical variables with health-related quality of life. A conceptual model of patient outcomes. JAMA. 1995;273(1):59–65. [PubMed] [Google Scholar]
- 2.Huysmans HA, van Ark E. Predictors of perioperative mortality, morbidity and late quality of life in coronary bypass surgery. Eur Heart J. 1989;10(Suppl):H10–2. doi: 10.1093/eurheartj/10.suppl_h.10. [DOI] [PubMed] [Google Scholar]
- 3.Carmon E, Keidar A, Ravid A, Goldman G, Rabau M. The correlation between quality of life and functional outcome in ulcerative colitis patients after proctocolectomy ileal pouch anal anastomosis. Colorectal Dis. 2003;5(3):228–32. doi: 10.1046/j.1463-1318.2003.00445.x. [DOI] [PubMed] [Google Scholar]
- 4.Crawford RS, Cambria RP, Abularrage CJ, Conrad MF, Lancaster RT, Watkins MT, et al. Preoperative functional status predicts perioperative outcomes after infrainguinal bypass surgery. J Vasc Surg. 2010;51(2):351–8. doi: 10.1016/j.jvs.2009.08.065. discussion 8-9. [DOI] [PubMed] [Google Scholar]
- 5.Albright EL, Davenport DL, Roth JS. Preoperative functional health status impacts outcomes after ventral hernia repair. Am Surg. 2012;78(2):230–4. [PubMed] [Google Scholar]
- 6.KARNOFSKY DA. Chemotherapy of neoplastic disease; agents of clinical value. N Engl J Med. 1948;239(8):299–305. doi: 10.1056/NEJM194808192390805. [DOI] [PubMed] [Google Scholar]
- 7.Schag CC, Heinrich RL, Ganz PA. Karnofsky performance status revisited: reliability, validity, and guidelines. J Clin Oncol. 1984;2(3):187–93. doi: 10.1200/JCO.1984.2.3.187. [DOI] [PubMed] [Google Scholar]
- 8.Hays RD, Bjorner JB, Revicki DA, Spritzer KL, Cella D. Development of physical and mental health summary scores from the patient-reported outcomes measurement information system (PROMIS) global items. Qual Life Res. 2009;18(7):873–80. doi: 10.1007/s11136-009-9496-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Searle SD, Mitnitski A, Gahbauer EA, Gill TM, Rockwood K. A standard procedure for creating a frailty index. BMC Geriatr. 2008;8:24. doi: 10.1186/1471-2318-8-24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Jett M, Sidney K, Blümchen G. Metabolic equivalents (METS) in exercise testing, exercise prescription, and evaluation of functional capacity. Clin Cardiol. 1990;13(8):555–65. doi: 10.1002/clc.4960130809. [DOI] [PubMed] [Google Scholar]
- 11.Organization WH. How to use the ICF: A practical manual for using the International Classification of Functioning, Disability and Health (ICF) Exposure draft for comment. Geneva: WHO. 2013. Oct,
- 12.Ingraham AM, Richards KE, Hall BL, Ko CY. Quality improvement in surgery: the American College of Surgeons National Surgical Quality Improvement Program approach. Adv Surg. 2010;44:251–67. doi: 10.1016/j.yasu.2010.05.003. [DOI] [PubMed] [Google Scholar]
- 13.Hollenbeak CS, Boltz MM, Wang L, Schubart J, Ortenzi G, Zhu J, et al. Cost-effectiveness of the National Surgical Quality Improvement Program. Ann Surg. 2011;254(4):619–24. doi: 10.1097/sla.0b013e318230010a. [DOI] [PubMed] [Google Scholar]
- 14.Dimick JB, Chen SL, Taheri PA, Henderson WG, Khuri SF, Campbell DA. Hospital costs associated with surgical complications: a report from the private-sector National Surgical Quality Improvement Program. J Am Coll Surg. 2004;199(4):531–7. doi: 10.1016/j.jamcollsurg.2004.05.276. [DOI] [PubMed] [Google Scholar]
- 15.Surgeons ACo. ACS NSQIP Operations Manual. 2016.
- 16.Center UoMAH. Clinical and Translational Science Institute: Clinical Data Repository. 2017.
- 17.Kuang J, Mohanty AF, Rashmi VH, Weir CR, Bray BE, Zeng-Treitler Q. Representation of Functional Status Concepts from Clinical Documents and Social Media Sources by Standard Terminologies. AMIA Annu Symp Proc. 2015;2015:795–803. [PMC free article] [PubMed] [Google Scholar]
- 18.Shao Y, Mohanty AF, Ahmed A, Weir CR, Bray BE, Shah RU, et al. Identification and Use of Frailty Indicators from Text to Examine Associations with Clinical Outcomes Among Patients with Heart Failure. AMIA Annu Symp Proc. 2016;2016:1110–8. [PMC free article] [PubMed] [Google Scholar]
- 19.Ruggieri AP, Pakhomov SV, Chute CG. A corpus driven approach applying the “frame semantic” method for modeling functional status terminology. Stud Health Technol Inform. 2004;107(Pt 1):434–8. [PubMed] [Google Scholar]
- 20.Hu Z, Simon GJ, Arsoniadis EG, Wang Y, Kwaan MR, Melton GB. Automated Detection of Postoperative Surgical Site Infections Using Supervised Methods with Electronic Health Record Data. Stud Health Technol Inform. 2015;216:706–10. [PMC free article] [PubMed] [Google Scholar]
- 21.Hu Z, Melton GB, Moeller ND, Arsoniadis EG, Wang Y, Kwaan MR, et al. Accelerating Chart Review Using Automated Methods on Electronic Health Record Data for Postoperative Complications. AMIA Annu Symp Proc. 2016;2016:1822–31. [PMC free article] [PubMed] [Google Scholar]
- 22.Skube S, Hu Z, Arsoniadis E, Simon G, Wick E, Ko C, et al. Characterizing Signals for Surgical Site Infections in Clinical Notes. Studies in Health Technology and Informatics. 2017;5 [PMC free article] [PubMed] [Google Scholar]