Table 6.
Error analysis of the annotation with disagreements
| Error | Dimension | Examples (Sentences) |
|---|---|---|
| Abbreviated terms | Biomedical & Procedure | "Events that occurred during follow-up were identified from hospitalization records, and ARIC and CHS study" (PMID25104519) |
| Standard codes | "Finally, the Apollo Data Repository provided data for ICDs" (PMID26961369) | |
| Medications | " ‘‘common’’ side effects, e.g. headache, to judge the relevance of side effects associated with AZA”. (PMID24177317) | |
| Use of NLP | "From this cohort, we identified 15,761 patients with HPI that was processed through a natural language processing algorithm…” (PMID25567824) | |
| Data | “Cohort with HPI data” (PMID25567824) | |
| EXC1 – irrelevant evidence | "190 patients completed the SCID assessment"(PMID25827034) | |
| EXC2 – Computational and statistical evidence | "The MCMC method" (PMID21931496) | |
| Missed keywords or criteria | Use of NLP | "The algorithm uses non-negated terms indicative of HF" (PMID17567225) |
| Data | "If data on weight and height were available” (PMID21862746) | |
| EXC1 – irrelevant evidence (financial) | "until termination of insurance coverage”. (PMID12952547) | |
| EXC1 – irrelevant evidence (ethical) | "To protect patient confidentiality, all personal identifiers are deleted” (PMID21051745) | |
| EXC1 – irrelevant evidence (location of the study) | "We randomly sampled outpatient clinical encounters from October 1, 2003 through March 31, 2004 at VA Maryland (VAMHCS) and at VA Salt Lake City (VASLCHCS) Health Care systems”. (PMID20976281) | |
| EXC2 – Computational and statistical evidence | "Characteristics were measured during the one-year baseline period (i.e., before time zero)”. (PMID20112435) | |
| Without co-occurrence with a biomedical, procedure, or medication terms | Use of NLP | "Humedica derives NLP items from text entries that correspond primarily to terms in two large dictionaries, SNOMED and MedDRA" (PMID26725697) |
| Data | "If the first record for a woman was either …" (PMID22071529) | |
| Term ambiguity | Biomedical & procedures events | "Only acute conditions occurring during the first 24 h of hospital admission were considered”. (PMID24734124) |
| Study design or IRB | "The nucleotide reference for this allele is guanine. 4″. (PMID26221186) | |
| EXC2 – Computational and statistical evidence | "More points mean a higher risk of hyperkalemia”. (PMID20112435) | |
| Neither biomedical nor procedure (e.g. Social status) | Biomedical & Procedure | "We created a binary variable for marital status, where “single” included those patients classified as divorced, single, widowed, or separated”. (PMID25091637) |
| A not clear statement of using standard codes | Standard codes | "Outcomes were evaluated by administratively coded data” (PMID26370823) |
| Assigning terms as biomedical & procedure vs. medications (e.g. substances) | Biomedical & Procedure/Medications | "The most recent fasting lipid profile in patients with dyslipidemia and glycosylated hemoglobin level in patients with diabetes” (PMID11388131) |
| Spelling and short forms | Medications | "Asthma meds refilled regularly”. (PMID12952547) |
| Without co-occurrence with supportive definition evidence | Biomedical & Procedure/Medications | "reports KD = 9100 for bupropion and KD > 10 000 for mirtazapine (vs 200 for nefazodone)”. (PMID22466034) |
| “More than or less than” value, but not directly relevant to phenotyping | Clinical measurable values | " ≥ 2 years of observation before the period of interest; n = 50″. (PMID23449283) |
| New keywords for the dimension | EXC2 – Computational and statistical evidence | Examples of new keywords describing “EXC2” are: risk score, inter-rater variability, custom-designed data entry template, predictor variable, Tukey multiple comparison test, Web-accessible, teleconferences, propensity-matched, machine-implementable rule, Illumina Omni1_- QUAD, Illumina 660 W, TaqMan, Illumina 660-Quad, and Illumina |