Abstract
BACKGROUND
Research and surveillance work addressing ectopic pregnancy often rely on diagnosis and procedure codes available from automated data sources. However, the use of these codes may result in misclassification of cases. Our aims were to evaluate the accuracy of standard ectopic pregnancy codes; and, through the use of additional automated data, to develop and validate a classification algorithm that could potentially improve the accuracy of ectopic pregnancy case identification.
METHODS
Using automated databases from two US managed-care plans, Group Health Cooperative (GH) and Kaiser Permanente Colorado (KPCO), we sampled women aged 15–44 with an ectopic pregnancy diagnosis or procedure code from 2001 to 2007 and verified their true case status through medical record review. We calculated positive predictive values (PPV) for code-selected cases compared with true cases at both sites. Using additional variables from the automated databases and classification and regression tree (CART) analysis, we developed a case-finding algorithm at GH (n = 280), which was validated at KPCO (n = 500).
RESULTS
Compared with true cases, the PPV of code-selected cases was 68 and 81% at GH and KPCO, respectively. The case-finding algorithm identified three predictors: ≥2 visits with an ectopic pregnancy code within 180 days; International Classification of Diseases, 9th Revision, Clinical Modification codes for tubal pregnancy; and methotrexate treatment. Relative to true cases, performance measures for the development and validation sets, respectively, were: 93 and 95% sensitivity; 81 and 81% specificity; 91 and 96% PPV; 84 and 79% negative predictive value. Misclassification proportions were 32% in the development set and 19% in the validation set when using standard codes; they were 11 and 8%, respectively, when using the algorithm.
CONCLUSIONS
The ectopic pregnancy algorithm improved case-finding accuracy over use of standard codes alone and generalized well to a second site. When using administrative data to select potential ectopic pregnancy cases, additional widely available automated health plan data offer the potential to improve case identification.
Keywords: ectopic pregnancy, algorithms, classification trees, information systems, sensitivity and specificity
Introduction
Ectopic pregnancy is the implantation of an embryo outside the uterine corpus. It is a potentially serious acute medical condition that can lead to substantial future reproductive morbidity, including subsequent ectopic pregnancy and infertility (Barnhart, 2009).
The public health burden of ectopic pregnancy in the USA has remained largely unevaluated for nearly 20 years. Major shifts in practice and treatment patterns, principally more use of medical treatment and care in ambulatory settings, have impaired the accuracy of the hospital discharge-based national surveillance sources (Centers for Disease Control and Prevention, 1995; Zane et al., 2002; Hajenius et al., 2007). The defined populations and electronic databases available in many US health plans and in other countries can help address this information gap and contribute to epidemiological research and surveillance. However, a remaining limitation is that reliance on the cases identified using standard diagnosis and procedure codes is likely to incorporate error. For example, these same codes are often applied to multiple types of utilization aside from diagnosis, including rule outs, follow up or when noting a history of a disease outcome or condition (Tu et al., 2007; Lix et al., 2008; Hoover et al., 2010; Yu et al., 2010). Undertaking medical record review to validate cases is labor-intensive and costly and, in some instances, medical records are not available. In many healthcare settings, the increasing availability of more detailed automated information on treatment, procedures, utilization and other aspects of care associated with a diagnosis may provide opportunities to improve the accuracy of ectopic pregnancy case identification.
The aims of this study were to evaluate the predictive value of ectopic pregnancy diagnosis and procedure codes available from automated health plan databases, and, through the use of widely available additional data elements, to develop and validate a case identification algorithm that could improve the accuracy of ectopic pregnancy case selection from automated data.
Materials and Methods
Study settings
This study was conducted at Group Health Cooperative (GH), headquartered in Seattle, Washington and at Kaiser Permanente Colorado (KPCO), headquartered in Denver, Colorado. Both US health plans are integrated delivery systems with multiple electronic databases, including an electronic medical record (EMR), that document enrollees’ health care. These two plans also are members of the HMO Research Network (HMORN), a collaborative network of 15 US health care plans that provides care to ∼13 million enrollees nationwide. The HMORN has developed shared resources to be used for research and clinical care improvement, including administrative data files constructed at each HMORN site using standardized data dictionaries and shared data extraction protocols (Platt et al., 2001; Vogt et al., 2004; Go et al., 2008).
All study procedures received human subjects review and approval at each institution.
Selection of potential ectopic pregnancy cases
Initially, from a pool of ectopic pregnancy cases identified at GH for an earlier investigation (Trabert et al., 2011) and based on funds available, we randomly selected 280 potential ectopic pregnancy episodes occurring in women aged 15–44 years, selected for the years 2001–2006. Diagnoses and procedures were captured from the inpatient, outpatient and external claims databases. A potential ectopic pregnancy visit was defined as one or more of the following: an outpatient clinical visit record or hospitalization discharge with an International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) diagnosis code of 633.x; a hospitalization with an ICD-9-CM procedure code for removal of ectopic fetus (66.0x, 66.62 or 74.3); or an outpatient visit with a Current Procedural Terminology-4 (CPT-4) code indicating ectopic pregnancy surgical treatment (59120, 59121, 59130, 59135, 59136, 59140, 59150, 59151) (Table I). Multiple ectopic pregnancy-coded visits within a 180-day period were considered part of the same episode. The diagnosis date was defined as the first visit date with relevant diagnosis/procedure codes in the 180-day episode window. Subsequently, again based on available funding, a random sample of 500 potential ectopic pregnancy cases occurring from 2002 to 2007 in the KPCO population was identified by applying the GH programs and using the HMORN-shared data dictionaries.
Table I.
ICD-9-CM and CPT codes used to identify potential ectopic pregnancy cases.
| Selection codes | Description |
|---|---|
| ICD-9-CM diagnosis codes | |
| 633 | Ectopic pregnancy |
| 633.0x | Abdominal ectopic pregnancy |
| 633.1x | Tubal ectopic pregnancy |
| 633.2x | Ovarian ectopic pregnancy |
| 633.8x | Other ectopic pregnancy |
| 633.9x | Unspecified ectopic pregnancy |
| ICD-9-CM procedure codes (inpatient) | |
| 66.0x | Salpingotomy and salpingostomy |
| 66.62 | Salpingectomy with removal of tubal pregnancy |
| 74.3 | Removal of extratubal ectopic pregnancy |
| CPT-4 procedure codes (outpatient) | |
| 59120 | Surgical treatment of ectopic, tubal or ovarian, with abdominal salpingectomy and/or oophorectomy |
| 59121 | Surgical treatment of ectopic, tubal or ovarian, without abdominal salpingectomy and/or oophorectomy |
| 59130 | Surgical treatment of abdominal ectopic pregnancy |
| 59135 | Surgical treatment of interstitial uterine ectopic pregnancy requiring total hysterectomy |
| 59136 | Surgical treatment of interstitial uterine ectopic pregnancy with partial resection of uterus |
| 59140 | Surgical treatment of cervical ectopic pregnancy with vaginal approach |
| 59150 | Laparoscopic treatment of ectopic pregnancy without salpingectomy and/or oophorectomy |
| 59151 | Laparoscopic treatment of ectopic pregnancy with salpingectomy and/or oophorectomy |
Data collection
Using the health plan EMRs, we conducted chart reviews to verify case status using a brief structured abstract form and trained abstractors. We collected information on the diagnosis at the index date. If the episode was determined to be ectopic pregnancy, we collected additional information on treatment and sites of clinical care. If the potential case was not ectopic pregnancy, information on the diagnosis at the index date was recorded along with why the ectopic code was present.
At GH, we identified potential predictors of ectopic pregnancy from automated data for the classification algorithm development. For each potential ectopic pregnancy episode, all of the individual ectopic pregnancy diagnosis and procedure codes were evaluated as potential predictors to be included in the algorithm (Table I). In addition, we identified 22 other variables available from the automated datafiles as potential predictors of ectopic pregnancy (listed in footnote, Fig. 1). These included age at ectopic pregnancy diagnosis; other diagnosis and procedure codes such as ovarian cyst or salpingectomy; medical treatment with methotrexate; site of care (ambulatory versus inpatient) and laboratory data such as pregnancy testing and serial hCG. Sources of these data included the pharmacy records (prescription fills), enrollment, laboratory, utilization and claims data.
Figure 1.
Algorithm for identifying ectopic pregnancy cases, Group Health development set. EP, ectopic pregnancy; EP episode, the 180-day interval following the first date with an EP code. (a) In addition to the three predictors that comprise the final algorithm, other potential predictors that were evaluated and were not incorporated into the final model included: year of diagnosis; age at diagnosis; the occurrence of each individual EP diagnosis or procedure code; a normal intrauterine pregnancy diagnosis and/or procedure code after the EP index date; any spontaneous abortion diagnosis and/or procedure code after the EP index date; any ovarian cyst code after the EP index date; any other non-EP diagnosis code occurring on any EP diagnosis date in an EP episode; any inpatient visit during an EP episode; any ultrasound performed during an EP episode; any pelvic/transvaginal ultrasound or obstetrical ultrasound performed during an EP episode; any pelvic computerized tomography or magnetic resonance imaging performed during an EP episode; number of pregnancy tests in an EP episode; any positive pregnancy test in an EP episode; number of hCG tests in an EP episode. (b) In boxes with EP, ‘N (%) not true EP’ refers to cases that were misclassified by the algorithm as EP when they were not true EP. (c)In the Not EP box, ‘N (%) true EP’ refers to cases that were misclassified by the algorithm as not EP when they were true EP.
Statistical analysis
For both sites, we calculated positive predictive values (PPV) and the associated 95% confidence intervals (CIs) based on the binomial distribution for the ectopic pregnancy-automated codes compared with the true case status from medical record review. PPV is defined as the proportion of women with positive ‘test’ results (in this instance, ICD-9/CPT-4 codes) who are correctly classified as true cases.
Using the additional variables identified from the health plan databases, we performed a classification and regression tree (CART) analysis to develop a case-finding algorithm at GH (Breiman et al., 1984). The CART analysis creates an algorithm by using a binary recursive partitioning strategy that classifies individuals as having or not having a true ectopic pregnancy on the basis of a set of predictors (Breiman et al., 1984). The model builds a decision tree that, at each partitioning step, evaluates all potential binary splits based on all possible predictors and then partitions the sample into two subgroups at the optimal split, defined as the split resulting in the best predictive accuracy. All cases in each subgroup are classified either as a true case or not a true case. The binary splitting of the data is repeated for each remaining subgroup until the optimal decision tree (algorithm) is developed. The CART analysis was performed using ‘rpart’ package in R version 2.9.2 (www.r-project.org) (R Development Core Team, 2011). The algorithm performance was summarized by calculating the algorithm sensitivity (percentage of medical record-confirmed cases that were correctly classified as ectopic pregnancy by the algorithm), specificity (percentage of cases determined not to be ectopic pregnancy by chart review that were correctly classified by the algorithm), PPV (percentage of cases classified as ectopic pregnancy by the algorithm that were medical record-confirmed cases) and negative predictive value (NPV, percentage of identified cases classified as not ectopic pregnancy by the algorithm that were determined not to be cases from medical record review). We calculated the 95% CIs for these performance measures based on a binomial distribution. We also calculated the algorithm misclassification proportion, that is, the percentage of all ectopic pregnancy cases that were incorrectly classified by the algorithm, and compared this with the percentage of ectopic pregnancy cases that were incorrectly classified when relying solely on the standard ectopic pregnancy diagnosis and procedure codes. Those potential cases with insufficient information in the medical record to determine true case status and those with uncertain case status after record review were excluded from the code evaluation and algorithm development.
Once we had developed and evaluated the algorithm at GH, we then applied the final algorithm to the KPCO validation set to assess its performance and generalizability to a separate population. The algorithm performance was assessed in the same way as at GH.
Results
The results of the medical record reviews of the 280 potential cases in the GH development set and the 500 potential cases in the KPCO validation set are summarized in Table II. After exclusion of those with insufficient information and uncertain case status, 242 potential ectopic pregnancy cases in the GH development set and 443 in the KPCO validation set were included in the analyses.
Table II.
Medical record review results of potential ectopic pregnancy cases.
| Results from medical record review | GH development set (n = 280) | KPCO validation set (n = 500) |
|---|---|---|
| n (%) | n (%) | |
| True EP cases | 165 (58.9) | 357 (71.4) |
| Not true EP casesa,b | 77 (27.5) | 86 (17.2) |
| Insufficient information in medical record to determine case status | 38 (13.6) | 57 (11.4) |
GH, Group Health; KPCO, Kaiser Permanente Colorado; EP, ectopic pregnancy.
aReasons for not being a true EP case at medical record review (not mutually exclusive): Rule out of EP (n = 30 at GH, n= 17 at KPCO); follow-up of prior EP (n = 3 at GH, n = 1 at KPCO); noting a history of EP (n = 50 at GH, n = 69 at KPCO).
bExamples of other conditions that were diagnoses for non-ectopic cases include normal or high-risk intrauterine pregnancy, spontaneous abortion, ovarian or corpus luteum cyst, and leiomyoma.
Accuracy of ICD-9/CPT-4 diagnosis and procedure codes for ectopic pregnancy
Of the 242 potential ectopic pregnancy cases identified at GH using the ICD-9/CPT-4 codes, medical record review confirmed 165 as true cases (PPV = 68%, 95% CI 62–74%). At KPCO, 357 of the 443 available potential cases were confirmed (PPV = 81%, 95% CI 77–84%) (Table II).
Ectopic pregnancy case classification algorithm
The CART analysis identified three main predictors from the larger group of variables that were assessed in the GH development set (Fig. 1): the number of visits with an ectopic pregnancy code during an episode; receipt of a tubal pregnancy code and receipt of methotrexate therapy. The strongest predictor was having at least two visits with an ectopic pregnancy diagnosis or procedure code within the 180-day ectopic pregnancy episode window. Of the 242 potential cases in the development set, 132 had two or more visits with one of these codes and thus were classified as cases by the algorithm. Only 9 of these 132 potential cases (6.8%) were not true cases based on medical record review and so were misclassified by the algorithm (Fig. 1). Thus, nearly 93% were correctly classified at this node in the tree. The remaining 110 potential cases were further classified as true ectopic pregnancy by the additional variable of an ICD-9-CM code for tubal pregnancy (633.1, 633.10, 633.11) and, lastly, by receipt of methotrexate treatment, as shown by moving down the tree in Fig. 1.
The sensitivity and specificity of the algorithm at both sites were excellent: 93–95 and 81%, respectively (Table III). The PPV and NPVs, respectively, were 91–96% and 79–84%. The algorithm's performance for the development and validation sets, relative to relying solely on the diagnosis or procedure codes for ectopic pregnancy, is summarized in Table III. When relying on the diagnosis/procedure codes to identify potential cases, ∼32% of potential cases at GH and 19% at KPCO were determined not to be true cases based on medical record review. In contrast, applying the case-finding algorithm with the additional predictors, the proportion of misclassified ectopic pregnancy cases decreased to 11% at GH and to 8% at KPCO. It should be noted that these proportions are not entirely analogous. The error accompanying use of the diagnosis/procedure codes is all in the direction of potential cases from the codes that are found not to be true cases on review; the algorithm misclassification, however, encompasses error in labeling a case as ectopic pregnancy when record review determines her not to be a case (as with the diagnostic codes) and labeling a potential case as not ectopic pregnancy when record review has determined a woman is indeed a case. Thus, in the latter instance, the misclassification can go in both directions.
Table III.
Accuracy and performance of the ectopic pregnancy case-finding algorithm versus ectopic pregnancy case selection using standard ICD-9-CM and CPT codes.
| EP algorithm classification | Medical record |
||
|---|---|---|---|
| True EP | Not EP | Total | |
| GH development data set | |||
| EP | 153 | 15 | 168 |
| Not EP | 12 | 62 | 74 |
| Total | 165 | 77 | 242 |
| KPCO validation data set | |||
| EP | 338 | 16 | 354 |
| Not EP | 19 | 70 | 89 |
| Total | 357 | 86 | 443 |
| Performance statistics (95% CI) | GH development data set | KPCO validation data set | |
| EP case identification using ICD-9-CM/CPT-4 codesa alone | |||
| PPV (%) | 68 (62–74) | 81 (77–84) | |
| Proportion of cases misclassified (%) | 32 (23–38) | 19 (16–23) | |
| EP case identification using algorithm | |||
| Sensitivity (%) | 93 (88–96) | 95 (92–97) | |
| Specificity (%) | 81 (70–89) | 81 (72–89) | |
| NPV (%) | 84 (73–91) | 79 (69–87) | |
| PPV (%) | 91 (86–95) | 96 (93–97) | |
| Proportion of cases misclassified (%) | 11 (7–16) | 8 (6–11) | |
EP, ectopic pregnancy.
aICD-9-CM/CPT-4 codes: ICD-9-CM diagnosis codes 633.x; ICD-9-CM procedure codes 66.0x, 66.62 or 74.3; CPT-4 codes 59120, 59121, 59130, 59135, 59136, 59140, 59150, 59151.
Discussion
Ectopic pregnancy is a potentially serious medical condition that may result in serious morbidity, and there is evidence from recent evaluations that its public health significance may not be diminishing (Van Den Eeden et al., 2005; Hoover et al., 2010; Trabert et al., 2011). As with many health conditions, research and surveillance activities addressing ectopic pregnancy may be advantaged by relying on automated health plan data (Centers for Disease Control and Prevention, 1995; Van Den Eeden et al., 2005; Hoover et al., 2010; Mol et al., 2010). Most recent estimations of ectopic pregnancy rates relying on automated data are based on use of the diagnostic and procedure codes we initially used to identify potential cases (Sewell and Cundiff, 2002; Calderon et al., 2005; Van Den Eeden et al., 2005). However, a potential limitation of these diagnosis and/or procedure codes is that they are likely to incorporate error. Our medical record review of cases selected using automated codes found that 19–32% of these potential cases were not true cases. A more accurate case-finding strategy thus has the potential to enhance future investigations of ectopic pregnancy. Using CART analysis, we developed and validated a case-finding algorithm for ectopic pregnancy that substantially improved the accuracy of automated case selection. The algorithm had excellent sensitivity and specificity, PPV and NPV and performed well when applied to a population separate from that used for the algorithm development. The algorithm was simple and incorporated predictors that are now widely available in many health care system databases.
The CART methodology we investigated offers a number of advantages. It is a non-parametric approach that makes no assumptions as to the underlying distribution of values of the predictors or the relationships between predictors and the ectopic pregnancy outcome (Breiman et al., 1984, Van den Bruel et al., 2007). Furthermore, all possible interactions between potential predictors and all possible cutpoints for categorical or continuous variables are evaluated at each branch of the decision tree to find the optimal, most accurate splitting criteria. The classification algorithm is readily interpreted and can be applied to other settings and data sets that include the variables that comprise the algorithm. In this study, the algorithm developed at one site performed well when applied to a separate study population.
A limitation of this study was that, as a pilot project, the available funds only allowed for selection and record review of a relatively small development set at GH. This may account in part for the differences in PPV at the two sites, as may different prevalence and coding practices. Fortunately, additional funding for a larger validation set in a second health plan allowed for further assessment of both the algorithm performance and generalizability (Bleeker et al., 2003; Barnhart et al., 2010). Evaluating the extent to which true ectopic pregnancy cases occur in the absence of receiving an ectopic pregnancy code also was beyond the scope of this project; we only identified potential cases that had received ectopic pregnancy codes. Further research validating the diagnosis and procedure codes and the algorithm in different populations and, optimally, with larger samples is needed.
To our knowledge, this is the first study to evaluate the accuracy of using standard diagnosis and procedure codes to identify ectopic pregnancy cases from automated data and to develop and validate a case-finding approach for refining automated case identification. Our results suggest that use of the algorithm may contribute to more accurate assessment of the public health burden of this condition and to ongoing surveillance and epidemiological research in this area. As these two health care plans are members of a nationwide consortium that can share programming and data elements across sites, this algorithm can be applied in these settings, which cover ∼13 million US enrollees. Given the wide availability of the variables identified, the algorithm is likely to be more broadly applicable, as well.
Authors’ roles
D.S. and V.H. were involved in the conception and design, and in securing NIH funding for the study. O.Y. was involved in the study design and conducted data analyses and interpretation. M.A.R. participated in the study design and obtained funding for the validation work. B.T. was involved in the study design and in the data acquisition and management. All authors participated in the analysis and interpretation of the data, in editing the manuscript draft, and in reviewing and approving the final version.
Funding
This research was supported by grants R03 HD052687 (PI: D.S.) and T32 HD052462 (PI: M.W.) from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), US National Institutes of Health (NIH), and the Kaiser Permanente Colorado (KPCO) 2009 Strategic Initiatives Fund (PI: M.A.R.).
Acknowledgments
The authors acknowledge the many valuable contributions to the project made by Jane Grafton, Patricia Yarbro, Linda Wehnes and Mary Sunderland at GH; and Heather Tavel, Karen Glenn, Mary Kershner and Marilyn Pearson at KPCO.
References
- Barnhart KT. Clinical practice. Ectopic pregnancy. N Engl J Med. 2009;4:379–387. doi: 10.1056/NEJMcp0810384. [DOI] [PubMed] [Google Scholar]
- Barnhart KT, Sammel MD, Appleby D, Rausch M, Molinaro T, Van Calster B, Kirk E, Condous G, Van Huffel S, Timmerman D, et al. Does a prediction model for pregnancy of unknown location developed in the UK validate on a US population? Hum Reprod. 2010;10:2434–2440. doi: 10.1093/humrep/deq217. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bleeker SE, Moll HA, Steyerberg EW, Donders AR, Derksen-Lubsen G, Grobbee DE, Moons KG. External validation is necessary in prediction research: a clinical example. J Clin Epidemiol. 2003;9:826–832. doi: 10.1016/s0895-4356(03)00207-5. [DOI] [PubMed] [Google Scholar]
- Breiman L, Friedman J, Stone CJ, Olshen RA. Classification and Regression Trees. Belmont, CA: Taylor and Francis; 1984. [Google Scholar]
- Calderon JL, Shaheen M, Pan D, Teklehaimenot S, Robinson PL, Baker RS. Multi-cultural surveillance for ectopic pregnancy: California 1991–2000. Ethn Dis. 2005;(4 Suppl 5) S5–20–24. [PubMed] [Google Scholar]
- Centers for Disease Control and Prevention. Ectopic pregnancy—United States, 1990–1992. MMWR Morb Mortal Wkly Rep. 1995;3:45–48. [Google Scholar]
- Go AS, Magid DJ, Wells B, Sung SH, Cassidy-Bushrow AE, Greenlee RT, Langer RD, Lieu TA, Margolis KL, Masoudi FA, et al. The cardiovascular research network: a new paradigm for cardiovascular quality and outcomes research. Circ Cardiovasc Qual Outcomes. 2008;2:138–147. doi: 10.1161/CIRCOUTCOMES.108.801654. [DOI] [PubMed] [Google Scholar]
- Hajenius PJ, Mol F, Mol BW, Bossuyt PM, Ankum WM, van der Veen F. Interventions for tubal ectopic pregnancy. Cochrane Database Syst Rev. 2007;1:CD000324. doi: 10.1002/14651858.CD000324.pub2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hoover KW, Tao G, Kent CK. Trends in the diagnosis and treatment of ectopic pregnancy in the United States. Obstet Gynecol. 2010;3:495–502. doi: 10.1097/AOG.0b013e3181d0c328. [DOI] [PubMed] [Google Scholar]
- Lix LM, Yogendran MS, Leslie WD, Shaw SY, Baumgartner R, Bowman C, Metge C, Gumel A, Hux J, James RC. Using multiple data features improved the validity of osteoporosis case ascertainment from administrative databases. J Clin Epidemiol. 2008;12:1250–1260. doi: 10.1016/j.jclinepi.2008.02.002. [DOI] [PubMed] [Google Scholar]
- Mol F, van Mello NM, Mol BW, van der Veen F, Ankum WM, Hajenius PJ. Ectopic pregnancy and pelvic inflammatory disease: a renewed epidemic? Eur J Obstet Gynecol Reprod Biol. 2010;2:163–167. doi: 10.1016/j.ejogrb.2010.04.014. [DOI] [PubMed] [Google Scholar]
- Platt R, Davis R, Finkelstein J, Go AS, Gurwitz JH, Roblin D, Soumerai S, Ross-Degnan D, Andrade S, Goodman MJ, et al. Multicenter epidemiologic and health services research on therapeutics in the HMO Research Network Center for Education and Research on Therapeutics. Pharmacoepidemiol Drug Saf. 2001;5:373–377. doi: 10.1002/pds.607. [DOI] [PubMed] [Google Scholar]
- R Development Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2011. www.r-project.org . [Google Scholar]
- Sewell CA, Cundiff GW. Trends for inpatient treatment of tubal pregnancy in Maryland. Am J Obstet Gynecol. 2002;3:404–408. doi: 10.1067/mob.2002.121623. [DOI] [PubMed] [Google Scholar]
- Trabert B, Holt VL, Yu O, Van Den Eeden SK, Scholes D. Population-based ectopic pregnancy trends, 1993–2007. Am J Prev Med. 2011;5:556–560. doi: 10.1016/j.amepre.2010.12.026. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Tu K, Campbell NR, Chen ZL, Cauch-Dudek KJ, McAlister FA. Accuracy of administrative databases in identifying patients with hypertension. Open Med. 2007;1:e18–e26. [PMC free article] [PubMed] [Google Scholar]
- Van den Bruel A, Aertgeerts B, Bruyninckx R, Aerts M, Buntinx F. Signs and symptoms for diagnosis of serious infections in children: a prospective study in primary care. Br J Gen Pract. 2007;540:538–546. [PMC free article] [PubMed] [Google Scholar]
- Van Den Eeden SK, Shan J, Bruce C, Glasser M. Ectopic pregnancy rate and treatment utilization in a large managed care organization. Obstet Gynecol. 2005;(5 Pt 1):1052–1057. doi: 10.1097/01.AOG.0000158860.26939.2d. [DOI] [PubMed] [Google Scholar]
- Vogt TM, Elston-Lafata J, Tolsma D, Greene SM. The role of research in integrated healthcare systems: the HMO Research Network. Am J Manag Care. 2004;9:643–648. [PubMed] [Google Scholar]
- Yu O, Nelson JC, Bounds L, Jackson LA. Classification algorithms to improve the accuracy of identifying patients hospitalized with community-acquired pneumonia using administrative data. Epidemiol Infect. 2010:1–11. doi: 10.1017/S0950268810002529. [DOI] [PubMed] [Google Scholar]
- Zane SB, Kieke BAJ, Kendrick JS, Bruce C. Surveillance in a time of changing health care practices: estimating ectopic pregnancy incidence in the United States. Matern Child Health J. 2002;4:227–236. doi: 10.1023/a:1021106032198. [DOI] [PubMed] [Google Scholar]