Abstract
Objective
To externally validate EPICON, a computerized system for grouping diagnoses from EMRs in general practice into episodes of care. These episodes can be used for estimating morbidity rates.
Design
Comparative observational study.
Measurements
Morbidity rates from an independent dataset, based on episode-oriented EMRs, were used as the gold standard. The EMRs in this dataset contained diagnoses which were manually grouped by GPs. The authors ungrouped these diagnoses and regrouped them automatically into episodes using EPICON. The authors then used these episodes to estimate morbidity rates that were compared to the gold standard. The differences between the two sets of morbidity rates were calculated and the authors analyzed large as well as structural differences to establish possible causes.
Results
In general, the morbidity rates based on EPICON deviate only slightly from the gold standard. Out of 675 diagnoses, 36 (5%) were considered to be deviating diagnoses. The deviating diagnoses showed differences for two main reasons: “differences in rules between the two methods of episode construction” and “inadequate performance of EPICON.”
Conclusion
The EPICON system performs well for the large majority of the morbidity rates. We can therefore conclude that EPICON is useful for grouping episodes to estimate morbidity rates using EMRs from general practices. Morbidity rates of diseases with a broad range of symptoms should, however, be interpreted cautiously.
Introduction
General Practitioners (GPs) increasingly use Electronic Medical Records (EMRs) to record information regarding the treatment of patients. North Western Europe, especially Sweden, Denmark, and the Netherlands, has a leading position in the use of EMRs in general practice. 1,2 Primarily used for patient care, EMRs can also be utilized for scientific research, particularly the estimation of prevalence and incidence rates (i.e., morbidity rates) in general practice. Morbidity rates in general practice may provide a good indication of the health status of the general population. This is especially true for countries where the main pathway to medical care runs through general practice. These rates are useful for monitoring health in the population as well as for developing and evaluating health care policy.
In order to estimate morbidity rates, diagnoses recorded in EMRs in general practice need to be grouped into episodes of care. We developed EPIsode CONstructor (EPICON), a tool for the computerized grouping of diagnoses in general practice. 3 The EPICON tool renders it possible to estimate morbidity rates automatically from EMRs in general practice. Results from a previous evaluation study indicate that the internal validity of EPICON is adequate. 4 The present study is aimed at examining the external validity of EPICON through the use of an independent dataset.
Background
A diagnosis in general practice can refer to a symptom or a complaint (symptom diagnosis), a syndrome (nosological diagnosis), or a disease (pathological/pathophysiological diagnosis). 5 In this article, we will use the umbrella term “diagnosis” to refer to any of these categories. In order to estimate the numerator of morbidity rates, the diagnoses recorded in EMRs need to be coded and grouped into episodes of care (all encounters for the management of a specific health problem), 5 since double counting may occur if episodes are not grouped. For instance, if a patient visits the GP initially with abdominal pain, and during a second visit a few days later the pain appears to be based on appendicitis, only the appendicitis should be counted in the morbidity rates. The abdominal pain has to be linked to the episode appendicitis.
Generally, two approaches for constructing episodes can be distinguished. In one approach, diagnoses are recorded in contact-oriented EMRs. To be able to distinguish between incidence and prevalence, an indication is usually given whether a diagnosis represents the start of a new episode (new) or is part of an episode that started in the past (ongoing). In this approach, diagnoses are grouped into episodes afterwards, either through manual review or by using EPICON. In the second approach, diagnoses are directly recorded into episodes by the GP in episode-oriented EMRs. 6 As far as we know, these episode-oriented EMRs are used routinely only in the Netherlands and in Malta. 7 In this respect, the Netherlands is a leading country in the Western world. In the Netherlands, all GPs rely on EMRs. Most practices use contact-oriented EMRs, while some of them use the new generation of episode-oriented EMRs.
We developed EPICON, a case-based application for grouping into episodes the diagnoses from contact-oriented EMRs. The EPICON application is based on a combination of logical expressions, a decision table, and former cases in which diagnoses from patients in general practice were grouped manually. These former cases were derived from 89 practices that participated in the second Dutch national survey of general practice (DNSGP-2). 8,9 The development of this application has been described in detail elsewhere. 3
The evaluation of EPICON, which can be qualified as a classification system, falls within the field of validating prognostic models. Different hierarchical levels of validation can be distinguished in evaluating prognostic models, starting with the internal validation (i.e., the performance of a system in the sample used to develop the system) as level 0. 10 In a previous study, we used a split sample approach to examine the internal validity of EPICON. 4 The dataset was split into two sets; one was used to develop EPICON and the other was used to test the developed system. Findings from the split-sample procedure showed that the internal validity of EPICON is adequate. Based on these results, EPICON has been brought into use to generate the yearly morbidity rates of practices with contact-oriented EMRs of the Netherlands Information Network of General Practice (LINH). 11,12
The next levels of validation refer to the transportability of the system, i.e., whether the system has the ability to generate accurate results in a sample taken from a population other than the one that was used to developed the system. In an independent validation, several components of transportability, including historical, geographic, and methodologic transportability, can be distinguished. 10
The goal of this study is to provide an independent validation of EPICON. We will use data from episode-oriented EMRs, in which GPs actually record diagnoses into episodes of care, as an independent dataset. Morbidity rates based on these GP-grouped episodes are considered the gold standard and will be compared to morbidity rates based on EPICON-grouped episodes. In this study, we will address the following research questions: 1)What is the deviation from the gold standard for morbidity rates that are based on EPICON-grouped episodes? 2)What are the causes of these deviations?
Methods
Dataset
We used data from six general practices that participate in a network of general practices in the northern part of the Netherlands (RNG) and that record all patient data using episode-oriented EMRs. 13 We consider this dataset to be independent because the data from these practices were not used in the development of EPICON. We used data from 2002 through 2005, although in the case of one practice, the data from 2005 were not available at the time of analysis. The dataset consists of a total number of 473,350 diagnoses which were coded according to the International Classification of Primary Care (ICPC, first edition). 14 The six practices participate in LINH, and this network requires that all practices record whether a consultation diagnosis is new or ongoing, a necessity to distinguish between incidence and prevalence. For the six included practices, this field was derived from the GP-grouped episodes. ∗
Episode Constructions
The GPs manually recorded and grouped the diagnoses for all their patients into episodes at the moment of the consultation. A second episode construction that regrouped the same diagnoses automatically was created by EPICON. It is possible for EPICON to group episodes differently from the GPs, which is called a misclassification. There are three types of possible misclassifications: Link failure (type 1) occurs when a diagnosis, which is linked to another diagnosis by the GP, is not linked by EPICON. A false link (type 2) originates when two diagnoses that were not linked by the GP are linked by EPICON. A wrong combination (type 3) occurs when both the GP and EPICON linked a diagnosis to another diagnosis, but the second diagnoses were different. 3 Misclassifications do not always cause differences in morbidity rates; they can only cause differences when they change a sufficient number of episode names (some misclassifications do change the sequence of links within an episode, but ultimately do not change the episode name).
Morbidity Rates
The next step in the study was to calculate two sets of prevalence and incidence rates: one set based on the GP-grouped episodes and the other set based on the EPICON-grouped episodes. The prevalence rate is defined as the proportion of the population with a particular disease during the period of one year, and the incidence rate refers to the occurrence of new episodes of a certain disease during the observed person-years at risk.
To establish the numerator of the prevalence rates, we counted, per episode name, the number of patients with at least one (new or ongoing) episode. When calculating the numerator of the incidence rates, we counted, per episode name, the number of new episodes. We used the mid-year population (i.e., the average of the population at the beginning and the end of each year) as the denominator, which varied from 24,067 in 2005 to 32,053 in 2003. Morbidity rates of diagnoses that occur only in the female (W, X) or male chapters (Y) are based only on the female or the male mid-year population. In total, we calculated morbidity rates for 675 different diagnoses.
Next, we compared the morbidity rates based on EPICON with the gold standard. In order to make this comparison, we distinguished between infrequent rates (less than one per 1000 patients per year according to the gold standard) and frequent rates (at least one per 1000 patients per year according to the gold standard), which divided the large number of rates into two approximately equal parts. The absolute differences were calculated for infrequent rates (absolute deviation), the relative differences, i.e., the percentages, for frequent rates (relative deviation).
Selection
We used the calculated differences to select a number of deviating diagnoses for further qualitative analysis. The criteria for selecting deviating diagnoses are based on sole outliers or structural differences:
- • Criteria for deviating infrequent diagnoses:
- a Absolute difference in at least one year < −0.2 or > 0.2 (sole outliers); or
- b Absolute difference in all four years < −0.05 or > 0.05 (structural differences).
- • Criteria for deviating frequent diagnoses:
- a Relative difference in at least one year < −15% or > 15% (sole outliers); or
- b Relative difference in all four years < −5% or > 5% (structural differences).
Qualitative Analysis
To clarify the causes of the observed differences, we conducted a qualitative analysis of these deviating diagnoses. The analysis included a detailed examination of both the documented grouping rules used by the GPs and the grouping methods used by EPICON (i.e., the algorithm, the decision table, and the cases). In addition, we interviewed one of the GPs and the coordinator of the registration network.
Results
Morbidity Rates
▶ shows the 2002 morbidity rates based on EPICON compared with the gold standard. Each scatter plot shows some hundreds of morbidity rates, named according to the ICPC. 14 (Because of the large number of rates, the plots show only part of all code names). For example, the prevalence of “other cardiovascular symptoms/complaints” (K29) in ▶a is 0.45 per 1000 patients per year in the gold standard (shown at the x-axis). The prevalence based on EPICON is 0.62 per 1000 patients per year which is a difference of 0.17 with the gold standard (shown at the y-axis).
Figure 1.
Deviation of morbidity rates (based on EPICON-grouped episodes) from the gold standard (i.e., morbidity rates based on GP-grouped episodes), 2002. a. Deviation from the gold standard for infrequent prevalence rates 2002. b. Deviation from the gold standard for frequent prevalence rates 2002. c. Deviation from the gold standard for infrequent incidence rates 2002. d. Deviation from the gold standard for frequent incidence rates 2002.
Figures 2–5 (Appendix 1, available as a JAMIA online supplement at www.jamia.org) show the results for the morbidity rates of 2003 through 2005. The figures show a similar pattern in each category. In general, the infrequent morbidity rates based on EPICON (▶a, ▶c, 2, and 4) deviate only slightly from the gold standard; the deviation of most rates is either 0 or close to 0. There are just a few deviating diagnoses, i.e., sole outliers (rates beyond the dotted lines) or structural differences (not clearly visible in the figures, but listed in the last column of ▶). Although the frequent morbidity rates (▶b, ▶d, 3, and 5) show more deviating diagnoses, with most of them occurring among the frequent incident rates (▶d and 5), the large majority of the frequent morbidity rates deviates not at all or just a few percent. With the exception of the diagnoses that comply with the selection criteria, there is no systematic bias, because the rates are proportionally distributed above and below the x-axis.
Table 1.
Table 1 Deviating Diagnoses per Category
| Deviating Diagnoses (ICPC) |
Selection Criteria |
||||
|---|---|---|---|---|---|
| Sole Outliers |
In all 4 Years | ||||
| 2002 | 2003 | 2004 | 2005 | ||
| Diagnoses related to death | |||||
| Euthanasia request/discussion (A20) | I > 15% | I > 0.2 | I > 5% | ||
| Death (A96) | I > 15% | I > 15% | I > 15% | I > 15% | I > 5% |
| Suicide/suicide attempt (P77) | I < 0.05 | ||||
| Cardiovascular diseases | |||||
| Acute myocardial infarction (K75) | I < −15% | I < −15% | I < −15% | I < −15% | I < −5% |
| Heart failure (K77) | I < −5% | ||||
| Stroke/cerebrovascular accident (K90) | I < −15% | I < −15% | I < −15% | I < −15% | I < −5% |
| Neoplasms | |||||
| Malignant neoplasm colon/rectum (D75) | I < −0.2 | I < −0.05 | |||
| Malignant neoplasm digestive other (D77) | I < −0.2 | I < −0.05 | |||
| Malignant neoplasm bronchus/lung (R84) | I < −0.2 | I < −0.2 | I < −0.2 | I < −0.05 | |
| Malignant neoplasm breast female (X76) | I < −15% | ||||
| Malignant neoplasm prostate (Y77) | I < −0.05 | ||||
| Diagnoses related to pregnancy | |||||
| Pregnancy (W78) | PI > 15% | PI > 5% | |||
| End points of pregnancy | |||||
| Abortion spontaneous (W82) | I < −15% | I < −15% | I < −15% | I < −15% | I < −5% |
| Abortion induced (W83) | I < −15% | I < −15% | I < −15% | I < −0.2 | I < −5% |
| Uncomplicated labor/delivery livebirth (W90) | I < −15% | I < −15% | I < −15% | I < −15% | I < −5% |
| Complicated labor/delivery livebirth (W92) | I < −0.2 | I < −0.05 | |||
| Adverse effects | |||||
| Adverse effect medical agent (A85) | P > 15% | ||||
| Possible symptoms | |||||
| Sweating problem (A09) | I > 5% | ||||
| Nausea (D09) | P > 15% | P > 15% | P > 5% | ||
| Rash generalized (S07) | P > 15% | P > 15% | P > 15% | P > 15% | P > 5% |
| General diagnoses | |||||
| General deterioration (A05) | I > 15% | I > 15% | I > 15% | I > 15% | P > 5% |
| General symptom/complaint other (A29) | I > 5% | ||||
| Complication of medical treatment (A87) | I > 5% | ||||
| Disease digestive system, other (D99) | I < −15% | ||||
| Other arterial obstruction/PVD (K92) | I < −15% | I < −5% | |||
| Cardiovascular disease other (K99) | I < −15% | I < −0.05 | |||
| Unrelated diagnoses | |||||
| Elevated blood pressure (K85) | P < 5% | ||||
| Hypertension complicated (K87) | I > 5% | ||||
| Epilepsy (N88) | I > 0.2 | ||||
| Chronic obstructive pulmonary disease (R95) | I > 15% | I > 5% | |||
| Syphilis male (Y70) | I < −0.05 | ||||
I = Incidence; ICPC = International Classification of Primary Care; P = Prevalence.
Out of all 675 analyzed diagnoses, 36 diagnoses (5%) were deviant. Five of these diagnoses were not further scrutinized because they included fewer than five misclassifications in a three year period; i.e., “haematemesis/vomiting blood” (D14), “fear of pregnancy” (W02), “unwanted pregnancy” (W79), “benign neoplasm female genital” (X80), and “genital herpes female” (X90). The remaining 31 diagnoses were analyzed in detail.
Causes of Deviations
▶ shows all deviating diagnoses. Similar diagnoses that deviate for the same reason are grouped together into one category. We found two main causes for deviations: differences in rules for grouping, and inadequate performance of EPICON.
Differences in Rules
Differences in the rules for grouping used by GPs and EPICON was the reason for the deviations found in the categories “diagnoses related to death,” “diagnoses related to pregnancy,” and some of the “unrelated diagnoses” (i.e., “elevated blood pressure” (K85), “hypertension complicated” (K87), and “chronic obstructive pulmonary disease” (R95)).
For instance, when looking at the incidence rates of “death” (A96) and “pregnancy” (W78), it becomes clear that the rates based on EPICON are higher than the gold standard, whereas the incidences of some causes of death (cardiovascular diseases and neoplasms) and end points of pregnancy are lower than the gold standard. This difference can be attributed to the fact that GPs use the rule that “death” (A96) or “pregnancy” (W78) should be linked to the cause of death or the end point, whereas EPICON uses the opposite rule. So EPICON classifies “death” (A96) as a separate, new episode that should not be linked to any previous diagnoses, such as “pregnancy” (W78).
The main cause for differences in the morbidity rates of “adverse effect medical agent” (A85) is a difference in the rule for naming the episodes. The EPICON rule is that the last disease code (ICPC codes 70–99) in time is used as the episode name for that episode. When there is no disease code in the episode, the last symptom code (ICPC codes 1–29) is used as the episode name. The GPs can choose every diagnosis as the episode name.
Furthermore, the interviews revealed that there are differences in handling the rules. The GPs, who group manually, vary in their use of the grouping rules, whereas EPICON groups automatically, and consequently, EPICON has no inter-doctor variation.
Inadequate Performance of EPICON
Differences in the category “general diagnoses,” “possible symptoms of adverse effects,” and some of the “unrelated diagnoses” (i.e., “epilepsy” (N88) and “syphilis male” (Y70)) are mainly due to problems in EPICON's performance. In some cases, EPICON displays linking problems, for instance, EPICON frequently failed to link possible symptoms of adverse effects, such as “nausea” (D09) to “adverse effect medical agent” (A85). In other cases, EPICON both displays link failures and creates false links. The main reason for this problem is that many different diagnoses can be linked to unspecified and general diagnoses. Moreover, the interviews showed that within the category of “general diagnoses,” a large inter-doctor variation exists in recording diagnoses as they are drawn from a reservoir of unknown or infrequently occurring diagnoses.
Discussion
In this study, we examined the external validity of EPICON, a system for grouping diagnoses into episodes whose purpose is to estimate morbidity rates in general practice. We used an independent dataset derived from GPs who record diagnoses using episode-oriented EMRs. Morbidity rates based on the GP-grouped episodes were considered the gold standard and compared to morbidity rates based on EPICON-grouped episodes. The results indicate that EPICON performs well for the large majority of diagnoses.
Only 5% (n = 36) of all analyzed diagnoses (n = 675) shows a substantial deviation in at least one year or a structural small deviation over four years. We found more deviations in rates for frequent than for infrequent diagnoses, because differences in episode construction have more effect on frequent than on infrequent rates. Except for these deviating diagnoses, we did not observe any systematic bias.
An explanation for part of these deviating diagnoses is a difference in grouping rules. Some of the rules used by EPICON for grouping diagnoses differ from those utilized by GPs. For instance, the GPs linked “death” (A96) to the cause of death, whereas EPICON is based upon a dataset in which it was decided to link “death” not to the cause of death. Both decisions have their advantages and disadvantages and it is not possible to claim that one rule is more valid than the other. In addition, we found differences in handling the rules and in naming the episodes.
Other deviating diagnoses are explained by inadequate performance of EPICON. This accounts, in particular, for deviations in unspecified and general diagnoses since many symptoms can be grouped within these diagnoses. The EPICON application is based on the probability that the cases in the training set were grouped. It does not use the same information in the grouping process as the GP does, such as age, gender, and the duration of the disease, which may cause misclassifications. Furthermore, it is possible that some cases did not occur in the training set. In all likelihood, however, EPICON will not be used for these unspecified and general diagnoses, because they are considered less important for epidemiological research. Should EPICON be used for these diagnoses, the resulting morbidity rates should be interpreted with caution. The EPICON application might be adjusted for these insufficiencies by: a) adding cases, and b) including variables such as gender, age, and duration of the disease into EPICON.
The strength of this study is that we used an independent dataset to examine the external validity of a prognostic model. There is not much research describing an external validation of a prognostic system. 15 Another strength is that we performed a quantitative as well as a qualitative analysis, providing insight into both the number of deviating rates and the main causes of these deviations.
A limitation of this study is that the criteria for defining deviating diagnoses are to some extent arbitrary, so should the criteria be altered, we will find less or more deviating codes. In addition, our judgement that the large majority of the morbidity rates deviates only slightly, is also subjective, although with the figures we provided, it is possible to judge deviations for oneself. Furthermore, our conclusions about the external validity of EPICON are based upon one test that was carried out in one setting.
This one test does provide some insight into the historical, geographical, and methodological transportability of EPICON. Regarding the historical transportability, EPICON was originally developed using data from 2001, and in this study, was applied to data from 2002 through 2005. The historical transportability of EPICON seems adequate as we did not observe any problem in the application to another time period. In the long run, however, changing medical insights that affect grouping rules may reduce the historical transportability.
Regarding the geographical transportability, EPICON is based on data from a nationally representative patient population, derived from 89 general practices throughout the Netherlands. 3 The six practices that were used for the external validation are located in the Northern part of the Netherlands. The patient population of these six practices is quite similar to the nationally representative population and we encountered no differences in episode-construction that could be attributed to the difference in geographical location. This cautiously indicates that EPICON is transportable to similar datasets in other regions or countries. We need more evidence, however, before we can draw any firm conclusions. If EPICON is applied to data from another geographical area, this ungrouped dataset should be compared first to the dataset upon which EPICON is based. For instance, the similarity of the age and gender distributions of both populations, and the frequencies of ungrouped diagnoses could be examined. The EPICON application should not be used if this comparison reveals large differences, such as many diagnoses of diseases that do not occur in the Netherlands.
Additionally, this study provides some insight into methodological transportability, because two different methods of collecting data were used. In this study we applied EPICON to data from episode-oriented EMRs whereas it was originally developed using data from contact-oriented EMRs. The EPICON tool is thus applicable to both contact-oriented and episode-oriented EMRs. A possible application for EPICON in episode-oriented EMRs is to discover differences in (handling) the rules used by different GPs. However, a limitation of the application of EPICON to contact-oriented EMRs is that the characterization of a diagnosis as either “new” or “ongoing” might be lacking. Although this field can be easily added to an existing EMR system, this solution does require extra recording for the GP.
These results regarding the generalizibility of EPICON need to be confirmed in cumulative tests across diverse settings. Furthermore, an important topic for future research is the extent to which differences in (the design of) EMRs affect the morbidity rates derived from these EMRs.
Conclusion
This study shows that the external validity of EPICON is sufficient for the purpose of estimating morbidity rates in general practice. Only a limited number of diagnoses (5%) deviates from the gold standard. There are two main causes for these deviations: inadequate performance of EPICON, and differences in rules. The latter cause seems particularly to apply to deviations in diagnoses related to death and pregnancy. The EPICON application performs less well when it comes to unspecified and general diagnoses, and hence caution is required when EPICON is used for these rates.
Footnotes
Operationally, an episode is a row of diagnoses that carry the same episode number. The first diagnosis of a new episode number was characterized as new; all other diagnoses were marked as ongoing.
References
- 1. European physicians especially in Sweden, Netherlands and Denmark, lead U.S. in use of electronic medical recordsAnonymous Health Care Res 2002;2:1-3. [Google Scholar]
- 2.Burt CW, Hing E, Woodwell D. Electronic medical record use by office-based physicians: United States, 2005. Health E Stats. 2006 Aug. http://www.cdc.gov/nchs/products/pubs/pubd/hestats/electronic/electronic.htm 2002. Accessed Jan 2, 2008.
- 3.Biermans MCJ, De Bakker DH, Verheij RA, Gravestein JV, Van der Linden MW, De Vries Robbé PF. Development of a case-based system for grouping diagnoses in general practice Int J Med Inform 2008;77:431-439. [DOI] [PubMed] [Google Scholar]
- 4.Biermans MCJ, Verheij RA, De Bakker DH, Zielhuis GA, De Vries Robbé PF. Estimating morbidity rates from electronic medical records in general practice: evaluation of a grouping system Methods Inf Med 2008;47:98-106. [PubMed] [Google Scholar]
- 5.WONCA Classification Committee An international glossary for general/family practice Fam Pract 1995;12:341-369. [DOI] [PubMed] [Google Scholar]
- 6.Okkes IM, Groen A, Oskam SK, Lamberts H. Advantages of long observation in episode-oriented electronic patient records in family practice Methods Inf Med 2001;40:229-235. [PubMed] [Google Scholar]
- 7.Lagasse R, Desmet M, Jamoulle M, et al. European situation of the routine medical data collection and their utilisation for health monitoring. EURO-MED-DATA. Final Report. Bruxelles: Université Libre de Bruxelles; 2001. December. Project No.: 1998/IND/2011. Contract No.: S12.107874.
- 8.Westert GP, Schellevis FG, De Bakker DH, Groenewegen PP, Bensing JM, Van der Zee J. Monitoring health inequalities through general practice: the Second Dutch National Survey of General Practice Eur J Public Health 2005;15:59-65. [DOI] [PubMed] [Google Scholar]
- 9.In: Westert GP, Jabaaij L, Schellevis FG, editors. Morbidity, Performance and Quality in Primary Care: Dutch General Practice on Stage. Oxon: Radcliffe Publishing; 2006.
- 10.Justice AC, Covinsky KE, Berlin JA. Assessing the generalizability of prognostic information Ann Intern Med 1999;130:515-524. [DOI] [PubMed] [Google Scholar]
- 11.Tacken MAJB. Quality of preventive performance in general practice: the use of routinely collected data[dissertation]Nijmegen: Radboud University Nijmegen; 2005.
- 12.Verheij RA, Te Brake JHM, Abrahamse H, Van den Hoogen H, Braspenning J, Van Althuis T. Netherlands Information Network of General Practice. Facts and figures on Dutch GP care. Utrecht/Nijmegen: NIVEL/WOK. http://www.linh.nl 2005. Accessed Dec 27, 2007.
- 13.Van der Veen WJ, Meyboom-de Jong B. Age and GenderIn: Jones R, Britten N, Culpepper L, et al. editors. Oxford textbook of primary medical care. Oxford: Oxford University Press; 2004. pp. 153-161.
- 14.In: Lamberts H, Woods M, editors. International Classification of Primary Care (ICPC). Oxford: Oxford University Press; 1987.
- 15.Stolwijk AM, Straatman H, Zielhuis GA, et al. External validation of prognostic models for ongoing pregnancy after in-vitro fertilization Hum Reprod 1998;13:3542-3549. [DOI] [PubMed] [Google Scholar]