External quality control and training of semen analysis in the Netherlands: starting point for further reduction of outcome variability

Abstract

Semen analysis is characterized by high levels of intra- and inter-laboratory variability, due to a low level of standardization, high subjectivity of the assessments, and problems with automated procedures. To improve consistency of laboratory results, quality control and training of technicians are important requisites. The goals of this study are to evaluate the results of an external quality control (EQC) program and standardized training by ESHRE Basic Semen Analysis Courses (BSAC) on the variability in manual assessments of semen parameters. We performed retrospective analyses of (1) the interlaboratory variability in the Dutch EQC program and (2) the interobserver variability in BSACs for concentration, motility, and morphology assessments. EQC data showed that the interlaboratory coefficient of variation (CV) for concentration assessment decreased (range from 24.0%–97.5% to 12.7%–20.9%) but not for morphology and motility assessments. Concentration variability was lower if improved Neubauer hemocytometers were used. Morphology assessment showed highest CVs (up to 375.0%), with many outliers in the period of 2007–2014. During BSAC, a significant reduction of interobserver variability could be established for all parameters (P < 0.05). The absence of an effect in the EQC program for motility and morphology might be explained by respectively the facts that motility assessment was introduced relatively late in the EQC program (since 2013) and that criteria for morphology assessment changed in time. BSAC results might have been influenced by the pretraining level of participants and the influence of external factors. Both EQC and training show positive effects on reducing variability. Increased willingness by laboratories to change their methods toward standards may lead to further improvements.

INTRODUCTION

The actual clinical value of semen analysis has been discussed for several years,1 due to, among others, a low utilization of published standardized processes2,3,4,5 and the ongoing debate on the predictive value of sperm parameters for fertility.6 Even the introduction of laboratory manuals by the World Health Organization (WHO) since 19807,8,9,10,11 did not result in the desired level of adherence and, over the years, the used methods remained highly variable between laboratories, especially for morphology assessment.2,3,4,12,13 Next to the standardization of used methods, another pitfall of semen analysis is the subjectivity of the assessment, characterized by intra- and inter-observer variability, as well as interlaboratory variability. As a result, the quality of male infertility diagnostics might be impaired, thereby possibly leading to an inappropriate choice of treatment.14

One strategy to measure the intra- and inter-laboratory variability and the lack of standardization is to implement internal quality control (IQC) and external quality control (EQC) programs. In different reports, EQC results showed a reduction of interlaboratory variability over time for the three main aspects of semen analysis: sperm concentration, morphology, and motility.15,16 IQC is an important requisite to improve the consistency of analysis results within one laboratory, for example, from one day of measurement to another.17,18 Both IQC and EQC are tools to evaluate the precision and accuracy of results, leading to a lower level of variability19 and improved standardization over the years.16

Training of technicians is another prerequisite to implement standardized methods and to minimize intra- and inter-observer variability. Reported immediate beneficial effects of semen analysis training were a substantial reduction of the variability in all aspects of semen analysis, reached within only a few days of training.14,20,21,22,23,24,25,26,27,28 On the long term, training showed increased awareness of the need for standardization and even significant changes in used methods.27,29,30

As shown, several previous studies evaluated the impact of quality control and training on the variability of semen analysis. However, these studies were performed before the introduction of the most recent version of WHO manual in 2010.7 Only two studies included limited data after this date as well.16,23 It is important to evaluate the impact on variability after 2010 since the implementation of the recommendations is still a challenge.16 In a previous study, we showed that there is still a wide variability in used methods for semen analysis in the Netherlands,4 even though both quality control and training have been offered for several years. Moreover, some of the recommended methods by the WHO were followed by rather small groups of the laboratories, for example, the used counting chamber (approximately 50%) and the staining method for morphology assessment (approximately 10%).4 Since the willingness of Dutch laboratories toward the WHO recommendations was rather low, we expect that this could influence the impact of both requisites on the variability of semen analysis results. The objective of this study is, therefore, to evaluate the impact over time on the variability of semen analysis results in the Dutch EQC program. Further, we investigated whether subgroups of WHO-recommended methods gave better results and whether standardized training during semen analysis courses lead to lower inter-observer variability on short time.

MATERIALS AND METHODS

EQC program

In the Netherlands, the Dutch Foundation for Quality Assessment in Medical Laboratories (SKML) distributes validated samples and digital data for the analysis of sperm characteristics, including the sperm parameters concentration, percentage of morphologically normal spermatozoa, and percentage of motile spermatozoa.29 Participants of the regular EQC program are mainly fertility laboratories, clinical chemistry laboratories, and microbiological laboratories. Since the start of the program, about 100 laboratories participate in the EQC program each year. Two samples for the assessment of concentration, morphology, and motility are distributed on a quarterly basis (i.e., March, June, September, and December) and should, preferably, be evaluated by the technician in charge. In the period of 2001–2018, 72 SKML distributions (144 samples) were assessed for sperm concentration and morphology as a part of the EQC program. Motility assessment was included in the program since 2013, resulting in 48 samples assessed in 24 distributes in the period of 2013–2018.

EQC semen samples consisted of pooled semen that remained after semen analysis or in vitro fertilization from men who visited the fertility laboratory of the Radboud University Medical Center (Nijmegen, the Netherlands) and the clinical chemistry laboratory of the Sint Antonius Hospital (Nieuwegein, the Netherlands). Semen used for concentration measures was prepared by density gradient centrifugation, diluted with Hayem's dilution (Boom NV, Meppel, the Netherlands), stored at 4°C, and sent to the participants in 0.4 ml aliquots. For morphology assessment, a small drop of semen (5 μl) was spread and air-dried on a microscopic slide. The participants were instructed about the preferred staining method. For motility assessment, semen was filmed in a Makler chamber (Sefi Medical Instruments, Haifa, Israel) at a magnification of 200× with phase contrast illumination (DM2000 light microscope, Leica Microsystems, Wetzlar, Germany). Each video consisted of two microscopic fields, each filmed for 40 s. Participants should evaluate motility of 200 spermatozoa of the samples according to WHO 1999 (i.e., rapidly progressive, slowly progressive, nonprogressive and immotile)8 or WHO 2010 (i.e., progressive, nonprogressive and immotile) laboratory guidelines.7 Results of all participating laboratories were included in this study, irrespective of their used methods.

Basic semen analysis courses (BSAC)

To evaluate short-term effects of training, results obtained during BSAC were used. The BSAC has been given twice a year since 2008 at the Radboud University Medical Center. It has been offered to laboratory staff aiming to reduce variability and increase standardization of semen analysis. Course content is provided by the European Society of Human Reproduction and Embryology (ESHRE) Special Interest Group for Andrology (SIGA), is based on the prevailing WHO laboratory guideline, and has been updated regularly in response to new findings and publications.30 The training includes all aspects of standard semen analysis and lasts four days. A more detailed description of the training schedule and course content is described by others.25

At the start of the course, participants assessed semen parameters (i.e., concentration, morphology, vitality, and motility) of two samples, without having had any training (pretest measurements). These parameters were assessed again for two samples after one day of theoretical and practical training (training measurements). At the end of the week, BSAC was completed with a practical examination, including assessment of all semen parameters of two more samples (exam measurements). It is important to note that different semen samples were used for the pretest, training, and examination and that the vitality results were not included in this evaluation study. All samples consisted of discarded semen from men who visited the fertility laboratory of the Radboud University Medical Center.

Statistical analyses

Results of the EQC program, in the period of 2001–2018, were presented as means and coefficient of variation (CV) per sample for all semen parameters. The results of all participating laboratories were used, irrespective of used methods. In addition, in the period of 2015–2018, CVs were presented separately for the improved Neubauer hemocytometer and Diff-Quik staining for, respectively, concentration and morphology assessments as compared to the other methods. Both improved Neubauer hemocytometer and Diff-Quik are WHO-recommended methods and most used in the Netherlands. During BSAC, participants analyzed two different samples during pretest, training, and exam. The mean of both measurements was calculated. Accordingly, the CVs of all measurements per course were summarized in boxplots separately for the pretest, training, and examination measurements. The Friedman test was used for testing whether the results of the three measurements were statistically different. All statistical analyses were performed using IBM SPSS Statistics for Windows, version 22.0 (IBM Corp., Armonk, NY, USA).

RESULTS

EQC program

Evolution of the interlaboratory CVs of sperm concentration, morphology, and motility assessment is shown in Figure 1. In addition, the mean values of these measurements over time are shown in Figure 2. There was a decreasing trend for interlaboratory CVs of sperm concentration assessment (range: 24.0%–97.5%, in 2001–2004), toward a more stable period in 2015–2018 (range: 12.7%–20.9%). The variability in mean values had a more or less stable course over the total period (range: 2.1%–59.0%). Morphology assessment was characterized by a declining trend in the means until 2006, where after multiple outliers (with a maximum of 375.0%) of interlaboratory CVs were shown in the period of 2007–2014. Since 2015, the range of interlaboratory CVs has been stabilized (43.9%–69.6%). Both levels of interlaboratory CV and mean (range: 15.0%–95.0%) of motility assessment variated constantly over the period of 2013–2018. In addition, evolution of the CVs using improved Neubauer hemocytometer (concentration) and Diff-Quik staining (morphology) versus the other methods is shown in Figure 3. Starting in the second half of 2016, the use of improved Neubauer hemocytometer was characterized by lower CVs (range: 5.2%–8.9%) compared to the group using other methods (range: 13.2%–26.9%). For morphology assessment over time, the range of CVs was comparable for laboratories using Diff-Quick staining versus laboratories using other staining methods.

Figure 1.

Overview of the CV of the EQC program results for each semen parameter. EQC consisted of four distributions each year with two samples per distribution. Each dot represents the CV (in %) of one EQC sample between 2001 and 2018 (sperm concentration and morphology, respectively) and between 2013 and 2018 (sperm motility). CV: coefficient of variation; EQC: external quality control.

Figure 2.

Overview of the measured mean values of external quality control program samples for each semen parameter. Each dot represents the mean value of the data supplied by all laboratories (one value per laboratory).

Figure 3.

Overview of the CV of the EQC program results for sperm concentration and morphology assessment. Each dot (improved Neubauer hemocytometer or Diff-Quik staining, respectively) or triangle (other methods) represents the CV (in %) of one EQC sample between 2015 and 2018. CV: coefficient of variation; EQC: external quality control.

BSAC

Results of sperm concentration, morphology, and motility assessment during pretest, training, and examination are summarized in boxplots in Figure 4. In total, 19 courses were offered between 2008 and 2018, where 15 participants per course evaluated the semen samples. Between pretest and training measurements, the CVs decreased significantly for all parameters (concentration: P < 0.01; morphology: P < 0.01; motility: P = 0.01). For concentration assessment, the initial improvement between pretest (median: 41.8) and training measurements (median: 23.2) was followed by a significant deterioration (P = 0.01) between training and examination (median: 35.0). For both morphology (P < 0.01) and motility assessment (P = 0.04), the CVs significantly decreased from pretest to examination. Differences between training and examination were not statistically different (P = 0.64 and P = 0.82, respectively).

Figure 4.

Boxplots of the CV of pretest, training, and examination measurements for each parameter, during 4-day ESHRE basic semen analysis courses (19 courses with 15 participants each). Pretest: results on day 1 of the course before any training; training: results during the training session; exam: results on the last day of the course during the examination; CV: coefficient of variation; ESHRE: the European Society of Human Reproduction and Embryology.

DISCUSSION

This study shows that, despite a low level of standardization, the variability in semen analysis results decreased in time in the Dutch EQC program, especially for concentration measurement. For concentration measurements, laboratories using the improved Neubauer hemocytometer showed lower variability compared to laboratories using other methods. This was not found for staining methods used during morphology assessment. During the Dutch version of the standardized BSAC of the ESHRE, a positive effect of training on variability was perceived.

The need of standardization of semen analysis among laboratories was emphasized previously.2,31 Previous studies showed beneficial effects of EQC and training on realizing standardization of used methods and reducing variability of semen analysis.19,25,26,27 Despite the long-standing presence of the Dutch EQC program and BSACs in the Netherlands, the used methods on fertility laboratories are still characterized by lack of standardization.4 On the other hand, this study showed that, even with a lack of standardization, EQC outcome showed a reduced variability of semen analysis results over time. Whereas other studies showed reduction of variability of all aspects of semen analysis during EQC programs,15,16 our study only showed a positive effect on sperm concentration assessment, especially in the period before 2015. When comparing results of laboratories using the improved Neubauer hemocytometer (WHO advised method)7 with the results of laboratories using other methods, reported CVs were lower since the second half of 2016 (Figure 3). These results emphasize, once again, the importance of standardization and suggest that even further reduction of variability can be realized.

Comparable improvements were not found in the assessment of sperm morphology if Diff-Quik staining was compared with other staining methods. Diff-Quik was used in the majority of laboratories, but still, variability in this group was high. This indicates that the origin of variability for morphology assessment may be found in other aspects such as the interpretation of the strict criteria. The successive WHO manuals7,8,9,10,11 describe different criteria and reference values which might result in some uncertainty among fertility laboratories, leading to an increased variability of morphology assessment results. This is especially visible in the results between 2007 and 2014, where multiple outliers are present during the introduction by the WHO10,11 of strict criteria. This assumption is supported by the reported decline of assessed percentages of morphologically normal spermatozoa caused by methodological adaptations based on guidelines.32 As a result of this trend, the value of the assessment of morphologically normal spermatozoa in counseling individual couples is under discussion in recent literature.32,33,34

For sperm motility, interlaboratory CVs were comparable during the total study period. An explanation for this is that the motility assessment was included for a shorter period (since 2013) in the EQC program than concentration and morphology assessment (both since 2001). It is possible that the effect of EQC on the variability of motility assessment did already occur before 2013, and therefore, this reduction is not visible in the EQC results used in this study. Furthermore, motility assessment in the EQC program was based on videos instead of microscopic examination, thereby limiting the influence of used methods (e.g., counting chamber and pipetting) on variability of the measurements. Compared to video assessment, it is known that real-time microscopic motility determination of cryopreserved semen in EQC programs showed a high interobserver and interlaboratory variability.35 Therefore, as long as WHO recommends real-time manual motility analyses,7 the actual value of EQC by analyzing video recordings will be lower than intended (lower commutability).

Next to EQC, another important requisite is training. BSACs have been offered since 1994, where the contents have been updated regularly based on new findings and publications.30 During standardized training courses, substantial reductions of inter- and intra-observer variability were reported, due to improved theoretical and technical skills of the participants.20,21,22,23,24,26 Furthermore, in the present study, a statistically significant reduction of the variability between pretest (baseline measurement) and training was seen for all sperm parameters. For concentration assessment, however, this reduction was followed by a significant increased variability of examination results, where the variability between pretest and examination results was not statistically different. This finding might be explained by the course contents and the manner the practical examinations of the course are set up. During pretest, participants were asked to perform semen analysis using their own equipment and methods. The next day, WHO/ESHRE recommended methods were taught and trained in order to maximize standardization. One hypothesis to explain the higher variability during examination is that examination stress might influence the results. Moreover, increased time pressure (all assessments need to be performed within a time interval of 2 h) can influence the results of this complex analysis.36,37 This trend in concentration assessment during BSACs, however, was not reported in a previous study.21

Strengths of our study are the big amount of data and the link between EQC and training data. It can be concluded that both EQC programs and training can be useful strategies to reduce the variability of semen analysis results. Further improvements seem to be blocked by a low level of standardization of used methods.4 Three strategies may improve semen analysis results. First, it might be useful to communicate the results of this study during BSAC and EQC program. Second, by introducing a continuous training program, the reported beneficial short-term effects of BSAC on the variability might be maintained as long-term effects. It has already been shown that long-term effects of BSAC were an increased awareness of standardization and a change in used procedures toward recommended methods.25,27,28 Short (on-site) refresher courses based on BSAC courses and WHO guidelines could be a solution to this. The need for such approach is supported by the fact that we received frequent requests for (additional) on-site training from the participants of the EQC program. Furthermore, video instructions or e-learning programs could be useful in this perspective. Third, further research could provide better insight in difficulties regarding semen analysis. For example, data from EQC or training sessions can provide information about difficulties in interpretation of certain morphological criteria which lead to subjectivity and thereby to variability. This information can be used not only to optimize training strategies (e.g., more focused on these aspects) but also to optimize recommendations (e.g., updating definitions of these morphological criteria). These three strategies will be subject of further research.

Overall, we conclude that reduction of both inter-observer and inter-laboratory variability is still an important challenge. Although many attempts were made, for example, by describing recommendations such as WHO guidelines, semen analysis is still dependent on the willingness of fertility laboratories toward the implementation of recommended or new methods. Approaches to reduce the lack of standardization of used methods did not yet result in the desired effect. Therefore, training, IQC, and EQC are the most useful tools at the moment to reduce intra- and inter-laboratory variability of semen analysis. BSAC should be provided as an initial training program,21 followed by IQC and EQC programs, regular refresher courses, and a course management system to maintain knowledge and to inform about new findings and publications. A combination of video instructions and an e-learning program could be a useful tool in this.

AUTHOR CONTRIBUTIONS

AMMW led the study and all authors contributed to the study design, data collection, and collation of the results. LL drafted the manuscript. LVDH and NJVV were especially involved in sample preparation and logistics of EQC. AMMW and LVDH organized the Dutch version of BSAC. DDMB, MAS, and WLDMN contributed to the interpretation of the data and were involved in the critical review of the paper. All authors read and approved the final manuscript.

COMPETING INTERESTS

All authors declare no competing interests.