Visual inspection of transvaginal ultrasound videos to characterize uterine peristalsis: an inter-observer agreement study

Abstract

Purpose

Contractions in non-pregnant uterine can be assessed by visual inspection of transvaginal ultrasound (TVUS). Many authors have used this method to extract features like contraction frequency and direction. However, visual inspection is a subjective method and the outcome is dependent on the sonographers and video analysts. In this study, we wanted to see which uterine feature is reproducible enough, in terms of inter-observer agreement, to serve as a reliable control for future research.

Methods

Six observers assessed 80 TVUS videos, and rated video quality, contraction frequency, direction and timing. One observer assessed operating time. A Fleiss’ kappa (κ) or an intra-class correlation (ICC) was calculated to determine the inter-observer agreement of all features.

Results

The inter-observer agreement in frequency was substantial (ICC = 0.68). Conversely, there was just slight to fair agreement in contraction timing and direction and in video quality: ICC = 0.26, κ = 0.17 and κ = 0.16, respectively. Overall, agreement among technical engineers was better than between medical professionals. The level of agreement was correlated with video quality, phase of the menstrual cycle and individual patient (all χ² with p < 0.00). The time to analyze one video ranged between 6 and 20 min.

Conclusions

This study shows that visual inspection of TVUS videos is a fairly reproducible method to assess contraction frequency. However, the operating time is too extensive to implement this method in daily practice. Automated methods could offer a solution for this problem in the future.

Introduction

Since Dickinson et al. [1] reported the presence of uterine activity outside pregnancy in 1937, researchers have recognized the possible benefits of learning more about this phenomenon. Since the early 1990s, transvaginal ultrasound (TVUS) has been used to identify and characterize uterine activity. Although other diagnostic methods like intra-uterine pressure [2, 3], hysterosalpingoscintigraphy [4–6] and magnetic resonance imaging [7–11] have also been used, the easy accessibility and its easy use during fertility treatment makes TVUS the most appealing method; this can also be concluded by the large number of authors using TVUS in their research [2, 12–31].

Many authors who have published on uterine contractions based their results on visual inspection, mainly by watching TVUS videos at original or accelerated speed, and identifying contractions by eye. Few of them assessed the reliability of this method in terms of inter-observer agreement in contraction characteristics. Considering the subjective nature of visual inspection, we think a good inter-observer agreement is crucial to guarantee the reliability and reproducibility of such a method. Fanchin et al. [15] and Zhu et al. [31] calculated the inter-observer agreement on contraction frequency between the two researchers in their studies. Van Gestel et al. [29] checked the inter-observer agreement between three observers for their direction classification system. To our knowledge, no authors have assessed agreement on contraction timing before, nor have they taken into account the operation time per video. In this paper, we tested the reproducibility of visual inspection of the TVUS videos by calculating the inter-observer agreement in contraction frequency, timing, direction and video quality to evaluate the possible implementation in clinical practice.

Methods

This study is a sub-analysis of the data set created within the WAVES study (Dutch trial register number NTR5264) in which the primary outcome was to test the reliability of electrohysterography (EHG) for uterine contractions measurement during different phases of the normal menstrual cycle in a prospective selected cohort. As visual inspection of ultrasound videos is a method most commonly used in the literature on uterine contractions outside pregnancy, TVUS examinations were performed in all patients as a comparison. Between November 2012 and February 2016, we performed TVUS examinations in 21 women. All women were well-informed about this study and gave written consent before participating. Female participants were included if they were healthy, between 18 and 40 years old and had a natural and regular menstrual cycle. Exclusion criteria were: pregnancy, positive chlamydia screening in the past, congenital or non-congenital anomalies of the uterus, known causes of infertility, use of drugs, mental disorders or a significant language barrier.

Data collection

Within the WAVES study, we performed TVUS examinations in four different phases of the cycle: menstruation, pre-ovulatory, 3 days after ovulation and 7 days after ovulation. The pre-ovulatory phase was defined as having a dominant follicle ≥ 16 mm. These four phases were identified as previous research shows that direction and frequency changes between these phases. Previous research showed that we can expect high amplitude, cramp-like contractions during menstruation, peristaltic contractions in a higher frequency around ovulation and a decrease of activity in the luteal phase [2, 32]. This pattern allows discharge of menstrual debris, transportation of spermatozoa to the dominant follicle and a quieter environment for nidation. During every ultrasound examination, two videos (2–4 min) of the uterus in the sagittal plane were recorded in 2D and digitally stored. We used two different ultrasound machines available at our outpatient clinic: the Accuvix V20 ultrasound machine with an EC4-9IS ultrasound probe and the WS80A with Elite ultrasound machine with a V5–9 probe (Samsung Medison, Seoul, South Korea). Three different, trained sonographers performed all the recordings.

Visual inspection

Six independent observers performed the offline analysis, once per video. Three of them were medical specialists in the field of gynecology, with experience in watching TVUS images. One of them had done visual inspection of contractions for research purposes before. Two of them had not. The other three observers were biomedical engineers, who had also been involved in research on ultrasound of the non-pregnant uterus, are familiar with the topic, and yet are not as experienced in watching ultrasound videos in general compared to the medical specialists. None of them had performed visual inspection of contractions before. We included engineers to evaluate the effect of experience on the reproducibility. All observers got the same instruction course on how to grade uterine contractions. Together, they analyzed three videos, which were not in the data set, as practice before starting their individual assessments. All videos in this research were blinded for the patients’ name, number and phase of the cycle by randomly numbering them from one to eighty. Observers watched the videos at high speed (four times the normal speed) using Virtual Dub 1.9.11 player (Avery Lee Cheyenne, Wyoming US), and were free to scroll back and forward through the video as much as needed. Every video contained a yellow vertical line in a fixed position in the video (Fig. 1). These lines were positioned by one of the researchers, based on the position in the video where contractions were generally best visible. All observers were asked to note the video quality, direction, timing and frequency of the contractions in a standardized spreadsheet. One observer also noted the time spent on analysis of every video.

Fig. 1.

Video still of TVUS recording with vertical fixed line. Timing of the contraction is noted when the contraction reaches this vertical yellow line (= an event)

Observers were asked to grade the quality of the video based on the ability to identify contractions: 0 = bad, 1 = intermediate, 2 = good. As an example, dark regions or out of plane movement would make the quality bad. The direction was classified, based on the entire video. We used the classification system that was proposed by van Gestel et al. [29], in which eight different direction groups are distinguished (Table 1).

Table 1.

Wave direction system used in this research (as proposed by van Gestel et al. [29])

Number	Wave direction	Description
1	CF	Waves from cervix to fundus
2	FC	Waves from fundus to cervix
3	CF/FC	Waves from cervix to fundus and waves from fundus to cervix, alternating
4	CF recoiling	Waves from cervix to fundus, with a reflection of the wave toward the cervix after reaching the fundus of the uterine cavity
5	Standing	Movement visible, but the activity is not conducted toward fundus or cervix
6	Opposing	Waves starting simultaneously in fundus and cervix
7	Random	Movement activity starting at various foci of the uterine cavity
8	No act	No intra-uterine movement visible

If a contraction was identified, the time at which the contraction reached the vertical line was noted in seconds and marked as an event. If there was a contraction, but it did not reach the vertical line, it was not noted.

Data analysis

The total number of events was noted per video and the contraction frequency (contractions/minute) was calculated by dividing the total number of events by the duration of the video in minutes. To calculate the inter-observer agreement in contraction timing, a dedicated software was used, created in Matlab (MathWorks, Natick MA, USA). The main goal of this software was to identify events as noted by different observers that we could consider the same contraction. Agreement would be a measure of how many of the six observers identified the same contraction. To determine which events were considered the same contraction, a time interval of 10 s was chosen. This interval was based on literature [33], which showed a maximum contraction frequency of six per minute in the most active phase of the cycle (generally around 3/min). Based on this value, we could assume that if one observer noted two events within 10 s, it would be very unlikely that those belong to different contractions. Therefore, one of those events would be clinically irrelevant. Subsequently, the maximum frequency of six contractions per minute indicated that two events noted by different observers within a 10-s interval were likely to belong to the same contraction.

The first step in our software was to detect those “double” events, noted within 10 s by a single observer. If the software detected two of these events, the second event was removed.

The second step was to compare the events of all six observers within every video. The software searched for the first event in time per video, and checked if any other observer noted an event within a 10-s time interval. If so, the events were considered to belong to the same contraction. Otherwise, the event was considered to be a contraction identified by one observer only. Thereafter, the software would search for the next event within that video and repeat this process within the video. This was repeated for all eighty videos. Figure 2 shows the steps the software took.

Fig. 2.

Flow chart of the dedicated software. Added are the results of level of agreement (six-point scale and intra-class correlation)

To confirm that our choice to use a time interval of 10 s was correct, we recalculated our results with the software using 19 different time intervals between 2 and 20 s. Figure 3 shows the results of this analysis. Logically as the time interval was decreased, our results showed fewer events that were removed and fewer events that were considered the same contraction. The 10-s interval proved to be a good cutoff, as the number of identified contractions stabilized from an interval of 10 s and higher, while the number of removed events was still relatively low (37 events removed, 1.7% of all 2186 events).

Fig. 3.

Number of removed events (x-axis) and number of identified contractions (y-axis) using time intervals between 2 and 20 s (red dots) in our dedicated software. A 10-s time interval gives an optimal balance between these features

The output of the software contained a data set with a number of identified contractions. Each contraction contained a minimum of one and a maximum of six events, dependent on the number of observers that noted the event within the 10-s time interval. This resulted in a six-point scale for agreement per identified contraction. Perfect agreement would be six observers who noted the same contraction, while the worst agreement would be if only one noted it. If zero observers noted any contraction within a video, agreement was perfect as well; they agreed the uterus was quiet.

Statistical analysis

We used SPSS statistics 23 (SPSS Inc, Chicago, IL, USA) to calculate the inter-observer agreement. Intra-class correlation (ICC) analysis was used for contraction frequency and timing, as both data sets consisted of continuous variables. Both video quality and contraction direction were categorical variables (ordinal and nominal respectively), and were analyzed by calculating a Fleiss’ kappa (κ) with calculations completed in Excel 2013 (Microsoft, Redmond WA, USA). Finally, we checked if the inter-observer agreement in timing (in terms of the six-point scale mentioned before) was related to video quality, phase of the menstrual cycle or individual patients. We used a Kruskal–Wallis test [measured in Chi-squared (χ²)] to achieve this (SPSS Inc., Chicago, IL, USA). We split the mean video quality data into four quartiles for analysis.

Results

Within the WAVES study phase II and III, a total of 21 patients were included. Six patients without a complete data set (not all phases of the cycle were included) were excluded from this study. In four patients, the quality of (some of) the ultrasound videos was not sufficient for visual analysis and so they were also excluded, leaving ten patients for analysis. In total, 80 videos were analyzed by 6 observers, which resulted in 480 observations. There were 66 videos with a 4-min duration, 16 of 2 min and 2 of 3 min. The median time spent on visual analysis per video was 13 min (range 6–20 min).

The observations resulted in a mean contraction frequency of 1.31 contractions per minute. Mean contraction frequency differed between the four phases of the menstrual cycle: 0.76/min during menstruation, 2.31/min pre-ovulatory, 1.73/min 3 days after ovulation, and 0.44/min 7 days after ovulation. These numbers did not change significantly after the software removed 37 events (mean: 1.28/min, phases: 0.75/min, 2.25/min, 1.69/min and 0.44/min, respectively). As shown in Fig. 4, the contraction direction was different between phases of the cycle. Fundus-to-cervix contractions were mainly present during menstruation (52/120 observations) and the pre-ovulatory phase (51/120 observations), while the number of observed cervix-to-fundus contractions increased pre-ovulatory (28/120 observations) and 3 days after ovulation (46/120 observations). Uteri without any activity were mainly seen at 7 days after ovulation (53/120 observations).

Fig. 4.

Contraction direction per phase of the menstrual cycle as classified by six individual observers (total of 480 observations, 120 observations per phase)

Table 2 shows the results of the inter-observer agreement analysis. The calculated Fleiss’ kappa values for quality [κ = 0.16, 95% confidence interval (CI) 0.12–0.20] and direction (κ = 0.17, 95% CI 0.15–0.20) both showed only slight agreement in the whole group, as well is in the subgroups. The engineers seemed to agree better than the medical professionals did. In contrast, the ICC calculated for frequency showed substantial agreement between all observers (ICC = 0.68, 95% CI 0.58–0.78), with a less pronounced difference between medical professionals and engineers. When we analyzed the frequency data set without the 37 events excluded by the software mentioned previously, these results remain more or less the same (ICC = 0.67 overall, 0.63 for medical professionals and 0.71 for engineers).

Table 2.

Levels of agreement between six observers divided into subgroups of medical professionals (n = 3) and engineers (n = 3): the closer to 1.0 the better the agreement

	Quality Fleiss’ kappa (95% CI)	Direction Fleiss’ kappa (95% CI)	Frequency ICC (95% CI)	Timing ICC (95% CI)
Medical professional	0.03 (− 0.06–0.13)	0.10 (0.05–0.16)	0.65 (0.51–0.75)	0.15 (0.11–0.20)
Engineers	0.25 (0.16–0.35)	0.21 (0.16–0.27)	0.73 (0.53–0.84)	0.33 (0.28–0.39)
All observers	0.16 (0.12–0.20)	0.17 (0.15–0.20)	0.68 (0.58–0.78)	0.26 (0.23–0.29)

CI confidence interval

Using our dedicated software, we identified 781 contractions, of which each could score an agreement of one to six. The mean agreement was 2.78 out of 6, which means that, on average, less than half of the observers agreed on the existence of a contraction. When calculating the ICC on the actual values of the events (time in seconds), the agreement is fair: ICC = 0.26, 95% CI 0.23–0.29.

The level of agreement per contraction (six-point scale) was correlated with video quality (χ² = 55.63, p < 0.01), phase of the menstrual cycle (χ² = 82.47, p < 0.01), and individual patients (χ² = 74.99, p < 0.01), which is depicted in Fig. 5. This figure shows that good agreement is seen significantly more in videos with higher quality (quartile 3 and 4). When arranged by menstrual phase, the most active phase (pre-ovulatory) gave the best agreement. Menstrual phases were not associated with level of video quality (χ² = 6.74, p = 0.08). Videos of some individual patients (i.e., patient 2) showed better agreement than others, though this dependency is also associated with video quality (χ² = 41.70, p < 0.01).

Fig. 5.

Mean agreement and standard deviation on six-point scale (agreement between six observers: 6 = perfect agreement, 1 = worst agreement) ordered by subcategories: a quality of the video divided into four quartiles = filled bars, b phase of the menstrual cycle = empty bars, c individual patients = striped bars

Discussion

Uterine contractions have been suggested to play a role in fertility by many authors in the last decade [34]. Reliable and practical measurement is key when we want to translate this knowledge to the clinical practice. Ultrasound imaging is non-invasive, relatively cheap, and easy to access for gynecologists. Therefore, this method has been popular in researching uterine behavior outside pregnancy. The most commonly used technique to analyze TVUS data is visual inspection of the ultrasound videos. This technique has been generally the same throughout the years: digitally storing ultrasound videos, re-watching them in normal or (more commonly) accelerated speed and recording the direction and frequency (and in some cases also amplitude) of contractions. The main disadvantage of visual inspection is the subjectivity of the technique, which might induce a lack of reproducibility. Unfortunately, inter-observer agreement of multiple contraction characteristics has never been thoroughly assessed.

Van Gestel et al. [29] studied the inter-observer agreement for their newly proposed system of contraction direction, which we also used in this study. They found a good kappa value of 0.83 between 3 observers. They did not mention any agreement in terms of contraction frequency or timing. Fanchin et al. [15] and Zhu et al. [31] researched the effect of uterine contraction frequency before embryo transfer in hormone-stimulated cycles. Both studies appointed two trained researchers as observers, and reported a high inter-observer agreement (κ = 0.75 and ICC = 0.988, respectively).

These results are better compared to our results on frequency (ICC = 0.68), with a substantial agreement in the whole group of observers, as well as in the physicians and engineers group. Our results on direction show slight agreement (κ = 0.17) and are not comparable with the results of van Gestel et al. This big difference might be explained by the fact that they reported in observers who were also researchers specifically dedicated to this subject. Our observers were not as well trained. This suggests that identification of direction takes more practice than counting the number of contractions.

To the best of our knowledge, agreement in contraction timing has never been assessed before. Timing was never used as an endpoint in studies, as it is not suggested to alter fertility like contraction frequency or direction. However, when comparing different methods to measure contraction activity (such as ultrasound and intra-uterine pressure, or electrical activity), timing is of the essence. Only when finding comparable patterns, can we prove that we are indeed measuring the same physical event.

Our data showed that the average agreement of all 781 identified contractions was 2.78 (six-point scale), which means that on average not even three out of six observers noted a contraction within the 10-s timeframe we set. The fair agreement we found statistically (ICC = 0.26) matches with these results. The degree of agreement seemed to be clearly correlated with video quality, phase of the menstrual cycle and individual patient. The influence of video quality seems clear: better quality makes it easier to identify contractions. This insinuates that the newest ultrasound machines with better options to optimize your image could give better results than the machines used in studies from the 20th century. During the menstrual cycle, we saw the best agreement within the phases that have relatively high contraction frequencies (pre-ovulatory: mean agreement 3.28/6 and ovulation + 3: 2.85/6, compared to menstruation 2.29/6 and ovulation + 7 1.98/6). The quality was not rated noticeably higher in these phases and the χ² test showed no significant relation. It seems that, if there are more contractions in a rhythmic pattern (peristaltic), it is easier to identify contractions and agree on timing. The level of agreement was also associated with individual patients, though these individual patients are also associated with video quality (χ² = 41.70, p < 0.01). In some patients, the uterus can be better visualized because of factors like favorable pelvic anatomy, less air in the intestines or a lower body mass index. This results in better quality and (indirectly) in better agreement.

Although unexpected, we consistently found slightly better results within the engineers group, than in the medical professionals group. It seems that being able to just note exactly what you see, without prior knowledge creating certain expectations, gives a more structured result and better agreement. Another feature that no author using visual inspection has addressed before is the time it takes for an observer to analyze a video. We found a range between 6 and 20 min per video. Although this is based on the efforts of one observer (medical professional), other observers confirmed this range. One observer even reported spending 30 min on one video. This feature is a serious limitation when trying to implement measurement of contractions to daily practice.

Within the WAVES study, we were looking for a non-invasive, easy to use comparison for our electrohysterography measurements. When taking its limited reproducibility into account, we think that contraction frequency based on visual inspection of TVUS can be a great tool. Yet outside research, this method is too time-consuming and fairly not reliable enough to implement in daily practice. To overcome the time-consuming issue, automated contraction identification in TVUS videos could be a solution. Fanchin et al. [15, 16] tried such a method based on m-mode (motion mode) videos. The downside of this method is that it only focuses on one small slice of the uterus and information about external influences (probe movement, respiration, etc.) is absent. Automated methods taking the whole uterus into account, such as speckle tracking techniques [35], might provide a great solution by combining the availability and low cost of ultrasound with fast and easily comparable analysis.

We can conclude that contraction frequency is the most reproducible feature of contractions in the non-pregnant uterus, when assessed using visual inspection of TVUS videos. Although the poor results of the inter-observer agreement in timing and direction could partly be explained by the lack of experience of the observers, we think these features are not reproducible enough to use in future research. The time necessary to analyze each video makes this method unattractive for use in daily practice. To make contraction detection based on TVUS viable for use in the clinic, automated methods are needed.

Funding

This research was funded by the Netherlands Organisation for Scientific Research (NOW) within the TTW domain (applied and technical sciences) in the form of a High Tech Systems and Materials grant (Grant number 13901).

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflicts of interest.

Ethical approval

This research was approved by the medical ethical committee of the Catharina Hospital (MEC-U protocol number NL52466.100.15).

Informed consent

Informed consent was obtained from all individual participants included in the study.

Data availability statement

The data sets generated during and analyzed during the current study are available from the corresponding author on reasonable request.