Intra- and Inter-Observer Reliability In Selection of the Heart Rate Deflection Point During Incremental Exercise: Comparison to A Computer-Generated Deflection Point

Abstract

This study was designed to 1) determine the relative frequency of occurrence of a heart rate deflection point (HRDP), when compared to a linear relationship, during progressive exercise, 2) measure the reproducibility of a visual assessment of a heart rate deflection point (HRDP), both within and between observers 3) compare visual and computer-assessed deflection points. Subjects consisted of 73 competitive male cyclists with mean age of 31.4 ± 6.3 years, mean height 178.3 ± 4.8 cm. and weight 74.0 ± 4.4 kg. Tests were conducted on an electrically-braked cycle ergometer beginning at 25 watts and progressing 25 watts per minute to fatigue. Heart Rates were recorded the last 10 seconds of each stage and at fatigue. Scatter plots of heart rate versus watts were computer-generated and given to 3 observers on two different occasions. A computer program was developed to assess if data points were best represented by a single line or two lines. The HRDP represented the intersection of the two lines. Results of this study showed that 1) computer-assessed HRDP showed that 44 of 73 subjects (60.3%) had scatter plots best represented by a straight line with no HRDP 2)in those subjects having HRDP, all 3 observers showed significant differences(p = 0.048, p = 0.007, p = 0.001) in reproducibility of their HRDP selection. Differences in HRDP selection were significant for two of the three comparisons between observers (p = 0.002, p = 0.305, p = 0.0003) Computer-generated HRDP was significantly different than visual HRDP for 2 of 3 observers (p = 0.0016, p = 0.513, p = 0.0001). It is concluded that 1) HRDP occurs in a minority of subjects 2) significant differences exist, both within and between observers, in selection of HRDP and 3) differences in agreement between visual and computer-generated HRDP would indicate that, when HRDP exists, it should be computer-assessed.

Introduction

The anaerobic threshold (AT) has been defined as the level of exercise oxygen uptake (VO₂) above which aerobic energy production is supplemented by anaerobic mechanisms (Weltman, 1995). While the "gold standard" in anaerobic threshold assessment has been a plateauing of lactate production during continuous exercise, or maximal lactate steady state (Aunola and Rusko, 1992), the anaerobic threshold has also been assessed using gas analysis (Wasserman et al., 1973) or lactate production during incremental exercise. It has been demonstrated that incremental exercise results in two breakpoints in lactate production (Skinner and McLellan, 1980). The first breakpoint has been identified as the aerobic threshold (Aunola and Rusko, 1992; Rusko et al., 1986; Ribiero et al., 1985). However, the maximal lactate steady state coincides with the second breakpoint (Davis et al., 1983; Aunola and Rusko, 1992). The AT has been shown to be highly correlated to endurance performance (Hagberg and Coyle, 1983; Sjodin and Jacobs, 1981). Improvement in AT has also been shown to correlate with improved endurance performance (Tanaka et al., 1986). However, all of the above methods require sophisticated laboratory equipment, tester expertise, or both that precludes widespread testing.

In 1982, a simple, non-invasive method of AT assessment was developed using progressive, incremental exercise and measuring heart rate during each stage (Conconi, et al., 1982). It was demonstrated that the slope of the heart rate/work rate relationship changed and this point (heart rate deflection point, or HRDP) occurred at the same work output as AT. The protocol of this original study was somewhat unique in that running distance per stage was maintained, speed increased, so that the subsequent stage duration was decreased. The term that has been used to describe this type of protocol is "acceleration phase (Pokan et al., 1999) Since the original paper (Conconi et al., 1982), an inverse deflection (increase in heart rate/work rate ratio) has been documented (Hofmann et al., 1997a; 1997b; Pokan et al., 1998), occurring more frequently in older subjects and those with documented cardiac disease (Pokan, et al., 1998). While the test-retest reliability of the test was initially reported to be extremely high (Conconi et al., 1982), subsequent studies have questioned its reliability (Ribiero, et al., 1985; Jones and Doust, 1995). Since its original validation, 14 studies have been conducted to determine the validity of the Conconi, or HRDP test in detecting AT (Carey, 2002). It is beyond the scope of this paper to examine the validity of this procedure. However, several authors have substantiated the validity of the "Conconi Test" (Ahmaidi et al., 1992; Cellini et al., 1986; Droghetti et al., 1985; Gaisl and Wiesspeiner, 1988; Kara, et al., 1996; Petit et al., 1997; Pokan et al., 1995; Ribiero et al., 1985) while others have refuted it (Bourgois and Vrijens, 1998; Francis et al., 1989; Kuipers et al., 1988; Tokmakidis and Leger, 1992; Vachon et al., 1999; Zacharogisnnis and Farrally, 1993). The use of the "Conconi Test" as a predictor of performance is also controversial, with some supporting (Cellini et al., 1986; Conconi et al., 1982; Droghetti et al., 1985,) and others refuting (Tokmakidis and Leger, 1992; Zacharogisnnis and Farrally, 1993) its validity.

Explanations for the discrepancy in results from different authors have been proposed (Conconi, et al., 1997; Hofmann et al., 1997a; Jeukendrup et al., 1997). Possible explanations for lack of validity for the Conconi test are the use of an "acceleration" protocol in which increases in speed result in a decrease in stage duration (Jeukendrup et al., 1997) and large physiological variability between subjects (Hofmann et al., 1997a). However, the theory that the HRDP is a result of the "acceleration" protocol has been refuted, since HRDP has been shown to occur after the beginning of the acceleration phase, and therefore cannot be its cause (Conconi et al., 1997).

The three objectives of this study are 1) determine the relative frequency of occurrence of HRDP, when compared to a linear relationship between heart rate and work rate 2) determine the intra- and inter-observer reliability of visual assessment of HRDP) and 3)compare visual and computer-assessed HRDP.

Methods

Descriptive statistics and maximal oxygen consumption (VO₂ max) test results are obtained in Table 1. As can be seen from this table, subjects participating in this study demonstrated a high level of cardiovascular fitness, with a mean VO₂ max 39.0 % higher than non-athletic males but slightly less than Olympic-caliber cyclists (Wilmore and Costill, 1994). Subjects were recruited through an advertisement placed on a popular website visited by the cyclists. This study was approved by the Institutional Review Board of the University of St. Thomas.

Table 1.

Descriptive statistics and VO₂ max test results. Data are mean (SD).

Age (yrs)	Height (cm)	Weight (kg)	VO2 max (ml.kg.-1min-1)	Max watts
31.4 (6.3)	178.3 4.8	74.0 4.4	57.0 (8.9)	374.7 (50.2)

All tests were performed on a Quinton Excalibur Sport electrically-braked cycle ergometer which is completely adjustable to simulate the riding position of road cycling. Subjects installed their own pedals and wore their clipless cycling shoes. Heart rate was visually observed during the last 10 seconds of each minute from a Polar Electro Vantage XL (Polar Electro, Woodbury, New York) heart rate monitor and recorded. The test started at 25 watts and increased 25 watts per minute to volitional fatigue. While gas analysis was performed on each subject, it was not part of this research design and will not be discussed further. A Minitab multiple linear regression macro was developed by the University of St. Thomas Quantitative Methods and Computer Science Department. This program was designed to determine either a linear or curvilinear relationship between heart rate and work rate such that squared error was minimized.. Descriptive data, correlation coefficients, t-values and p-values were calculated using the Statistix software program.

On two separate occasions, three observers were given scatter plots of each subject, with watts displayed on the x-axis and heart rate on the y-axis. Observers were instructed to 1) initially determine if the data points best resembled a single line or two straight lines with a "breakpoint" 2) if a breakpoint was determined to be present, observers were asked to select the breakpoint as that point representing the last heart rate before the deflection. While inverse deflections (increase in the heart rate/work rate) have been identified subsequent to the original Conconi test (Hofmann et al., 1997,a; 1997b; Pokan et al., 1998), observers were instructed to treat this data as linear because it is contrary to the original validation (Conconi et al., 1982). Also, some scatter plots displayed a flat heart rate response in the early stages of exercise (less than 100 watts) before the linear phase of the heart rate/work rate relationship. Observers were instructed to exclude the use of these data points in their analysis. For intra-observer comparisons, results were classified as either linear (both observations determined to be linear), breakpoint (both observations determined to be breakpoint), or discordant (one linear, one breakpoint). For inter-observer comparisons, results were classified as either linear (100% agreement among observers, both observations), breakpoint (100% agreement among observers, both observations), partial agreement (2 of 3 or 3 of 3 observers determine a breakpoint on at least one of 2 observations). Two of the three observers were naïve to the Conconi Test, while this author, who was familiar with the Conconi Test, served as the third observer. While there is some evidence that experienced observers may be more accurate and reliable in identifying the heart rate deflection point (Heck et al., 1988), it was thought that knowledge of the Conconi theory may bias the observers in selection of the heart rate break point.

The null hypothesis states there are no significant differences 1) among observers 2) between observers 3) between visually-assessed and computer-assessed breakpoints. Level of significance was set at p < 0.05.

Results

Intra-observer

Of the 219 paired observations of scatter plots (3 observers, 73 subjects), 93 (42.5%) showed linear agreement, 76 (34.7%) showed breakpoint agreement, and 50 (22.8%) were discordant, with one of the paired observations determined to be linear and the other demonstrating a breakpoint (see Figure 1). Of a total of 438 observations, 236 (53.9%) were determined to be linear by these observers. Table 2 contains t-values, p-values and correlation coefficients for paired comparisons for each observer. While correlation coefficients for paired observations were significant for all 3 observers (p < 0.05), differences for paired observations for each observer also were significant (p < 0.05), demonstrating a lack of reliability within observers. The mean difference for intra-observer comparisons for power output and heart rate was 15.2 watts and 4.7 beats per minute, respectively.

Figure 1.

Intra-observer comparison.

Table 2.

T-values, p-values and correlation coefficients for intra-observer comparison.

Observer	T-Value	P-Value	Correlation Coefficient
One	2.06	0.048	0.895
Two	3.00	0.007	0.851
Three	3.83	0.001	0.911

Inter-observer

For inter-observer comparisons, 23 of 73 comparisons (31.5%) demonstrated linear agreement, 12 of 73 (15.1%) demonstrated breakpoint agreement, and 38 of 73 (53.4%) demonstrated partial agreement (see Figure 2). Table 3 contains t-values, p-values and correlation coefficients for inter-observer comparisons. Two of three inter-observer comparisons for subjects showing either partial breakpoint or breakpoint agreement resulted in significant differences (p < 0.05) in the selection of the breakpoint, demonstrating poor agreement between observers. The mean differences for power output and heart rate was 20.3 watts and 6.4 beats per minute, respectively.

Figure 2.

Inter-observer comparison.

Table 3.

T-values, p-values and correlation coefficients for inter-observer comparison: Partial or breakpoint agreement.

Observer	T-Value	P-Value	Correlation Coefficient
1 vs 2	1.19	0.260	0.867
2 vs 3	4.56	< 0.001	0.890
1 vs 3	3.56	0.044	0.914

Visual vs computer assessment

Table 4 lists t-values, p-values and correlation coefficients for breakpoint assessment for each observer compared to computer-assessed breakpoint. When the heart rate/work output relationship was computer-assessed, 44 of the 73 subjects (60.3%) were determined to be linear. For those subjects in which there was agreement of the existence in HRDP between observer and computer assessment, two of the three observers selected breakpoints that were significantly different (p < 0.05) from that selected by the computer. The mean differences for computer-assessed and visually-assessed power outputs and heart rates were 17.3 watts and 5.3 beats per minute, respectively.

Table 4.

Comparison of visual and computer assessment of breakpoint.

Observer	T-Value	P-Value	Correlation Coefficient
One	3.56	0.002	0.709
Two	0.66	0.513	0.718
Three	4.51	< 0.001	0.564

Discussion

The rate of computer-assessed HRDP of 39.7% reported in this study is in agreement with rates found in some studies (Ahmaidi et al., 1992; deWit et al., 1997; Heck et al., 1988; Kuipers et al., 1988; Lucia et al., 2002; Vachon et al., 1999) but somewhat less than others (Ballerin et al., 1996; Conconi et al., 1982; Droghetti et al., 1985; Grazzi et al., 1999; Hofmann et al., 1997a; Pokan et al., 1998; Ribiero et al., 1985). There are 2 possible explanations for this discrepancy: 1) the original study (Conconi et al., 1982) assessed the HRDP on a track by increasing running speed, maintaining distance per stage, and thereby progressively decreasing stage duration ("acceleration" phase). Most studies using this Conconi protocol have identified HRDP in a high percentage of cases (Ballerin et al., 1996; Cellini et al., 1986; Conconi et al., 1982; Droghetti et al.,1985; Vachon et al., 1999). Only one study using a "fixed time" protocol has found such high incidence of HRDP (Grazzi et al., 1999). When the same subjects have been tested using both a "fixed time" protocol and the Conconi protocol, the incidence of HRDP has been significantly different (Pokan et al., 1999; Vachon et al., 1999). However, Conconi (Conconi et al., 1996) has argued that the HRDP is not protocol-dependent and that HRDP occurs before the "acceleration" phase and shortened stage durations. Further research examining the incidence of HRDP using both a "fixed time" and "acceleration" protocols in the same subjects is needed to clarify this issue. 2) it is assumed that most studies have determined HRDP by visual assessment unless stated differently. Visual interpretation may lead to errors in assessment of HRDP (Bodner and Rhodes, 2000). If HRDP is visually determined, it should be performed by experienced observers (Ballerin et al., 1996). Some studies that have used computer assessment of HRDP have found a relatively low incidence of HRDP (Kara et al., 1996; Francis et al., 1989), while others have found a high incidence of HRDP (Hofmann et al., 1997a, Pokan et al., 1998).

In those subjects that were determined to exhibit HRDP by computer assessment, the mean heart rate as a percent of maximum heart rate was 87.7%. While this percentage may be considered somewhat high, others have found the HRDP at even higher percentages of maximum heart rate, ranging from 90% to 95% (Grazzi et al., 1999; Hofmann et al., 1997a; Kara et al., 1996; Ribiero et al., 1985; Zacharogisnnis and Farrally, 1993). In those studies in which the same subjects were asked to perform continuous exercise at HRDP, fatigue developed prematurely (Bourgois and Vrijens, 1998; Heck et al., 1988; Kuipers et al., 1988; Jones and Doust, 1997). This would indicate that HRDP may overestimate lactate threshold.

Differences in intra-observer reliability obtained in this study are both supported (DeWit et al., 1997) and refuted (Ballarin et al., 1996) by others. All three intra-observer comparisons were significantly different (p < 0.05), with a mean difference of 15.2 watts and 4.7 beats per minute. Differences may be explained by relative experience of observers in assessing HRDP. Two observers in this study were naïve to the Conconi theory of HRDP and the third observer, while familiar with this theory, had no experience in visually assessing HRDP. However, it was felt that previous knowledge and experience with the HRDP may have biased the results.

The inter-observer reliability measured in this study was determined to be low due to significant differences in two of the three paired comparisons between observers. This indicates that HRDP is not apparent and may be interpreted differently between individuals. The mean difference between observers was 20.3 watts and 6.4 beats per minute.

The finding of significant differences between visual and computer assessment of HRDP in two of the three observers may indicate that visual inspection may not be a valid method for assessment of HRDP. This is contrary to the results of others (Ballarin et al., 1996), but again may be a reflection of differences in experience of observers. The mean difference between visual and computer assessment of HRDP was 17.3 watts and 5.3 beats per minute. It may be argued that this mean difference is relatively small and is of no practical significance. However, given that a computer program is needed to determine if a HRDP exists, it seems prudent to use the computer-assessed HRDP.

Conclusion

The following conclusions seem warranted as a result of this study:

• 1) The frequency of occurrence of HRDP (39.7%) is relatively low and should be applied on a "subject by subject" basis.

• 2) Intra-observer differences in selection of HRDP were significantly different, indicating poor reproducibility of selection of HRDP.

• 3) Inter-observer differences in selection of HRDP were significantly different, indicating poor agreement between observers.

• 4) Significant differences between visual assessment and computer-assessed HRDP indicates the need for the use of a computer program to select HRDP, if and when it is present.

Biographies

Daniel G. CAREY

Employment:

Assistant professor in the Department of Health and Human Performance at the University of St. Thomas in St. Paul, Minnesota, USA.

Degrees:

PhD.

Research interests:

Anaerobic threshold and body composition assessment.

E-mail: dgcarey@stthomas.edu

Robert L. RAYMOND

Employment:

Professor of statistics in the Quantitative Methods and Computer Science Department of the University of St. Thomas in St. Paul, Minnesota, USA.

Degrees:

PhD.

E-mail: RLRaymond@StThomas.edu

Bridget A. DUOOS

Employment:

Associate professor of Kinesiology/Biomechanics in the Department of Health and Human Performance at the University of St. Thomas in St. Paul, Minnesota, USA.

Degrees:

PhD.

E-mail: baduoos@stthomas.edu