The validity and reliability of an integrated approach for quantifying match physical-tactical performance

Abstract

This study aimed to: (1) develop an integrated approach to quantifying match physical-tactical performance and (2) comprehensively examine the validity and reliability of this novel approach. Both UEFA qualified coaches and performance analysts (n = 30) participated to verify the scientific robustness of this new method. The percentage of correct responses were used to verify the validity of the integrated approach and the minimum acceptable agreement was set at 80%. Two well-trained groups of observers analysed a randomly selected English Premier League match for inter- and intra-observer reliability using the kappa statistic. A high degree of validity was demonstrated as the mean percentage of correct responses by all participants, accounting for 91.8 ± 4.3% for all, 92.2 ± 4.7% for out-of-possession, and 91.6 ± 5.7% for in-possession physical-tactical variables. Inter- and intra-observer reliability were found to be strong (κ = 0.81) to almost perfect (κ = 0.94), respectively. Additional analyses demonstrated that there was a nearly perfect correlation between data derived from the novel filter used for the present study to capture high-intensity running and those obtained from the filter of the commercial data provider (r = 0.99; P < 0.01). The data demonstrates that the integrated approach is valid and reliable regarding the quantification of physical-tactical performances. Therefore, it is now possible to unveil unique high-intensity profiles of elite players related to key tactical actions. This may help coaches and practitioners better understand the physical-tactical performances of players, as well as effectively translate physical metrics into training.

INTRODUCTION

Football (Soccer) is a complex sport as a myriad of technical, physical and tactical parameters have an influence on a player’s match performance [1]. To reduce this complexity, researchers have adopted a reductionist approach, where they typically analyse either physical or technical performance in isolation [2]. Using time-motion analysis, a plethora of research has quantified the physical profiles of elite players during match-play and how it is influenced by other factors such as formations, positions and context [3–9]. Despite low-intensity activities dominating football, high-intensity actions are of greater importance as they are associated with critical situations [10]. Therefore, an increasing attention has been paid to high-intensity activity as it enables practitioners to benchmark current requirements and prepare players for the physical requirements of modern match-play [2]. Nevertheless, authors have traditionally quantified physical metrics in isolation or effectively ‘WHAT’ distance players have covered during matches without any context. This ultimately limits the coaches understanding of the most pertinent performance elements [2]. According to a systematic review by Castellano et al. [11], only ~30% of papers considered tactical or technical variables alongside physical metrics. As players’ performance can be influenced by all of the factors above in isolation or collectively [12–14], physical metrics should be integrated with tactical and technical factors to obtain a more holistic understanding of football performance.

Despite hundreds of papers focusing on physical match performance, some studies have incorporated physical, technical and tactical metrics within their method [4, 15, 16]. However, data are still not comprehensively integrated but instead aggregated within the results. Only a single study has been published, which has fused the high-intensity physical-tactical activities of elite players [17]. This study revealed unique high-intensity running profiles as the data was associated with the tactical purpose of the physical action. For instance, full-backs cover ~10% of their total distance in high-intensity running ‘overlapping’ during transition/attacking play [17]. Moreover, forwards will typically ‘run in behind’ the defence at high-intensity to create an offensive threat. This clearly unveils the modulatory factors of the physical efforts or ‘WHY’ players produce high-intensity running efforts in matches. This innovative methodology provides not only how much distance players cover but also how they perform their tactical roles during match-play. Thus, this approach seems to provide more complete information on players’ physical data with tactical purposes to coaches and practitioners than the traditional approach.

Although Ade et al. [17] should be commended on such an insightful approach, there are some limitations. This approach does not incorporate other metabolically taxing actions such as accelerations/decelerations and changes of direction [18], so understanding the comprehensive physical demands of match play is limited. However, since this methodology still requires a manual coding process, which is very labour intensive, contextualising high-intensity running actions with tactical purposes seems to be a starting point [2]. Moreover, a major drawback of this original work was the lack of objectivity within the coding process whereby some crossover could occur for selected physical-tactical categories. For instance, a player could produce a high-intensity action ‘driving through the middle’ of the pitch but this does not specify if this is ‘with or without the ball’ or ‘supporting play’. To solve this problem, a modified version of this original approach was proposed [2]. The variables within the initial integrated approach were adapted and some of them were merged to simplify the method. Yet, some variables provided limited information regarding the actual tactical purpose of the action, and simply indicated their direction and location. Therefore, to enable this approach to be fully accepted by the academic and applied domains, a more systematic and intuitive version of this method was warranted. Moreover, scientific disciplines require a robust verification of any novel methodological approach’s validity and reliability before collecting and analysing data [19]. Therefore, this study aimed to: (1) develop a systematic integrated approach to quantifying match physical-tactical performance and (2) comprehensively examine the validity and reliability of this novel approach.

MATERIALS AND METHODS

Participants

Two populations of participants were involved in the present study: 15 UEFA qualified coaches (age: 27 ± 5 yr; range: 20–38 yr; mean ± SD) with an average experience within the football industry of 5 ± 6 yr (ranging 1–22 yr), and 15 performance analysts (age: 24 ± 5 yr; range: 20–34 yr; mean ± SD) with an average experience within the football industry of 3 ± 1 yr (ranging 1–4 yr). Participants worked for a variety of professional teams at different competitive standards (English Premier League, Championship, League 1 and 2, Women’s Super League and others). All participants provided their informed consent before commencing the study and were informed that they were free to withdraw at any point. Prior to data collection, ethical approval was granted by the local Ethics Committee of the appropriate institution.

Systematic Validation Process

Stage 1: Establishment of New Variables and Video Clips

Two additional physical-tactical variables (Move to Receive/Exploit Space, and Support Play) were created through extensive discussion with a highly qualified working group that composed of football science and UEFA qualified coaching staff (all working group members had > 10 yr experience within the football industry and were selected due to their expertise and experience in this specific area) to provide more information on the tactical purpose of various high-intensity efforts. Table 1 illustrates the refined version of the integrated approach.

TABLE 1.

A Systematic Integrated Approach to Quantifying Match Physical-Tactical Performance

Variables	Description
In Possession
Run with Ball	Player moves with the ball either dribbling with small touches or running at speed with fewer ball touches.
Over/Underlap	Player runs from behind to in front of the player on the ball or receiving the ball.
Push up Pitch	Player moves up the pitch to play offside and/or to squeeze to a higher line.
Break into Box	Player enters the opposition’s penalty box in an attempt to receive the ball (typically from a cross – ball in front and wide).
Run in Behind/Penetrate	Player attacks space behind, overtakes and/or unbalances the opposition defence (typically ball is behind).
Move to Receive/Exploit Space	Player moves to receive a pass from a teammate or to create/exploit space (typically comes short or moves wide to receive ball).
Support Play	Player supports from behind/level by trying to engage in offensive/transition play (typically during fast transitions).
Out of Possession
Close Down/Press	Player runs directly towards opposition player on/or receiving the ball, or towards space or players not on/ receiving the ball (typically blocking passing channels).
Interception	Player cuts out pass.
Recovery Run	Player runs back towards their own goal to be goal side when out of position.
Covering	Player moves to cover space or an opposition player while remaining goal side.
Other	All other variables that could not be categorised by the above.

Note: All variables were quantified during high-intensity running (e.g., speed threshold of >19.8 km·h⁻¹).

To enable this approach to be validated, video clips of all physical-tactical actions were derived from randomly selected games. From a total of 1,500 physical-tactical actions, the working group above randomly selected 150 of these actions to be clipped from video footage. Five examples for each variable were then randomly selected with various degrees of difficulty to represent gold standard responses (20 out-of-possession and 35 in-possession clips), and they were approved by the working group. Five clips per variable was deemed to be the optimal number for offering not only variety but it also enabled participants to complete the trial in a reasonable amount of time (30–45 min). The latter point is a major barrier previously experienced in this type of work given the busy schedule of staff working within elite football. To enable participants to find the correct player producing the high-intensity effort, a visualisation software (Viz Libero software, Bergen, Norway) was used to initially draw the participants attention to the player in question. All of the selected video clips were then randomly placed into presentation slides for the validation process.

Stage 2: Pilot Testing and Observer Training

Prior to the validation study, a pilot test was established by showing the presentation to two advanced football experts to verify the video clips as the gold standard responses (They unanimously agreed with all clips). Additionally, for the purpose of reliability tests the principal coder of games underwent approximately two months of training to achieve mastery of the approach, thus minimising errors when coding. Each high-intensity effort during match-play was then viewed using wide-angle video footage to manually tag the action with a relevant tactical purpose label (Table 1).

Stage 3: Establishing Validity

A one-on-one session was undertaken in a quiet location to enable the participants to fully concentrate. At the beginning of the session an answer sheet relating to the classification of each variable was provided to participants. Each participant was asked to tick the appropriate box after watching each clip. The presentation was shown to participants, initially introducing the integrated approach followed by numerous video examples of various physical-tactical actions and their associated definitions. Only after participants fully understood the concept and the associated variables, were they presented with the 55 test clips. Video clips were played as many times as possible when participants were unclear with the action. To ensure standardisation, the verbal explanation of all variables was similarly delivered to participants throughout the study and care was taken to ensure their decisions were not influenced.

Stage 4: Inter- and Intra-Observer Reliability

The inter-observer reliability of the integrated approach was assessed by two observers coding the first half of a randomly selected match. Two familiarisation sessions were undertaken to discuss each variable and understand the coding process. The descriptions of all variables in the integrated approach were provided and each variable was verbally explained with visual examples. The observers used the same descriptions throughout the process (Table 1). Intra-observer reliability was undertaken by the researcher coding the first half of a randomly selected match twice with a minimum of 7 days separating each observation. Coding was performed independently in a quite location for a maximum of 2 h with breaks every 30 min to guarantee optimal concentration levels [17].

Novel High-Intensity Filter

High-intensity efforts associated with tactical actions were isolated using a novel filter developed for this study (Figure 1) as the data provider could not disclose the proprietary developed filter used. To validate the novel filter, the present study compared the data derived from an established company (TRACAB, ChyronHego, New York, USA) with the new filter. Both filters operated on the premise that high-intensity running was defined at a speed threshold of > 19.8 km·h⁻¹ [20], but the dwell time for the established filter was not disclosed while the new filter used a minimal dwell time of 1 s.

FIG. 1.

The data processing and filter phase prior to integrated classifications.

Statistics

Data are presented as the mean ± standard deviation. All statistical analyses were conducted using IBM SPSS Statistics for Mac OS X, version 26 (IBM Corp., Armonk, N.Y., USA). The percentage of correct responses were calculated for each variable by dividing the correct responses by the total number of reference clips, then multiply this by 100 to verify the validity of the systematic integrated approach with 80% acting as the minimum acceptable threshold [21, 22]. Differences between UEFA coaches and performance analysts were determined using independent t-tests. Effect sizes for the meaningfulness of the difference were determined as follows: trivial (≤ 0.2), small (> 0.2–0.6), moderate (> 0.6–1.2), large (> 1.2–2.0) and very large (> 2.0–4.0) [23]. The Pearson correlation coefficient was calculated to determine the relationship between the data derived from the novel filter and those from an established data provider, in addition to the percentage of correct answers and years of experience. According to Hopkins et al. [24], the magnitudes of the correlation coefficients were regarded as trivial (r ≤ 0.1), small (r > 0.1–0.3), moderate (r > 0.3–0.5), large (r > 0.5–0.7), very large (r > 0.7–0.9), nearly perfect (r > 0.9), and perfect (r = 1.0). To measure inter-and intra-observer reliability, the kappa statistic was used. Kappa magnitudes were assessed as follows: none (0–0.20), minimal (0.21–0.39), weak (0.40–0.59), moderate (0.60–0.79), strong (0.80–0.90), almost perfect (> 0.90) [25]. Statistical significance was set at P < 0.05.

RESULTS

Validation of the Integrated Approach

The total mean percentage of correct responses by all participants was 91.8 ± 4.3% (range: 80 to 98.2%). The mean percentage of correct responses for out-of-possession variables was 92.2 ± 4.7% (range: 80 to 100%) whilst that for in-possession variables was 91.6 ± 5.7% (range: 74.3 to 100%). The most identifiable tactical action for out-of-possession variables was ‘Recovery Run’ with 98.7 ± 5.2% for UEFA coaches and 97.3 ± 7% for performance analysts whilst it was ‘Run with Ball’ for in-possession variables with 97.3 ± 7% and 98.7 ± 5.2%, respectively. In contrast, the least identifiable tactical action for out-of-possession variables was ‘Covering’ with 81±14% for UEFA coaches and 84±11.2% for performance analysts with ‘Move to Receive/Exploit Space’ for in-possession variables (80±16.9% and 86.7±14.5%, respectively). No differences were found between correct responses for UEFA coaches and performance analysts, except for ‘Push up Pitch’ (ES: 0.4, P<0.05). The comparisons between correct responses for UEFA coaches and those for performance analysts for out-of-possession and in-possession variables are shown in Figure 2 and 3, respectively.

FIG. 2.

Total mean percentage of correct responses for out-of-possession variables. Asterisk denotes small effect size

FIG. 3.

Total mean percentage of correct responses for in-possession variables. Asterisk denotes small effect size; Hash denotes trivial effect size. ^ΔDifference between UEFA coaches and performance analysts.

Reliability of the Integrated Approach

There were 241 physical-tactical actions during the first half of a randomly selected match for inter-observer reliability. Out of 241 actions, 202 actions were agreed between the two independent observers (Table 2). The agreement was 83.8% and the kappa statistic of 0.81 reflects a strong level of inter-observer consistency. Regarding intra-observer reliability, the same physical-tactical actions (n = 241) were used. Between the first and second trial by the researcher, 228 actions were agreed (Table 3), corresponding to 94.6% of agreement with the kappa statistic value of 0.94, which is interpreted as an almost perfect intra-observer reliability.

TABLE 2.

Inter-reliability depicted as the number of tactical actions recorded by the 2 independent observers.

Observer 2	Close Down/Press	Interception	Recovery Run	Covering	Run with Ball	Over/Underlap	Push up Pitch	Break into Box	Run in Behind/ Penetrate	Move to Receive/Exploit Space	Support Play	Others	Grand Total
Observer 1
Close Down/Press	50		1				2					2	55
Interception	1	1											2
Recovery Run	4		38	7									49
Covering	8		4	40									52
Run with Ball					8								8
Over/Underlap						0							0
Push up Pitch							4						4
Break into Box								2					2
Run in Behind/Penetrate									16				16
Move to Receive/Exploit Space				1					2	24			27
Support Play									3	1	16		20
Others					1					1	1	3	6
Grand Total	63	1	43	48	9	0	6	2	21	26	17	5	241

TABLE 3.

Intra-reliability depicted as the number of tactical actions recorded by the one observer.

Second trial	Close Down/Press	Interception	Recovery Run	Covering	Run with Ball	Over/Underlap	Push up Pitch	Break into Box	Run in Behind/ Penetrate	Move to Receive/ Exploit Space	Support Play	Others	Grand Total
First trial
Close Down/Press	55												55
Interception		2											2
Recovery Run	2		46									1	49
Covering	1		1	50									52
Run with Ball					8								8
Over/Underlap						0							0
Push up Pitch							4						4
Break into Box								2					2
Run in Behind/Penetrate	1								14	1			16
Move to Receive/Exploit Space	1								1	23		2	27
Support Play										1	19		20
Others											1	5	6
Grand Total	60	2	47	50	8	0	4	2	15	25	20	8	241

Relationship Trends

A trivial relationship existed between the participants’ experience and the mean percentage of their correct responses for all variables (r = -0.08, P > 0.05). However, a nearly perfect correlation was found (r = 0.99, P < 0.01) between the high-intensity distance covered by players using the novel filter and the distance covered from the filter of the commercial data provider (Figure 4).

FIG. 4.

Relationship between data derived from the new filter and commercial provider’s filter. Line indicates linear trend and dashed lines the 95 % confidence intervals. The y=x line denotes the line of identity.

DISCUSSION

This study aimed to develop a systematic integrated approach to quantifying match physical-tactical performance and to verify its validity and reliability. The findings indicated that the physical-tactical variables within this approach demonstrated a high degree of validity. This was evidenced by the strong agreement between the responses of both the UEFA qualified coaches and performance analysts versus the gold standard responses (~92%). Moreover, the inter- and intra-observer reliability of this approach was found to be strong (κ = 0.81) to almost perfect (κ = 0.94), respectively. Additional analyses also indicate that the novel high-intensity filter used for the integrated approach correlates to a nearly perfect magnitude with values attained from the filter of the commercial data provider (r = 0.99).

The physical demands of football have been widely examined in the literature, but they have been quantified in isolation without taking tactical and technical performances into account [8, 9, 26, 27]. A limited number of studies have contextualised the physical match performances of elite players. The integrated approach was initially proposed by Ade et al. [17] and a modified version of this method was adapted by Bradley and Ade [2]. However, there are some limitations associated with some of the physical-tactical actions used such as ‘Run the Channel’ and ‘Drive Inside/Through the Middle’. These two variables provide limited information on the actual tactical purpose of the actions and simply indicate their direction and location. Players could run down the channel ‘exploiting/creating space to receive the ball’ or ‘supporting’ teammates when the ball is advanced forward etc. Thus, through a deep discussion with the working group of experts, new variables such as ‘Support Play’ and ‘Move to Receive/Exploit Space’ were created to provide more information on the tactical purpose of the actions.

The integrated approach represented in this study has been adapted from previous research [17]. However, this previous study did not investigate the validation but did examine, to some extent, the reliability of the initial model. A robust verification of validity and reliability for any novel methodological approach needs to be verified before collecting and analysing data [19]. Validity typically refers to the extent to which a method adequately measures what it claims to measure, and it can be achieved by independent verification from experts [22]. Thus, this present study initially examined the validity of the variables within the integrated approach by evaluating UEFA coaches’ and performance analysts’ opinions versus the gold standard responses obtained from an expert group. The validity of the integrated approach was confirmed with ~92% agreement, indicating that the categories within the methodology have appropriate definitions and demarcations between tactical actions. Thus, this innovative approach could potentially be used for valid data collection and analysis. As this innovative approach still needs intense manual work, an automated solution should be the next iteration for researchers (e.g., machine learning).

The present study followed a systematic validation process adapted from Brewer and Jones [28]. The validation process in the present study demonstrates some similarities with previous research that developed a novel instrument to measure tactical/technical performance [29–32]. For instance, validity was established through communication with experts within the field of study [29, 32]. Nonetheless, regarding statistical analysis for validation of performance indicators, different methods were used. Whilst Gong et al. [30] and Fernandes et al. [29] used the Aiken’s V to assess the validity for performance indicators, Sarmento et al. [32] and Larkin et al. [31] performed no statistical analysis. As no statistical evidence is required for content validity [22], this could potentially explain the disparity in methods of analysis. Since an agreement of 80–85% is the minimum threshold for validation verification [21, 22], again the physical-tactical variables within the integrated approach are valid (~92% agreement). However, additional research is needed to independently verify the findings of this present study before the area can fully accept this new paradigm. As the detailed definitions are now contained within this method, research teams should try to adopt this approach to test independently its merits and limitations.

Both UEFA coaches and performance analysts appeared to be able to answer correctly regarding the reference clips for all physical-tactical variables. Additionally, there was no correlation between UEFA coaches/performance analysts’ experience versus their total mean percentage of correct responses. Thus, as this method was intuitive to all participants, understanding this innovative approach does not necessarily require extensive levels of experience within the football industry and advanced football intelligence. The terminology used for the integrated approach has been adapted from coaching language; therefore, UEFA coaches and performance analysts seem to intuitively understand the physical-tactical categories within the approach.

However, some variables performed more favourably than others when calculating the percentage of correct responses. For ‘Covering’, this was consistently lower compared to the other out-of-possession variables. The definition of ‘Covering’ referred to covering space or a player whilst being goal side of the ball, while ‘Recovery Running’ referred to running back toward your own goal when not goal side of the ball [2, 17]. However, occasionally players tend to cover space or an opponent player when the ball is just past them but they are in a good tactical position. Due to the numerous permutations of these actions and the very brief introductory period during the validation study, these can create some discrepancies. Since there was this type of action in one of the clips for ‘Covering’, this may explain why it had a marginally lower percentage of correct responses for both UEFA coaches and performance analysts. In possession, ‘Run in Behind/Penetrate’ and ‘Move to Receive/Exploit Space’ showed relatively lower percentages of correct responses. This could be due to the increased degrees of diversity in the in-possession variables, or due to more unique tactical actions players attempt when in possession of the ball to trick the opposition team. However, the key demarcation between these actions relates to a player attempting to overtake opponent players in behind [33]. Consequently, more refined differentiators between variables, particularly these actions are warranted. Whilst some variables had relatively lower percentages of correct responses, all variables are > 80%, which implies that all variables are acceptable for validity purposes [22].

It is important to assess not only validity but also reliability of performance indicators since only valid and reliable variables are reliable in performance analysis [34]. Thus, the present study examined inter- and intra-observer reliability. The data demonstrate strong inter- (κ = 0.81) and almost perfect intra-observer reliability (κ = 0.94). This finding was in accordance with previous studies that developed a novel tactical match analysis system [29, 30]. Additionally, similar results were found in the study by Ade et al. [17], reporting excellent inter- (κ > 0.8) and intra-observer reliability (κ > 0.9) of the initial integrated approach. It is worth noting that the present study was the first comprehensive study to examine the validity and reliability of a novel approach for amalgamating physical metric with tactical purposes. The application of this innovative method could generate new insights in the area of football performance. For instance, it could be determined ‘HOW’ teams/players change their physical-tactical behaviour during the congested fixture period or just after the most intense periods during match-play. Nonetheless, it is recommended that only one analyst code games when analysing them manually using this approach as the number of observers, the observer’s experience, and the quality of the observer could influence reliability [35].

Since the present study used a novel high-intensity filter to calculate distances at high-intensity (> 19.8 km·h⁻¹), validity of this filter should be determined. Some optical tracking systems such as ChyronHego have been validated using gold standard technology such as Vicon [36]. Therefore, the data derived from the new filter used in the present study were compared with that from the manufacturer. A nearly perfect correlation (r = 0.99) was found; however, the distance covered at high-intensity running (> 19.8 km·h⁻¹) using the novel filter was systematically lower than that obtained from the commercial data provider. This may be explained from two viewpoints. Firstly, a different algorithm was used for the new filter as companies do not typically disclose their algorithm for their established filter. Another perspective is that the dwell time for the manufacturers filter is different from that of the new filter used for this study. The findings from Varley et al. [37] support this notion as different filtering methods and dwell times can significantly affect the measurement of physical movements such as high-speed running and sprinting. Whilst some studies have assessed the accuracy of the most widely used tracking technologies in elite team sports, comparing them with VICON as the gold standard [36, 38], different filters within optical tracking systems have yet to be compared with the gold standard. Thus, further research is warranted to compare different filtering methods within optical tracking systems to the gold standard to examine their accuracy.

Limitations

There are some limitations that should be addressed. First, the novel integrated approach does not include other intense actions such as accelerations/decelerations that are more frequently performed during match-play, and do not reach high-intensity speed thresholds [18]. Therefore, these should be added into the method in future to understand more complete match physical demands with context. Additionally, the present study only focused on the categorisation of a single primary action because the majority of high-intensity actions are singular. However, football is complex and requires players to execute not only a singular action but a series of actions whilst producing high-intensity running (e.g., a player produces a ‘covering action for 3 sec and then ‘presses’ for 1 sec) [2]. Although the number of hybrid actions performed during matches is very small, hybrid actions would add more transparency and insight to coaches and applied practitioners.

CONCLUSIONS

The present study verified the validity and reliability of the integrated approach. Although more refined differentiations for some categories are required, the findings demonstrate that this innovative approach appears to be valid and reliable. The new data derived from the integrated approach could aid practitioners’ and coaches’ understanding of the physical demands in relation to tactical aspects of the game.