Prognostic validity of a subjective coach assessment and motor performance tests for talent selection in football: science boosts coaches’ eye!

Sieghartsleitner, Roland; Zuber, Claudia; Zibung, Marc; Conzelmann, Achim (8 February 2018). Prognostic validity of a subjective coach assessment and motor performance tests for talent selection in football: science boosts coaches’ eye! In: 10. Jahrestagung der Sportwissenschaftlichen Gesellschaft (SGS) der Schweiz: "Leistung im Sport". Conference Abstracts. Magglingen. 8.-9.02.2018.

Text BookofAbstracts_SGSTagung2018.pdf - Published Version Restricted to registered users only Available under License Publisher holds Copyright. Download (326kB)

Introduction

Talent selection in football is an inevitable consequence of limited resources within clubs and national federations. Whenever it comes to this undesired necessity of selecting the most promising football talents for any development program, sport science advocates the use of multidimensional approaches [1]. This means that an evaluation of young football talents based on the single dimension of current competitive performance is insufficient to determine a players potential for future performance. It would rather be necessary to consider various endogenous (e.g., motor performance, biological age, psychological features) and exogenous (e.g., social support, characteristics of training history) predictors of relevant areas, which should ensure a more precise prediction of future performance [2].
Whilst there is undoubtful benefit of multidimensional models for talent selection from a theoretical point of view, there seems to be a lack of knowledge according to their practical application [3]. First, some of the potential dimensions of interest cannot be operationalized in an evidence-based way, which could be taken into account for the use in a quantitative decision-making process within talent selection (e.g., social support or training history). Second, the procedure of integrating single measures from several dimensions to an overall assessment is challenging as well as vigorously debated [4]. Therefore, no unified understanding of delivering applicable solutions for overall assessments from multidimensional data has evolved yet.
Resulting from these methodological limitations according to the use of multidimensional approaches for talent selection, clubs and national federations typically reduce the complexity of their assessments to fewer or single dimensions. Most frequently, they rely on subjective coach assessments solely [5]. Within several talent development programs, objective data from motor performance diagnostics is added [6]. In doing this simplification, there is uncertainty about the value of these two dimensions for decision-making about the future of young players. For that reason, the current research investigates the prognostic validity of the subjective coaches’ eye and objective measurements from motor performance diagnostics and compares their usefulness for talent selection in football against each other.

Methods

The sample consists of 122 talented football players (all born in 1999), from several regional youth squads throughout Switzerland (including 16 players with at least one nomination for the Swiss U17 national team; 13.1%). During the season 2012/2013 (U14 age category) all of the players took part in a test battery consisting of nine tests to determine motor performance: dribbling and agility, ball control, shooting, juggling (all from [7]), YoYo-IR1 [8], core-strength endurance [9] and counter movement jump as well as 40m-sprint. In addition, club coaches carried out a visual scale estimation procedure to rate players’ current game performance (visual scale between 0 and 100). With a concordance coefficient of rtt=.89 (n=140) the inter-rater reliability for this subjective coach assessment can be described as satisfactory [10].
Data analysis used binary logistic regressions (BLR) from SPSS (version 24) and receiver operating characteristics (ROC) from MedCalc to determine the capability of a) the subjective coach assessment, b) the motor performance tests, and c) a combination of those two to identify later U17 national team players three years before (at U14 age category). Within this procedure, BLR first calculated the
likelihood for each individual to be categorized as U17 national team player. Afterwards this likelihood was used to create a ROC. The area under the curve (AUC), as an index for measuring the quality of ROC, and its standard error were finally used to compare the ROC models against each other [11]. In addition to this immediate statistical comparability of the model quality, ROC enhances the statistics from BLR by the descriptive reference to sensitivity and specificity, better known as proportion of correctly selected talents and correctly deselected non-talents in terms of talent selection. Finally, ROC creates the possibility to ensure the most efficient selection threshold (known as Youden Index, which maximizes the sum of sensitivity and specificity), as an additional benefit over BLR.

Results

According to the results of the BLR analysis, all three models are significant and show model fits (Nagelkerkes R2) from .38 to .62 (subjective coach assessment: χ2=28.2, df=1, p<.01, R2=.38; motor performance tests: χ2=39.9, df=9, p<.01, R2=.52; combined model: χ2=49.7, df=10, p<.01, R2=.62). Further the AUC [95% confidence intervals] for the subjective coach assessment reaches .86 [.78; .92], the one for motor performance tests shows .91 [.85; .96] and the combination of both dimensions displays an AUC of .94 [.88; .98]. The nonparametric approach of comparing AUCs [11] indicates a significant higher AUC of the combined model, compared to the subjective coach assessment (p=.02). Further comparisons of AUC do not show any differences (coach assessment vs. motor performance tests: p=.31, motor performance tests vs. combined model: p=.32). The subjective coach assessment model uses its most efficient selection threshold from ROC by picking 36 out of the total 122 players, containing 13 correctly identified U17 national team players (sensitivity: 81.2%; specificity: 78.3%). The model from motor performance diagnostics selects 25 players, 13 of them were correctly identified national team players (sensitivity: 81.2%; specificity: 88.7%). Finally, the combined model bets only on 24 players, containing 14 true-positive national team and 10 false-positive non-national team players within (sensitivity: 87.5%; specificity: 90.6%).

Discussion

The common subjective coach assessments of talent development programs in football [6] seem to be a powerful tool for talent selection in terms of prognostic validity. Around 80% correctly selected talents and correctly deselected non-talents show the ability to buffer the biggest problems of the unwanted selection. On the other hand, the use of scientific and objective motor performance diagnostics further boosts the value of coaches’ eye to a remarkable extent. Whilst there is no significant difference between the models of single predictors, the combined model is able to take almost 90% correct decisions three years before the selection to U17 national team and outperforms the subjective rating. Although prognostic validity of coach assessments [5] and motor performance diagnostics [7] have already been researched, the head-to-head comparison of the two most common assessments in talent selection within youth football adds value to the scientific discussion.
Furthermore, the current results seem to be ambiguous in relation to earlier presented believes in benefits from multidimensional approaches [1]. On the one hand, the single use of subjective coach assessments or motor performance diagnostics shows already substantial prognostic validity to predict the U17 national team membership of fourteen-year-old players. Contrariwise, integrating both predictors by means of simple BLR into a two-dimensional decision led to a noticeable improvement of decision quality. Therefore, a benefit of even more appropriate, presumably nonlinear statistical methods for handling data from multiple dimensions for overall assessments in talent selections seems to be indicated. Although first steps in this direction have already been taken (e.g., person-oriented methods [2, 4]), there is the need to further develop these methods and make them more applicable to practitioners in the field (clubs and federations). This may contribute to a further improvement in deciding about the potential of young sporting talents future performance.

References
1. Vaeyens R, Lenoir M, Williams M, Philippaerts RM (2008) Talent identification and development programmes in sport. Current models and future directions. Sports Medicine 38 (9): 703–714.
2. Zuber C, Zibung M, Conzelmann A (2016) Holistic patterns as an instrument for predicting the performance of promising young soccer players – a 3-year longitudinal study. Frontiers in Psychology 7:1088. Available: http://journal.frontiersin.org/article/10.3389/fpsyg.2016.01088/full.
3. Fuchslocher J, Romann M, Rüdisüli R, Birrer D, Hollenstein C (2011) Das Talentidentifikationsinstrument PISTE - Wie die Schweiz Nachwuchsathleten auswählt. Leistungssport 4 (2): 22–27.
4. Zibung M, Conzelmann A (2013) The role of specialisation in the promotion of young football talents: A person-oriented study. European Journal of Sport Science 13 (5): 452–460.
5. Christensen MK (2009) "An eye for talent": Talent identification and the "practical sense" of top-level soccer coaches. Sociology of Sport Journal 26 (3): 365–382.
6. Höner O, Leyhr D, Kelava A (2017) The influence of speed abilities and technical skills in early adolescence on adult success in soccer: A long-term prospective analysis using ANOVA and SEM approaches. PloS one 12 (8): e0182211.
7. Höner O, Votteler A, Schmid M, Schultz F, Roth K (2015) Psychometric properties of the motor diagnostics in the German football talent identification and development programme. Journal of Sports Sciences 33 (2): 145–159.
8. Bangsbo J, Iaia FM, Krustrup P (2008) The Yo-Yo intermittent recovery test. A useful tool for evaluation of physical performance in intermittent sports. Sports Medicine 38 (1): 37–51.
9. Rosser T, Müller L, Lüthy F, Vogt M (2008) Basistests SUISSE Sport Test Konzept: Validierung einer sportmotorischen Basistestbatterie für den Schul- und Nachwuchssport. Schweizerische Zeitschrift für Sportmedizin und Sporttraumatologie 56 (3): 101–111.
10. Zuber C, Conzelmann A (2014) The impact of the achievement motive on athletic performance in adolescent football players. European Journal of Sport Science 14 (5): 475–483.
11. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics 44 (3): 837–845.

Interest & Impact

Downloads

7 since deposited on 08 Mar 2018
1 in the past 12 months

Citations

5 Citations in Web of Science ®
6 Citations in Scopus

Search

in Google Scholar™

Services

Actions (login required)

Edit item

Actions (login required)

Edit item