When it comes to training load management, sports scientists are constantly trying to manage the tension between simplicity and complexity, and between usability and fidelity. The acute: chronic workload ratio (ACWR) came under attack for taking an overly simplistic analysis approach to a complex problem. At the other end of the complexity spectrum, there is trepidation about “black box” structures; those which are opaque, non-intuitive, and difficult to interpret in relation to decision-making (Teng et al., 2022).
Sports scientists have a desire, a need in fact, to understand the results that machines output. We strive to “unpack the black box” (Malone et al., 2017), to understand the inner workings of force plates and tracking technologies. Our interventions and recommendations are built upon trust; trust that is itself built upon the ability to reason our decision making.
It was unsurprising then that artificial intelligence (AI) transparency and black boxes came up as a key topic for sports science practitioners.
In my series so far on AI in sport, Zone7 has added to the discussion on the AI Dictionary for sports scientists, sports science data inputs, and the importance of contextual factors. As we move onto the analysis part of the data pipeline, they have kindly agreed to answer practitioner questions relating to analysis, transparency, and validation.
Thanks to Zone7 CEO & Co-founder, Tal Brown, and Performance Director, Rich Buchanan, for sharing their thoughts!
Firstly, can you give an overview of Zone7’s approach to data analysis?
First principles - access to comprehensive datasets that touch on what we, and the practitioners think are key important inputs. Data should be in digital form and collected through validated technologies.
Be technology-agnostic and reliably transform workload, personal history and other inputs from multiple trusted sources into a robust multi-team data lake that reflects data quality as well as volume.
Have a quantified sense of how consistent and clean different datasets are, so that corrupt or sparse datasets can be identified and bypassed, as discussed earlier in the series.
Careful care is given to the completeness of the timeline. In other words, identify and have clear criteria on how to deal with data gaps in the timeline. Some examples are soccer/football players that leave for international fixtures, or temporary lack of data due to malfunctions or decisions by practitioners.
Reflect the environmental needs of both the organisation’s and individual user’s attitudes towards risk. This is manifested in digital data articulating the relevant context, as well as calibration to conform to the environment’s preferences.
Perform data analysis efficiently with an understanding that the human performance space is multifactorial.
Dynamically adapt and learn over time to improve injury risk forecast accuracy as variables change and more data is collected, meaning the system’s sensitivity and specificity are expected to improve.
Appropriately translate data trends and clearly communicate actionable insights for the human end-users to apply as deemed appropriate.
Does Zone7 operate with a ‘black box’ approach? How do you address concerns relating to this?
It is important that the underpinnings of machine learning (ML)/ deep learning (DL) solutions deployed in a medical and sport science environment be as transparent as possible, in terms of how the algorithm models have been developed and validated and by whom.
Interpretable black boxes describe big data being ingested, transformed and analysed by ML/DL processes, resulting in insights in which humans can interpret and act upon as deemed appropriate.
When systems are created “in-house”, it is typically easier to access these insights and the reasoning behind them. However, commercial data science solutions typically do not provide their source code and the human operator might be left with bottom line insights that are difficult to reason through.
To avoid the pitfalls of working with a black box, Zone7 adheres to two key principles:
The validation process should be as transparent and strict as any in-house solution
While source code may not be provided, explainability components are an absolute must to provide operators with the ability to “reason through” insights before implementing.
Where have you seen black box projects fail?
There are two underlying dangers of working ‘blindly’ with a black box system:
Insufficient help or context is provided to understand the reasoning driving the risk assessments, hence the practitioner must either spend considerable time ‘reverse engineering’ or go on blind faith.
For the system to overstep and encourage insights to be interpreted as causality related. Insights must be clearly tagged with such context and robust validation must be provided if causality is suggested.
Correlation is not necessarily an explanation of causality. Yet, it is important to acknowledge that ingested datasets should be representative of what is trying to be interpreted as possible causality by the algorithms. Such an example in the sporting context is a Global Positioning Systems (GPS) and accelerometry dataset, which represent an insight into an athlete’s mechanical and locomotive output when participating in their specific sporting activities. A common example (rightly or wrongly) is the broadly accepted notion that external workload datasets from player tracking technologies indicate some level of causality relationship towards amplifying performance potential and understanding injury risk.
Presenting the algorithms with datasets that have no known causality or relationship to the output would be unwise. It is therefore important that domain experts apply their professional knowledge when deciding what datasets are important for algorithms to examine, in order to seek a deeper understanding of potential causal relationships. It is also not uncommon for datasets collected by commercially available solutions (e.g., strength assessments, GPS) to be ingested along with proprietary/in-house datasets (e.g., strength and biomechanical assessments).
How does Zone7 balance the need to keep the company’s Intellectual Property (IP) private with the desire to be transparent?
Transparency has multiple levels, and each organisation or practitioner may require a different level depending on their experience and/or needs. Zone7 operates with the following principles at the core of our engagements with clients:
Detailed validation studies (a.k.a. retrospective analysis reports) are shared openly with clients. These provide an opportunity to learn how the algorithm is tested against other datasets.
Custom validation studies may be created for clients, providing them with full control over inputs and outputs. Results from these projects include all the raw output data so clients can understand how key metrics are calculated.
Transparency is also about explainability of the results - all of Zone7’s products include an explainability layer, which breakdown the “why” - what specific parameters are driving risk at any moment. These are both visualised and provided as data for clients to engage and investigate.
Explainability can also be generated by simulating modifications of controllable variables and asking a model "what if" questions. This is also a key part in building trust and is shared openly with the environment operators both retrospectively and in live form.
For clients with in-house data science teams, we also provide documentation and detailed workshops about the internal workings of the platform. That said, Zone7 is not an open source platform, hence source code is not provided.
We go into more detail in our validation study here.
Machine learning techniques are said to be at risk of overfitting – how does Zone7 reduce this risk?
Overfitting refers to cases where an algorithm’s results are skewed or biased to show ‘better’ results. This is typically associated with a (usually unintentional) leakage of data from the dataset used to test the algorithm and generate results, and the dataset used to train the algorithm.
Zone7 avoids overfitting by adhering to strict operational principles of absolute separation between training, validation data, and test data. When creating validation studies per client, this principle is manifested by never including the client’s data in the training dataset.
One of the claims frequently made against ML is that it is not actionable. How does Zone7 turn algorithms into interventions?
The holy grail of data science is to drive impact through action. To make this happen, any predictive technology must provide the following:
Agree on the desired metrics both for retrospective analysis and for live impact on the environment (e.g., how we measure player availability and changes to it)
Accurate and trusted forecasting
Contextual explainability
Provide the operator with potential modifications to controllable variables that will hopefully drive towards the desired impact.
Zone7 provides all three of these components to ensure expert practitioners are empowered with a time efficient and deeper insight into what the data they are collecting is indicating. Insights include detailed risk forecasting profiles, reasons behind each forecast as well as load management scenarios that are ‘human readable’ and can be easily interpreted.
Gavin Benjafield, Assistant Coach & Performance Director of Los Angeles FC, recently shared how he actions the Zone7 insights day-to-day at LAFC in a webinar that is available to watch here.
How does your analysis support your clients, but also potentially the wider sports science community, in understanding injury risk forecasting and training load management?
Through interrogation of the data lake, we are able to explore and disseminate insights to specific clients, as well as the wider community. We frequently engage with clients and industry experts to learn what the industry needs and is curious about. That’s how this series came about; starting with the questions and comments that came from the sports science community.
By providing answers to these questions, significant or not, we feel we are able to contribute to the sports science landscape. As scientists, we are driven to share our findings to help improve knowledge and further understanding.
We’re trying to focus on those questions most relevant, particularly at time-critical moments. We explored pre-season injury statistics prior to this season’s pre-season, then invited one of our industry experts, Damian Roden, to provide his personal perspective on the findings. Recently we explored injury risk for players involved in this unique men’s World Cup, by analysing injury trends from other mid-season tournaments.
Thanks to Zone7 for sharing their thoughts on transparency in sports science AI.
Keep a lookout for our next pieces in this series, which will focus on the system outputs and thereafter, the application of this information in the applied setting.