Football AI Tutorial: From Basics to Advanced Stats with Python

almost two years ago I published my first football AI video on this channel back then I showed you how to detect and track ball players and referees on football pitch I learned a lot in the meantime so today we are taking this project to the next level I'll show you how to use Sly embeddings to divide players into teams how to use the keyo detection and homography to create video game style radar view we'll also use the extracted dat to calculate some Advanced stats like ball trajectory and vorono diagram illustrating team control over the pitch

and most importantly you don't need to be an AI Guru to follow this tutorial I'll guide you through the whole process all you need to know is some python Basics but before we start let's get one thing straight I'm from Europe and where I live we call the sport football first thing first let's lay out the plan the project is quite complicated so I drawn a diagram illustrating all the models and tools we will use today the Row video will be split into frames and each frame will be processed by two models object detector and

keypoint detector both models will be fine-tuned on custom data sets the object detection model will detect players goalkeepers referees and the ball while the keyo detection model will detect 32 characteristic points on football pit since we'll know the positions of these points on both the video frame and the Real Pit we can later use them for perspective transformation detections for all classes except the ball will then be passed to B track assigning each of them unique Tracker ID and tracking their movement on subsequent video frames next we will use embedding analysis to divide the players

into two teams we will use S to generate embeddings U map for dimensionality reduction and K means to create two clusters of players then using previously obtained results of keyo detection will perform perspective transformation in two directions from Peach plane to frame plane and from frame plane to Peach plane the former will allow us to project virtual lines onto the video the more accurate the predictions of our keypoint detection model then more closely the virtual lines will overlap with the actual lines on the peit on the other hand by projecting the positions of ball and

players from the frame onto the peach will be able to create a radar view known from video games showing the actual positions of players on the pit it should also be possible to draw the exact path of ball movement as well as vorono diagram illustrating the control of individual teams over the peach okay enough of the talking let's write some code let's start by detecting ball goalkeepers players and referees to do this we'll use YOLO V8 object detector trained on custom data set we have a dedicated tutorial on this channel showing how to train such

models so make sure to watch it if if you want to learn more this time however we'll focus primarily on aspects specific to analyzing football game the original data for this project comes from dfl Bundesliga data shootout competition organized by kagle around 2 years ago as part of this competition the organizers released around 80 30C videos from 20 different matches today we'll use this data to fine tune our object detection and keyo detection models I prepared a data set of almost 400 images which I uploaded to roboplow Universe each image has been annotated with classes

ball goalkeeper player and referee notice that goalkeepers and players are separate classes this is because goalkeepers wear different colored uniforms than the Outfield players the additional information that a particular player is a goalkeeper will help us later when dividing players into teams the original images which has a resolution of 1920 by 1080 are then subjected to post-processing including rescaling to a square format of 1280 by 1280 they're also stretched to fill the entire space after the transformation they look like this now one of the key challenges ahead is reliable ball detection the ball is small

fast moving object it can be blurry in frames where it moves particularly fast making it harder for the object detector to accurately located life sport events often have cluttered backgrounds particularly in football when the ball is high in the air amid spectators not to mention other objects on the peach that may look similar and confuse our model when preparing a data set we need to make sure that all these anomalies are included this will help to train a robust model capable of handling all these edge cases later on now that the data set is prepared

is time to train the model by the way all the models necessary for this tutorial are already trained and available publicly so if you want to skip the training part and jump ahead feel free to do so using the chapter time stamps in rof flow Sports repository you'll find a readymade Google C template that we'll use for model training Google collab has a free tier allowing you to use GPU for a few hours this should be just enough for us to train our model for free before we start we need to make sure that the

secret stop located in the left sidebar of Google collaps UI contains our roof flow API key we'll need it to download the detection data set from roof flow Universe to get it log in into your roof flow account or create one if you don't have it yet and then expand the settings dropdown from the left panel and select API Keys now copy it and paste in Google collaps Secrets tab now we need to make sure that our environment is GPU accelerated to do this I run Nvidia SMI command if your output looks like mine you

are all set if not in the top bar click run time and from extended drop-down select change runtime type choose Nvidia T4 accept and restart the notebook next it's time to install necessary python dependencies for the training of object detector we only need two packages ultral litics to train the model and roof flow to download the data set and deploy the fine-tuned model this cell may take few seconds to complete but when everything is ready it's time to download the data set the link to the data set is in the description below the video Once

you open it click data set and then select the version you want to use in my case it's the latest one then click download data set and in the export popup select YOLO V8 as the output format checking show download code click continue and after a few seconds a code snippet will be generated which we copy and paste into Google collab we execute the cell and after a few seconds the data set will be downloaded to our coding environment we can confirm that by opening the data Explorer located in left panel of Google collab UI

in the data sets directory we see football player detection data set divided into test train and valid subdirectories we can also view data yl where among other things we can see class names and paths to the sub directories and finally it's time to start the training the most important thing here is to select a 1280 input resolution by default model trains with 640 input resolution this means that every image and every video frame that runs through the model is first rescal to 640x 640 squares in our case this means that the amount of pixels in

the input image is reduced by four to five times and this is the problem because of our ball detection you see on the raw image the ball barely occupies few dozen pixels and after such a rescaling the amount of information in the image May no longer be sufficient to reliably detect the ball for this reason we increase the input resolution from 640 to 1280 pretty much keeping the amount of pixels the same and boosting the accuracy for ball class detection but nothing in the world is free except of my tutorials so we need to pay

the price and the pric is slower training and slower inference since the input resolution increases the amount of memory needed during the training we need to adjust our batch size accordingly batch size is the number of samples processed in one forward and backward pass of training algorithm if you're training the model on more powerful GPU card you might be able to use larger batch size I set mine to six training such a model will take take at least an hour to complete so I'll use the magic of Cinema to speed up the process by the

way on the right side of collapse UI you see few charts one of them is GPU Ram where 18 to 19 gab of 22 available is already allocated this is the value you need to keep in mind when choosing custom Training hyperparameters you see that we are already in red just slightly larger batch size and we would be done out of memory exception and training [Music] kiled when the training is finished we can Benchmark our model to make sure everything went well a popular metric used in such situations is map mean average Precision it measures

the average Precision across all classes and different IOU thresholds providing a single number to represent the overall performance of the object detector here we can see the global map value and per class value the model does quite well at detecting goalkeepers players and referees and as expected worse with ball class but don't worry we have few tricks up our sleeve that we will use to clean up the ball detections the last step is to upload the model to roboff flow Universe this way we don't have to worry about storing and versioning of our weights we

simp upload the training results to universe and load the model when we need it at the end a link to rlow universe is display where we can find our uploaded model now it's time to use this fine tuned model to build something cool I debated it for a long time whether to use regular IDE or jupeter notebook for coding in this tutorial and in the end I chose Google collab it's just a lot easier to play with different visualizations and I assume that a lot lot of viewers don't have gpus so it should be a

lot easier for them to play with the code there however if you prefer to run your code locally all the code along with installation instructions is in roof flow Sports repository you can find the link to this repo and the collab that I'm using in this tutorial in the description below okay let's start with our Google collab setup before we do anything else we run Nvidia SMI command to confirm that we have access to GPU and after few seconds we should see the familiar output if you see something like this you're all set then we

install our python dependencies this time I need fre python dependencies G down inference GPU and supervision I use G down to download files from Google Drive inference to pull and run models from rlow universe and supervision to process videos and among other things perform really nice visualizations uh this installation should take few seconds to complete I sped it up just a little bit to save some time next we install Sports repository this is the repo where we store a big chunk of our Sports demos uh and utilities we'll use some of them uh for visualizing

the outputs and last but not least we uh pull five files uh those are video files from the original Bundesliga kagle competition and we will use those um as our uh Source videos for demos um so we already downloaded four of them here is the fifth one and at the very end uh I'm setting up an environment variable that will be consumed by the inference package uh telling it which uh execution provid onx should use so in that case I would like to use Cuda our environment is ready now we can play with some models

okay so let's create a new text field and call this section ball players goalkeepers and referees detection and now in the code field we will will import get model from inference package that we just downloaded and user data from google. cab now we will use users data get to extract the value of roof flow API key secret that we set up in Google collab you can see that beforehand I added this secret to Google collab secret step now I'm just retrieving it and I'm also setting up another constant called player detection model ID and I'm

copying the ID of the model that we ftuned for roof flow Universe the link to that page on robow universe is in the description below now I'm creating an instance of model calling the get model function and providing my model ID and my roof flow API key as arguments let's just break the line because it's quite long and execute the cell cell may take a little bit of time to execute because we are pulling the weights from roof flow universe and loing them into the memory I hope that in few seconds we should be we

should be done yeah now I'll create another constant called Source video path and I will just set its value to one of uh the path to one of the files that we uh downloaded from Google Drive just few seconds ago and I'm importing supervision as SV the first utility that we will use from supervision is called uh get video frame generator and we just pass uh the path to our source video file over here and it allows us to Loop over frames of the video here I'm just using the next uh to get the first

frame from the generator and I plot it on the screen awesome now it's time to run uh our uh model on that frame so I'm calling player detection model infer passing my frame and confidence threshold as as arguments and that function Returns the list so I'm just picking the first element from that list and then parsing the result into supervision detections this the class from supervision library that allows us to manipulate the actions and visualize them and do all sorts of useful stuff I'm initiating the most basic annotator from supervision called box annotator that draws

boxes and I'm creating a copy of the source frame and I'm annotating that copy with my bounding boxes passing the copy of the frame and detections to it I run the frame uh run the cell um ah of course um I messed up uh we need to pass the annotated frame to the plot image cool we see the bounding boxes now beforehand I prepared a list of colors that I would like to use for visualization so I'm just passing that list to color palette and passing that as an argument running the cell once again and

the colors are updated awesome now it would be awesome to display more information about our detections to do that I'll use another annotators um another of annotators from supervision Library called label annotator I'm passing the same list of colors um those will be used as Bank background for those labels and I'm setting the text color for black now uh below my box annotator I will run uh the label annotator so once again uh I'm passing the annotated frame along with detections and I press shift enter now what we see is the default uh uh label

uh it's just the class name if I would like to show more information I can do it as well um so I would like to show the class name and the confidence levels so detections class that was created from paring the inference results stores that information so you can just um get class names and confidences from there use zip to Loop over both of them at the same time and pars our labels I just need to pass labels as another argument to label annotator shift enter once again and yep we see uh both class names

and confidence levels now let me show you how to go from processing of a single frame to processing of the whole video first thing first I Define new constant called Target video path uh where I defined the location of the result file and of course I would like our Target file to have the same FPS and the same resolution as the source one so to do it I first create an instance of video info this is one of the utilities available in the supervision that allows you to extract the information about the video I pass

the path to our source video file so that we extract The Source information and then I create another um instance of another utility from supervision package called video sync that allows me to save videos to the hard drive I pass the Target location of the video and our video info object this way we Define the location and the FPS and the resolution and then instead of getting just the first frame from our frame generator we Loop over the frames even important tqdm to get this nice loader visualization showing how many frames we already processed and

how many frames are still to be processed then I indent all the code that previously was there to process a single frame to now process every frame in the for loop I open the video sync with the wh statement and at the end of the nested for Loop I call video sync right frame to pass the currently processed frame into the video sync now I just run the cell I speed up the footage so that we don't need to wait so long for it to finish processing but when it's completed we can download the result

file we just open the file explorer quick download and after few seconds the video should be on our hard drive here's the result as expected players goalkeepers ball and referees are being uh detected we see bounding boxes and labels with probabilities so this is the most basic visualization of the results now let's try to do something more advanced our visualization so far was very simple and to be honest a little bit boring so I plan to make it a little bit more interesting uh by using more advanced supervision annotators uh to be precise I will

use ellipse annotator to annotate uh players goalkeepers and referees and triangle annotator to annotate ball okay so we go to the top of our cell and we start by defining new constant called ball ID it will store the class ID for ball class now I I'm renaming my ball annotator into ellipse annotator and I'm no longer calling box annotator Constructor I'm calling ellipse annotator Constructor uh below I'm commenting out uh label annotator we don't need it for now but I'm initiating new one uh call it triangle annotator um and here I will use a nice

yellow color um to annotate the ball I'm calling color from hex and I'm passing the hex value of that yellow color I can also Define the geometry of that triangle uh those are in pixels now just after parsing our detections we will divide them into two groups so the first group are only detections for a bow class and the second group are all other detections we can uh simply filter them out using uh class ID so here you will have all classes but ball now I'm commenting out the label parsing and in the an ation

section I need to clean up a little bit so I'm renaming my box annotator into ellipse annotator and I'm passing uh different detections those filtered out and instead of calling label annotator I'm calling triangle annotator and here I'm passing on the ball detections now when I run the cell we see new visualizations already looks a lot more interesting um the color coding still the same but instead of boxes we have ellipses and we have the triangle over the ball but the triangle is quite close uh to the ball um doesn't look so good but supervision

have a utility that allows you to expound Boxes by a specific number of pixels so what we will do is we'll expand boxes for ball class by 10 pixels and that should move away the triangle marker a little bit higher so when you run the cell we see that the triangle is no longer touching the ball it's just over the ball that's a lot cooler now we remove one value from our color uh list and that's because we no longer use the ellipse or previous box annotator to annotate the ball and ball had class zero

so the first one uh in the list um and to adjust for that we move all classes for remaining detections uh one down so if players had class ID 2 now they have class id1 and on top of that we use non-max suppression to remove overlapping detections and we use class agnostic true to do it regardless of whether they are from the same class or not it'll just improve the quality of predictions so here is our result uh after change annotators so I made it side by side so on the left of course we see

the the initial annotations and on the right we see the Nuance I hope that you already see that I was aiming for this kind of video game Vibe with ellipses below the players and this characteristic uh triangle marker over the ball uh now let's add some tracking the next step is to add tracking so for that step to have an sense we need to have a way to display our tracker IDs and we will uncomment label annotator to do it we remove this extra color from the um beginning of color list uh the one that

we also removed from from ellipse annotator and we update text position from the default top left uh to the bottom center so we want to display the text kind of like in the middle of those ellipses now we also initiate by track this is the tracker supported by supervision uh and we will use it to track all detections but the ball so we call Tracker update with detections and pass all detections variable but once again this is the variable that stores all detections bable now we also uncomment the labels parsing and instead of displaying class

and confidence now we will only display Tracker ID and put this kind of like hashtag prefix in front of it and we put back uh label annotator annotate call to to our annotation section pass our annotated frame all detections and our par labels and when we run it we see ellipses with tracker IDs awesome so here's the video result for that step we successfully managed to track goalkeepers players and referees and with our next step we are taking that to the next level as we will divide players and goalkeepers into two teams in theory dividing

players into teams is simple we could for instance run our object detection model crop boxes containing players and then calculate the average color of pixels within the crop however this approach have few drawbacks first besides the player the crop also includes the back ground like the grass the stands or even other players second the proportion of pixels representing the player and the background in the crop varies depending on player's pose third different lightning in different areas of peach affect the average color value all this makes the color based approach unreliable it may work for some

scenes but it will definitely fail for others therefore I've decided to use an approach that might seem over engineered but works perfectly in practice some time ago on this channel I showed you how to use clip image embeddings to look for similar images today we'll use a similar strategy to look for players from the same teams embeddings capture a semantic meaning of an image making it robust to variation in Po occlusions and lightning conditions this time instead of clip we'll use clip it's more computationally efficient making it a better choice for realtime applications for each

crop containing a player will generate a 768 dimensional SE Gip embedding Vector then using umap we'll project this Vector down to threedimensional space umap is a dimensionality reduction technique that helps us visualize High dimensional data in low dimensional space while preserving the relationships between data point finally will trade K means to divide players into two teams K means is a clustering algorithm that helps group similar data points together it will find two clusters in our threedimensional embeddings representing that two teams trust me it sounds complicated but it's actually quite simple so first thing first we

need to collect a sufficient set of player crops for us to use to train our team classification model to do it we once again uh create an instance of frame generator but this time instead of uh looping over every frame what we will do is we'll use stright argument uh to skip every 29 frames and return every 30 frames uh just like before we will use a for Loop to Loop over frame in frame generator uh we will use qdm to get the nice loader uh and uh maybe add a description um saying that we

are collecting uh crops I also created uh a list called crops before the for Loop and we will append our crops to that list now inside the for Loop I'm doing uh everything that we already done in the previous sections so I'm calling the infer function in our player detection model pass the frame pass the confidence threshold uh extract the first value from the list and and pars that result into supervision detections um then once again I'm running non suppression to remove uh the overlapping detections setting the uh IOU threshold to 1 uh 0.5 um

once again uh making it class agnostic next we'll filter our detections by class so previously we uh already done that for ball class but now we will do that for players so in our model uh player have ID to so we just uh use that player ID to filter um detections by class ID and now what we do is we uh do lease comprehension to Loop over detections the player detections that we already have and use one of the utilities from supervision package that allows us to crop out um parts of the image based on

the box definition so I execute that uh cell just to make sure that everything works correctly um the FPS of that video is is 25 and we are striding uh every uh 30 frames uh so that's up to 750 frames so so far so good looks like we get all the frames to make it a little bit easier for us to use that uh I wrap all that code into utility function call it extract crops now I just update the parameter in the frame generator so that we don't use the global value and we use

the the value that is passed as the argument to extract crops function and Below we can test if it still works um it should so let's just wait a few seconds for it to finish yep now we can just display the count of our crops so alt together we got 478 crops uh we can plot sample of that set let's say 100 of them just to visually confirm that we have uh everything we want and here's the result of uh crop collection so we see that there are players from uh green and red team now

we need to build another algorithm to automatically uh sort them into those two teams so I prepared a code snippet uh that will allow us to load sigp model that we'll use to uh create embeddings for those crops uh the code uh is pulling a SE lip uh from hugging phase and uh loading it into the memory um and also making sure to load the model on the GPU if the GPU is available and now we need to run this model on those crops and get our embeddings so the first thing that we need to

do is to convert our crops from uh open CV format that we have right now into the pillow format so supervision is using uh open CV as as the engine internally and uh sigp is expecting those crops to be in pillow so we used Le comprehension to convert crops um from numai arrays into pillow images now we also use uh chunked uh function from more e tools package to split those crops into batches so I set the batch size to be 32 I pass our crops and the batch size to to Chun function and as

a result I will get batches um I initiated the list called Data where we will store our embeddings now we just Loop over the batches use sqdm to get the nice uh loader showing us how many batches were already processed and if you ever used Transformers package you know that models there usually have the model and the processor and the processor is responsible for pre and postprocessing the data and the model just just executes so here we uh pass our batch to pre-processor that creates uh inputs in the format expected by the model then we

just pass those inputs into the model get our outputs um and then what we need to do is is from those outputs get our embeddings so embeddings are located in the last hidden uh state but that um layer have uh higher Dimension so it's not one by 768 it's more than that so what we need to do is we just average out across the second dimension so that uh in the end we will get those nice vectors uh and when everything is done I just append those embeddings into uh the data and use n aray

ah the model was running uh in the in the training mode so it was calculating the uh gradients and that's why we got that exception so we will just call uh with torch no grat and indent everything inside that uh and this will make sure that the model uh don't calculate gradients we need to restart the the upper cell that extract the crops because we already over the crops uh from from open CV to p and right now um you would get an exception uh here so we need to restart the upper cell and go

back here we already have the crops we calculate embeddings so every uh every batch was uh 32 by 768 and when we concatenated them we got 478 by 7 uh 68 so that means that we have 478 examples that exist in this very high dimensional space with 768 dimensions and um that's very hard to to imagine of course so we will use um map to reduce those Dimensions uh and in the end we'll get 478 examples in free dimensional space and when we will have that we will just pass those examples into the like very

popular and basic clustering algorithm called uh K means um that allows us to predefine the amount of clusters of course we talk about football we have two teams so we have two clusters uh so for now we initiate uh both the umap reducer and the clustering model and once that is completed we will um use the umap to to reduce um the amount of Dimensions so we just call feet transform method that method is doing exactly what uh you think it's doing uh so first it trains uh um map and then once it's completed it

uh also run uh kind of like the inference the the projection uh on the input data and as expected we got 478 by three um now we take those project Cs and we pass them into the clustering model this time calling the method feed predict this one will first train the clustering model and then run the predictions and now we can um take a look at the sample of that data so it returns as either ones or zeros depending on so this is pretty much the label of the of the cluster um so we can

treat it uh as class ID for example now we can easily filter out crops based on uh that uh that team ID so I just zip over the crops and clusters and when the cluster is equal zero I I keep them uh and store them into the team zero uh Leist now when we display that Leist um so we display those crops similarly as we did before uh we will see that we'll only have players from one team so this is how the result looks like we successfully divided uh players into into two teams here

is the red team now we need to plug in our team classification uh code into into our uh tracking demo so instead of using the code that we just wrote I will import team uh classifier from Sports repository like internally that class exactly what we just did I just showed you what is inside but it's a lot easier to to use it instead of our code because I don't need to uh run every step manually I can just run it all uh nicely packed into the team uh classifier so at the very top of our

uh tracking uh code I uh call extract crops pass the path to our source video file um and uh then pass those crops to our team uh classifier so all that will happen before we even start tracking um this way we will have the model trained and ready to use uh when we uh want to process frames now what we need to do next is we need to divide our all detections this is the uh the object that we created a few minutes ago it stores all detections except the ball um and from that uh

group of detections we we cut out even smaller uh portion uh those that are only the players and what we do next is we uh do exactly what we did before so we convert the crops uh we we crop out those detections from um the whole frame and then pass those crops uh to our team classifier to the predict method and as a result we will get uh Team IDs we assign those team IDs to to class IDs so when we run our annotators it will automatically Peck uh the color and coding from the one

that we have so far uh so previously it it colored goalkeepers with blue and players with pink and now when we run it uh that color encoding is now uh converted into teams so we have pink team and blue team but we don't have the goalkeepers goalkeepers are still not assigned to the team so how do we do that um to do it we will need to write a short her ristic uh the assumption is pretty much that if we take the positions of all players from both teams and we will average those positioned out

we'll have kind of like the centroids of both teams and goalkeeper should be closer to its own team's centrate so the assumption is pretty much that the defending team is closer on average to their goalkeeper than the attacking team on average because individual players for example uh could be closer but the average of the team uh should be should be exactly how I described it so we will calculate uh this uh inside this helper function called resolve uh goalkeepers team ID uh this function will take uh player detection uh detections and goalkeeper detections so remember

our player detections already have class ID uh where where that class ID Express the team ID so uh players have either zero and one depending on the team that they are belonging to and goalkeepers for now have the the regular class ID something that uh does not relate to their team and now we need to convert boxes into points so uh to be able to calculate the average position of the player we we cannot have boxes we need to have some some exact location of that player and what we will do is we will convert

those boxes into a single point that is in in the middle of the bottom edge of the bounding box supervision has a utility that allows you to do it very efficiently so now we have two n arrays goalkeepers XY and players XY and those nire arrays store coordinates of both goalkeepers and players uh so next what we will do is we will calculate the Cent rates of both teams so we filter out players based on their class ID like I just told you class ID right now Express uh the team ID uh and when we

do it uh so filter out the the num arrays based on that condition and then calculate the mean we end up with the average XY coordinate of the team um and to wrap it up we just Loop over the goalkeepers because it's quite possible that on single wide uh angle uh frame we have more than one goalkeeper so we Loop over the goalkeepers we calculate the distance uh from the goalkeeper to uh centroid of Team zero and centrate of team one and then what we do is we just append uh value to goalkeeper team ID

list uh and we just do an if statement uh if if the team zero is closer then it's zero if team one is closer then it's then it's one and at the end we convert that least into n array because our class IDs and we will once again uh use class ID to to store information about the team is an amp array so so we need to return the N array so the utility is is ready we just go back to our code uh and um call that function uh passing the goalkeepers detections and player

detections and the result of this function will be saved to uh goalkeepers detections uh class ID now what we can do is we can uh take the the players detections and goalkeeper detections and kind of like merge them together into a single uh detections object and then we will we will use that object to pass into our annotators and visualize them uh so the idea is that right now also goalkeeper should have the uh color of the team uh assigned to it and so we need to put all detections back instead of player detections in

all of those annotators and once we are ready we can run the cell so of course once again it will uh collect the crops it will train the team classifier so all of that will take uh a little bit of time um but it should be almost there um ah the merriage function in detections object takes a list and we passed players detections and goalkeepers detections individually so we just need to wrap those values in the list and pass them as a single list instead of indiv objects so we restart the cell once again [Music]

um and in just few seconds we should get the visualization yeah so the goalkeeper is the most left player on that frame and it looks it got assigned to a pink team uh which just by looking at the frame seems correct we don't have the referees visualized uh yet uh because uh they are not part of the all detections anymore so what we will be doing is we just at one more filtering statement uh this time we filter by referee ID and we um add those referee detections uh to our all objects after we assign

teams uh to players and goalkeepers so now we just we just uh have a list of three elements instead of list of two elements uh so we run the cell uh once again and after a few seconds we should see uh both teams and uh referees uh on the frame awesome and here's the result when we run it for entire video works really well um there are believe I believe no accidental team swaps um the only moments where we lose somebody is when it gets occluded and it's no longer detected at all so yeah really

really uh robust solution before diving into key points detection let's take a quick deter to understand homography homography is a mathematical transformation that Maps points from one Lane to another when taking a picture of flat surface like a football field from an angle it will be distorted due to perspective the further away something is the smaller it appears homography allows us to undo this Distortion and get a bar eyes view this is critical for our football AI because it enables us to accurately track player positions ball movement and other key events regardless of where the

camera is positioned and how it's moving moving to perform this transformation we need to calculate a homography matrix this 3x3 Matrix encapsulates the geometric relationship between the two planes it is computed using corresponding sets of points in both the original image the camera perspective and the desired top down view typically we need at least four such Point pairs to solve for homography Matrix once we have the Matrix we can apply it to any point in the original image to find its corresponding location in top down view when the camera is static using homography for perspective

transformation is simple we determine the position of corresponding pairs of points on source and Target Planes once and since the camera doesn't move we can apply the same homography Matrix for each video frame we even showed this approach in our vehicle speed calculation tutorial cameras at sport events however are often placed in various locations and angles and frequently pan teed and zoom during the game this Dynamic nature makes it challenging to determine the position of corresponding points between the video frame and real world to overcome this challenge we will train YOLO V8 keypoint detection model

to automatically identified specific characteristic points on football field keypoint detection is a computer vision task that involves identifying specific points of interest in image or video key points represent distinct features or landmarks such as facial features body joints or object Corners in our case we'll use keyo detection to detect characteristic points on the football peit since the camera can pan and zoom in and out freely following the action we rarely see the entire field therefore we need to Define our points densely enough so that at any time even when the camera is tightly following the

action at least four characteristic points are visible this requirement is related to homography we need at least four reference points to run it I defined 32 characteristic points on the pit including Corners penalty areas goal areas penalty spots and the center circle labeling these images took forever to complete so if you appreciate the effort and would like to see more AI projects like this make sure to subscribe and leave a like it motivates me to keep going now back to the video Once the labeling was completed I applied post-processing steps where we rescaled each image

to 640x 640 squares by stretching them to new format I trained over 10 versions of this model and in my test stretching the frames performed far better than maintaining the original aspect ratio and using latter box speaking of model training let me quickly show you how to do it once again I remind you that all models used in this tutorial have been already trained and are available publicly on roof flow universe so if you want to skip this section feel free to do that using chapter timestamps however if you never trained keypoint detection model before

I highly encourage you to stick around in the rlow Sports repository you can find a link to Peach keypoint detection model training Google collab so if you want to reproduce my experiment or Port this demo to another sport or a completely different use case you're all set the notebook starts with standard environment preparation before we start we need to add roof flow API key to Google collapse secret stop it will be necessary to download the peach keypoint detection data set from roof flow Universe next let's make sure our Google collab is GPU accelerated to do

this I run Nvidia SMI command by default my notebook runs on Nvidia T4 but due to the large amount of computation required to optimize the model I'll upgrade my GPU to Nvidia a100 I click runtime and then select change runtime type from the drop-down when the popup appears select A1 100 and click save this GPU is available only for collab Pro users still you should be fine with Nvidia T4 it will just take a lot more time to complete the training next we install the necessary python dependencies similar to object detection model training we only

need two packages ultral litics and roof flow once the installation is complete we can proceed to download the data set in roof flow Universe we select the desired data set version and then click download data set button when the popup appears select YOLO vate as data export format and check show download code after clicking continue we see a code snippet that we can simply copy to Google collab we paste it press shift enter to run the cell when the download is complete our data should be visible in data sets directory we can confirm this by

opening the data Explorer in left sidebar the football player detection data set is divided into three subdirectories and each of them uh in typical yellow style contains another directories images and labels when we open data we can see something new however it contains a flip index section where corresponding pairs of keyo indices are defined indicating which key points should be swapped when an image is flipped horizontally this information will be used when creating horizontal flip augmentations once we have the data we can start the training I'll be training YOLO v8x poose for 500 EPO and

I know what you're thinking 500 EPO like do we really need that here's a small example where key Point detection results are marked in pink and the lines projected using homography are in blue we will build this literally in a few minutes but in the meantime we can see how the projected Lines no longer closely overlap with the lines on the actual football Peach all due to minor keypoint detection errors that's why we need to train a bit longer and optimize our model as much as possible to speed up the process I set my batch

size to 48 I remind you that I'm running my training on Nvidia a100 if you don't have access to such a beast you'll need to adjust this value accordingly to the amount of vam you have also make sure to disable Mosaic augmentations Mosaic augmentations combine multiple images into one during the training increasing the diversity of your training data those augment ations work great when you expect multiple objects in various configurations to appear in the image in our case there's always one object in the frame the peach in my test I noticed that Mosaic augmentations sometimes

misled our model making it expect multiple Peaches on one frame you can disable this feature by setting the probability of Mosaic augmentations to 0% training this model even on Nvidia 100 takes a lot of time so see you in a few hours just kidding let's use the magic of Cinema to speed up the process once again after the training let's see how the model performs on some images from validation data set it looks promising regardless of perspective the model provides good quality predictions except of one outlier on the third image from left in the top

row the pink dot seems to be in rather random place we'll see how the model handles images from our test videos finally I upload my model to roboplow Universe I can now easily deploy it in browser through the roof flow API and locally using the inference package let's plug it into our football AI demo okay so let's start by creating uh another text cell and call this section uh peach keypoint uh detection uh similarly as before we copy the ID of our model from roof flow Universe uh so we previously did that for object detection

model now we do it for peach detection model uh we create a constant and assign uh the copy value to that constant and then uh call the get model function that we imported from inference at the very beginning of uh of that tutorial uh pass our roof flow APA key along with the model ID uh press shift enter and that should uh download the model from roof flow universe and load it into the memory we didn't need to extract the roboff flow API key from uh secret step uh because we did it already so we

just uh had that value all the time uh in in the memory so now we do exactly what we did uh before for object detection but uh now we do it for uh keypoint detection so we create uh frame generator once again we extract the first frame uh from that frame generator and this playay that frame uh on the screen uh and now we uh run the infer method of our Peach uh detection model pass the frame and confidence threshold and uh parse it this time not into the detections object from supervision but into key

points object from supervision um and once we do that we have access to uh whole array of uh annotators dedicated for uh keypoint detection one of those annotators is a vertex annotator this one is pretty much using simple dots to visualize all the anchors of the skeleton uh so we just set the color and the radius and uh exactly like we did before we we copied the frame uh assign that copy to to new variable and then uh run vertex annotator annotate uh passing that uh annotated frame which is the copy of original frame and

our key points objects to it and we run that and and just like before uh I forgot to swap the variables in exactly the same place so so here's the result and what we can noticed immediately is that there are some key points that are in the right place and there are some key points that are kind of like randomly located so just like object detection results uh key points also have confidence and uh that confidence can be on the skeleton level and on the anchor level so when we take a look at the result

um esec specifically at the confidence values we see that there are some anchors that have very low confidence we can filter those out so I just set out the threshold uh for anchor confidence at around uh 0.5 and now uh I will use that filter to remove uh key points that uh have have lower values so I just pick the first skeleton from XY uh filter uh anchors from that skeleton based on that uh confidence threshold then I create new keypoints object call it frame reference key points uh and assign uh those filtered out uh

key points positions to that uh to that uh new object I call it a frame reference key points because uh I prepare already for the homography so we'll have frame reference key points and Peach reference keypoints and we'll use them to transform perspective uh later on so here are the filter out uh anchors those look really well uh they are almost exactly in the places that we would expect them to be uh and here is the result for the entire uh video what is also important that the anchors are stable um if the if the

model is undertrained they tend to vibrate uh and jump from one place to another it happens from time to time but it's uh very subtle and we'll be able uh to work with that result uh in the next stage so as we discussed to perform homography we need to have pair of points that exist both in source and Target uh planes so we already have points on a source plane uh because those are the point that we uh acquire using keypoint detection now let me show you points on the target plane so inside uh Sports

repository we have an object called soccer uh peach configuration and that uh class stores information about coordinates uh of our 32 points uh on a real uh football pitch so when we print out config vertices we see that we have a list with 32 values those values are tups and uh describe the coordinates in centimeters of all 32 points we also have something called edges uh and those describe which vertex is connected to which vertex on top of that uh in the sports report we also have uh annotators uh we will use few of them

today but uh the simplest one is draw Peach uh so we can pretty much print out uh the uh the layout of the peach based on the uh configuration that we pass uh uh so let's do that um let's display that result uh in the collup now we have everything we need to uh transform perspective uh except of the actual code that will do it uh so let's write a quick utility that will perform that transformation so uh I'll call that class view Transformer um and it will take uh Source points and uh Target points

um uh into the Constructor inside the Constructor we need to make sure that um those points uh that are stored in N arrays are in float 32 format so I'm just uh running as type on both the source and Target and then uh I'm uh creating uh a field uh called M and that will be our homography Matrix and we'll calculate that homography Matrix using using uh open CV method called find homography uh this is the method that allows us to pass more than four points uh to calculate the homography uh we do it because

uh when we will run our code uh we will have uh sometimes more sometimes less points visible on the frame as we already said we need at least four but we would like to be able to have more than four and and that matter allows us to do it so uh find homography we we calculate our homography Matrix and then inside transform points we'll use um that homography Matrix to move our points from uh one plane to another uh because homography Matrix is 3x3 Matrix uh we need to expand the dimensionality of our points because

our points only exist on Tod plane so so we reshape that um nire rering points so that it will have three dimensions then we will run perspective transformation with uh those points with expanded dimensions and our homography Matrix and then once the uh Point uh transformation is completed uh we will uh remove that one extra Dimension uh so now I just uh copied our uh key points uh example that we uh already wrote and we will add um transformation uh to that example uh so what we will do is we will uh first of all

uh have a transformation from Peach uh to camera perspective so we will take all the points we have and uh we also have edges as we uh uh just uh explored so we will transform those points and edges uh from uh the peach perspective to the to the camera perspective um so to visualize those edges we will use Edge annotator that is also part of supervision uh library and now we will uh perform the the point transformation so uh we already have a frame reference uh key points uh as I told you it was part

of the preparation for that uh transformation right now and now we convert our config vertices first into n array and then we filter them using the same filter as we filtered to to remove the Redundant key points from the key Point detection result so the idea here is that uh we have out of the box we have 32 key points and then we remove some of them because they are not visible on the frame and we would like to remove the same key points from uh frame perspective and from Peach perspective so that we end

up with pair of points that exist on both planes and then pass those points into Transformer view view Transformer to to perform that transformation so once we do it then we take all the points that exist on the on the peit plane and we'll transform them uh to the camera view uh this way we'll be able to to draw points that don't even exist there but most importantly you will be able to draw the lines connecting the points that exist on the plane and that don't exist on the plane because they are not visible so

I'm just doing a little bit of uh NPI magic here to um to have the same shape that is expected by key points object and supervision and then uh I uh call Edge annotator annotate method passing the annotated frame and my frame all key points and as a result we get a nice visualization where the pink points are the detections coming from our key points detection uh algorithm and the blue points and the lines are the projections uh performed uh using our view Transformer now now let's go ahead and perform a transformation in uh the

other direction so let's transform points from a camera view into the uh peach perspective so to do it I uh went ahead and I copied our uh detection code that we wrote uh a few minutes ago I'm removing uh from it the part that was responsible for visualizing the results uh on uh camera frame uh but I'm adding uh code that is responsible for keypoint detections so at this point uh I have um the code that is a mix of our object detection example and key points detection example uh and the idea is that we

take our detections uh that we acquire using the object detection model and move them to Peach perspective using our uh view Transformer so uh first thing first we um we acquire the position of ball uh on the frame uh and we do it the same way as we do it with did it with goalkeepers and players few minutes ago so we call get anchor coordinates and we pass a bottom Center as uh the desired anchor so once again this utility converts boxes into points uh and we pick uh the point on the bounding box that

we are interested in and we chose the the middle of the uh bottom edge of the bounding box uh now we import uh draw Peach uh annotator the one that we used already a few minutes ago go and draw points on Peach annotator so so the draw Peach annotator will draw the entire layout of the of the football Peach and uh draw points on Peach will will U allow us to visualize the position of projected uh object uh on the pit so first thing first we we call the draw Peach uh annotator past the config

this will result in in visualizing the the layout of the of the football Peach and then we call draw points on Peach uh pass the config once again but most importantly pass Peach ball uh XY so this is the uh the projected position of the ball uh that is already processed by view Transformer and now I'm just I'm just passing uh color configuration radius configuration so so just uh a few things that will impact how how the position will be visualized uh and I'm just calling plot image passing the Peach um and we will see

the result in just a second uh because it's the same example that we copied uh from the from a few minutes ago it it also uh collects the crops and calculate embedding so uh that will take a little bit of time before we will see the result but it should be yeah it's it's exactly what we expected uh we see the ball uh near the edge of the of the football field so we have the ball already U projected now let's project players so I just copy those two lines uh that I previously used to

project ball position and now I take player detections acquire bottom center uh position for all the players and I uh and I transform them uh from U camera perspective to Peach perspective and I also uh paste those two lines once again this time uh to project the position of uh referees um so right now we have three variables we have Peach ball XY we have uh uh Peach players XY and we have Peach referees XY and just to remind you in the peach player XY we already have both the players and the goalkeepers and now

I just copy and paste uh the uh draw points on Peach call um uh to visualize uh both players and referees and because we would like to visualize players from uh both teams with different colors uh then that's exactly what I do I filter out uh peach uh players XY based on the uh class ID of of the player as you remember we use class ID to store information about the the team ID and then uh I uh visualize both of those thems with different colors and finally I visualize also the position of referees uh

here we don't need to do any filtering we just uh pass a different call um and that's it and when we run it uh we should see a visualization with uh players with from both teams because we will run a team assignment uh as well as referees and the ball projected on the pit so we just need to wait few few seconds uh for that uh training to complete and and uh once that happens we should see we should see the visualization and and there it is awesome here's the allaround result I uh put both

the um Peach visualization and the frame visualizations all together uh so that we can see how how the ball is moving on both visualizations at the same time and and players are moving from one side of the pach to another yeah it it it looks really well now we can easily convert this uh radar visualization into voronoi diagram uh that uh Express the control of the team over portions of the field so to do it uh we just comment out the the visual Iz ation that we just created and we use draw Peach voronoi diagram

uh function available in supervision just as well and uh it will be actually a lot easier to to make that visualization so all we do is we pass the config of course and then we pass uh teams uh as separate arguments so we have team one and team two so to do it we do exactly what we did above we we filter out Peach players XY uh coordinate base on uh class ad values uh then uh we pass information about the the expected color uh configuration so we use the same colors as we did above

um so different color for team one and for team two um and we're almost there and finally we pass uh the peach that we generated above and once we run it of course as usual we need to wait for the uh Team classification model to uh finish uh training uh we should see the uh the generated uh VY uh diagram for our ster field it's taking a little bit of time to complete uh stay patient and there it is and finally I put together uh pretty much the same visualization that I did before for the

uh radar U visualization but this time with vulo diagram updating uh accordingly to position of players uh on the pach um yeah I mean it's it's kind of me mesmerizing to look at that result uh both at the same time this video is a lot longer than I initially anticipated but there is one more item on our to-do list that I promised you that we'll explore and this is B trajectory analysis so I keep my promises but instead of me coding that piece in front of you we'll just go through the code that they already

written and I will just explain it step by step so here is our ball trajectory analysis code it's actually quite simple because we have almost everything we need uh we detect ball using our player detection model we detect uh field uh key points using our um key points detection model and then we perform a perspective transformation that's all we did uh so far but to make those results a little bit more stable I created a Quee where I store my um homography matrices for a Time window equal to five and I average out values in

my homography Matrix and I put it back to a view Transformer this way when we have those uh small vibrations of key points uh we can we can medicate that problem uh by just averaging out the values in the uh homography Matrix and then I just put values uh that I transformed from uh one plane to another uh inside the list and that list stores coordinates of the ball on the football field and most of the time uh we only have one ball and that's great uh but sometimes we have known and even better sometimes

very rar but sometimes we have more than one ball so now what we need to do is we need to clean up those results so what I'm doing is first of all I just uh remove all the entries where I have more than one ball I just assume that if I have more than one ball I don't know which one is it so I just I just remove all of them in reality it would be possible uh to figure out which ball uh is the right one but uh turned out uh I don't really need

to do it um that don't happen very often so I don't lose a lot of data this way uh next uh I just visualized the path uh that I got and here is the here is the small problem that we have so most of the time we get the correct puth but sometimes we don't uh so sometimes we uh get those spikes um and that's because the ball the one ball that we actually detected was not the correct ball so those are those frames where we didn't detect the right Ball but we detected the wrong

ball um and it's still in the data so we need to clean it up and uh I decided to clean it um using uh this uh replace outlier uh based on distance function so this function pretty much uh takes an uh distance threshold so this is the allowed movement of ball between uh detections and if uh the ball uh moved uh more than that allowed distance I just discard um that detection uh and turns out this works really well uh if we run this function on our uh puth uh and I set the distance to

be 5 m I think it's more than enough uh the the path we got is actually really really really clean we covered a lot so far but before we wrap it up let's discuss some important considerations that make or break your football AI project our models have been fine-tuned on broadcast footage captured from above with good view on entire Peach significantly different perspectives such as as low angle shots or those from behind the goal may lead to unexpected Behavior due to changes in object appearance and scale in such cases it would be necessary to expand

our data sets with images taken from those New Perspectives and retrain the model moreover when the camera is low or players clustered tightly occlusions become more frequent those can easily disrupt our bite track tracking algorithm it might be necessary to apply reidentification mechanisms or trackers that consider object appearance such as Massa while homography effectively Maps ground level objects to 2D Peach representation it struggles with objects in the air this is most notable with long passes or high shots where the ball trajectory appears curved on a radar view instead of true straight line addressing this may

require incorporating 3D ball tracking or trajectory estimation models the overall quality of our perspective transformation hinges entirely on the accuracy of our keypoint detector even minor errors in leadmark localization can lead to skewed player position or distorted pitch visualizations one of the tactics we applied to solve this problem was averaging out our homography Matrix across time window but depending on the specific case we may need to come up with more robust strategy last but not least for live applications the entire AI pipeline will need to run efficiently to keep up with game Space the proposed

solution is only proof of concept and we are currently running at around 1 frame per second far from required 30 fps to apply this approach on life sport events it would be necessary to optimize all models by choosing smaller architectures or using Advanced strategies like quantization to speed up the inference today's project barely scratch the surface of AI in sports analytics we tackled key challenges like player tracking and perspective transformation but there is still so much more to explore from Advanced player metrics to model that predicts what will happen next on football field leave your

questions and ideas in comments below and join me for upcoming Community session where we will talk more about this project and Sport analytics in general I'm curious if you have any ideas how to apply strategies we explore today to solve problems outside of sports if you enjoyed the video make sure to like And subscribe and stay tuned for more computer vision content coming to this channel soon my name is Peter and I see you next time bye [Music] [Music] he [Music]