Notes from refreshing the rating algorithms.......

The purpose of this post is a whistle stop tour of the work I’ve been doing over the Summer to refresh the ratings with updated data and hopefully improve on the results from the last 4 seasons. As discussed in the previous post, it has definitely been a case of refreshing what I already had and not rebuilding ratings and therefore, if everything goes wrong next season, it won’t be due to changes I’ve made this Summer! I think my one concern with refreshing my ratings has been the fact that it invalidates the live results of the ratings but having spent time looking at the ratings and looking at what others do in the market, refreshing ratings is common place and therefore, I’m not concerned.

I have shown over the last 4 seasons that I am able to build new ratings from scratch and ensure the rating algorithm makes a profit each season. Refreshing ratings by tweaking some rating weights based on updated data isn’t going to suddenly mean that my ratings turn bad overnight and of course, bringing in more recent data allows me to refine the algorithm somewhat and hopefully make it much more current than what I was using previously.

I’ve made some notes as I was going through the work for this and I’ll share these below. Of course, the notes mean more to me than all of you but in the interests of transparency, I’m happy to share them on the blog. People can ask questions if they like but I’m not going into any great detail about the impacts on results as quite simply, I don’t want to get bogged down. The odds I’ve used in the work to refresh the ratings are not the same odds I’ve quoted for the bets when the ratings were live and therefore, I’d be comparing chalk and cheese. What I would say is that I wouldn’t make any changes to any ratings or weightings without working through the consequences on all live results to date and obviously, I wouldn’t make changes unless it improved the results of the ratings and the systems.

I would say that these ratings are now in the best state they’ve been in since the first season they went live. Given the first live season is usually the most potent for my ratings, I’m really confident and excited that next season, we’ll reap the rewards of the work I’ve been doing this Summer.

I’ll quickly discuss some generic changes I’ve made before looking at each algorithm in detail

Removal of the minimum odds stipulation on some rating algorithms for Home teams – Quite simply, I was never too comfortable playing in the area of very short odds on Home teams as historically, I was finding it very hard to make 10%+ return from this subgroup of bets. What I found through when I was refreshing the ratings was that this subgroup had made a profit overall based on the last 4 seasons of results and therefore, I was missing out a potentially lucrative subset of bets. Last season was an amazing season for this group of bets but even before that, 2 of the previous 3 seasons had made a profit. This was a simple change to make although it is a very selective change and only applies to some underlying systems.

Removal of the minimum odds stipulation on same rating algorithms for Away teams – Same comment as above really and I was never comfortable playing short priced away bets. This was less clear cut than the Home bets though but basically, I think there are better ways for my ratings to filter out the poor value low odds away bets than just me applying an across the board exclusion. This does vary by rating algorithm though, so I’ll make it clear which algorithms are impacted.

Increased volume of Home bets on every algorithm – Quite simply, due to the fact my rating algorithms were traditionally built on data from 2000-2009, there was a clear bias towards Away teams. This made its way into my ratings where around 70% of bets were Away teams and 30% were Home bets. By flushing through more recent data (especially last season), the bias has reduced somewhat and we will see more Home bets next season on every algorithm. Away teams will still make up the vast majority of bets though and given their higher long-term expected returns, I’m comfortable with this.

Over estimation of Home bias in my rating factors – Again, this is sort of linked to the previous point but my models were putting too much weight on how teams played at Home and this then meant when they played away from home, the expectation was they could produce similar performances. Hence, my models favoured Away teams who were strong at home when in reality, this wasn’t coming through in the results. Based on the last 4 seasons of results, by reversing this factor, it turned the factor from a losing factor (backing all bets was losing when compared to no adjustment) to a winning factor (adjusting for this negatively improved the returns). As always, not sure I really understand why this pattern changed from the early 2000s to the recent years. I suspect it’s simply the case that weaker teams find it more difficult to perform away at better teams than historically. The market knows this, my model didn’t!

Over estimation of long-term form – By long-term form, I mean performance in the last 6,8 and 10 games. My model uses a rating factor for all 3 and what I found was that by reducing the weightings on these factors, it increased the returns. Again, seems a more recent phenomenon based on my dataset but should see an improvement next season.

Under estimation of recent form – By recent, I mean the last game played. I think I fell into the trap of thinking that recent game form wasn’t a good indicator of future performance and that in all likelihood, the market would have this rating factor sewn up but as I’ve found out this Summer, this is definitely not the case. Only makes a very small difference overall but when used in conjunction with some of my other rating factors, I thought the results were much better than I anticipated.

More appreciation of the favourite-longshot bias – When building my ratings years ago, I had it in my head that backing short-prices was a losing strategy long-term whereas betting outsiders was much better as clearly, you get much better value on outsiders don’t you? Well, as I’ve found over the last few seasons, if simply backing all favourites, you can’t lose that much money long-term whereas if you back all outsiders, you will lose badly! Hence, although there are much better returns to be found backing outsiders, it’s a riskier place to play and therefore, ignoring all value bets at short-odds is not the right thing to do! I had lots of filters on systems to ensure that as I filtered the bets, I excluded those at short-prices which ensure the ROI increased as I filtered bets out. This is a flawed strategy I think and as I saw last season, the Home bets that made it through to the higher combined systems were rubbish whereas all the nice value bets at shorter odds did well on the systems. Some of these filters have been removed where I could see I had depressed the profits over the last 4 seasons.

I think that covers off the generic changes I’ve been making. Some were specific things I went looking for, others were things that jumped out as I refreshed the data and started moving the rating weights up and down. Some were surprising, some were obvious with hindsight but overall, I’m much more comfortable with where the algorithms are now.

Algorithm 1 (systems 6-8)

•          Removed the Home odds filter on one of the underlying systems. This will see a handful of Home bets appear on system 6 at very short odds (1.10+) but they can’t appear on systems 7 or 8.
•          Removed the Away odds filter on one of the underlying systems. This will see a handful of Away bets appear on system 6 at short odds (2.30+). Again, they can’t appear on system 7 or 8.
•          Refreshing the ratings has saw an increased number of Home bets relative to Away bets than previously. Over the 4 live seasons, split moves from 62/38 in favour of Aways to 59/41 in favour of Aways.
•          Reduction in average odds from 3.16 to around 3.00 which is a consequence of the above movements
•          Weightings shifted slightly on all 16 rating factors but nothing significant to note.
•          Overall, turnover should be increased slightly but as always, moving from backtested turnover to live turnover isn’t always as easy as you think. Historically, I’ve tended to see reduced turnover when systems have gone live compared to what I expected.

I’ve done a quick and dirty analysis of the games that would have appeared on system 6 last season based on the old algorithm compared to the games appearing under the new algorithm.

Homes – Profitability increased from 4% ROI to 9% ROI and turnover is up 60%

Aways – Profitability increased from 2% ROI to 8% ROI and turnover is down 20%

This analysis is not perfect as the odds quoting isn’t the same in both datasets, so there will be flaws in this analysis. In addition, the ROI under the new rating algorithm will always be better than what I achieved last season, the same way it tends to be better during backtesting I think. Some of the bets that make its way onto the new algorithm wouldn’t have been bets on a Thursday evening when I gave the bets and interestingly, I can see a handful of teams where the odds have swung so much from the Thursday evening, we’d be on the opposing team in the updated dataset!

If people want to follow pretty much the same algorithm as last season, I suggest they don’t back any Home bets below 1.75 or any Away bets below 2.75.

I haven’t done any analysis at all on TFA value ratings and therefore, assumptions will need to be made that the bets that are different hasn’t changed the picture too dramatically with these value ratings. Given I suspect the vast majority of the bets that have changed is where value is less than 5% (I assume this to be the case but may not be), then I don’t think it should impact the value ratings at all in terms of the higher bands.

Algorithm 2 (systems 21-22)

• Removed the Home odds filter on one of the underlying systems. This will see a handful of Home bets appear on system 21 at very short odds (1.10+) but they can’t appear on system 22.
• Removed the Away odds filter on 3 of the underlying systems. This will see a decent number of Away bets appear on systems 21 and 22 at short odds (1.80+).
• Mix between Home and Away bets remains unchanged from previous rating algorithm.
• Slight reduction in average odds by around 0.2pts (3.4 to 3.2)
• Weightings shifted slightly on all 16 rating factors but nothing significant to note.
• Overall, turnover will be increased slightly but always, moving from backtested turnover to live turnover isn’t always as easy as you think. Historically, I’ve tended to see reduced turnover when systems have gone live compared to what I expected. 

I’ve done a quick and dirty analysis of the games that would have appeared on system 21 last season based on the old algorithm compared to the games appearing under the new algorithm.

Homes – Profitability increased from -9% ROI to 6% ROI and turnover is up 100%

Aways – Profitability increased from 6.8% ROI to 16% ROI and turnover is unchanged

I think this algorithm looks like it has changed lots with the movements in profits but a lot of the increased profitability I think is driven from the refreshing of the ratings. I didn’t set out to change too much at all and apart from a few tweaks to odds filters (especially Away odds), the algorithm was pretty much untouched. Given the fact the ratings are badly outdated, I think this makes it appear as if the ratings have changed a lot but nothing too much to worry about.

Similar to the comments on algorithm 1, if people want to follow the same algorithm as last season, I would remove all Home bets less than 1.91 and remove all Away bets less than 2.75. This will take away a chunk of turnover next season and of course, I really hope it takes away a massive chunk of profits next season too since I’ve done all this work!

I haven’t done any analysis at all on TFA value ratings and therefore, assumptions will need to be made that the bets that are different hasn’t changed the picture too dramatically with these value ratings. Given I suspect the vast majority of the bets that have changed is where value is less than 5% (I assume this to be the case but may not be), then I don’t think it should impact the value ratings at all in terms of the higher bands.

Algorithm 3 (systems 31-33)

• Removed the Home odds filter on 2 of the underlying systems. This will see a handful of Home bets appear on system 31 & 32 at very short odds (1.10+) but they can’t appear on system 33.
• No change to the Away odds filters on any of the underlying systems.
• Slight increase in mix towards Home bets for Home/Away bets
• Average odds have reduced slightly (by 0.04) across all the bets for the last 3 seasons – will be driven by some short priced homes appearing
• Weightings shifted slightly on all 16 rating factors but the big change was on short-term form as there was a definite issue on this algorithm with this rating factor. I have fixed it now and it has increased profits on Homes significantly over the last 3 live seasons.  
• Overall, very little changes to this algorithm at all which is probably not surprising given the fact it is the strongest algorithm out of all my algorithms

I’ve done a quick and dirty analysis of the games that would have appeared on system 31 last season based on the old algorithm compared to the games appearing under the new algorithm.

Homes – Profitability increased from -1% ROI to 12% ROI and turnover is up 20%

Aways – Profitability increased from 9% ROI to 11% ROI and turnover is down 10%

I think the changes are minimal on this algorithm although a lot of work has gone into the Home bets to try to understand what went wrong last season. I think I have pinpointed the issue and it was dragging down the profits last season and even over the previous two seasons. I’m confident this algorithm will remain the most consistent algorithm going forward and can build on its previous successes.

Similar to the comments on the previous algorithms, if people want to follow the same algorithm as last season, I would remove all Home bets less than 1.91.

I haven’t done any analysis at all on TFA value ratings and therefore, assumptions will need to be made that the bets that are different hasn’t changed the picture too dramatically with these value ratings. Given I suspect the vast majority of the bets that have changed is where value is less than 5% (I assume this to be the case but may not be), then I don’t think it should impact the value ratings at all in terms of the higher bands.

Algorithm 4 (systems 41-42)  - Significant changes on this algorithm

• New filters set up for Home and Away bets as old filters weren’t optimising profits at all based on the results over the first 2 seasons
• Split of Home/Away bets is unchanged although the actual Home/Away bets appearing will look different to the last 2 seasons
• Average odds look very much in line with what this algorithm had previously
• Significant changes to all weights across all rating factors as around 30% of the factors were loss making over last 2 seasons
• Overall, some massive changes to this algorithm and it will look a much different algorithm next season.

I’ve done a quick and dirty analysis of the games that would have appeared on system 41 last season based on the old algorithm compared to the games appearing under the new algorithm.

Homes – Profitability increased from 6% ROI to 12% ROI and turnover unchanged

Aways – Profitability increased from -2% ROI to 10% ROI and turnover unchanged

This algorithm is the one I spent most time on and although I haven’t started again, there are a lot of significant changes to the algorithm from last season. However, when I analyse things like average odds and splits of Home/Aways, the data looks very similar to what the algorithm had before.

I think this algorithm needed a complete overhaul as the first two seasons weren’t great and I’ve spent a lot of time trying to understand where the issues were and tweaked a lot of things. I’ve not done any further analysis to understand how the system performs when cross referred to system 31 for example, so system 31-41 carries a fair bit of risk this season I suspect. Systems 31-42 to 33-42 will look like much different systems to last season which to be fair, isn’t a surprise considering the systems were loss making last season.

I haven’t done any analysis at all on TFA value ratings and therefore, assumptions will need to be made that the bets that are different hasn’t changed the picture too dramatically with these value ratings. Given I suspect the vast majority of the bets that have changed is where value is less than 5% (I assume this to be the case but may not be), then I don’t think it should impact the value ratings at all in terms of the higher bands.

Draw Algorithms (D1 & D6)

I flushed through the data from last season and I think it impacted about 10-15 bets on each algorithm. These 10-15 bets were removed and replaced with others but I don’t think the profitability moved at all. I think given the fact that most of the dataset remained unchanged from when these draw systems were built last Summer, I didn’t expect many changes at all.

Euro Rating Algorithms  (E1 & E6)

Same story as with the Draw algorithms although there were more changes here than on the Draw systems. There appeared to be a bit of a bias towards Away bets last season in the Euro leagues and my refreshed ratings have picked up more Away bets than we actually saw last season live. Not a huge fan of this as I quite liked the fact we had much more of a Home bias on these Euro systems last season than the previous ratings but the impacts are fairly small. 95% of the bets are unchanged as we have moved from last season’s algorithm to the updated one.

Euro Draw Ratings (DE1 & DE6)

No changes as I only built them a few months ago and they have no live results!