Forecasting Score | Sport |

.0311 | MLB |

.0696 | Club Soccer Matches |

.0962 | NFL |

.1296 | NBA |

.1917 | World Cup (men) |

.2035 | My RLCS Forecasts |

.2502 | Tennis (women) |

.2715 | World Cup (women) |

.3135 | NFL Playoffs |

.3228 | March Madness (men) |

.3244 | Tennis (men) |

.3604 | NBA Playoffs |

.4568 | Club Soccer League |

.4806 | MLB Playoffs |

.5232 | March Madness (women) |

# RLCS Predictions Evaluation

What good are predictions if we don't know how well our model performs? Anyone could come up with a model or algorithm that spit out probabilities, but not all predictions will be that accurate. So the next question is: how should we measure the accuracy of our model? One might think of finding the average number of matches I accurately predicted (where any event probability above .5 would be treated as a 1, and any event probability below .5 would be treated as a 0. Using this somewhat naive approach, I get an "accuracy" score of above 70%. However, this does not take into account the magnitude of the probabilities (for example, 52% chance of an event happening should not be treated the same way as a 74% chance of an event happening). It would be nice to keep this magnitude issue in mind when evaluating the model.
One way to address this issue is to bin the predictions I calculated using the model, and then compare them with the actual scores that have been realized. The following plot shows this:
My (binned) predictions looks promising. The closer the forecasted chance of winning to 50%, we see that the actual win percentage is fairly close to the forecasted chance of winning. I do want to note that I have only used the model on 1 partial season of RLCS. Thus, we will get a clearer picture at how "accurate" my model is once I have added more data to test the model. Another thing we can do is use a (modified) Brier Test. This will show us how much better (or worse) my model performs when compared with a baseline model. The higher the score, the better. The baseline model usually used is giving a 50% chance of an event occurring for every event or state. The following table shows how well our forecasts perform when compared to forecasts from fivethirtyeight.com of other sports: