RLCS Predictions Evaluation

What good are predictions if we don't know how well our model performs? Anyone could come up with a model or algorithm that spit out probabilities, but not all predictions will be that accurate. So the next question is: how should we measure the accuracy of our model? One might think of finding the average number of matches I accurately predicted (where any event probability above .5 would be treated as a 1, and any event probability below .5 would be treated as a 0. Using this somewhat naive approach, I get an "accuracy" score of above 70%. However, this does not take into account the magnitude of the probabilities (for example, 52% chance of an event happening should not be treated the same way as a 74% chance of an event happening). It would be nice to keep this magnitude issue in mind when evaluating the model. One way to address this issue is to bin the predictions I calculated using the model, and then compare them with the actual scores that have been realized. The following plot shows this: predicted versus real accuracy My (binned) predictions looks promising. The closer the forecasted chance of winning to 50%, we see that the actual win percentage is fairly close to the forecasted chance of winning. I do want to note that I have only used the model on 1 partial season of RLCS. Thus, we will get a clearer picture at how "accurate" my model is once I have added more data to test the model. Another thing we can do is use a (modified) Brier Test. This will show us how much better (or worse) my model performs when compared with a baseline model. The higher the score, the better. The baseline model usually used is giving a 50% chance of an event occurring for every event or state. The following table shows how well our forecasts perform when compared to forecasts from fivethirtyeight.com of other sports:
Forecasting Score Sport
.0311 MLB
.0696 Club Soccer Matches
.0962 NFL
.1296 NBA
.1917 World Cup (men)
.2035 My RLCS Forecasts
.2502 Tennis (women)
.2715 World Cup (women)
.3135 NFL Playoffs
.3228 March Madness (men)
.3244 Tennis (men)
.3604 NBA Playoffs
.4568 Club Soccer League
.4806 MLB Playoffs
.5232 March Madness (women)