Most fans of the CFL will agree that, to win on a regular basis, a team must have good quarterbacking.
But what, exactly, is “good quarterbacking?” Is it better to be able to launch a 50-yard pass downfield, or put the opposition on the defensive with the ability to run? Maybe it’s most important to be able to strategize at the line of scrimmage by calling an audible.
Or maybe it’s even more important to be the game manager and leader, methodically marching the team down the field with two minutes left on the clock.
It’s easy to see that, while most fans will agree that good quarterbacking is critical to success, those same fans may not agree on what qualities are most important in a good quarterback.
In an attempt to measure quarterback performance, the QB Efficiency Rating metric was devised in 1971 (the CFL adopted the QB Efficiency Rating as an official statistic beginning in 1987).
The purpose of this article is to assess how well the QB Efficiency Rating has measured quarterback performance, as it relates to team performance.
Once that link has been established, I will then attempt to show how the historical distribution of quarterback efficiency ratings can lead us to some interesting insights, including why the Montreal Alouettes need Anthony Calvillo more than the Saskatchewan Roughriders need Darian Durant.
The QB Effiency Rating combines four different quarterback metrics – pass completions, passing yardage, touchdowns and interceptions – into the following formula:
Some critics of the QB Efficiency Rating argue that the formula does not consider quantifiable skills such as the quarterback’s ability to run and qualitative skills such as the quarterback’s ability to think. Despite these perceived flaws, the QB Efficiency Rating has evolved into a widely accepted statistic to judge and compare quarterback performance.
But just how much does the QB Efficiency Rating tell us?
In order for a statistic to be valuable, it must be accurate, understandable and predictable. Obviously, if a statistic is not accurate or cannot be understood, it has no value whatsoever. However, what does it mean for a statistic to be predictable?
Let me explain.
I mentioned above that good quarterbacking is needed to win in the CFL on a consistent basis. Therefore, any measure of quarterback performance (whether it be the QB Efficiency Rating or some other statistic) should give us an indication if a given quarterback rating is more likely to result in a win or a loss.
To examine the predictability of the QB Efficiency Rating, I compiled every single QB Efficiency Rating and game result of all CFL regular season games played during the nine-year period 2002 to 2010. All in all, that’s a total of 1,360 QB Efficiency Ratings and team scores!
What did I find?
Well, the first thing I noticed was that the QB Efficiency Ratings are normally distributed (Figure 1).
In other words, there are relatively very few excellent scores and very few terrible scores. Most of the time, the QB Efficiency Rating assembles around an average score, which in this case is 86.34. The bell curve shape in Figure 1 is a very important and useful statistical distribution which we will use to make some conclusions later on.
The bell curve “shape” of the distribution above, represented by the curved red line, allows us to make some interesting conclusions about the QB Efficiency Rating, including:
- 68% of the time, CFL quarterbacks will have a QB Efficiency Rating of between 57.75 and 114.93; and
- 95% of the time, CFL quarterbacks will have a QB Efficiency Rating of between 29.16 and 143.52.
Having established that the QB Efficiency Rating statistic is normally distributed, I decided to test the link between quarterback performance, as measured by the QB Efficiency Rating, and team performance, as measured by wins and losses.
The QB Efficiency Ratings for all winning and losing teams are presented in Figure 2 below. The winning teams’ ratings are represented by the solid red line and the losing teams’ ratings are represented by the dotted red line.
Based on Figure 2, we can conclude that there is, in fact, a link between the QB Efficiency Rating and team performance. In other words, there is a degree of predictability with the QB Efficiency Rating, which is required in order for this statistic to be relied upon.
Back on Oct. 18, 2009, Quinton Porter of the Hamilton Tiger-Cats was 29 of 44 for 506 yards and three touchdowns. This equated to a QB Efficiency Rating of 127.65. The Tiger-Cats lost that day to the Alouettes by three points, as Calvillo performed to a QB Efficiency Rating of 155.39.
As Figure 2 above shows us, a strong QB Efficiency Rating does not guarantee a win, because (among all the other variables), the opposing quarterback might just be a little bit better.
Because of this, I thought a more insightful analysis might be to take all of the “winning” quarterbacks for each of the 680 games from 2002 to 2010, and compare their QB Efficiency Ratings to the “losing” quarterbacks. The difference between the two QB Efficiency Ratings will be termed the “Marginal QB Efficiency Rating”.
For example, in the Oct. 18, 2009 game between Hamilton and Montreal, Montreal was the winner and the Marginal QB Efficiency Rating of Calvillo was 27.74 (Calvillo’s QB Efficiency Rating of 155.39 less Porter’s QB Efficiency Rating of 127.65).
Plotting these results (once again, they are “normally” distributed with a bell-curve shape) gives us the graph presented in Figure 3 below:
Figure 3 shows us that, on average, the winning quarterback will have a QB Efficiency Rating of 27 points higher than the losing quarterback. The properties of the bell curve above also tell us that:
- A team has an approximate 10% chance of winning a game if the opposing quarterback outperforms its quarterback by a QB Efficiency Rating of at least 13.51;
- A team has an approximate 5% chance of winning a game if the opposing quarterback outperforms its quarterback by a QB Efficiency Rating of at least 25.06; and
- A team has an approximate 1% chance of winning a game if the opposing quarterback outperforms its quarterback by a QB Efficiency Rating of at least 46.73.
Now, what does all of this have to do with Montreal needing Anthony Calvillo more than Saskatchewan needing Darian Durant?
Well, plotting Montreal’s Marginal QB Efficiency Ratings in its wins and losses from 2002 to 2010 results in the graph presented in Figure 4 below:
I’ve included some gridlines that will help us read the graph. I’ll come back to the yellow and blue stars in a moment.
As we would expect, when Montreal’s quarterback (which has been mostly Calvillo during the period of analysis) has a higher QB Efficiency Rating than the opposing quarterback, it has a greater probability of winning.
(The solid red line – representing the estimated distribution of Montreal’s QB ratings when Montreal wins – is to the right of the dotted red line, which represents the estimated distribution of Montreal’s QB ratings when Montreal loses).
I’ve highlighted two points on Figure 4 above with yellow stars. You will notice that when the Marginal QB Efficiency Rating is zero (in other words, when the two opposing quarterbacks have the exact same QB Efficiency Rating), Montreal is more likely to lose the game (the dotted red line is above the solid red line).
It would appear then, that based on this data, Montreal has been very reliant on its quarterbacks (especially Calvillo) to substantially outperform the opposing quarterback in order to win.
In fact, looking at the graph above, it would appear that Calvillo (or any other Montreal quarterback) needs to have a QB Efficiency Rating about 10 points higher to have an equal ‘chance’ to win the game (see the bright blue star highlighted in Figure 4 above, at the intersection of the two normal distributions).
Now let’s take a look at Saskatchewan’s results in Figure 5 below:
Once again, you will notice that a higher Marginal QB Efficiency Rating is more likely to lead to a team win.
That is, when Saskatchewan’s quarterback has a higher QB Efficiency Rating than the opponent, Saskatchewan is, more often than not, going to win the game. And once again, I’ve labelled the Marginal QB Efficiency Rating of zero with a yellow star.
However, you’ll notice in Figure 5 above that there is only one yellow star, compared to the two yellow stars we saw for Montreal in Figure 4 above.
The reason that there is only one yellow star above is because the solid green line (representing the distribution of QB Efficiency Ratings when Saskatchewan wins) happens to intersect the dotted green line (representing the distribution of QB Efficiency Ratings when Saskatchewan loses) precisely at a Marginal QB Efficiency Rating of zero.
In other words, when Saskatchewan’s quarterbacks perform at the same level as its opponent, Saskatchewan is just as likely to lose the game as it is to win the game.
For Saskatchewan, it would appear that outperforming the opposing quarterback has not been as critical to its success as it has been to Montreal’s. And this is why it would seem that Montreal needs Calvillo more than Saskatchewan needs Durant.
