In this age of sports analysis and analytics, it can be fascinating to see how the media picks and chooses what analysis to run with and which to mock and ignore.
One fascination that I don’t get is the lock-step acceptance of everything that comes out of the company known as Pro Football Focus.
They’re cited endlessly and their stats are treated as the end-all. Football writers seem to love their stats, using them in their articles as ironclad proof.
I did an interview with the founder of Pro Football Focus back in 2011. Even then I was a bit leery of their methods and tried to express that in the “subjective angle” question.
Unless you’ve been under a rock for the last couple of days, Pro Football Focus was on your radar as one of their writers, in an ESPN Insider piece wrote the following:
Tom Brady, Peyton Manning, Drew Brees and Aaron Rodgers. The elite quarterback Mount Rushmore has been in place for a few years now, a comforting constant in an NFL of consistent turnover and change. But it might be time to wipe one of those four faces off our mountain of elite play. The Tom Brady of 2014 no longer belongs on that monument.
and in his concluding section, he writes:
but there is little doubt at this point that we are witnessing his decline in action. Brady is no longer an elite quarterback. He remains very good, but if the decline continues at the same rate, it won’t be long before that is no longer true.
This story took the sports media by storm, and has been easily the number one topic on sports radio, television, and on the internet. Everyone is talking about it.
The writer, Sam Monson, has made the media rounds talking about his opinion, so it’s a win-win for him, ESPN, Pro Football Focus and the football media at large. An NFL discussion in the first week of June!
It almost – almost – feels like this piece was a response to the Peter King MMQB piece a week earlier which talked about Brady and his performance before and after the 2008 knee injury and how, in Brady’s words, “You know, you don’t have to suck when you get older.”
The point of this post however, is not to debate whether Tom Brady is on a swift decline or not. (I think that has been addressed by the likes of Tom E. Curran, Ron Borges, and Christopher Price.) It is to explore the dangers of blindly relying on data and conclusions that we have no idea if they are actually accurate and pertinent or not.
What Pro Football Focus Is (And What They Aren’t)
It is important to note what Pro Football Focus is. Actually, first we’ll define what they are NOT. They are not taking raw numbers and data and crunching them into new and exotic formulas to provide a different sort of insight into player performance. This is not sabermetrics for football.
No, their methods are different. They are a UK-based company, who obtain games through NFL Rewind and sit and watch and grade each player on each play. Their dedication to this is admirable, as I can’t imagine sitting down and doing this kind of deep grading for every play, every game week after week.
I suppose there is some value in this data, in a big-picture sort of way. Stats on items like dropped passes, QB hits, things like that are likely extremely accurate. In my interview linked above, founder Neil Hornsby said that PFF’s value is this:
- Who was on the field – in 2010 this was 99.83% accurate but we didn’t double hand most games then – this year we do so I’m predicting well in excess of 99.9%
- What position they played (at a level which allows us to provide formation as well as package information)
- What they generically did (block, pass route, cover, pass rush etc.)
- A measure of how well they achieved what they attempted to do (obviously we don’t know their assignments so this is what we use)
The last part is the gotcha and this is where it is dangerous to put too much stock in the Pro Football Focus stats.
The Dangers In The PFF Method
Last August, Bill Belichick talked about the dangers of watching film and making conclusions based on it.
It might even look to us like somebody made a mistake but then we look at it more closely maybe somebody besides him made a mistake and he was trying to compensate. I think we need a little closer analysis a lot of times. Sometimes the play calls or what was called on the line of scrimmage might be something that we’re not aware of. That could happen in any game. You think a player did something that he shouldn’t have done but maybe he got a call, a line call or a call from a linebacker or he thought the quarterback said something so he did what he thought was the right thing or maybe it was the right thing but that call shouldn’t have been made or should have been on the other side. But yeah, I think we need to be careful about what we’re evaluating.
So sometimes even the team itself doesn’t know exactly where things broke down and who did what wrong. Belichick then went on to talk about watching opposing team’s game films and the impossibilities of knowing what happened:
But believe me, I’ve watched plenty of preseason games this time of year and you’re looking at all the other teams in the league and you try to evaluate players and you’re watching the teams that we’re going to play early in the season and there are plenty of plays where I have no idea what went wrong. Something’s wrong but I don’t…these two guys made a mistake but I don’t know which guy it was or if it was both of them. You just don’t know that. I don’t know how you can know that unless you’re really part of the team and know exactly what was supposed to happen on that play. I know there are a lot of experts out there that have it all figured out but I definitely don’t. This time of year, sometimes it’s hard to figure that out, exactly what they’re trying to do. When somebody makes a mistake, whose mistake is it?
Bill Belichick doesn’t have it figured out. But Pro Football Focus does? They can provide a grade on every play?
Another problem is that the NFL just recently added the coaches film to Game Rewind, so before that, the PFF graders could not even see the entire field. I don’t know if they currently even utilize the overhead game film, or just rely on the standard HD game telecasts. If it is the latter, they cannot see every player on the field for every play…so how can they grade what they can’t see? (And actually, the All-22 film doesn’t come out until mid-week, which is after PFF has posted their initial grades- so they’re not using it, at least in their first gradings.)
There HAS to be a subjective element in the grading process. They have to be making conclusions based on conjecture and assumption or what they “think” the player was attempting to do or was assigned to do on any given play.
On their own grading page, they explain their “rules” for making their grades:
• DON’T GUESS — If you’re not 95 percent sure what’s gone on then don’t grade the player for that play. The grades must stand up to scrutiny and criticism, and it’s far better to say you’re not sure than be wrong.
It is, however, crucial that this is not seen as an excuse to shy away from making a judgement. What we definitely do not do is raise or lower the grading because we’re not sure. Giving a grade of -0.5 rather than -1.5 for a player on an individual play because you’re unsure is the wrong grade to give. If the grader is 95 percent sure of the severe fault on the play, the grade is -1.5. If, however, the grader is unsure of his judgment, the correct grade is 0.
A couple things I don’t like here. How does the grader know whether they are 95% certain or just 90%? How many plays per game are going ungraded because a determination cannot be made?
Later, in the section which asks How subjective is the Grading?
Just like with the more mainstream statistics, there are occasions when the choice is difficult. But the difference on our site is this: If a guy is going to be upgraded or downgraded on a judgment call, we let it ride. We simply make the comment and then put in a 0.
Again, how often is this happening? It seems like it wouldn’t take many “0″ grades to skew the data.
Lastly, I hesitate to bring this part up, but part of me wonders the qualifications for doing this work. It feels like me taking a job to to play-by-play film breakdown on the Premier League. What are the football coaching or scouting backgrounds for these UK analysts making these grades? Is there anyone on staff with an NFL background?
Why Such Devotion?
From all of this, the national media are using PFF stats as gospel? Why? Are the simple +1.2, -0.7 ratings so damn attractive that they are accepted without question? Is it just an easy way for the media to rate players without doing a lot of work themselves?
Honestly, I don’t know. As mentioned above, I do feel there is some merit and value to the work that Pro Football Focus is putting in. I just don’t get the slavish devotion to their grades that I see when I read many NFL articles.
Again, this is not taking actual numbers and using them to come up with new stats to use in analytics. This is not taking passes complete and passes attempted and breaking it down into the various lengths of throws and spots on the field. This is sitting down in front of the monitor, forming an opinion and making up their own stats and advanced formulas based on stats garnered from what they think is happening on each play.
I believe the NFL media as a whole needs to be a little more judicious in how they use these stats instead of blindly accepting what comes out of the PFF factory.
Some worthwhile sites with NFL stats and analytics include: