Looking outside in: from Luton Town, England to the New England Patriots
I recently compared the player evaluation innovations of Dallas Cowboys executive Gil Brandt to those of the auto insurance industry with the introduction of credit data to individual risk underwriting and pricing. As Al Davis and I are both want to do, I’ll now extend the comparison and push the football/insurance analogy further down the field. Starting with the story of an NFL fan who grew up a few miles south of the Scottish border (also the origin of the world’s smartest canine breed), I’ll illustrate how the application of new, super granular analytical tools has dramatically increased the ability of NFL teams and insurance companies to evaluate individual performance, predict outcomes, and manage risk.
Got Game Score?
Neil Hornsby was bitten by the professional football bug in the early 80’s. A statistical consultant by trade, as a fan of the NFL Hornsby, became dissatisfied with the limitations of standard football statistics. Neil began grading players on his own in 2004 and he officially launched the website Pro Football Focus (PFF) in 2007. With assistance from a small army of like-minded NFL fanatics, Hornsby found a large and growing interest from NFL teams, players, agents, the media, and fans in a deeper set of performance evaluation tools for the most popular sport in North America.
While teams had reams of information, PFF offered a level of analysis that went well beyond traditional football statistics. A trained PFF analyst reviews and grades every individual player on every play in every game. That’s more than 3,000 evaluations per game, more than 750,000 evaluations per season. PFF then aggregated individual play grades to provide complete game and full season scores for every NFL player. Traditional game metrics offered the number of carries and yards gained for backs, the number of catches and yards gained for pass catchers, the number of attempts, completions, interceptions, and yards gained for quarterbacks, and of course touchdowns and fumbles for all offensive skill position players, but not much else, and barely anything at all for the beasts of the trenches.
Anybody who’s spent any time watching the game knows that it’s won or lost at the line of scrimmage and that all yards, catches, and touchdowns are not created equal. If you want to understand the drivers of success or failure in the game of football, you must study the leading indicators of those traditional outcome metrics. You have to understand and then model the behaviors that lead to those outcomes. Beneath the surface is a complex system of individual behaviors and interactions – it is at that level where one can truly begin to effectively evaluate player performance.
When PFF first started, they were met with a tremendous amount of skepticism from the NFL establishment, and outright ridicule from most of the media – they were from freaking England for Pete’s sake! What do the Brits know of American football? But as their work gained traction with the fanbase, the perception of the organization and their product amongst industry experts changed.
In 2014, Cris Collinsworth, a former NFL player and part of NBC’s Sunday Night Football broadcast team, bought a majority interest in PFF. No doubt influenced by Collinsworth’s endorsement, NBC Sports began using PFF’s Game Scores as part of its broadcast in 2016, and it wasn’t long before the deepest level of PFF analytics was reserved for their newest customers: the individual franchises of the National Football League. Still skeptical or not, teams didn’t want to miss out on an opportunity to better understand individual player performance, and they weren’t about to risk the death spiral of adverse selection. Hornsby’s player evaluation methodologies had broken through. PFF was officially part of the NFL establishment.
Refining How to Score the Game
Outcomes-based evaluation has long been the focus of the auto insurance industry too. Other than the traditional class characteristics used for decades to group drivers of like age, experience, gender, marital status et al for rating – an approach we could aptly compare to the annual NFL Scouting Combine, otherwise known amongst cynics as The Underwear Olympics. For many years the industry has depended on relatively infrequent (and expensive to source) outcome data from accidents and moving violations to differentiate the expected performance of individual drivers.
Tickets and accidents are the interceptions and fumbles of auto insurance. They’re really the completions and touchdowns too, although some may struggle with that comparison at first. These are all historical intermediate outcomes that provide insight into the ultimate focus, the target outcome of each field: wins in football, and losses in auto insurance. If you’re struggling with the positive/negative juxtaposition, think of it as losses in both. The point is, if you have enough data, you can model the expected value of losses in the NFL given a historical profile of completions, first downs, fumbles, interceptions, and touchdowns, and you can model the expected value of losses in auto insurance given the historical profile of moving violations and claims. But you can do both better if you incorporate a larger data set containing the behaviors that lead to those intermediate outcomes and, more importantly, those that lead to wins and losses.
Moving violations and accidents for the average driver are, like touchdowns for the average NFL player, infrequent. They are predictive – they do help us to determine the likelihood of future claims – but because they only happen once every five years or so, they don’t provide us with a lot of data, and our ability to differentiate is limited by their infrequency. So much so that we were forced to look to the insurance industry’s version of The Underwear Olympics (driver classification) to buffer the predictive signal. Telematics is the PFF of auto insurance. With telematics, insurers have access to a data set of predictor variables that are orders of magnitude more frequent than violations and accidents. Just as the PFF analytics platform offers NFL teams super granular predictive insights like the frequency that a particular offensive guard allows disruption of the backfield flow when shaded on the outside shoulder by a defensive undertackle, a scaled telematics platform can offer the frequency that a particular driver uses his mobile phone while driving in moderate traffic at 120% of the posted limit. These things happen more frequently than do touchdowns and auto accidents, and each is a single example of a multitude of high-frequency predictive insights that can be derived from these contemporary analytics platforms.
Open the Door to Deeper Insights and Intelligence
A key threshold that PFF had to cross in successfully commercializing a new and deeper level of predictive insights was establishing these new metrics in the first place. People familiar with the game knew about interceptions, fumbles, completions, touchdowns, sacks etc. and directionally understood how they related to the likelihood of victory. They weren’t as familiar with the importance of the first step, hand placement, arm extension, sight direction, initial and counteractions, and all the rest of the truly fundamental movements players make and how they relate to success. PFF had to hypothesize which of these fundamentals were important predictors of success and then train and test models using the fundamentals as predictor variables. Then they had to get the rest of the world to buy into their science.
A similar threshold had to be crossed in auto insurance with credit data, only in that case the fundamentals had already been defined by another industry for a different reason. Creditors had used things like bankruptcies, late payments, cancellations, the number of open accounts, and even the number of inquiries to determine credit risk for decades before anyone even thought of using this data to predict insurance risk. The variables and credit score models had been developed and trained on huge data sets in a strict regulatory environment. When the insurance industry realized that these variables were highly predictive of insurance losses too, they only had to test the predictive power of each of the already established events on actual loss data and then build a new model using the best of those variables; that’s how the insurance score came to be.
The age of telematics is a mashup of these two. Like credit, we are using a new-to-domain source of data to realize a superior prediction of insurance risk but, in this case, no one had already defined the meaningful behavioral events for us. Neither God nor the banking system handed us a definition of “hard braking” or “rapid acceleration” the way the latter did with bankruptcies and late payments. Like PFF, we have to use our intuition to brainstorm predictor variables (both in a broad category and refined definition), and then build, train and test models on the outcomes we’re trying to predict: insurance losses. Because we now have lots of telematics data, and some of us have the claims data associated with those telematics users, we’re building and deploying telematics-based loss models that add unprecedented predictive power to the traditional models used by the industry today. Best of all? Unlike when credit was introduced to the auto insurance industry, and unlike when PFF was introduced to the NFL establishment, nearly everyone agrees that how, how much, where and when you drive is intuitively related to the likelihood of your having an accident, and therefore what you should pay for your auto insurance.
The Amazing Race: Scores and Scores and Scores of Data
Since 2014 The NFL has gathered in-game player-tracking data through tiny sensors in players’ shoulder pads. Beginning in 2016 teams had access to only their own player’s data, but earlier this year, the NFL’s Competition Committee agreed to release in-game player-tracking data on every NFL player from 2016 and 2017 to all 32 teams. Some teams have dedicated more resources to analyzing the data than others, and thus figure to realize more value. That has led to push back from less-invested clubs about distributing any data because of the potential competitive impact. But after years of debate with league officials, the full data set is expected to be released this summer and the race to create a competitive advantage will be on. It is going to be fascinating.
A similar race has been happening in the insurance industry. Here at Arity our strategy includes originating telematics data with our proprietary software, while also adding data to our platform through the relationships we’ve established with large mobile app providers, automotive OEMs and telecoms. Our experience, domain expertise, access to both predictor and target variable data, and scale position us nicely to offer real value to the National Insurance League. And while my 46-year-old shoulder can’t quite zip the 15-yard out pattern anymore, our team of more than 200 experts can take your insurance player evaluation and development, game planning and strategy to a whole new level. Huddle!
