There's a lot of sound and fury in sabrmetric circles today, and it's the doing of Bill James. He wrote an article for SABR's Baseball Research Journal titled "Underestimating the Fog," and he's gotten a lot of folks pretty riled up in the process. We don't have access to the essay itself, so here's a link to an article that discusses it.
Summarizing the summary of another person's work is a mug's game, but here I go anyway: James argues that sabrmetrics has fallen into a bad habit. Sabrmetrics has long relied on the principle of persistence to establish whether a given phenomenon is an actual skill or just a coincidence. For example, James says, players with better-than-average stolen base totals and batting average marks one year tend to also post better-than-average totals and marks in subsequent years. They do it in a consistently measurable pattern, so therefore it's a skill.
"Clutch hitting," on the other hand -- hitting for a higher average late in the game, in close games, with the game on the line, etc. -- does not persist. A player who hits .390 in "clutch situations" one year may hit .225 the next year and .281 the year after that. If this player really had a clutch-hitting skill -- for whatever reason, he becomes more focused and more productive in tight situations -- he should be able to do it every year (and frankly, he should be able to do it every at-bat). Because he doesn't, his performance in those situations is considered simply to be luck, coincidence, the alignment of the stars, or what have you. It is transient.
This approach has been used by numerous sabrmaticians, including James, to discount a number of hallowed "truths" held by traditional baseball folk (platoon differentials, catchers' ERA, etc.). In the best traditions of science, they insist of running the numbers and conducting the experiment to see if reality conforms with theory and legend. In many ways, it's the fundamental principle of sabrmetrics, and it has taught us scads of previously unrealized things about baseball.
Only now James isn't so sure. He recently revised his opinion of his own 1988 study that discounted platoon differentials. He thinks that there was a whole lot of noise and static in the numbers he looked at, and that that noise drowned out patterns that might be there. In the platoon differentials situation, he thinks that the randomness, luck and noise might be ten times as loud as whatever signal might be coming through, and that may be why we can't make out the signal.
James's essential point, I think, is summarized in this line: "We ran astray because we have been assuming that random data is proof of nothingness, when in reality, random data proves nothing." In other words, just because we can't see a pattern where we were expecting to find one doesn't mean the pattern isn't there. It may simply mean we're not looking hard enough, or not looking at it from the right angle, or that it simply isn't measurable by our current means. But the absence of measurable evidence is not, in itself, proof of anything. Failure to locate is not the same as non-existence.
James concludes his essay with a metaphor of a sentry standing guard on a foggy night:
...A sentry is looking through a fog, trying to see if there is an invading army out there, somewhere through the fog. He looks for a long time, and he can't see any invaders, so he goes and gets a really, really bright light to shine into the fog. Still doesn't see anything.
The sentry returns and reports that there is just no army out there -- but the problem is, he has underestimated the density of the fog. It seems, intuitively, that if you shine a bright enough light into the fog, if there was an army out there you'd have to be able to see it -- but in fact you can't. That's where we are: we're trying to see if there's an army out there, and we have confident reports that the coast is clear -- but we may have underestimated the density of the fog. The randomness of the data is the fog. What I am saying in this article is that the fog may be many times more dense than we have been allowing for.
Let's look again; let's give the fog a little more credit. Let's not be too sure that we haven't been missing something important.
The reaction to the James essay has not been difficult to make out -- the signals are loud and clear. The Primer thread that links to the summary article is quite a read, featuring full-scale rants, discourses on optimum bullpen use, and the always-amusing antagonism of Backlasher. More enlightening commentary can be found (not suprisingly) from TangoTiger, and I'm reliably informed that our very own Craig B is planning an article responding to the James piece in an upcoming edition of The Hardball Times.
James probably does overstate his case somewhat, and there's a key objection that both Tango and Craig have already raised: is there any practical difference between a signal that doesn't exist and a signal that is so small and quiet that it has no effect in practice on a baseball field? They suspect there is not, and I'm inclined to think they're right. But I think the James essay is nonetheless important, for two main reasons.
The first is that it shines a light on an increasingly unfortunate attribute in sabrmetric discussion. Many people who are attracted to the intellectual rigour and accomplishments of sabrmetrics find themselves turned off by what they perceive as an odd mix of condescension and zealotry on the part of some of its proponents (ironically enough, James himself is somewhat infamous for his personal quirks in this respect). Hang around Primer or other sabrmetric-friendly neighbourhoods long enough and you soon get a sense of people who believe that those who disagree with them are really too stupid and naive for words. Primer in particular has fallen into the bad habit of linking to articles from decrepit sportswriters and old-school managers and, essentially, laughing at them.
This stems in part, I think, from a degree of self-satisfaction in some sabrmetric quarters, a quasi-Enlightenment assumption that reason rules here in the light while superstition governs out there in the darkness. You could argue that some folks in the sabrmetric community have gotten a little carried away with themselves. The problem is that this sort of attitude is both infectious and lethal to honest inquiry, insulating the inquirer from the effect of differing viewpoints. An article like James's can serve as an antidote to the smugness that can come with insularity. The most prominent practitioners of this art will argue that they don't have to be pleasant or open in their approach to their work; they only have to be right, and they are. I respectfully disagree with that attitude as a sound pedagogical approach.
The second is that what James is doing here is turning sabrmetric principles on sabrmetrics itself. A prime danger faced by the sabrmetric community, which is in ascendance throughout baseball as never before, is that it will become as stubborn and set in its beliefs and methodologies as the people it used to attack. One of the chief attractions of sabrmetric thought is that it challenges conventional wisdom: "Is this really true? I know it's been accepted as true for decades, but do the facts back it up?" The first sabrmetric writer I read and enjoyed, Rob Neyer, applied that skeptical curiosity to numerous aspects of baseball and showed that many of them were not supported by the evidence. That's the lifeblood of the genre, and it must be protected.
It's important, it's critical, that sabrmetrics always be willing to re-examine its own conclusions, revisit its old attitudes. It's easy to shine the light on the fog, see no army and say "Good old light -- never let me down yet." The desire to always shine brighter and brighter lights on a subject is the engine of inquiry. James is, I think, telling sabrmaticians never to grow too fond of or too complacent about their lights.
Scientific inquiry doesn't have an axe to grind or a philosophy to expound; it's not defensive or angry. It simply wants to know if something is true or not. There is a strain of thought in sabrmetrics -- not a strong one, but not invisible either -- that seems to think baseball evolution ended when Oakland hired Billy Beane as General Manager. Thankfully, the bulk of sabrmetrics has lost none of its rigour and honesty, and is willing to always question the questioner as well as the subject. Essays like "Underestimating the fog" -- flawed as it might be -- will, I hope, ensure that that rigour continues.
