Let's say that we all agree to watch the same 3 games one weekend. Our job will be to keep track of all BIP, and, by IF/OF, say "was that a routine play or not".
Routine would mean that even putting Manny Ramirez ( or Derek Jeter!!.. just kidding you thin-skinned Jeter lovers... ) at SS or CF would make that play.
Then
, we compile our numbers. Let's say that 50% of all balls in play are routine. This will form our basis.
Then, you go to MGL and David Pinto, and say: "hey, I tracked all these balls, and half of them were routine... does your system consider them easy outs?".
MGL or David might respond that: "No... in the case of IF, I can't get the out conversion rate higher than 92% on those specific balls". This to me is a large margin of error. What happens here is that those 50% of the balls are the noise. They tell you absolutely nothing about the fielder, and should be thrown out. This is one reason the year-to-year r is not higher for fielding metrics.
On the other hand, if MGL or David responds: "Yes... because there are other things that I know about those plays, the hand of the batter, the tendency of the pitcher, the hardness of the ball hit, I was able to classify most of those easy outs as easy out... my average out conversion rate on those balls is 96%", then I'm ecstatic! This tells me that, at least on the "noise balls", we can feel comforted that all this noise is not getting in the way of other signals.
This doesn't tells us about our ability to properly evaluate the signals, but at least, we're one step closer.
So, who's up for a little work opening weekend?
https://www.battersbox.ca/article.php?story=20050203100739999