The Scouting Report - Compared to UZR (March 23, 2004)
I'm just going to jot down a quick note here. I compared the UZR for the 160 players between 2002 and 2003 who I had enough Fans' balloting on. The r was .58. I then compared the Fans' Scouting to the 2003 UZR. The r was .47! That is tremendous! The 2002 and Fans against 2003 was an r of .63. In terms of weighting, you give about 2 weights to UZR to 1 weight to the Fans.
Now, the caveats. It's possible that the fans were aware of the performance of the players in 2003, so that they had an unfair advantage to UZR. Then again, if I asked you, what would you rather have to establish a player's true talent fielding level as of Oct 1, 2003: knowledge of the 2002 UZR or knowledge of the fans' observations from 2002 to 2003, which would you choose? It's alot closer than you think.
It's also possible that the fans were aware of UZR, and made their selections accordingly. I doubt it though. There were large discrepancies among many notables. Fans love Griffey, Ichiro, JT Snow, Jeter, Mondesi, Wells, Hunter, Izturis far more than UZR does. On the flip side, fans don't care for Rondel White, Bell, Eckstein.
What this gives us is something useful. Now, instead of regressing Izturis' sample UZR towards the population mean (zero), we can now regress it towards the Fans' mean (in this case about +10 runs above the avg SS). How much to regress I haven't figured out yet.
--posted by TangoTiger at 04:03 PM EDT
Posted 4:23 p.m.,
March 23, 2004
(#1) -
J Cross
Nice, you can tell something by watching. Maybe for the mid-season judging you could also ask voters which defensive metrics they're familiar with (UZR, RF, ZR etc.) and see how well the metrics they known correlate with their votes (compared to how those metrics correlate to all votes?).
Posted 5:21 p.m.,
March 23, 2004
(#2) -
tangotiger
What will be really telling is the 2002/2003 UZR v 2004 UZR and the 2002/2003UZR+Fans v 2004 UZR.
If the fans have been unduly influenced by the UZR, then by definition they have nothing more to add to them, and therefore the two r's above should be identical. I'll be happy if the Fans would add to the UZR regression as much as it did in my little study here.
What I'll do is list the players that the Fans and the "true talent UZR" (already regressed) disagrees with the most. If this is meaningful, like Cesar Izturis, it would mean that the 2004 UZR (or ZR) will regress towards the Fans' somewhat.
Posted 7:35 p.m.,
March 23, 2004
(#3) -
David Smyth
I think J Cross has a good point. Why not simply ask each voter to answer a few questions about what def metrics they are familiar with, and how much they were aware of such rating for the players they voted on.
Posted 1:23 a.m.,
March 24, 2004
(#4) -
Kyle S
Are UZR totals available throughout the season? If not, it would (obviously) be great if you could collect a bunch of fan data before they were published, then compare 2004 fans vs 2004 UZR. J. Cross's idea should probably be employed anyway.
I should know this, but how well do year y UZRs correlate with year y+1? Given the nature of luck in fielding (as well as the greater effect of age on changes in fielding true talent year to year), I would guess it's probably a lot lower r than hitting LWTS?
Posted 4:29 a.m.,
March 24, 2004
(#5) -
MGL
I should know these off the top of my head by now, but the y-t-y "r" for batting lwts for players with around 550 PA per year is .675.
For UZR, it is based on UZR "opportunities" of course and not PA's or games. For around 400 opps (around the same number of games as 550 PA, or around 130 games, for an "average" fielder), the "r" is around .450. For a SS or CF'er, for 130 games (550 PA) per year, the y-t-y "r" is around .525. For 2B and left or right field, it is around .475, and for a 1st or 3rd baseman, it is around .350.
[an error occurred while processing this directive]
Posted 9:50 a.m.,
March 24, 2004
(#7) -
tangotiger(e-mail)
I should know this, but how well do year y UZRs correlate with year y+1?
Uh... this is what I said in my intro:
I compared the UZR for the 160 players between 2002 and 2003 who I had enough Fans' balloting on. The r was .58.
I guess that wasn't clear enough. The year-to-year r for UZR (between 2002 and 2003) for the 160 players that qualified was .58.
***
How did you convert Izturis' scouting report to runs? Was that just an example to make a point? Also, do you (or will you) weight all the skills evenly?
In this example, I weighted all traits evenly. That's not what I will be doing in the future.
Will you (or have you)regressing your results vs. UZr so as possibly to establish the importance of your categories?
I have done this, but my sample size is too small, and there is alot of interdependence. Some traits end up with negative coefficients, which makes no sense.
Could positional averages be used to create a profile of who might succeed at a certain position? At least average quickness, reactions and hands are necessary conditions to being even a below average SS.
This is what I will be publishing soon. I will show the average rankings by position. Off the top of my head, the avg SS was above average (relative to all fielders) in every category, and to about the same degree. C/1B was by far the lowest in speed. 2B do not fare so well.
It'd be cool to be able to conclude that Mientkiewicz has the profile of a defensively better than Rivas 2B. Or to figure out which position Jeter is best suited to switch to.
That's also my next assignment. I will compare each player to each positional average, and figure out where he is most similar (and dissimilar). In the meantime, I have provided you with each player's most similar comp, so that should give you some idea. Since Jeter's most similar comp is Bobby Abreu, we know that one, or both, of these guys are playing out of position. My guess is that it's Jeter.
Does anyone know if teams do this sort of analysis on their 20-80 scouting?
I should hope so. Isn't this where most teams are spending their efforts? It would shock me if teams only collect scouting data, but don't have a systematic process to handle it. If they are not doing what I'm doing (at least at the minor league level), then those teams should call me!