Tango on Baseball Archives

The genius of Paul DePodesta (February 4, 2004)

--posted by TangoTiger at 11:22 AM EDT

Posted 2:45 p.m., February 4, 2004 (#1) - Craig B
That's terrific.

Posted 2:58 p.m., February 4, 2004 (#2) - tangotiger
I used to want to be like Tim Raines. Now I want to be like Paul DePodesta.

Not something that I should admit.

Posted 3:27 p.m., February 4, 2004 (#3) - jto
me too! awesome.

Posted 3:46 p.m., February 4, 2004 (#4) - Matt D
"the fact that 90% of the player population in major league baseball is replaceable by someone who makes less"

I thought this 90% figure was a bit extreme. If it is true, is it due to A)Many (if not most) players are overpaid or B) Many (if not most) players are overrated, or C) Some combination of A and B?

Posted 4:16 p.m., February 4, 2004 (#5) - tangotiger
90% of the player population in major league baseball is replaceable by someone who makes less

That's a true statement, but a little deceiving.

Take the case of Albert Pujols and Mark Prior. Based on their salary of last year, you can say that they are in the bottom 10% of the salary scale. So, every player who makes more than Pujols (which is 90% of the player population) is replaceable by Pujols. Well, in some weird way, yeah.

How about looking at it from a more realistic perspective: flip 90% of your roster, and expect to replace them with players who make less, but perform the same (for each one). Yes, you can flip Jeter for Nomar. Yes, you can flip Alomar for Reyes, etc, etc, all the way down the line. You can't flip ARod, you can't flip Pujols, or Prior, etc.

But, is there a cost to flipping? If you want Reyes, you give up Alomar, and some cash. So, while you can say that 90% of the players are replaceable for players who make less, they are not readily available, and it's going to cost you money.

Posted 4:38 p.m., February 4, 2004 (#6) - tangotiger
I should say that 90% of a single team's roster might be flippable, but not every roster can be flipped at the same time. I mean, I flip Jeter to get Nomar, and then Bos flips Jeter to get Tejada, and then Bal flips Jeter to get... at some point, you can't flip.

Posted 9:44 a.m., February 5, 2004 (#7) - Arizona
But more to the point: The A's believed Tejada was replaceable by Crosby and made no effort to sign him. They believe Chavez is NOT replaceable by someone cheaper, and will make a serious effort to sign him. Other teams, they believe, use money inefficiently by overpaying for a Tejada, Giambi, Damon, Koch, Guillen.

Talking about Tejada/Jeter/Nomar misses DePodesta's point, I think.

Posted 10:08 a.m., February 5, 2004 (#8) - Chuck Oliveros
DePodesta denigrates scouts in his piece and, I suspect, any traditionalists who read this article (if they would indeed read it) are likely to be turned off. Now, as tangotiger and others have pointed out, an efficient organization requires both performance analysis and scouting. The problem in the past, IMO, has been that there haven't been any rational scouting standards. DePodesta talks about the scout raving about the guy who has all the tools that keeps swinging at the slider in the dirt. That struck me. A well-trained scout would think that this aspect of the player's observed performance could be significant and would include it in a report. The problem isn't that the process of scouting is flawed. It's that most organizations apparently don't have a system of scouting in which scouts are trained and according to which they are evaluated. However, such an approach is as likely to meet resistance as have the performance-oriented standards.

Posted 10:42 a.m., February 5, 2004 (#9) - tangotiger
I don't see how I missed the point if I said that you would flip Alomar for Reyes (which is essentially the point you are saying about flipping Tejada for Crosby).

***

Tejada is an above average player. If Tejada would have offered the A's 3 years at 5 million$ each, what do you think they would do? If DePodesta's point is that there are alot of overpaid players, fine. Crosby is an excellent prospect, and may be close to Tejada's equal, but there's a large margin for error there. I would bet that the A's wouldn't mind having Tejada at 5 million / yr, even though that is much more than Crosby will make.

Posted 12:11 p.m., February 5, 2004 (#10) - Craig B
DePodesta denigrates scouts in his piece and, I suspect, any traditionalists who read this article (if they would indeed read it) are likely to be turned off.

No kidding. That's the point. DePodesta doesn't give a flying f**k about what "the traditionalists" think. He, and the Oakland A's, can't afford to.

In fact, a central part of DePodesta's whole point is that what a "traditionalist" thinks of what you are doing doesn't matter in the slightest, unless they are actually part of the organziation you are trying to change. If you are aggressively re-engineering processes, then every concession you make to a "traditionalist" is a step in the wrong direction.

If they are, then you need to co-opt them, plain and simple. You can't change an organization overnight, but you make changes gradually and leverage your successes.

The problem isn't that the process of scouting is flawed

It is. It always has been. It's not completely broken, but it's badly flawed, a theme that Lewis (via his discussions with and about Beane) hammers again and again in Moneyball. Read Dollar Sign On The Muscle if you don't believe me. Scouting has in the past fetishized "The Good Face" to the point that subjective and non-baseball criteria have overwhelmed baseball criteria as the starting point for scouting analysis. That is stupid, and the reason nothing was ever done about it was - if you ask me - because baseball has had a lousy corporate culture as a result of being treated like a plaything by its owners.

such an approach is as likely to meet resistance as have the performance-oriented standards

If your scouts don't like it, let them leave. Make them leave. You think there is something special about scouts? There are hundreds of guys who post to Primer alone who would make fine scouts. Scouts are the dime-a-dozen guys. No organization worth its salt gives a rat's ass about what the replaceable employees think - and your average scout is as replaceable as a guy on the grounds crew. Sure, your cross-checkers have valuable experience, just like the head groundskeeper. You want to try to keep a guy like that happy. But if scouts are resisting necessary changes to process, they're in the way and should be brushed aside.

There's nothing in the DePodesta attitude to the place of scouts that is inherently problematic. New scouts, or adaptable ones, should be able to handle it just fine. There are those who will resist it, because they have been infected with a sick corporate culture. The only solution is to throw those guys out on their ass.

Posted 12:40 p.m., February 5, 2004 (#11) - tangotiger
I think my favorite line was the one that DePodesta quoted, about if you weren't doing this already, how would you do it.

There's an incredible inertia, simply because most teams don't know WHY the heck they are doing what they are doing. To believe in sabermetrics is to believe that you can properly identify and balance performance-based results and tools-based observations. And, the only way to do it is to have a systematic methodology.

I want scouting information, I need scouting information, but my guess is that the way that the scouting information is compiled makes it less than ideal.

If I had scouting information, instead of regressing the performance-based results of my players towards the population mean, I'd be able to regress them towards players with similar tools. Once you have a convergence of tools and performance, then you've got the pinnacle to sabermetrics.

My feeling is that most teams would rather dip their toe (as opposed to jumping) into sabermetrics because:
1 - they think it's only about statistical analysis
2 - there's inertia
3 - they're afraid to pay for it

Sabermetrics, as a process, can probably identify 5 to 10 wins that you can maximize in some way. That's at least 10 million$ of value right there. But why would an owner believe a guy named Tangotiger of all things that there's 10 million$ of inefficiencies that can be tapped?

What baseball owners should want is geniuses like Beane/DePodesta or Ricciadi/Law, guys who believe in doing things differently, adhering to a set justifiable philosophy, and controlling things from the top.

Posted 2:27 p.m., February 5, 2004 (#12) - Craig B
why would an owner believe a guy named Tangotiger of all things that there's 10 million$ of inefficiencies that can be tapped?

Because he's a smart guy who understands business and knows that inefficiencies are everywhere. That's what John Henry did in Boston - stopped buying the line that you have to do it a certain way "because it's baseball".

It can be done. It won't always be done, but it certainly can be done.

DePodesta's real point is that it's not enough to have a smart guy like Tangotiger crunching data for you. What needs change is not the details of player acquisition, but the whole operation and the way it does business...

Posted 3:55 p.m., February 5, 2004 (#13) - tangotiger
I agree, and that's why I don't think a team that's going to just dip their toe in the sabermetric water will be enough. They are either in, or they're out.

Regarding my "named Tangotiger" comment, I meant that you need Paul DePodesta (himself, not Michael Lewis) to write the book, otherwise, it's just a whole bunch of people giving good advice online, without having the "proven" credentials. That's also the ironic part, since the "proven veteran player" is usually the guy that sabermetrics would say is being overpaid and should be avoided.

Posted 5:50 p.m., February 5, 2004 (#14) - Mike Green
Surely, the situation is a little more grey than this. There are definite weaknesses in the traditional scouting approach, and any intelligent organization would not accept it holus bolus IMO.

But, I can't agree with the assertion that scouts are as replaceable as grounds crew members. Keith Law has said that we have no fielding stats that generate reliable projections. How is a team to evaluate whether a prospect has the fielding ability to play a particular position without relying to a significant degree on the subjective evaluation of scouts? I don't agree that the judgment required to perform this kind of subjective evaluation is possessed widely. I've been watching and thinking about baseball for many years, and I know that I could not do it well.

But, as for changing the methodology that scouts use, I'm right there. For instance, the regular use of a stopwatch and a zone fielding chart for prospect fielding evaluation would be something on my list. And yes, changing the methodology can involve some disagreement and perhaps a few departures.

Posted 8:21 p.m., February 5, 2004 (#15) - MGL
Great article! He really writes well too. I'm listening to the video lecture right now. This guy looks about the same age as my son!

One of the funny things is that it appears as if Depodesta did and may still be trying to reinvent the wheel in a lot of the things he does. For example, he talked about creating the run expectancy matrix as if it were a great 21st century innovation when in fact, it has been constructed at least since Hiden Game...

Posted 10:02 p.m., February 5, 2004 (#16) - MGL
In the video lecture, Depo again (Beane did in Moneyball as well) alludes to the fact that the A's use a player "rating" system or metric similar to what we talked about in the above thread on digitizing the events in a game. For example, a line drive is socred as .8 of a hit (or .5 runs or whatever) and no attention is paid to the actual outcome (hit or out). Same thing for every other type of batted ball. Ideally that is a great "system" and ultimately the optimal system. In order for it to work, though, you have to have almost perfect data. For example, if you assigned .5 runs to every line drive, regardless of the actual outcome, you would end up overvaluing weaker hitters who hit "weaker" line drives and vice versa for stronger hitters, etc. IOW, if you are going to use a system like that, you had better have fine distinctions in your data (like a 80 mph line drive versus a 60 mph line drive). I have no idea what kind of data the A's have or use, but I am a little skeptical of their being able to utilize this kind of a system, at least at the present time until that kind of data becomes avalaible...

Posted 10:53 p.m., February 5, 2004 (#17) - tangotiger
Yeah, I was a little annoyed about the lack of citations on things like run expectancy and Markov chains, making it almost seem like it was A's-invented.

Then again, it *is* a presentation, and he's not doing a history lecture, or writing a magazine article.

Posted 11:09 p.m., February 5, 2004 (#18) - Danny
I think it's possible that DePodesta really didn't know, at the time, that other people had used these methods. He then came up with his own run expectancy chart. His article and lecture were simply detailing his time in MLB, and his vision, so I don't see much of a problem with not citing previous work.

Posted 12:58 a.m., February 6, 2004 (#19) - MGL
It didn't really bother me about not "citing" other sources. Actually I guess maybe it did, as I couldn't imagine doing that kind of lecture and not explaining how I heard or read about these things and then decided to duplicate and refine them myself. The way he explained it comes across as kind of silly to someone knowledgable about sabermetrics.

Danny, I can't imagine that as he got into sabermetrics, he didn't read about RE charts and the like.

Anyway, he seems like a real smart guy. I've upgraded the level of respect I have for the A's organization...

Posted 6:26 p.m., February 6, 2004 (#20) - Rob H (homepage)
DePodesta's comment about meeting mgmt of some company and investing in them because "They're smart and the CEO's built like a Greek god!" reminded of something he said at BA (see homepage). The following quote sheds some more light on DePodesta's view of scouting:

For example, if an investment bank is going to invest a great deal of money in a particular company, it will not invest millions of dollars by just looking at the balance sheet and never meeting with the management. Conversely, it will not make the investment by just meeting the management and not checking the balance sheet. We�re all making million-dollar investments on these players. What we�re trying to do is get the most comprehensive picture we possibly can.

So, while DePodesta may denigrate some scouts, he almost certainly believes scouts can serve a useful purpose.

Posted 9:30 p.m., February 6, 2004 (#21) - MGL
I wholeheartedly believe in a combination of scouting and obejective analysis, however, to some extent objective analysis ALWAYS trumps scouting.

As well, the usefulness of scouting is really only limited to young/inexperienced players and old or injured (presently or in the past) players. Basically scouitng fills in the holes/gaps/questions/uncertainties of the objective analysis. When I evaluate a player (almost always in the context of a projection), I like to build a "story" on the player, which ideally includes scouting/injury reports...

Posted 2:09 p.m., February 9, 2004 (#22) - Mike Emeigh(e-mail)
I wholeheartedly believe in a combination of scouting and obejective analysis, however, to some extent objective analysis ALWAYS trumps scouting.

A player with a .250 level of *real* talent derived from past performance performs at a .270 level this year; real improvement or just a fluke? The objective analysts would suggest that his likely level of performance would be around .252, or .255, or something else closer to .250 than .270 next year. But the scout watches game tape of the hitter last year and this, and picks up something he's doing differently, whether it's laying off the hard slider out of the strike zone, driving pitches the other way rather than trying to pull everything, opening up his stance and hitting line drives rather than weak popups and shallow flyouts. Or maybe he's seeing that there's nothing really different, and that he just had a couple of extra good weeks where those little loopers were dropping in.

Objective analysis shouldn't trump scouting, nor should scouting trump objective analysis. They complement and supplement each other. Objective analysis is derived from aggregate data about players with similar performance levels, and provides a *likely performance* estimate for the player; scouting gives you information that can tell you whether a *particular* player has a good chance of being different from the aggregate. The GM needs to evaluate both sets of information.

-- MWE

Posted 2:47 p.m., February 9, 2004 (#23) - tangotiger
Sure, they both need to be evaluated. The question is how much weight to give the scout.

You have to know:
1 - How good is the scout at evaluating what he sees (qualitatively)
2 - How persistent is it what he sees

If the scout sees that Dwight Evans has changed his stance, and is finally hitting the way he should, can the scouts qualitative observation be very confident after 20 or 30 or 50 PAs?

And, even if the scout is right... is what he sees going to be what he's going to continue to see for the next 100 PAs?

***

In terms of results-based performance, you would regress a player's performance by:
200 / (200+PA)

This means that if you have 200 PAs, then you would regress the player's performance 50% towards the league mean, because you have that much uncertainty to account for. If you had 1800 PAs, you'd regress 10% towards the league mean.

But, what about scouting, or tools-based analysis? How reliable is it, and how good can it get?

I would say that a trained scout can do something like:
50 / (50 + PA), with a max of 200 PAs or so.

That is, what I can tell you about a player with 200 PAs of stats (50% regression), a scout can tell you with 50 PAs of observations (50% regression).

Give a scout 200 PAs, and we'd regress his observation 20% towards the mean, and you'd have to give me 800 PAs of results to give you the same confidence.

But, after that, there's not much more that the scout can give you. There's too much deception that the scout can't process out.

***

Note: I have no idea if I'm right, and what the boundaries or break-even points are. This is just a guess as to how I think it works.

***

Optimally, I would have the scouts mark things down in a systematic way so that it becomes part of the performance-based results. Like, where was Jeter, how long did it take him to react to a ball hit, how fast did he get the ball out of his glove, how did his 1B handle the throw, etc, etc. Some of these things are qualitative, and some are quantitative.

The key is to try to quantify as much as you can, and in a systematic fashion.

Posted 3:06 p.m., February 9, 2004 (#24) - tangotiger
Just to try to give you a better idea as to what I'm thinking:

regression(performance) = 200 / (200 + PA)
regression(tools) = 5 / (5 + PA^0.333)

So, here's how much regression you would need with various levels of PA:

Results Tools PA
95% 70% 10
80% 58% 50
67% 52% 100
50% 46% 200
25% 37% 600
20% 35% 800
10% 29% 1800
5% 24% 3800
2% 19% 9800

I kind of fixed 100 tools PA to be equal to 200 performance PA. I don't know that they are.

Take the case of say Shane Spencer, and his remarkable Sept. After 50 PAs, a tools-based analysis might have determined that he's really a slightly above-average MLB hitter (say a .350 OBA guy in a league of .340), and would have regressed their analysis 58% towards the .340, for a .344 "true" OBA. So, the scout thinks he's seeing a .350 guy after 50 PAs, but we make a best guess that he's actually seeing a .344 true player.

Results-based, we'd see Shane Spencer as Barry Bonds or Jim Thome, and claim him to be a .440 player. But, we regress that 80% towards the mean, to a .360 player. Again, our best guess would be that he's a .360 player after 50 PAs.

What would be better is to merge these two as being not independent. If a scout's observation says that he's a true .344, then we don't regress Shane towards .340, but towards .344. And, having the extra reliability of the scout, now we don't regress only 80% towards the scout-driven mean, but say 90% towards the scout-driven mean. In this case, his .440 becomes .354.

So, that becomes our best guess, merged, result. Based on scouting, and how we think it's reliable, and based on performance-results, and we have a pretty good idea of its reliability, we come up with a true talent level of .354 for Shane.

This is where I see the next advance of sabermetrics/scouting (if it hasn't already occurred behind the curtain).

Posted 3:45 p.m., February 9, 2004 (#25) - Rally Monkey
Scouting and other info could be used to create player templates for regression.

Say Prince Fielder, he'd be regressed against big fat slow guys with lightning quick bats.

When I was doing my MLE's and adding in a regression to the mean feature, I asked myself why in the world anyone would regress Prince's stats the same way you would for a 150 lb no hit SS like Ozzie Smith Chavez.

Posted 4:33 p.m., February 9, 2004 (#26) - tangotiger
Voros, MGL, and Nate Silver, among others, use the player's body type and position to do their forecasting.

Posted 11:23 a.m., February 10, 2004 (#27) - studes (homepage)
BTW, a friend of mine heard him speak at an investor conference, and said he was super. Great speaker. DePodesta is originally from the East Coast, and said, in response to questions, that he wouldn't mind going back to the East Coast. Something like "If the Expos move to D.C., they could get me cheap."

Also, he said that Michael Lewis originally was just working on a NY Times Sunday article when he was covering the A's. It wasn't going to be a book. Evidently, they cut him off when they realized that he was going farther (after about a month of observation) and DePodesta claims that he must have gotten a lot of his information based on other sources.

Posted 3:24 p.m., February 10, 2004 (#28) - MGL
But the scout watches game tape of the hitter last year and this, and picks up something he's doing differently, whether it's laying off the hard slider out of the strike zone, driving pitches the other way rather than trying to pull everything, opening up his stance and hitting line drives rather than weak popups and shallow flyouts. Or maybe he's seeing that there's nothing really different, and that he just had a couple of extra good weeks where those little loopers were dropping in.

Mike, the big (huge) flaw in that kind of thinking is that a lot of what the scouts think is a true change in ability or a reflectionof true ability is luck itself! How do you think players with trye .250 BA's hit .350 over 10 or 12 games or .270 over a season? It's not just bloop hits falling in! It is that they lay off that slider, drive pitches the other way, etc. Thise things flucuatue as as well - not jusy whether the bloop hits fall, the liners get caught, or the long flies just make it over the fence or not. There are all kinds of levels and manifestations of luck. What makes you think that a scout or anyone for that matter can watch a player and figure out to any degress of certainty what is random fluctuation and what is true ability?

When I say that objective analysis "trumps" scouting what I mean is that while objective analysis has it's limitations (even in a perfect form), it makes no mistakes! Scouting on the other hand, definitely has it's limiations too, even in its prefect form, but in addition, and not insiginificantly, it makes lots of critcal errors, due to the ignorance and superstitions of the scouts, and the "illusions" naturally created by the eyes and the brain...

Posted 5:40 p.m., February 10, 2004 (#29) - Mike Emeigh(e-mail) (homepage)
When I say that objective analysis "trumps" scouting what I mean is that while objective analysis has it's limitations (even in a perfect form), it makes no mistakes!

This assertion is dead wrong. I'm not aware of *any* objective analysis technique that we use that is *not* based on the assumption that individual performance can be inferred from aggregate performance, and the paper I've linked in the homepage discusses some of the problems with that inference. Walt Davis has discussed this at some length in past threads.

-- MWE

Posted 9:52 p.m., February 10, 2004 (#30) - AED
Mike, I'm not sure what you're talking about. Using aggregate data in the way that is described in that paper is completely different from using known distibutions of talent as priors in Bayesian inference.

Posted 10:39 p.m., February 10, 2004 (#31) - MGL
Mike, you don't understnad what I mean by a prefect objective analysis making "no mistakes" yet having limitations?

Posted 10:43 p.m., February 10, 2004 (#32) - MGL
If I tell you that MLB player A had a .370 BA in 100 AB's, you can tell me exactly what the best estimate of his true BA is. That is what I mean by a "prefect" objective analysis (you can't do any better than that with no more information) and there are no "mistakes" in that estimate (again, it is a perfect estimate given the information we know).

Can I not make an obtuse comment without being accused of being "dead wrong?"