Using a Model During the Auction
April 19th, 2010 by Robert Dixon in TheoreticalIn the comment threads Chris Liss said:
"I don't know every step my brain makes. I just look at the facts, and form impressions which I translate to a loosely ordered list and trust myself to get the job one at the auction. I'm not sure why you're convinced that making these calls ahead of time is necessarily helping you. "
That is a fair question and I am going to answer it.
In the course of preparing for the auction Bill and I will project every player. Using techniques like these we will use those projections with our model to convert them to prices. Before the auction starts we are both comfortable with the value of every player in the pool and how we arrived there. This is not to say we are equally confident on all of those values, but more importantly we know how confident we are in each of them. Also, our pool will add up to the appropriate total. As each player comes off the board we record what price he went for and how that reshapes the pool. In other words, while we started with a fixed price for each player we are constantly adjusting for as money is spent in the auction. We know how much people have overpaid or underpaid for the total talent purchased and what kind of discount that implies for the remaining talent. I don't think a general axiom like "People tend to overpay early which opens the door to good value late especially on lower price players" comes close to this even if that is what usually happens.
But beyond that, we can answer many more questions instantly mid-auction such as:
How much of the overpay or underpay has been on hitting vs pitching?
What positions are going over or under our values?
What stats are people overpaying for and by how much?
There is a lot of talk about the many things that go on during an auction and how those change the values of players. For me, an auction is almost exclusively about getting value, so I think all of those other factors are dwarfed by the overpay/underpay tracking. However, to whatever degree you want to adjust on the fly for anything else a model is just as versatile as the human brain, and much more accurate and precise.
For example, in the CR auction I thought closers were consistently going for too much. This did not surprise me at all because I have never been in an auction where I did not think closers were overpriced. Just because I go into an auction not expecting to own any closers doesn't mean I should ignore valuing them. Players only have value in relation to the total pool. Besides, if I went in like this I would be unprepared to fully take advantage when a closer does go cheap. If you had a different overall view, and felt that while saves were overvalued you were not comfortable giving up on the category out of the gate, the model can still be helpful. Simply adjust the value of a save to see what the implied price is during the auction as the first few closers go and from there you can still look for relative value in closers.
Everyone is irrational. When we let our instincts take over in the heat of the moment all we are really doing is finding out which players we loved too much to pass on. If you think your subconscious is better at picking talent than your conscious mind then it might be worth it. However, by turning it over to spur of the moment gut calls you sacrifice the ability to precisely reflect your views with your pricing. You also are stuck with rough impressions of how things are going instead of the exact view a versatile model can provide.
I would like to close with an analogy that I think is much more apt to FBB than how Tom Brady throws a football. If a casino allowed you, an experienced blackjack card counter, to use your laptop while you played would you? Or would you say that since counting cards has been good enough for you in the past you will stick to it? Or perhaps say that you have played blackjack successfully for so long that you don't even have to count cards anymore? Or sigh and say that you can see where some people might like the crutch of a computer since they won't deal down to the last card of the deck it can never tell you exactly what is coming next and so can't give you any real value?
I know my answer.
Loading ...
I'm not sure why Chris is arguing that everyone should be freestyling their bidding at the auction, but then again I don't actually think he is. I do think he's claiming that his head his better suited to the appropriate analysis and flexibility for the auction then any numbers-bound model he can imagine. That's fine for him. It isn't for everyone.
I think what you're describing above, Robert, is right. It has the right adjustments, the money tracking that is key, and the speculative/arbitrary/informed decisions one has to make about the value of closers vs. speed vs. the rest of the room, etc to adjust on the fly.
I think it is still an open question about whether the cost of having all that info during the draft is actually a help or a hindrence, but certainly if you could reduce the friction of info acquisition and management (as Chris claims he's doing by living in his head) it would be a big plus.
I don't know if you've experienced the way a Tout Wars auction is run, but the pace is astounding. There is no time to do anything but bid and enter the winning price, if you need to, on your sheet. The typical pattern is: Papelbon 14, 20, 22, 23, going once, going twice, going three times, 24, going once, going twice, going three times, sold. 24. By the time you try to look something up, the player is sold.
I keep a plus minus count of the money during Tout to track whether too much or too little has been spent vs. expectations, and it's easy enough to see whether categories are over or under. The years I used a computer I was sunk, but maybe that's my keyboarding. But my point isn't about intuition vs. the model. It's what parts of your model actually help you, and which parts give potentially misleading information or a false sense of security, and how they might lead you to make poorer decisions than you might if you used a different model.
It seems to me that's the issue we're lurching toward.
If you think your subconscious is better at picking talent than your conscious mind then it might be worth it. However, by turning it over to spur of the moment gut calls you sacrifice the ability to precisely reflect your views with your pricing.
Why are "your views", i.e., the projections you settled on before the auction, important? What basis do you have for believing them? Are your views of players – the ones that inform O1, O2 and O3, and the probabilities of each any good? If so, how would you know?
Let's put it differently. Let's say I get together with a friend and rank the NFL teams 1 through 32 in terms of likelihood of total wins. We analyze the teams, give our reasons and rank them. And then we have two choices – either go with our top-10 or Vegas' top 10 to win a prize. Which do we choose? Ours? Why? Just because we decided on them at some prior time? Just because we think it? Thinking it does not make it so. Don't we need a basis to trust our thinking beyond it just being "what we think?" Do we have expertise in this? Have we done as much research as the oddsmakers? Why go with our picks?
Don't get me wrong, I'm not saying your actual projections are any worse than anyone else's. I just wonder why it's important to you to stay true to them. As opposed to synthesizing the variables on the fly and going with what your gut tells you.
When we let our instincts take over in the heat of the moment all we are really doing is finding out which players we loved too much to pass on.
Not me. I pass on a lot of players I really really wanted to own and thought I would own, but for someone else's surprisingly keen interest in them.
If a casino allowed you, an experienced blackjack card counter, to use your laptop while you played would you?
Of course I would. No idea what that has to do with the current debate, though. As I've explained exhaustively – it's not possible to do what I do and somehow use your model. They cannot possibly go together. Your model is an analytic tool. I'm am not doing analysis when I make my bids. It's apples and oranges. One cannot use both. One must choose one or the other.
But beyond that, we can answer many more questions instantly mid-auction such as:
How much of the overpay or underpay has been on hitting vs pitching?
But only relative to your values. Which depend on your projections. Which for some reason you trust, but I have yet to hear why.
It sounds more and more like you have to make *some* kind of projection because otherwise you don't have inputs for your model. The cart is before the horse. The tool requires a certain framework, so you envision the task to fit the tool rather than the tool to fit the task.
Wish I could edit one of the responses – basically, I said I exhaustively explained that my method is incompatible with Robert's model, which is true, but I also earlier said that maybe I did do "unconscious" projections. Which means Robert's blackjack analogy at least has a basis. I think the truth is that "unconscious" and vague projections would not work with Robert's model because (a) to the extent I tried to articulate the stats, they'd be false, i.e., not really a reflection of the information I've absorbed for translation into bidding and (b) I'm not sure they exist in the form of a one line projection anyway – but a vague matrix of many possibilities.
So I retract the dismissal of the analogy – I still think it's inapt, but I gave Robert a basis for making it.
Chris, if your evaluation of a player is but a vague matrix of possibilites, which of those possibilities dictate your bid, and how do you decide on which possibiity that is?
Peter, thanks for the feedback.
My first entry into the fray was an article that gave my background and then went on to give the simple starting point that you need to price the entire pool of players before the auction. Without this you have no way of tracking how much over or under players are going. This is the same as saying no player has a value in a vacuum, but only in the context of the pool of talent your league is using.
Chris, you are missing something huge. Yes, my success or failure will depend on how well I have projected and priced players. That is true for everyone. What I achieve is playing the game as well as I possibly can with the information I have. I strive to translate my understanding of the players into correct bidding with absolute precision in order to maximize.
What I achieve is playing the game as well as I possibly can with the information I have. I strive to translate my understanding of the players into correct bidding with absolute precision in order to maximize.
But how do you know you're translating the information you have into correct bidding? It seems you know how to translate the projections you make pretty well into correct bidding, but how do you know (1) That you're prioritizing your information correctly when you create your projections; (2) That the projections you settle on are really an accurate reflection your information or (3) that you're informed enough to make this translation (even if perfect) worthwhile?
Chris, if your evaluation of a player is but a vague matrix of possibilites, which of those possibilities dictate your bid, and how do you decide on which possibiity that is?
It's not conscious, so I don't know specifically. Each possibility (depending on its likelihood and magnitude) is given some weight. I see a range, and when the bidding starts, i join in until it gets too high or until I get the player. It's not a one to one correlation between one scenario and my decision to bid. Because at auction time, there are many possible scenarios for every player. I don't choose one arbitrarily or assign an arbitrary probability number to each. I just have a sense of the range as it applies to a player and let myself bid or not bid according to it (also relative to what other players with other ranges are going for, too). I keep refining my sense of the "market" as the auction progresses – which I suppose is my way of keeping track of over and underbids. I don't do it mechanically, but by constantly updating my sense of what's (relatively speaking) a good or bad relative price as well. I think I'm internalizing the market to an extent – pricing it in as we go.
We have officially reached the nonsense point.
You bid when it feels right and stop bidding when it gets too high? Really? The bidding is "too high" but you won't quantify things.
Everyone who puts projections into numbers is subject to your "but they aren't right!!!" mantra, but somehow your intuition and gut just 'get you there'. I think you have backed yourself into a corner with a lot of statements that do not mesh consistently or rationally.
Robert, I explained exhaustively both what goes into a gut call and why I think synthesizing the data into an impression is more apt than doing backward looking analysis while predicting the future.
You have yet to even supply a good reason why you should believe your own projections other than that they happen to be what you came up with. You don't claim to be an expert or to have particularly unique or important information to generate your projections. You don't have a justification for why you assign the percentages to your O1, O2 and O3. Yet it's still very important for you to translate your projections precisely.
It sounds to me you just know how to build models and are determined to use one whether there's any rhyme or reason to doing so or not.
Robert and Chris, I've said this before, I'll say it again…
You're saying the same thing. You do the same things. You do them in the opposite way.
Robert, Chris takes offense that you think your way is more analytical. You don't acknowledge the arbitrariness of some of your decisions. They may be the correct decisions, but they are suffused with subjectivity, like basing your prices on projections.
Chris, Robert wants you to acknowledge that you implicitly have projections and prices and a draft tracking system, which you actually have done. He has also asked you do acknowledge that if you used his system with your system you would do better, which you have justificably resisted.
So, I don't think this is the nonsense point, this is the point where the two sides recognize that they don't recognize themselves in the other side, even though it is clear they are both there. Hmm, maybe that is the nonsense point.
I think Chris has presented himself as a bit of a fbb magic eight ball. Instead of shaking him, you need to give him tons of inputs. Instead of a random answer, he gives you something that is '3%' off 'the ‘right answer’'. But more than that, he can't really tell you how or what he does, other than the inputs he studies. We don't know whats going on inside.
I do think this is an intellectually comprehensible argument, and I agree with Chris actually it is unfair to say he is making projections or even expectations. All he knows is he is studying inputs, and his Braingut ™ synthesizes out an answer. So maybe a model wouldn't help him, because he can't really say how he turns his inputs into bids. He's like a medium, channeling the answers of his highly developed Braingut.
That being said, I agree a lot of Chris' complaints strike me as somewhere between unpersuasive and nonsense, particularly when he says stuff like 'mistakes are not necessarily a net negative', he wouldn't benefit from a spell checker in a writing analogy, or that mathematical methods aren't good for analyzing the future, or moving from presumptions about facts to valuations.
I don't really see where the conversation goes from here, because Chris's methods are so alien from Robert's and Bill's. I would personally like to hear more from Peter about how the different analytic non projection approaches work.
I'd also be curious to discuss how we can test the power of various methods, without having to play 10 leagues a year for a few hundred years and seeing who wins.
ERIC: I'd also be curious to discuss how we can test the power of various methods, without having to play 10 leagues a year for a few hundred years and seeing who wins.
When I was learning to play Hold 'em I had this computer game called Turbo Texas Hold 'Em that let you set up situations at the table and let you play thousands upon thousands of iterations of particular hands. Because of the nature of the beast, a fantasy auction is somewhat more complicated, but I think working out the rules for an automated auction system would be very challenging and rewarding (if there are any programmers out there interested in taking up the task). To test, for instance, whether there is an advantage of Stars v. Scrubs or not, over thousands of drafts, would be of great interest.
All of a sudden, I can think of a commercial model for it, too. Forget I mentioned it.
And Peter- I think Robert has acknowledged that projections and % likelihoods for each outcome are very subjective. It's just a place where we have nothing better to plug in but our best guesses, so thats what we plug in.
And I don't see how Chris can claim to be more analytical. Didn't he just write a whole post saying what he was doing was synthesis, the 'opposite of analysis?'
I know, Eric, Chris has said a lot of stuff. He even said to me he doesn't do projections, but then if you talk to him he has specific portraits of every player living actively in his imagination. I think that's tantamount to a projection. If he's drafting off the 2011 cheat sheet, what are the prices next to those names? Those are projections, I think. If he doesn't think so, that's cool. He doesn't have to agree with me.
I know that Robert has acknowledged that projections and percentage outcomes for each outcome are subjective, but I don't the he sees yet that using the best available projection statlines to derive draft values is baking error into his draft approach (at least as far as I understand it),
The primary reason for this is that there isn't a single fixed ideal outcome in a fantasy draft. The decisions you make change not only the values of the players relative to your team, but relative to everyone else that is in the auction at the same time. So, if everyones' projections are equally icky, but everyone is trying to predict the same thing, then the one who does the best conversion of prediction to actual value (fantasy points) would win.
But in a fantasy league some teams are valuing saves, and some are not. Some are valuing steals and some are not. Some put more value on power, while others prefer to value pitching. This means that while everybody is traveling to the end, they're taking (sometimes wildly) different routes, and it turns out that the end isn't necessarily one place, but rather lots of different places.
So, you can perfectly price all the players for one trip, but if everyone else is going on a different trip your prices won't be nearly as much help to you. To be clear, I'm not arguing against price lists or trying to figure out the market or even having a model. Some don't need these things, I guess, but I find them useful. I think it is very important, however, to understand just how mutable all the prices are, and why they might be different for reasons that will hurt as much as help you.
Peter, the point that you are pricing one trip is important. At the end of the day, all the points matter evenly. It doesn't matter that everyone is pricing differently. They are all using their intuitive/inate models for this. But valuing the stats consistently across the players, and then adapting to the changes that occur in the auction is the goal.
As you adjust your model to reflect that the pool is overspent by 20%, having that happen in a systematic fashion maintains the integrity of your initial assumptions, and makes the prices you bid for players consistent given your views.
Modelling isn't about having different/better/worse views that other people, it is about systematically valuing your view, looking at all the players with the same objective view.
I understand models, I don't understand the "adapting to changes that occur in the auction that is the goal." Just like the bands of risk and the guesswork that goes into creating them, this seems to open the process up to the flexibility and improvisation that is necessary during the auction. But then that also makes it subjective and intuitional. Which is why I'd like to go back to the walkthrough of the model.
When we left off, I had asked how the bands of risk were created for each player projection, and whether complementary bands were created for other players to account for the various scenarios? That is, if you assign a 25 percent risk that Jason Heyward will crap out after 150 AB, do you also create a 25 percent chance that some other player (Diaz, Schafer, Cabrera) will get more AB? And how actually interrelated are all these actual calculations?
The answer to this Heyward/other Braves scenario, is that yes, a quant would ideally account for a Heyward fizzle in the numbers of all the other Braves. Or at least I would, and I don't see why one wouldn't if taking this probabilistic approach.
I totally agree that you would Derek, and that Robert would too, though he hasnt said so yet.
But let me be the devil's advocate here. If you create a 25 percent scenario where Heyward fails and you give AB to Diaz, do you then split those AB as if he were the good Diaz we have faith in and the worthless Diaz we've seen rather move of? And at what point are you divvying up those hypothetical AB?
Once you're projecting guys for only 150 or even 300 AB, the warp and woof of sample size hugely outweighs talent. In my model, derived from Regression Analysis primarily, if you don't have a history of 250 AB the formula basically gives you the constant in each category. And from my looking at CHONE, it works the same way to the exteme (Sean basically gives the 2000 players who might play in the majors this year, but probably won't, the constant.). This is great for predictive accuracy, but it's crap in terms of differentiating players and talent.
So, what point is slicing and dicing on Heyward when we have hundreds of players we simply assign a constant value? But that's the question I'm asking those who do it. Maybe I'm missing some level of accuracy that hasn't shown in the numbers I'm looking at.
I'm not trying to prove myself right. But I do want to find out what the most right we can be is.
Peter
That being said, I agree a lot of Chris' complaints strike me as somewhere between unpersuasive and nonsense, particularly when he says stuff like 'mistakes are not necessarily a net negative', he wouldn't benefit from a spell checker in a writing analogy, or that mathematical methods aren't good for analyzing the future, or moving from presumptions about facts to valuations.
Here's what I ask, and I think I'm done beating a dead horse for a while. (1) Don't nitpick throwaway comments that are peripheral to my thesis. Refute the thesis itself if you disagree. Why do you find my "limits of analysis" post unpersuasive. Can you give me examples of how analysis has been used effectively to predict the future. (And I don't mean conditionally as in: "if earnings are x, then the stock price should be y.").
But more importantly (2) – just consider that analysis (which is but one mode of processing information) might not be optimal here. Just consider it. And obviously the model is just an analytic tool and is only optimal to the extent analysis is the optimal way to attack this problem.
And I'll leave one final thought. The task is to spend $260 on the players most likely to produce the best numbers for you. The best numbers are those most likely to help you win a 5 x 5 league.
That's it. It's not to apportion $2600 among 280 players. That's just one approach that might or might not be optimal.
You don't have to agree with how I do it, but my impression is that you don't even see anything beyond analytic thinking as a possibility. Even though I have explained in detail my method you either conclude that it's a "magic 8 ball," i.e., unserious, or it really is analytic thinking just that I'm not acknowledging it.
So I'm asking you to consider another possibility. If you are unwilling or unable to do that, there's really not a whole lot I can do.
Chris, our arguments themselves are based on mathematics and rational thought. Since you reject those as the process for evaluating the game, I suppose I should be completely unsurprised that you reject our arguments.
As to why nobody will tackle your grand thesis, it is untestable, unverifiable, unaccountable, and unanalyze-able. How exactly should we go about discussing it?
Chris,
I agree that the goal is to win the most often. It seems that there are probably many routes to this goal. And in all games with incomplete information, a mixed strategy is probably ideal.
For this to be an educational discussion, you don't all have to have the same opinions. While I tend toward the mathematical side of the argument, I would like to better understand how both sides work, and then I can take what would work best for me away from it.
As it is, what I understand about your side is:
1) You study the palyers at great detail
2) You probably have as good an understanding of how the context of team, injury, career arch and other aspects of the player value change as anyone
3) You then, in your head, in a process you can't or don't want to explain synthesis this information into action.
So if one studied as much as you, how do they synthesize the information? and what makes the way one person synthesizes better or worse than the next person?
These types of things would greatly help in trying to understand your approach.
I'm sorry if magic 8 ball struck you as unserious. Perhaps 'black box' is better? My point was intended as what Rep said above, that you study and take a ton of inputs, and spit out an output but we don't know whats going on inside.
I don't think all those comments were 'throwaway' comments, and many of them still make my head spin. I'm always willing to let you clarify, like when you said 'i wasnt saying baseball wasnt a math game.' As soon as you said that's not what you meant, we dropped that point and moved on.
As far as your central thesis goes, it is a very difficult one to refute a priori because it really comes down to "I study everything in sight i consider relevant, and figure stuff out but I'm not precisely sure whats going on inside."
Is it possible this is the best approach to fbb? Definitely. But I don't think its possible to figure that out from a discussion, which is what I was getting at before in my earlier question about testability.
I appreciate Eric and Rep – that you are at least considering that there is an alternative. But all I can tell you is the facts I feed into the algorithm (all the prep work), and the outputs you can see from the CR auction in this one case. It's mysterious. Tom Brady doesn't know how he knows what to do – he just prepares, sees the situation and responds. Artists, writers, musicians don't always know where that tune comes from or why the character they created ended up taking a certain turn – acts of creation are not always knowable in the sense that a mechanical analytic model is. I think being good at this game takes some imagination and creativity.
I would assume my process develops via a kind of natural selection – the strategies that bear fruit survive, the factors that seemed to work as indicators are emphasized, the ones that didn't are deemphasized until they prove otherwise. The key is in weighing each factor (ground ball rate, K rate, walk rate, park, health, etc.) in proper proportion in each circumstance. For example, last year I bought Todd Helton for $7 in mixed Tout because I thought the most pertinent fact about him was that he was healthy again, not his age, his poor 2007 and 2008 seasons or his downward trend. That factor had extra weight for me, and so I got him. But with another player, the big factors might be ground ball rate and fastball velocity. Even though the pitcher had mediocre other peripherals and pitches in a tough division. It depends – one sees factors in combination and weighs different ones differently for different players and circumstances. Not every fact gets the same weight for each player. How I synthesize the factors is constantly evolving, too – I think most people overemphasize K rate in isolation, for example. I think even moderately strong GB/FB rates can be very important if paired with other factors. But it all depends on the entire picture of the player.
Basically, I pay lots of attention to everything – all the facts on the input side and all the results on the output side. I try to minimize bias, i.e., my arbitrary guesses and opinions that projectors value so dearly on the input side, and self-congratulation or just remembering my good results on the output side. I want to give the system real feedback. But I mostly stay out of my own way. Just study for the test, and trust myself to perform when the names are called.
One other major advantage I have is that this is my job. So I'm making calls on the radio and in print that I'm checking up on constantly. So instead of just my players, I'm making calls on everyone all the time and so my system gets a ton of feedback.
This is as well as I can explain it. The test for this and any model is in the performance. If Robert is somehow really good at generating projections (I don't know why he'd be, but maybe he is), then his model will work well, and he'll win more than his share of leagues over time.
Chris- the problem we run into now is that while trying to look at your approach and consider its merits, we are unable to determine how decisions are made. This makes it impossible to evaluate the merits of it.
As to your arguments about the "mathematical solutions", you also aren't articulating why they fail. You continue to say it is not the best way to reach a decision on how to bid on players. That the approach you use is better. But the question we keep coming back to is WHY??? And until you can offer some kind of insight to this process, you have created an impasse. I would like to know how you make decisions, but respect the fact that you say it is not at the conscious level.
The math guys claim they are happy to discuss the how and why of what they think. I find this to be valuable and would like to partake in discussions about their process. But you just saying they are wrong without offering a better solution will only cause arguments and lead to unfruitful discussions.
The reason the mathematical solutions are likely to fail (though not necessarily) is that they lack a sound basis. They're based on dollar values which are based on projections, which are guesses. Robert assigns a 10 percent breakout scenario for Heyward for example in calculating his dollar value. But the projections under that scenario are just a guess, and the 10 percent is just a guess. It might really be 22 percent for all he knows. And combined with the guess as to what stats a breakout entails, it's just incredibly arbitrary.
So while Robert might actually be very good at these estimates – maybe the breakout stats are close and so is the 10 percent, what reason do we have to believe that? Robert himself concedes he's no baseball expert, doesn't have any special knowledge about the players. So why is he trusting his estimates which inform his entire system?
That's why the approach is likely to fail in my opinion – it's got an arbitrary foundation. The math isn't going to save you if your projections aren't good.
My method is all about refining my sense of which players will exceed their commonly accepted past indicators and which will not. There are many variables, and I try to evaluate them in combination, depending on the case at hand. The key is to understand what the facts are and how these variables work together generally and constantly refine one's understanding based on one's results. Over time, one gets a keener sense of the meaning of the information. Peter was right when he said that everyone has access to all the info now, but not everyone really knows how to read and interpret it.
So you do analysis of the past, then synthesis these thoughts in a subconscious process (incorporating what you have observed, and your perceived relative value of the atributes and how they relate) into a real time bid?
But you think that if a person was to try to quantify how these attributes relate to each other, using their best understanding, they would underperform their subconscious mind?
Somewhere in your process you identify that Felix Hernandez is one of the best pitchers in the AL. You aren't doing that by physically looking at him. You are looking at his past statsand preformance I assume. And saying he is likely to continue to preform as he has in the past, with some considerations for the changes in the year?
You claim not to be sure whether you would bid the $1 more for a player without being in the heat of the moment. But each dollar more you bid implies you think the player is slightly better. If that is a steady everyday player, they you by deciding to bid (taking an action) you are effectively stating you think their year is worth slightly more to you. If that player is a volatile one, you are stating you think the combination of those possibilities is worth one more. But if someone were to take a systematic approach to making those choices, then observe the results and incorporate that feed back into the system? It seems that is what you are claiming to do on a sunconscious level. This is a paradox for me. How can you claim that you implicitly do it to improve your game, but then deny that explicitly doing it has any value?
All the modellers are doing is trying to ensure that they value all the attributes the same for all the players. I understand your objection to the forecasting part of the process, whether it be singular or a range. But I have a hard time with the argument that consistently valuing these attributes is not optimal.
Somewhere in your process you identify that Felix Hernandez is one of the best pitchers in the AL. You aren't doing that by physically looking at him. You are looking at his past statsand preformance I assume. And saying he is likely to continue to preform as he has in the past, with some considerations for the changes in the year?
I do look at him pitch. And read reliable scouts who are more knowledgeable than me. I even ask real baseball players what their impressions are while facing a player sometimes. So it's not all stats. Some of it is background – like Jeff Francoeur being a football player and getting to baseball late – maybe his development curve is two years behind normal and he could be a late bloomer. There are lots of factors, and it differs from player to player.
But there is analysis of past performance, no doubt. I look at a stat line and *analyze* where the luck/skill fault lines likely lie. I use analysis to gather facts, and once I have all my facts, I put them together (synthesis).
The problem with the systematic approach – (or the conscious movement from facts to stats) is that much is lost in translation. It's very hard to know how much "Todd Helton's back feels better this spring" is worth in stats. And there's so much information to convert into a stat line, much of it not easily quantifiable that so much is lost during that step. Even if you use a range of probabilities as Robert does, it's very hard to know what probabilities to assign to each range.
I find it far easier to have an impression players, and bid or pass during the auction based on market history and league parameters. I know that Helton's feeling healthy, and when someone says "6" (after seeing Derrek Lee go for $22 (last year)), I can say "7". I also have a roughly ordered list in front of me as a reminder.
I understand what you're saying, but to make it systematic, I must write down precise projections for every player, and in doing that I feel much is lost. I don't know what "return to health" or "increased ground ball rate with slightly above average K rate" works out to exactly stats-wise in a given park. I could guess. Or I could just be aware of all the pertinent facts and let them guide my bid or pass decision.
The market also helps me – I know who I think will be relatively cheap, and if they are, I'll end up getting them.
Chris have you ever done a silent auction? Do you think your methods would work if you were forced to bid a number in a vacuum, rather than just bidding $1 more than the someone else bid?
I think a silent auction would be harder for me – at least the way my system is currently calibrated. That said, I have been known to jump bid on players I'm high on, so that the other owners have less time to make a decision. But I'm better at recognizing value when I see it than knowing exactly where to set the line. At least given my current experience. I think I could adapt, but it might take some practice before I got it down. Or maybe I'd have to switch up how I do things.
Chris Liss, what confuses me is why my approach of recognizing all the factors that go into the projection and value and trying to refine them mathematically is invalid, but your approach of looking at the same list of factors and intuitively deciding which ones matter the most is valid?
I wrote the articles I did because you and Peter said that things like players who get their value from their upside and fluctuating values in an auction can't really be represented in a model. Those articles were about the technique of representing those things with a model instead of taking the primitive first approach you sketched out in the comments.
Chris Liss's views here are very interesting to me. I am an academic philosopher who plays both FBB and hold'em, and I think I recognize the style of assessment Chris is using here; it parallels a theory of morality called 'moral particularism'. Moral particularists claim that morality cannot be reduced to principles; there is no mechanical way to determine whether or not a particular fact is relevant to the moral situation.
The analogy that is commonly made is to reasons for belief; the fact that we see an object before us that appears blue is normally considered to be a reason to believe that there is a blue thing before us. But if the dominant light source is red in the room we're in, or we've just taken a drug that affects visual perception of color, we might not think that we see a blue thing is a reason to believe there is a blue thing there, and there's no principled way to determine what things are relevant to that belief.
I really think, having read all of the analysis on this blog, that something like this is the difference between Chris and the quantitatives. If you have a moment, read this page on moral particularism: http://plato.stanford.edu/entries/moral-particularism/
I think sections 3 and 7 are particularly enlightening. This might help Robert et al at least understand what Chris is saying, if I'm right about Chris's views. Chris, please let me know if this is something like what you mean.
Robert,
I said that using a set of projections to price players was problematic and misleading for a couple of reasons. I also said that too rigid reliance on a list of prices didn't take into account the dynamic nature of prices during the auction. I also said that I use projections and I create a price list in advance for each of my auctions. I thought I was arguing you guys away from assumptions and certainties that I think are less important than it seemed you do, though now I'm not so sure of that. Maybe, but maybe not.
I don't think I was saying that the dynamic nature of the values couldn't be modeled, just that buidling them the way you seemed to be doing it (based on the projections) was either too rigid in its math (because it was too trusting of spurious details) or not acknowledging of the level of subjective decisionmaking that you were actually incorporating in your model. I thought that argument bridged some of the differences between the anaysis vs. intuition debate ging on then, which struck me as not very important on the whole, but fascinating in the margins.
It struck no receptive chords, but I just wanted to try to clarify what my role was.
Peter, as someone who trades for a living my entire career is using models. If all I did was read a number off a screen and completely forget the assumptions that went into it and follow it like a monkey, I'd have lost my job years ago. When I make the model myself I have an even better knowledge of its strengths, weaknesses, and limitations. The stance that since your inputs are imperfect modeling is useless is flat out untrue.
I think it is obvious that better knowledge of the players helps in FBB. I still don't get why you or Chris doubt that a better model and ability to use it also helps.
Robert,
This isn't your fault, we've been distracted, but I'm still waiting to hear about an aspect of your model that contains something that wasn't built into the models of fantasy baseball valuation created 15-20 or so years ago by the first wave of fantasy baseball analysts. I'm sure you have some. I would like to hear about them.
I'll say this again. I believe in modeling. I think modeling is important. But it's also important to know what to model. I just don't believe that using a pricing model built on statline projections makes the most sense. A lot of the early part of this discussion grew out of Bill's comment about the poor state of fantasy baseball analytical work. Maybe he was right. Maybe you are right. But we haven't heard anything from either of you that actually backs that up with anything other than your rigid assertion that modeling is the way. Forget that Chris disagrees. That's his right, isn't it?
The conversation gets interesting when you actually present some complete ideas. I don't know about Chris, but I reacted to the beginning of the presentation poorly. But I want you to follow through on it. I want to hear what you have to say, because I got to a certain point with all this in re fbb and eventually retreated, as did just about everyone else I know who does it.
Why is that? Doesn't that interest you? Do you think we might have information you don't yet understand? Or do you think we live in the bush of ghosts? This is where the conversation should go, I think.
Best,
Peter
I would really like to get off this treadmill and get back to something that really addresses the issues of the game.
Hi Nathan, thanks for sharing! A poker playing/fbb playing philosopher seems just the person to sort out this mess.
Interesting stuff. I will have to give it a few more reads before I say anything much about it.
Let me ask you this, though- while you think Chris's comments parallel moral particularism, what do you think of applying this style of thought to fantasy baseball? I don't think anyone would argue that morality makes a crappy subject for models. Do you think this kind of thinking makes for a good approach to fantasy?
Well, part of the confusion so far is that there are (at least) two ways of reading Chris, depending on what exactly he thinks can't be captured by models. Sometimes Chris seems to mean (1) the considerations relevant to projecting MLB players can't be fully captured bya model, and sometimes Chris seems to mean (2) the considerations relevant to translating a particular statline into a particular amount of value, relative toa certain league-and-team situation, can't be captured in a model. (He may of course think both of these things are true.)
For both of these questions, the questino about proejctions and the question about translating a statline to fantasy value, I think teh notion of being captured in a model is very similar to the notion of being reducible to principles. In fact, I think that probably one could capture all the relevant considerations in a model if and only if the relevant considerations are fully explainable by principles. So I'm just going to talk about wehtehr or not we can capture all the relevant considerations in principles (cause thats' how I think); if you want to think instead in terms of capturing the relevant considerations ina model, be my guest.
Uh, so I haven't answered your questino yet, you'll notice. But I think getting clear on the exact details of the question is useful, partciluarly so when people get to the kind of dialectical roadblock that seems to have developed between Robert and Chris.
I think the answer to (2) is clearer than the answer to (1). Remember the case of the retrospective fantasy draft; we need a ton of informatino about the rough distribution of stats in the league (which we could havethrough konwing people's strategies in terms of which stats they're focusing on and which stats they're dumping), but it seems like with that information the question of how many roto points Derek Jeter is worth to us is a matter of number-crunching.
But (1) is a much tougher question, and I think I might actually come down on Chris's side to some extent. That is, it is plausible to me that the variety of factors that go into determining a particular player's performance in a given year are so many, and the interaction between them so complicated that attempting to capture them with principles seems doomed to failure. There's actually some empirical support for this, at least with regard to current projection systems; there was a study, though with a small sample size, done on the Hardball Times by David Gassko who picked the players he thought were likely to over or under perform their projections, and the results were quite good. See that here: http://www.hardballtimes.com/main/article/man-vs.-computer/ Now, there are many complications with this kind of study, but I'm inclined to think his result is plausible.
This is not to say that projection systems are useless, or anything like that. The systems are clearly our best instrument at figuring out how the entire league will do; that is, if Chris had to project every player in MLB, I claim that CHONE or ZiPS would kick his ass. But what I do want to say that there's something to be said for the 'genius' approach to drafting. For some players, in some cases, we have a strong intuition that the projection systems are failing to appreciate some relevant information.
The problem is that because statistical quantification is conceptually tied to analysis of groups, not to individuals, it's functionally impossible to quantify our intuitions with any degree of precision. It would be a case of misleading precision, similar to giving the percentage of people who response 'yes' to a poll down to five decimal places when the error bar is +/- 2%, to assign a particular stat line to the player about which we have the strong intuition.
There are, of course, more complications than this, even, but I'll leave it here for now; this is already super-long.
Wow, my formatting sucks and I have a bunch of obvious typos.
Sometimes Chris seems to mean (1) the considerations relevant to projecting MLB players can't be fully captured bya model, and sometimes Chris seems to mean (2) the considerations relevant to translating a particular statline into a particular amount of value, relative toa certain league-and-team situation, can't be captured in a model. (He may of course think both of these things are true.)
Definitely feel much more strongly about (1). To the extent (2) is true, it's more because players have different values depending one their distribution among the fantasy teams, but one could probably narrow down their value given a "neutral" or "average" distribution, whatever that means.
And there's a high probability CHONE would kick my ass on league-wide projections off the top of my head. But if I did make projections (which would be a poor translation of my actual baseball knowledge for the same reason Robert's or anyone else's is), and you put them in Robert's model, I'd bet on mine over Chone's.
Because I'd take chances in them, and so Robert's model would draft the players I liked for $1 more than CHONE, and steer clear of guys I didn't like.
If you graded mine vs. theirs, they'd win easily in kind of global accuracy contest. But that's a different game entirely. And if I were playing that game, I'd probably go the cowardly route, too since that's what would be rewarded. (Actually, I do this for football – and used to do poorly every year because I'd submit my actual DRAFT cheat sheet which wins leagues than a much more timid top 20 by position graded cheat sheet which you have to do to win that contest. Because an upside pick that doesn't pan out really hurts in that format.
But CHONE would probably beat my cowardly picks in a global contest, too – just I'd at least have a better chance and lose by a smaller margin.
To be clear, what I meant when I said that CHONE would 'kick your ass' at projecting the whole league is that CHONE would much more accurately project the league as a whole, which is what you said here.
Clearly, your advantage over CHONE (if you have one, which you probably do) in terms of fantasy baseball is the fact that you leverage the information advantages you have in regard to specific players. Which is again what you just said.
I think we're on the same page here, I just wanted to make sure my rhetoric wasn't misinterpreted.
Chris Liss, what confuses me is why my approach of recognizing all the factors that go into the projection and value and trying to refine them mathematically is invalid, but your approach of looking at the same list of factors and intuitively deciding which ones matter the most is valid?
Robert – I have explained this a few times, but here's one more try:
You do not know that you are recognizing all the factors that go into projection.
Like me, you have access to a certain amount of factual information about players. Some of it is easily quantifiable like his past stats, and some of it is not, like his manager's opinion of him, his injury history, his likelihood of being traded to the NL, his physical tools, his pedigree, that his cousin died in a car crash, etc. Some of it is extremely relevant, some not so much, but it is not necessarily obvious which is which in a given case.
The notion that you could take this vast amount of data (quantitative and qualitative) and reduce it to a single stat line (through averaging three outcomes that you choose for him) with any reliable degree of precision is mind-boggling to me, and that's assuming there was no variance AND it was possible from his past stats to know his EXACT baseline – at least over the last three years. (Which it's not). You cannot know that you are weighing the information correctly in terms of the stats it truly warrants. Is the 10 pounds of muscle he put on worth 5 extra homers and cost him three steals? Or will it not affect his speed? No one knows.
I cannot make an reliable conversion, and your O1, O2, O3 method does not do it, either. Unless you have some algorithm that creates projections out of facts (and you don't claim to), AND that algorithm has been uncannily accurate in the past, you are flying blind.
I don't see how one can dispute this.
But the good news for you is: so am I!
The difference is I know this.
So what I do is not even attempt what you're doing any more and simply observe the game and come up with theories, reasons, possibilities, hunches, ideas, impressions and put the players on a roughly ordered cross-off list. I do this because I know I need to keep track of the pool as the auction goes and to see generally where the players fall.
I know my method is imperfect, that I will likely pick some players who don't pan out, but that's okay. If I do I will note why (was it bad luck, or did my analysis of them miss something important? If it's the latter, then that information will be incorporated into next year's or next draft's preparation). When I like a player, I make sure I consider other similar players I liked in the past and was wrong about and cross-examine my impression through the prism of my prior mistake. And when a player I liked goes bad, I like to go back into my mind set before the season, review the reasons I liked him, really get back into the mindset of liking him, then fast forward to what happened and how bad he was and try to revel in that cognitive dissonance so that I can learn from it.
Doing it this way, I will get better and better at identifying the key factors that affect unanticipated player growth. And that is my edge. Moreover, I don't have to evaluate these guys in a vacuum, i.e., predict their exact stats, but only relative to each other, and to their market prices as the auction progresses. Which I think is easier.
I also have experience doing auctions, know what players typically go for in different formats, understand how changing league parameters affects the player pool and the proper pricing of players, etc. That's also important (and I think that's what you're getting at with your gaming background and pricing model). But i think you need to separate it from your flawed projections in order to benefit from it.
Nathan, insofar as I grasp the article on particularism, I think there are some parallels. I am certainly a particularist when it comes to player valuation, though I do think there are general principles that I observe (though not necessarily strictly).
I actually majored in philosophy, and I always liked Kant's "act only on that maxim that you would will to be a universal law." Except I could not always understand when to apply it universally and when to apply it universally to someone in my situation. And then the issue always came down to in what way does my situation merit unique consideration, and in what way does it not. Seemed like one would therefore have to decide whether to apply that principle on a case by case basis.
Not sure what that says about my drafting philosophy.
Chris,
Yeah, of course there are some general principles you observe, but the point of being a particularist is that you think the principles have serious exceptions, but you can't put the exceptions into the principles. X might be a property that, in most cases, means that a hitter is going to improve, but not in every case, and to figure out in which cases X means that the hitter will improve, you need to look at all the facts about the hitter in question.
I think you guys have been over that ground plenty. Robert is well aware he is 'flying blind', he just thinks its better to try to quantify his best guess (and you don't have to stop at 3 outcomes) instead of not quantifying them at all and trusting his brain to synthesize an answer when the moment comes.
Your typical counter argument here usually boils down to something along the lines of 'once you quantify it and write down a number, you are locked in' but there's nothing stopping Robert from doing stuff on the fly in the auction too. He's just thought more consciously and precisely in advance. As he's noted in several instances, he's well aware of the limitations of the inputs.
I think its time to move on to new ground guys.
Chris, it sounds like your Grand Unifying Theory is that since I insist on attaching numbers to my impressions of players I will always be less accurate than you. The problem with your G.U.T. is that since you don't put numbers on things there is no way to really compare.
Peter, thanks for the post. I think that is a good push back in the right direction.
Part of the problem here is I built my model from scratch. I did not go back and study what you guys did. I don't know where we diverged. That is partly why i was beginning where I did and working my way up.
What I suspect is that the devil is in the details. There is a lot to put into a model and how you apply it. Something like bad baselines can make it completely worthless. I'd be curious to know if the early push with models included things like I discussed in the valuation of upside players or did you guys always just use median projected stats and convert them? If the answer is yes, then I think that alone goes a long way to explaining why it didn't pan out.
This is clearly steering the convesation in a good direction and I'd be happy to have another thread to discuss the models you've used in the past and ours.
Robert, you might not be less accurate than me if for some reason you're great at translating facts into numbers. I don't see why you would be, but who knows? I'm just saying once we acknowledge that there is no sound basis for the numbers one generates, why bother making that move at all? If your answer is – because then I can put them into my model, then the cart is before the horse. The model should serve to translate well-founded numbers – it's just a tool. One should not begin with shaky assumptions just to create proper inputs for the model.
My GUT outputs could be wildly inaccurate, too. Probably were at one time. But I've refined them in the way described above, and they're better now. i suppose it's possible to refine projections, but as I've detailed before, there are serious limits to what one can do with them given the outlier problem and also just the loss of important and difficult to quantity information issues.
The goal is not to have a great set of projections, but to make the proper buys at auction. So even if you created award-winning projections, they might not be optimal for fantasy drafting. Anyway – let's move on, I agree. I buy that quant analysis is useful in understanding the parameters of the league and how it affects player value. I just would like to see it divorced from specific projections so people can benefit from it.
By the way, "Grand Unifying Theory" is much better than BrainGut which Eric tried to trademark.
I'm aware :<