Lessons of Greatness: Making the Informed Choice

Annie Duke: Decision Making for Experts—How to Bet on the Right Outcome

Mar 08, 2021

📣 Announcements

Thank you for an amazing launch of Season 2! I hope you enjoyed the first episode with Tim Ferriss and our Clubhouse shows—in the past two weeks, we’ve hosted “Starting Greatness with Steve Blank” and “Starting Greatness in Austin” where top Austin leaders (and surprise guest, Michael Dell!) joined us. Clubhouse is a new platform we’re trying out and I’m excited to continue hosting more shows with the Floodgate team this coming month.

In fact, we have two new Clubhouse shows this week! Mark your calendars for:

🌊 Top Lessons from Floodgate Alumni happening Monday, 3/8 (today!) at 6pm PT
🎙 Starting Greatness: Future of Work happening this Thursday, 3/11 at 5pm PT

Along with Clubhouse, we’re also launching the Starting Greatness YouTube channel where you can find teasers and video snippets from the podcast episodes. Subscribe to our channel here.

✨ So you never miss a beat, subscribe to this newsletter and follow @m2jr (me) and @floodgatefund on Twitter.

Now, it’s go time with Annie Duke.

👋 Introducing Annie Duke

Annie Duke is known by many as the Duchess of Poker. Annie's two books, Thinking in Bets and How to Decide are instrumental for me when I face a crucial decision under massive uncertainty.

Annie is a colorful person with a lot of good stories to tell. She started as an academic, graduating from Columbia and then pursuing a PhD in Psychology at the University of Pennsylvania. But one month before defending her PhD dissertation, she decided she no longer wished to pursue academia and left school. Naturally, she decided instead to become a Professional Poker Player, amassing 4.2 million in winnings over time. She then embarked on a career as an author writing about her decision process ideas that contributed to her success in poker. Many of her discussed frameworks for poker are applicable to any life situation that involves massive uncertainty, particularly the kinds that startup founders are faced with regularly.

In our interview, she's going to talk about how you can be the best possible leader in the chaotic world of startups.

✨ Follow Annie @AnnieDuke on Twitter.

✨ Find Annie’s book Thinking in Bets here

✨ Find Annie’s book How to Decide here

🎙 The Full Episode

Listen on Spotify
Listen on Apple Podcasts
Read the full transcript of the episode here
Teaser below:

🌟 Highlights from the Episode

Golden Line:

“Let's think about what a great decision is. And a great decision is a forecast of the future. So the better your forecast of the future, that's how we can tell that a decision is great.” —Annie Duke

Journey from Poker Player to Decision Theory Expert:

Getting into the decision strategy and decision-making cognitive science was actually a circle back for me. I started off my adult life in a PhD program in Cognitive Science at the University of Pennsylvania at a National Science Foundation Fellowship and was planning to go off and become a professor, to tell you the truth. Right at the end when I was going to go off and become a professor as planned, something I didn't have on my bingo card then was that I was going to get sick and end up in the hospital for a couple of weeks. And it happened right around the time that I was supposed to go out for all my job talks. So I needed to take some time off to recover and during that time, I honestly needed money. That's when I started playing poker. During that year when I was playing poker, I just ended up really falling in love with the game.

I really, really loved the problem that you're trying to solve in poker, which is how are you making decisions in these environments where there's very limited information, there's quite a strong influence of luck. During the last 10 years of my poker career, so 2002 to 2012, I had already circled back into the more academic work. 2012, I retired from poker really to focus on that part of my life. It was actually what I was mostly spending my time on anyway at that point. I was more playing poker on television by then because I was really interested in this topic. Ended up writing, Thinking in Bets and then now I've got the new book, How to Decide coming out. I fully circled back into that more research and academics and what has resulted in a lot of consulting, as well.

Principles for Evaluating Decisions:

Let's think about what a great decision is. A great decision is a forecast of the future.

The better your forecast of the future, that's how we can tell that a decision is great. We can think about if you were omniscient or you had a crystal ball, obviously all things being equal, you would be an amazing decision maker because you would be able to say, "I'm omniscient. I know what all the different ways that the world is going to go and then therefore I can make decisions about that." If you're omniscient, I'm not saying that you know that the coin is going to flip heads, but what I'm saying is that you know that the two possibilities are the coin could be heads or tails and that it will land that way some percentage of the time.

What that means is that a great decision process is one in which you have identified as well as you can, given the limitations of not having a time machine and not being omniscient. That you have identified as well as you can with a reasonable set of possibilities are for any outcome that you're considering. That you have a really good sense. In other words, you can see the luck. I'm not saying you can control it but you can see the luck in the sense that you have a good idea of how probable any of those particular outcomes are.

Your goal is, I want to be able to see two things, one is what's the expected value and what's the variance or the volatility.

How to Decide:

In How to Decide, my new book, I talk about memory creep. And this is a really big problem. First of all, the problem in itself is an issue because once you know the outcome, what happens is that certain aspects of the decision get highlighted and certain ones of them get low lighted that help you to sort of square the decision. In other words, to make the outcome make sense in relationship with the decision. If it's a good outcome, you'll tend to remember the bullish parts of the decision process. And if it's a bad outcome, you'll tend to remember the bearish parts of the decision process. But then we lay on top of it this problem of we misremember, we aren't particularly good at, in retrospect, really accurately reconstructing what our state of knowledge was at the time that we made the decision.

There's this great example that I feel collides these two problems of resulting and then hindsight bias or this memory creep problem, which is really just the 2016 election. So very briefly, obviously Clinton lost the Rust Belt. Michigan, Wisconsin and Pennsylvania. Across the three States, it was about a total of around 80,000 votes, across three States. It was a razor thin margin, but the first thing is we can see the resulting problem because everybody's agreed that there was a horrible decision that she didn't invest more time in those three States. Why was she in Florida? Why was she in New Hampshire? Why was she in Arizona? Why was she in North Carolina? So, here we can see the resulting problem in the sense of, we know that it was a bad outcome for her.

And we're assuming that her decision-making around where to campaign was bad, but there's nothing more crowdsource than decision-making during a presidential campaign. We can see that right now, there're a billion things written about Biden's strategy or Trump's strategy and whatever and comparisons. So I actually just searched. And the first article that appears critiquing like saying that it was a really bad strategy on her part appears on November 9th of 2016, which I don't know if people remember, but the election was on November 8th. So obviously if it was a terrible decision, you would think that all of the brain trust of every political analyst in America would have figured that one out in advance. And why would we assume that she somehow would have figured out something that nobody else did? So we know that, for just that reason alone, we've got a problem, but then we have this other problem.

I actually pitched this particular story to an editor at one of the big three newspapers. I won't say which one. And they said, "well, I'm not going to publish this because all of my friends knew about it. We were all talking to each other and I read a billion things about it. And so your thesis is wrong." And I literally said, "I did the Google search." So if you all were talking about it, it was this really big secret in journalism that apparently nobody thought would be interesting to write about, even though it was an incredibly contrary take at the time to say that she was making a mistake. And the funny thing was, that there were a couple of articles that I could find that were about those three States, but they were criticizing Trump.

I just want to separate this from journaling. People talk about having a journal, which feels like something on top of the process. If you have a good decision process, the record will be created naturally. Part of the process, it will just appear because you can't have a good decision process without the record without naturally creating a record. So, journaling, I feel like people are like, "Oh, I got it. I have to do this extra thing. And I need to write all this stuff down." After we've already made the decision or even during, or whatever, that's not an extra step.

Knowledge Tracking:

First of all, like a lot of the ways that you think about how do I improve the look back, in order to understand when I get an outcome, what it means and the way you think about how to solve that retrospectively actually opens up how you do it prospectively, like how you create a better decision when you're actually making the decision, which is actually the order that the book goes in, it starts with this retrospective problem and then says, "Aha! But now we know how to make a good decision. So let's actually get that into our decision process." So the problem when we're doing the look back is two-fold, as we've just said, the first is that we forget that there's a whole bunch of other ways the decision could have turned out. That the branch that we ended up observing, might've been like incredibly low probability.

And we forget to realize, I have to put this in context of the multiverse. Because there's all sorts of universes that can unfold. And I happen to be on one timeline. So let me try to think about the other timeline. So that's the first piece to getting this kind of retrospective look, which helps us to populate those four cells that we talked about. That different relationship. And then in terms of the knowledge problem, we're trying to figure out, what did I actually know at the time of the decision? Because my decision can only be as good as the knowledge that I had. So what we want to do is think about what did I know before the decision. And in other words, what informed the decision that I made?

What was the decision? What was the outcome? And then what revealed itself after the fact, and then as you look at what would reveal itself after the fact, now you can say to yourself, could I have known this beforehand? Now for most things, actually, the answer is going to be no. And that's where we really get into the problem by saying Clinton should have known that there was a polling error in exactly three States, but not nationally and not another state. And she should have known this thing that nobody else on earth knew. And obviously that's silly and it can get even worse because we can remember we did know it, which is what happened with the editor that I talked to. He said, "No, I knew it." I'm like, "You're at a magazine. I mean, a newspaper. Why didn't you publish this opinion then?"

Once we have what revealed itself after the fact. We can think about that, which category does it go in? Could I have known about it beforehand? And that you're going to get a yes or no answer. If you could have known about it beforehand, try to figure out what happened in your decision process, that you didn't spot it. If it was something that was totally knowable beforehand, and then don't beat yourself up too much about it. Just make sure that you make sure that it's something you're looking for in the future. Now, then there's going to be another category, which is, I couldn't have known about it beforehand. It wasn't something reasonable for me to know about beforehand. And that is going to fall into two categories. One is going to be, it's never going to be knowable. Because it could be, for example, the outcome itself.

How Founders Can Take Advantage of the Six Steps of Decision-Making:

I know we talked about a decision as really just a prediction of the future. Ideally we'd like to have a crystal ball. So we're kind of trying to get as close to that as possible. And when we think about that, when we talk about these two different influences, which is luck, you have a particular option. And that option has a set of things that could occur that are associated with it after you've chosen the option, that's where the influence of luck occurs. Because you can think about what those possible futures are, but you have no control whatsoever about which one you actually observed that particular time. That's the influence of luck. But we have this other problem which is the hidden information or incomplete information.

And we can think about, we can be really amazing in terms of what our process is. We're going to forecast things and we know, we're going to think about the different options and we're going to forecast them. And we're going to think about what the expected value is. But if what's informing our decision isn't high quality, that house is going to fall down. Everything about this process that you're doing is trying to motivate you to deal with this informational problem. Because that's the foundation on which this whole decision house is built. And we have a problem with that foundation.

The first is that a lot of the stuff we believe is inaccurate. So let's call that cracks in the foundation. And the second is that we don't know very much, it's like a super flimsy foundation. So we can think about the universe of stuff we know versus a universe of stuff we don't know. The universe of stuff we know is like a speck of dust on the head of a pen it's like Whoville. And then the universe is stuff that we don't know is like the size of the actual universe.

So what we're trying to do is how do we get stuff out of the universe of stuff we don't know into the universe of stuff we know to repair our foundation? Because we can solve both problems by exploring that universe of stuff that we don't know. So we can add that tore, we're going to find ways to repair the inaccuracies, and that's where we're going to find new information. Anything that I talk about is really focusing on that particular problem. So even when we talk about what are the steps for a good decision? The six step process. It's really meant to start getting you to ask questions, to seek knowledge, to explore the universe of stuff you don't know in a much more objective way, because we don't take random walks through that universe.

We take very specific walks where we're particularly shining lights on stuff that confirms our beliefs and on people who believe the same things that we do. And information sources, i don't know a lot of Bernie supporters who are spending their days watching Fox news. And I don't know a lot of Trump supporters who are spending their days watching MSNBC. There you go. That's kind of what we're trying to solve for. Let's now step back and say, okay, so what does a really good decision process look like? Well, it's got to be an accurate prediction of the future. Is it accurate as we can get. We're going to usually be pretty far off, but small differences aren't really good. If we can make small improvements, we're really good. It starts with identifying the reasonable set of possibilities. So we wanted to do, and by reasonable, I mean, I don't want you to sit there and go down a rabbit hole of what if an asteroid hits Russia.

Why are you spending your time on that? You're trying to decide whether to release a particular piece of code. Let's not worry about that. What are the reasonable set of outcomes that actually matter for the decision that I'm making. And then you want to go farther than that and you want to look at them and say, what is your preference for those outcomes? And that really just has to do with the payoff, what's going to get you the biggest bang for your buck of those possible outcomes that I'm considering if I choose this particular option. How much is it going to advance me toward my goal? How much of it is going to advance me away? And we can think about that in a lot of different ways. It could be, for example, if I'm a founder, I'm trying to maximize how much information I'm going to get from any option I choose.

I can think about what of the set of possible outcomes is going to informationally give me the heaviest lift? It could be like if I'm trying to hire someone and turnover is just a mess, I may be very particularly worried about how long the employee is going to stay with the company. We can think sort of broadly or narrowly about what those outcomes are and what payoffs we really care about. But we want to think about, what's the magnitude of the payoff, good or bad. Then we want to think about what's the probability of those things occurring. That's obviously incredibly important because otherwise we don't really know how to compare the good to the bad. It could be that there's like some disastrous outcome in there, but it's like a really, really low probability.

Likewise, there may be something that's very low probability on the good end, but it's got a big enough payoff that it outweighs the bad. So unless we combine these two things, the payoffs and the probability, it's very hard for us to think about what the quality of that option is. And then basically the rest of the steps are just to rinse and repeat. What are the options I have? Let me go through the same process. And then I can compare these things together to figure out which option is going to be more likely to get me to advance toward my goal in a way that has obviously the appropriate risk associated with it. Because you always have to be thinking about those extreme downside outcomes.

Daily Decision-Making:

I think that we generally think about decisions as doing something new, as opposed to staying in the course. I mean, I'm sitting here and talking to you, that's the status quo right now. I could get up and leave. I mean, but I have to think about what the consequences of that would be. I think that would be very weird and it wouldn't be really something that I didn't want to do, but I could do it.

Mike: I would not hold it against you.

Well, it would be strange though. It would be, Annie left. But we can think about that in terms of even jobs. I'm in a job and I don't like it, but so why don't people then go find another job? Well, because that feels like a decision and what if I make that decision and I make that change and it doesn't work out. We're going to... Because we view that as a decision, we think we somehow decided to have things not work out, whereas, because we don't view it as a decision to stay in the job that we're already in.

If it doesn't work out, we are hard on ourselves. And this actually comes generally into this sort of world of like, when we think about the interaction between the resulting problem and this kind of status quo versus innovation problem, things get really interesting. So let's go back to the Pete Carroll example. This actually will be fun. Pete Carroll did something that was really strange, at least to the public, on the first play. Very weird. And we know what happens in that case when it fails, he's an idiot, but let's do the thought experiment, it succeeds. What do the headlines look like?

We get these big extremes, heads or the tails. Idiot, genius.

Lucky Advice:

To be clear, I'm not downplaying the role of luck. Luck is the greater influence here. The issue though you don't have any control over luck. So pay attention to the thing that you do have control over. And everybody is subject to. I mean, that's the thing. It's not like "I'm differently subject to luck than other people." Luck is a problem for all of us to deal with and problem, or maybe, a mitzvah, whatever, like it exists. But the whole point is that you can't, you have no control over that anyway. You have to focus on the thing that you do actually have some control over. I didn't have any control over when I was born or who I was born to or how tall I was or what my talents were, but I do have control over the way I decide about all those things.

I didn't decide that I wanted to have a career as a singer, given what luck dealt me. I think that that separation is really, really important. So I just want to make clear, I'm not downplaying the role of luck. Obviously luck has a huge influence in the way that people's lives turn out. It's just, I can't do anything about it.

💌 Enjoyed this Episode?

If you enjoyed this episode, we’d be grateful if you could leave us a review on Apple podcasts.

Questions / Comments / Have a favorite quote or moment from the episode? Send us an email at greatness@floodgate.com

Starting Greatness Newsletter

Discussion about this post