- JJ Rorie
The Importance of Decision Science in Product Management
Updated: Mar 5, 2022
Episode 003: In this episode, Adam Thomas shares insights on how product managers can make better decisions.
"Decision science seeks to understand and improve the judgment and decision making of individuals, groups and organizations. If that sounds familiar to what you're doing as a product manager, it should."
Â
Episode Transcript
JJ: Hello and welcome to Product Voices. So how many decisions does a product manager and their team make every day? A dozen? 100? A gazillion? Yeah, it's probably about that many. So here's the thing. We will never have all the data, all the answers, but we must make decisions and keep progress moving for our products. I personally consider judgment to be one of the most foundational skills of great product managers. And of course, a subset of judgment is being able to make good decisions. There are a few things more central to the product manager role than making choices and decisions for our products. My conversation today is with someone who mixes the hard topics of product management with context from culture and behavioral psychology to guide product managers towards making better decisions.
Adam Thomas is a technologist in Harlem, New York, focused on strategy, team, organization, and product management. He has over 15 years of experience and thinks product management is about improving the decision fitness of the people around you. I love that, Adam improving the decision fitness of the people around you. Adam, thank you so much for joining me. I'm excited about our conversation.
Adam: Thank you for having me, JJ. I am quite excited about our conversation today as well.
JJ: You and I've spoken, obviously, and so I love your take on this. So I'm so excited to get your thoughts out to the audience as well. So lay the groundwork for us. Give us an overview of decision science and why it's so helpful in product management.
Adam: All right, we'll do yes. Let's start with the definition. What is decision science? Well, decision science seeks to understand and improve the judgment and decision making of individuals, groups and organizations. If that sounds familiar to what you're doing as a product manager, it should, because as folks that have no discrete output, we don't turn in code, we don't turn in screens, we don't turn in slides, although we can do all those things. That's not what we're responsible for. The thing that we're responsible for is enhancing the decision making of the teams around us. And since we don't have that discreet output, our best bet is to stand in front and own this process.
Because at the end of the day, when a launch doesn't go the way it should, the eyes will fall onto you. Not your engineer, not your designer, not your product marketing manager, not the salesperson. It will fall on you. So it's best for us as product managers to stand up and own this process. And owning that process looks like you spending time prioritizing these issues based on business and technological need. Right. There are things that people in the tech side of the business won't see, and the things that people on the business side won't see. Your job is to sit in those rooms, help build those bridges, and then lead them into solving the most important problems that your company team organization faces. When we're doing this well, this becomes a cycle, right? Sometimes people know this is build, measure, learn. But whatever the cycle is, the longer you work in the cycle, the better and faster you get it. Understanding these problems and using the right tools to solve them.
JJ: One of the things that you said there, Adam, is the first part of your response. By the way, thank you for that kind of definition and overview. It's really important, but I absolutely love the way you couch that product manager managers have no real discrete output. It's all about decisions and it's all about helping make the best bets and make the best decisions within the team. And I don't think I've ever really thought of it that way. I think it's such an impactful way to think about the product manager role is that like you said, we don't deliver the code per se. We deliver the experience and the decisions behind it. That's a fascinating way to look at it. And I think that's a really important way to frame this conversation.
Adam: Yes. And the anti pattern is one I see so much. And younger product managers, including the teams I'm managing now, and of course, teams have managed the past is. When you're not as confident in that point of view, you start to over index on what I call generating that's outputs, right? You start to find yourself budding your way into engineering meetings, trying to figure out what the code base looks like, and you start to go into design meetings and start to say, hey, you should move that green box just to say you have a piece of that decision, but you're not helping the decision making, you're just putting yourself on the decision.
What ends up happening is they'll create documents, they'll overload sprints, they'll point to the delivery board, whatever that delivery board is, and say, I did this, I did this, I did this. But in six months to a year, when you look back, there's nothing there, right? That has moved the needle forward.
JJ: So what got you interested in the first place in utilizing decision science? Because you obviously have a tremendous understanding of the psychology of things and what the real crux of the PM role is. Tell us a little bit about the kind of background or the triggers that made you get really interested in utilizing decision science and product management.
Adam: It's a really weird frame for most folks, but I started my career in mainframes and one of the things that I think mainframe teams do better than any of what they would call distributed teams or what most people see as technology teams on this side of technology is risk management. Mainframes are often in charge of really high risk information, think banks, think governments, so on and so forth. And the first thing they drill into your head is, albeit it slowly is how to make decisions in order to protect that information. And I think compared to other people types of technology like the ones that we generally work in for those mostly listening to this podcast. They can go a little bit overboard.
That said, when moving over to this side of technology, that foundation helped me a lot with noticing I had a lot of time and understanding dealing with risk management. And so sitting in Sprints, once I made the transfer over to this side of technology, I started noticing the difference between decisions that move the needle and decisions that didn't. On top of that, my research and risk management started with conversations with those folks on the sell side that had to figure that out because I worked on the street at a company called the TTCC, which is actually the home of derivatives. So I had nothing to do with that. But it was. And in talking to the person that actually created them or was on the team that created them in the conversations, he would guide me on things to read and understand as looking at risk, which led me to people like Nassim Taleb, books like Black Swan, which again help me notice the last thing which led to this merging was an interest in marketing.
Started my career as a founder. And the interesting thing about being a founder is you have to put on a bunch of hats for everything. One of those hats was marketing for me, and so I was building my first company called the Gamer Studio. I had to learn how to get the product out there because no one else was going to do it. At first it was me, then it was a few writers. But in general, the things that didn't fall onto other people fell on me. Right. And so in my rush to try to understand it, I found Seth Godin and his work. And through my career, I have to shout out Seth at his work, because not only did I find his blog and then started reading his books, I also went through his MBA program.
Eventually I got to meet Seth as a coach in his MBA program. And one thing or and I still am a coach. And one thing that Seth is big on is noticing. Right. And so when you combine the risk side, which is kind of on this, you know, kind of the conservative side of making sure things don't blow up with the noticing of marketing, which is really thinking about the optimist side, and then you combine that with some other reading in terms of product and decisioning. With people like Annie Duke. Right. You start to I think it starts to breed this thing in terms of noticing for me. Right. How did that turn into how is that practical?
So I have a story that I usually tell folks that combines all that into one little piece of thing. Right. You can take with you. So when I was a younger in my PM career, I was working on I had a product, a printing product. And once we started making a move to a different location, there was some local hardware stuff that had to be done. I started getting 03:00 a.m. Phone calls. And at first because then 1s where I was working, I ended up being level three support as the PM. That's how it was structured. So the phone calls eventually got to me. Regardless of what happened. And so at first, I did what younger PM's would probably do, right? I started looking at the code. I started making documents. I started calling the vendor, and I'm sure their head started to explode. That didn't do anything. Spent weeks getting 03:00 A.m. Phone calls, which, of course, made my work at 09:00 A.m.. It's not good to wake up three A. My boss is also on the call, especially if it's bad. And so they're not happy. And so nothing changed until I sat down with the operations people and I started looking from both of those angles. Hey, what risks are you seeing? Walk me through what happens. What is the path of this leads to this phone call that you're going to give me at 03:00 A.m. Today. And then that started helping me bridge that gap of noticing what decisions were being made. And all it took was one day of sitting down with the information, of talking to call it stakeholder management. Right. Talking to the people that led to that decision, of me getting called that path and figuring out and working backwards. All it took was one document that took me 20 minutes to write two screenshots that led to me never being called at 03:00 A.m. Again. Right. And so the important thing there was understanding the decision science that was happening. My job wasn't to make screens. It wasn't to build decks. It wasn't to get involved with the engineers and what they were doing or even operations too much. It was just noticing what decisions were being made and then adjusting the decision making process.
JJ: That's fascinating. I love that. It's such a good example, a good story of how when looking at the right things and realizing where you can add value and where you should be adding value, that sometimes it just clicks and otherwise we just spin our wheels and you'd probably still be getting those 03:00 A.m. Calls if you hadn't done that. So I love that example. Let's turn to something I'm really excited to hear more about something that you've been working on and have introduced to the world called your survival metrics framework. So tell us about that survival metrics. What is that and what is that framework? And how has that come about? And tell us a little bit about that framework.
Adam: Sure, JJ survival metrics. It feels like my life's work over the last year and some change. But much like the story I told, it's really a bunch of experiences coming together and just crystallizing into this definition. Survival metrics are way to help you stay agile. Specifically, the fourth piece of the Agile Manifesto, adjusting or reacting to change instead of following a plan, which is where teams tend to get stuck. Right, we look at our roadmaps. We look at the plans that executives tell us to do. We look at what we feel are things that have to get done. And we say we don't have time to react to anything. We got to ship A, B and C over the next year, sometimes in some businesses, even over two years, 2s and then they'll still call themselves agile. You. Right. It's not true. Right. What's missing is really a story. What's missing is there's a fear around Pivoting because there's not a story. There's not a story of change that they can and tell that they feel safe telling. Right. And so it's easier to CYA instead of placing that story down. And so what survival metrics ultimately is, is a way to tell that story. It's giving you the tools to tell that story and helping you start that investigation. So when that change happens, you and the team are agreed that when we see these things, 1s we're going to shift.
So the official definition is survival metrics help a product team determine if an initiative is worth investing in, pivoting, or stopping completely. It's a forcing function to prevent product teams from suffering due to sunk cost. Fallacy. Survival metrics put re resource allocation and company initiatives both implied. Think the politics of it and direct. Think of data in front of the team before a project begins and again at regular intervals. And giving that team permission to act quickly and a story to follow.
JJ: I love this and I'm going to let you keep going. But this resonates so well with me is that I have struggled I think everyone in product has struggled over their careers with inertia, with getting people to change. And even in agile environments, like you said, we're pivoting and changing and evolving is supposed to be a core tenant. It's so hard once you get going down a path, any path that it's so hard to pivot and to change. So I love this idea of almost giving us permission to change and the story behind the change. So I love that. So continue on. Tell us a little bit more about survival metrics.
Adam: Yes. And I want to double click there. Right. A lot of the fear around change is fear that we all experience not just the product person, but the engineering person or the design person. That's usual. That makes sense. But also the executive. Right. They're looking for signals because they're so far removed from what's happening on the ground and ultimately thinking about what's happening. At that level. Right. The decisions that are made on the ground ultimately affect them just as much. They're scared of change, but only because the change for the most part, it's mostly because the change change doesn't feel like they've been considered when the change happens. Right. They feel like they're getting a change just thrown on them. The thing that Survival Metrics helps do unconsciously is trained the product team to start asking questions that open up stakeholder management so they can understand what drives how the stakeholders act. Right. So when you're crafting that change story, when you need to change, it's way easier. You can speak to why the change happens and why that it affects that executive just as much as it affects you.
JJ: Yeah. And then I suppose again, it's just human nature. If somebody is presenting us something and they can clearly articulate why it matters to me and they don't center it around them or anyone else for that matter, but centered around me, then that's going to make it easier for them to embrace. That makes a lot of sense.
Adam: Bingo. Survival metrics creates that clear picture of what can go wrong while that project is in motion. We do that by spelling out potential limitations early. As I mentioned before, going to ask questions like what's scaring you right now, right. Or what are some signals that would tell you that would help you go ignore everything else and let's move forward, forward with this. Or maybe a question around. If you saw this thing, you would want us to stop immediately. Right. Going around and asking these types of questions which correspond to stop, pivot and invest. Right. Give us the opportunity to create the inputs for that story by spelling out those potential limitations. Then you have the opportunity to translate that into metrics that are usable right now. You have something to track against.
Once those stories are out and once you communicate that, you don't have to spend time figuring this out. When you hit those lines, as the company gets comfortable with the methodology, they're able to pivot a lot easier. I think this is super important, especially for startups. The thing that kills businesses more than anything else is bad ideas. Right. Things that people probably know aren't going to work, but we just go because of inertia. And so having that clear picture laid out, things that will make us stop, things that will make us pivot, things that will make us invest gives us an idea that story that we can use to go ahead and make changes when we need to. Yeah. Again, even in the organizations that seemingly have a culture that's not terribly risk averse, that they tout themselves as innovative and agile and flexible, they still fall prey to this, if you will. So I think this is just really just a great approach at things. nd I'm sure most people out there, including myself, are always working on improving their decision making. How would you advise someone to get started? Like, what steps would you tell them to take to work on their decision making skills. I think the first thing is more than anything, you need to find a place for reflection. And I think at its best, that's what the agile methodology is trying to get us to do is take a second reflect, understand what are we doing? How is it affecting our long term vision? The negative feedback loops are what you want, right? And before I move forward, I want to make sure it's very clear the difference between a negative feedback loop and a positive feedback loop. For team and.org design, negative feedback loops are what you want because negative feedback loops means that you're taking time to reflect. It means there's a break in the process and scrum. This is a retro, and this is why retros are very important. Right. To teams. It stops what you're doing so you don't continue on the path. Because if you continue on the path, it's a positive feedback loop. And while those are good for things like that's really the basis of machine learning for teams that in decision making, it's an anti pattern. So the first thing, the first thing that someone should do is really look at ways to create time for reflection. Retros are extremely important in this regard. I think the next thing is you want to create a decision Journal, right? You want to write your major decisions down through a decision Journal, because what a decision Journal is, is it's a tool? I have one in front of me from a blog called Farnum Street. And inside of this little booklet, what it tells you to write is what was your decision that you made? What's your mental or physical state? What's the situation? What's your problem? Statement. Net. And what are the variables. What are the variables, the complications that exist in front of you after that, write the range of outcomes and what you expect to happen. And by writing all of that down before you make the decision, when you are reflecting, you have the time to pick this decision Journal up and go, how real was this? How far away was this from what actually happened? And is it chance? Was it chance? Was it life? Was it something I didn't see? And adjust how you think about your decisions. I think it's an extremely important way to make you a better decision making because you can't lie to your past self if it's written down. You've said it, it's here. All you have to do is go back and reflect. And so, lastly, thanks about the future. Right. Build on that reflection time. Build on the ability to write your decisions down and reflect by. Scenario planning, giving yourself opportunities to scenario plan, think about the future. My favorite way of doing this is Premortems, which is an opportunity when you sit down with your team and go, what's the worst that can happen? Alright, let's write that down. Ok. What's realistic, right? You voted out and then you're in the meeting saying, how do we get rid of these unforced errors? How do we get through these unforced errors? How do we avoid these unforced errors that may happen? And then this can create a great loop for your decision making. Like the reflection time, the writing things down, and then also scenario planning and pre-morteming. In order to make yourself better at making those decisions
JJ: That is such tremendous advice and frankly reflection and retrospectives and even Premortems I've used before, probably not as consistently as I have. I've never done a decision Journal. I'm going to start that. That is fascinating and such really, really good advice. So I think that's a tremendous way to advise someone to get started. And one of the things that I think really kind of underlies all of what you just laid out is being intentional. You have gotten where you are, Adam, because you have been very intentional about improving certain areas of your skill set. And then of course, you coach others to do that as well. And I think that a good learning just in itself is for product managers to be intentional. And I think those reflections and journaling and Premortems are all very good ways of being intentional about improving ourselves.
Adam: What a great word. Right. Intentional. And it's so hard as a product person to be intentional because of all the forces around you trying to pull you in different directions, some see and some unseen. And the thing that the worst thing about it is for the things that see, there's two or three things that are unseen by that person looking at your direction. Right. The engineering person can see the architect pulling you in the direction of the designer pulling you in the direction they don't see sales pulling you in the direction. Right. And so it's super important for every PM to get more intentional, to slow down, to speed up, because that's the biggest trap of the work that we do is moving way too fast and looking back at the end of the year and having nothing to show for it. Yeah, absolutely. Okay, so we're going to link to your resources on the podcast website, Productvoices.com. You can also find a lot of Adam's resources and writings at his website, TheAdamThomas.com. We'll link to that as well on ProductVoices.com.
Before I let you go. Are there any other go to resources that you would suggest folks reading up on or listening to or getting involved in to help in this area? A couple that we went ahead and mentioned. Right. I think reading Nassim Taleb Super important reading. Annie Duke, her work is super important here. Seth's work is great on this, even though it's marketing. But I think it's the through mind there for the statistic people. How to measure anything. How to measure anything is extremely important in terms of thinking, how to turn things into metrics. Right. That means you're going to have to bone up on those statistics, ladies and gentlemen. 1ut it's really, really a good read. And I think lastly, Farnam Street, the blog is tremendous in terms of figuring out these things. And of course, my writing and not a selfless plug or selfish plug. But I have a newsletter that comes out every two weeks that talks about these things in depth. And so you can get more links. You can read a story, you can hear examples from my writing and get better at this stuff.
JJ: Awesome. And that newsletter is at TheAdamThomas.substack.com. Is that correct?
Adam: Indeed. I'm like everyone else these days with substance.
JJ: I love it, though. I love it. Okay. So, TheAdamThomas.substack.com again, we're going to link the things that Adam has mentioned. We're going to link that we're going to link his website on productvoices.com. You can go directly to his website at theadamthomas.com.
Adam, this has been a tremendous conversation. Thank you so very much for sharing your wisdom and your insights. I've loved the conversation. I've certainly jotted down a few things that I'm going to do differently and make better decisions. So, Adam, thank you again for joining me.
Adam: JJ, it's been a pleasure
JJ: Thank you all for listening to Product Voices. Hope to see you on the next episode. Thank you for listening to Product Voices, hosted by JJ Rory. To find more information on our guests resources discussed during the episode, or to submit a question for our Q and A episode, visit the show's website, Productvoices.com, and be sure to subscribe to the podcast on your favorite platform.
Resources
Ask a Question