JJ Rorie
- 1 min read

AI Platform Products: What You Need to Know to Be Successful

Updated: Apr 11, 2022

Episode 005: Prerna Kaul, Senior Technical Product Manager at Amazon:

"You're now seeing every industry being touched by AI products. But at the same time, how do you ensure that what you're building is extensive in some form so that you're not limited to any particular domain or limited to any particular set of goals or objectives? So there's almost a tension between moving fast and solving for one use case in a particular industry, but also going broad and making sure that you may be targeting a vertical, but you're not targeting one particular goal or objective. You are ensuring that in the future, when you have different objectives, those can also be taken care of within that platform.

Episode Transcript

JJ: Hello. Welcome to Product Voices. Artificial Intelligence is such a fascinating topic. AI can simulate the cognitive functions that human minds perform, such as problem solving, learning, reasoning, even social intelligence. Some of these innovations that are happening these days around AI are just amazing. There can also be some learning curves around how to do it successfully ethically as well. So AI is a topic that you probably are hearing a lot about. I personally am not an expert on AI products and platforms, but I'm just fascinated by the possibilities.

So I'm really excited to be chatting with my guests today on this really cool topic. Prerna Kaul is Senior Technical Product Manager at Amazon, where she has built AI / machine learning products in the payments, healthcare and Alexa domains. She's also worked at Deutsche Bank, Citi Group, and Walmart Labs where she has built industry leading product innovations.

Prerna, thank you so much for joining me.

PRERNA: Thank you for having me, JJ. How are you?

JJ: I'm good. I'm so excited about this. I'm going to learn so much from you.

So let's just jump into tell me just kind of the foundation. How would you describe an AI platform.

PRERNA: Sure. Yeah. So I think to put it simply, there isn't much of a difference between an AI platform and pure software platform idea being that any platform product needs to be extensible, scalable and reusable. Those are the three pillars that one would go by. I think the nuance of the difference with respect to AI is really going into some of the intricacies of building a model and deploying it into production and maintaining it, and refreshing and upgrading it year over year, week over week for a variety of different use cases.

So I've spoken extensively with product managers who have launched AI products in the retail space. For example, they are building things that are leveraged within a checkout flow for shopping related use cases, or building something in the smart assistance base, which is something that I'm working on. In addition, you're now seeing every industry being touched by AI products, and it really depends on kind of your use case. But at the same time, how do you ensure that what you're building is extensive in some form so that you're not limited to any particular domain or limited to any particular set of goals or objectives? Right. So there's almost a tension between moving fast and solving for one use case in a particular industry, but also going broad and making sure that you may be targeting a vertical, but you're not targeting one particular goal or objective. You are ensuring that in the future, when you have different objectives, those can also be taken care of within that platform.

So I'll give you sort of a deeper example there. The ML flywheel really consists of different aspects in terms of exploring, preparing, training, testing, evaluating, deploying, and updating a model. So how does one ensure that you're efficiently doing all of those things while also meeting the business objectives? The core of what an AI platform is really capturing the efficiency and the reusability of going through all of those steps while also meeting the business objective.

JJ: Wow. That's an amazing explanation of that. Also quite intimidating, I think. And that's why I'm so excited to speak with folks like you who've done this, who know what's doing, because you tell me that. And I know there are brilliant people out there like you working on this. But to me, that sounds almost overwhelming. Obviously, there's a lot of great minds working on this. But here's a question for you. You mentioned this a little bit, but is there a difference, or is there a way that product managers should think differently about AI products versus just other software products?

PRERNA: Yeah, absolutely. I think that the key difference really going back to fundamentals of product management. The key difference lies in how we view the customer and who the customer is. So in this case, my focus has as an AI platform product manager been on ML scientists as the customer. We might think of other software platform products as having different sets of customers. So, for example, if I'm launching something on the Shopify Ecommerce platform, I may have small businesses as my customer. But in this case, you will see that with AI products, the platforms that we're launching are very even with sort of customer facing BTC use cases, there are two sets of customers. There is a customer who we're building things for at the end of the day and launching magical experiences for them. But there's also the scientists for whom we need to solve a ton of unique challenges that perhaps with other software products, we might not have to. So to keep it quite simple, this is still a nascent space in terms of the tooling, the availability of tooling for ML scientists.

Every company that is making a foray into the space be it the Ubers of the world or Amazon, Walmart, Facebook, and Google. They each have a unique set of challenges in terms of providing their ML scientists with reliable tools, the right toolkit to actually launch models more quickly, ensuring that there is some consistency in evaluating whether their model is performing well, and then ensuring that they can upgrade the model. These are like intricacies that one would not think would be a challenge as far as other software platform products are concerned. But here really the key or sort of the nuances building something for the BTC customer or another type of customer while also keeping the ML scientists front and center. So that is the nuance that I found to be quite interesting here.

JJ: Yeah, that is interesting. Making sure that you keep that scientist in mind while keeping the end consumer in mind, or vice versa. And that's not an easy task, I'm sure, because I can imagine those two constituents being quite different in their needs.

PRERNA: Absolutely. And the scientists, of course, I've worked with a number of different great scientists within Amazon with my previous work as well. I think the challenge that they will find is they are quite focused on solving innovative, solving for an innovation, solving for interesting ideas, but bringing them back to the customer problem too is where the product manager really comes into play. Sometimes you have to often ask this question of so what is the business impact of what we're doing here? So that's another nuance that I wanted to call out that may not be present in other software products.

JJ: Yeah, absolutely. So that kind of ties into my next question, which is, have you seen some empathy building approaches that have worked for these types of platforms and for aligning an AI and ML organization.

PRERNA: Absolutely. J so I think taking a step back, you and I have spoken about this earlier. We've talked about how there is a power in having a product mind your product thinking in a highly technical organization. And one of the things that I see that is unique in AI ML role is you are often working with folks who are very driven to solve high technical challenges, and you really need to take a step back and bring in that approach of, well, are we solving the business problem? So we've spoken about that earlier. And I think the interesting aspect of this is that there are some things that are new in terms of the sort of low level details of what you might need to do in order to build a successful AI ML product.

But then also there are things that are not different. They're just one and the same in the sense that you also need to continue to do those user interviews as a product manager, in which case talk to your scientists. In the past, what I've done is I've talked to we have about five to seven ML scientists in our organization. I've spoken to every one of them, have a 1 hour interview with each of them trying to really understand the life cycle of what they do in order to build an AI ML product and deploy it in production and maintain it week over week. And I've recorded those interviews, and I've dived deeper into them and listen to them over and over again, really to get into head space of an ML scientist that is nothing different from what a product manager would do in any other organization. So that's why I say it's unique, but it's also not so different.

Another thing that I would say in terms of sort of an organizational alignment is having a feedback loop. So one of the unique things that I've seen here is we often don't. We will go down the path of and I'm speaking at high level here, but we will go down the path of coming up with an idea and executing on a solution. But then having that reflection of and which brings me back to the initial point on how to be a good AI platform product manager. What's unique and what's different about this pieces, bringing it back as a feedback loop. So the next time around you execute on the same idea and have to align your organization or you are more effective, more efficient, more productive, and you get to the right solution quicker.

JJ: Yeah, I love that. I think that's really great advice and good techniques. I also like how you said in some ways it may be nuanced, but it's very similar to gauging any customer base or customer target and love the technique of recording and actually talking to those scientists and then listening again and again because there's so many different things that you can hear each time you listen to a conversation. So I think that's a great technique.

I have maybe a little bit more of a tactical type of question. I'm curious how A/B testing in this space works. Like how do you ensure something works or how do you ensure success.

PRERNA: I think there are layers of A/B testing within the AI/ML world. One happens offline and the other online. So when I say offline, I'm speaking about the experiments that scientists do. We typically almost never launch a model in production without having conducted some sort of an offline experiment with a set of features, training and testing the model on a set of data, production data, and then coming up with offline results of what we believe the precision and recall of the model is. Precision and recall are typically the metrics through which we gauge performance of a model. There are others, but these are the most commonly used ones. I can go into details of what the others are, but probably less relevant to this conversation.

So what you will see is there are two layers. Number one is offline, which is an experiment that a scientist would do, and the second is online. Once they deploy the model in production. How A/B testing works in the spaces. It's similar. Again, as I said, not so different. The only nuance here is how do you make sure that when you're doing an A/B test, you are clear on what is the new model bringing to the table in terms of features and then what metrics you're testing them on. Right. So you may be testing for a business objective or business oriented metrics such as how much revenue share did I increase? But you're also testing on the performance of the model and understanding and knowing what you changed.

There are nuances in terms of explainability of features that we may talk about with ML scientists, where we talk about what are the new sets of features we're launching. Is this a true AV test? Is this an Ae test? Are these two different models altogether? Do we understand what objectives we're trying to hit? What are the success metrics?

And then finally, what are the data sources? Are the data sources the same in both cases? Are you pulling the data offline versus online from the same places? So these are some questions that we ask that are sort of unique. But also you'll see that there's a pattern here. The AB testing is largely very focused on making sure that you have the metrics right. The success metrics are defined and you're measuring both sets of versions on a similar you're clear on what is the A and what is the B. In that sense, how you ensure success I think is largely, you might be surprised largely about communication with the scientists and the engineers involved because a lot of it is just making sure that we fully understand the problem and what it is we're launching and we are aligned on our mental model and thinking around it.

JJ: That makes so much sense. I love that. And by the way, I'm going to ask you to come back and be a guest on a part two episode, because I could spend all day long talking about this stuff with you and go into lots more detail. But that's a really great response to that question. I've always been a little curious about that.

Let me take it a little in a different direction slightly when thinking of product managers in this space. What are some common traits and strengths that you've seen in successful AI and machine learning product managers?

PRERNA: Absolutely. And I've had the opportunity to meet many in my career so far, and I will say that and I'm quite grateful that I had the chance to interact with some great product managers, both peers as well as seniors in my career. I think a few common threads that I've seen is there is this aspect of resilience with AI ML product managers that's unique because I do see that there could be longer lag times as far as launching ML products is concerned. So those lag times could be introduced by the fact that we may not have enough data to actually train or test a model and deploy it in production. Or it could also be introduced by the fact that AI is new to an org. So getting different stakeholders within the org aligned and sold on the idea of AI/ML. That is one sort of situation that I've also seen happen.

In addition, there are complexities and nuances that only the scientists understand stand, but then they may not have the know how to make sure that what they're launching meets the right sort of business objectives and fully understanding the business problem. So there is this aspect of resilience for sure. Second thing I will say here is AI/ML product managers have to be uniquely curious about the problems that they're solving. It requires just another level of really, really understanding the challenges, the customer problems, diving deeper into all of the intricacies and nuances of what you're solving where with software products. I think because largely the sets of inputs and outputs are well defined, there may be less of that. But here I found that it is definitely a curiosity bug that one needs to carry that one may not succeed without.

Third thing I will say is and this is again common trait to all product managers, but also for AI. And it is true that partnering well with your engineers and your scientists. It's mission critical. I've seen some great product managers who partner so well with their scientists and engineers but may not have a technical background. They really just understand how to communicate with their stakeholders and engineering partners and they're able to build that trusting relationship within the team to really launch some great, successful products. And I've definitely found that to be quite true in this space as well.

JJ: I love it, love the advice. My final question for you is one that I like to ask all of my guests, and it comes down to kind of resources that can potentially be shared with the listeners in case they want to augment the learning. So in your journey to becoming an AI product expert, what investments have you made in yourself as you grew and learned along this journey?

PRERNA: I will say that there is definitely a spectrum as far as investments is concerned. On the one side, you will see a lot of what needs to be done in order to be successful in this space is more about intuition. Getting an intuition around ML AI, so you're able to bring that thought leadership to the table. That's one end of the spectrum, other end of the spectrum is really diving deeper into the technical aspects and the details and talking hard skills for both sides. I would have two sets of recommendations. On the one hand, on the intuition side, I have found some great books that we're particularly useful to me.

In my journey. So I was fortunate actually, when I was at University of Toronto, where I had the opportunity to learn from some stalwarts in the AI space, including Geoff Hinton. He's a Nobel Prize, Shivon Zilis, who was an active speaker as a venture capitalist in the Creative Destruction Lab at the University of Toronto, and Professor Ajay Agrawal, who's also a thought leader who kind of brings in that market for machine intelligence perspective. So they gave me exposure to a lot of great courses and books as far as understanding the intuition of AI.

Two books that I would recommend are Super Intelligence by Nick Bostrom, and that's something that I actually have a book club around. We're reading that book together and really just discussing it as a topic and Humans Need Not Apply, which is a book by Jerry Kaplan, and that really gives you the idea of how our world will change when AI ML is the norm. And then how will wealth work? How will income work? How are we thinking about the core capabilities and expertise that people can offer versus AI? So it really goes into some of those aspects. So I will say reading books around just understanding the intuition will be super powerful. I found it to be very interesting and I also am a big fan of Sci-fi, so I just love learning about this stuff.

On the other hand, other end of the spectrum, there are courses, there is material that you can learn, including Andrew Ng's famous AI on Coursera and multiple topics that you can pick up on Udemy as well. And I'm happy to link those resources for the audience, but I will say there is an investment that is required on both ends. In addition, I think one must not forget that learning from the people around you and continuing to refine your thoughts through experts that you meet on a day to day basis is perhaps as true as ever for AI as it is for any other PM rule.

JJ: Those are wonderful, wonderful resources. And yes, we will have links to those on Product Voices.com. So you'll be able to find information about Prerna and connecting with her and also will link to some of those those resources that you mentioned there.

Prerna, this has just been such a fun conversation. Thank you so much for sharing your wisdom with us. I have learned a lot. I know the listeners have as well. Would love to have you back and talk more about this because it's just such a fascinating topic and love your perspective on it. I've really enjoyed our conversation. Thanks again for being here.

PRERNA: Absolutely. It's my pleasure.

JJ: And thank you all again for joining us on Product Voices. Again, you can find out more information on Prerna and the links and resources that she discussed on Productvoices.com and thank you for joining us. We hope to see you on the next episode.

Resources

Connect with Prerna

Ask a Question

AI Platform Products: What You Need to Know to Be Successful

Episode 005: Prerna Kaul, Senior Technical Product Manager at Amazon:

Recent Posts