I’ve been talking a lot about how I use AI tools at the Product Mind Lab, but today, we’re talking about the level of intention and thought needed to build these tools for product builders. It’s my pleasure to have the Yana Welinder, founder and CEO of Kraftful on the Product Mind Lab this week. Kraftful is an AI assistant that helps 50,000 product teams build better products by listening to users at scale.
One of the trickiest things about building with Generative AI especially is that it is more probabilistic than it is deterministic. When you’re building products, that can sometimes mean the AI gets a little “too creative” which means to the user, the quality of the solution isn’t consistent. My first question to Yana was how she and her team at Kraftful try to account for the probabilistic nature of GenAI, and it made a lot of sense. She said:
“Some of that has been around just breaking down tasks into lots and lots of small steps. So really limiting the model's creativity, making it do the one thing we wanted to do and, you know, be smart, but not creative.”
Using task decomposition is a great way to help reduce the variance in how different the final output will be, and Yana brought up how important that is for a product that leverages several models that change quite frequently. You don’t want to build your product in a way that the models are regularly breaking the product.
Another important component of insuring consistency despite model upgrades and integrating new models Yana brought up is evaluations (often called “evals” for short).
“At different times, we have a combination of different models…you run evals on every step, which means that we can really make sure that the model behaves in the way we would expect it to, at least on a difficult data set.
Beyond making sure the product is consistent and high quality, I was curious about how Yana and the Kraftful team are intentional about the product decisions that account for the human tendency to think AI is smarter than them, or automatically right. While powerful models, like o3 from OpenAI are really great at reasoning, they are not automatically right about everything. So, it’s important to make sure there is a human in the loop and the human knows they are in control.
“We've really thought deeply about kind of building around the product workflow and thinking about what are the things where we can really make PMs be leveraged and make it really easy for them to make decisions and then made it so that when there. is a decision step, any kind of like human decision step to be made, it's very clear that that's what they need to do.”
I also asked Yana about what the future of Product Management looks like with AI being able to handle more and more of what PMs do, and she said:
AI is going to be performing as well as humans at all the tasks, and superintelligence is exceeding humans at everything, including creativity. With that, with AI at that level, AI is going to autonomously be building products. I think realistically, we have to acknowledge that that's going to happen.
At the same time, I do think that we humans have a drive to create. Yes. That drive to... create things for other humans based on our empathy and kind of the desire to just see people use the thing we kind of built and exercise our product craft in that way. I think that's going to continue. So I think that both of those worlds are true in the future and the path there is going to be really interesting. I also think that the future is very soon.
While the future isn’t certain, it’s pretty clear that product builders need to improve their own clear thinking and creative thinking to be able to continue on the “product path” as a career for whatever that will look like in the future. I encourage people to focus on what it means to hone their craft, and continue to learn about AI and the AI tools out there to understand where your place is in the future of Product. In the short term, I hope this Substack helped shine a light on building products that leverage AI, and some of the intentional decisions that come with it.
Thank you so much to Yana Welinder for being on the Product Mind Lab.
Stay curious.
P.S.: I’m a fractional product leader and consultant, and I recently wrapping up a GenAI initiative at a GovTech startup. If you’re looking for a AI savvy product leader who can amplify themselves with AI, I have limited fractional hours available. Please reach out here to learn more.
P.S.S.: The fifth live rendition of Jumpstart Your AI Career on Pearson/O’Reilly will be February 24. Register here.
Author note: Kraftful and Yana are not sponsoring this Substack, and I have not been offered anything in exchange for this interview. I was excited to do this interview to share how people are building AI and building with AI.
Share this post