---
title: "Lenny's Podcast — 2025 Q3 合集"
date: "2025-01-01"
source: "Lenny's Podcast"
url: "https://www.lennysnewsletter.com/"
---

<div v-pre>


# Lenny's Podcast - 2025 Q3 (18 episodes)

This file contains 18 articles/episodes.

---

## [1/18] I’ve run 75+ businesses. Here’s why you’re probably chasing the wrong idea. | Andrew Wilkinson (co‑founder of Tiny)

**Andrew Wilkinson** (00:00:00):
You don't want to walk into the gym on day one and try and deadlift 300 pounds. So when someone comes to me and they're a first time entrepreneur and they say, "I'm going to make the next great AI company," I think that is the equivalent.

**Lenny Rachitsky** (00:00:12):
I feel like you've actually started and run more companies than maybe anyone else in the world. What is your best advice for coming up with a great startup idea?

**Andrew Wilkinson** (00:00:20):
Charlie Munger, Warren Buffett's longtime business partner, has this amazing quote.

**Speaker 3** (00:00:24):
Fish where the fish are.

**Andrew Wilkinson** (00:00:26):
The biggest mistakes I've made have been going into business models where other people have repeatedly failed and thinking, I can do this better.

**Lenny Rachitsky** (00:00:34):
It's so funny to watch you on Twitter. Clearly you've become AI obsessed.

**Andrew Wilkinson** (00:00:38):
It's like having the world's most reliable employee who costs $200 a month and works 24/7. So many knowledge work jobs are going to change massively. I think the fundamental question is, do all jobs just become a single prompt?

**Lenny Rachitsky** (00:00:54):
Today, my guest is Andrew Wilkinson. Andrew is the co-founder and CEO of Tiny, a holding company that's often called the Berkshire Hathaway of the internet. They own over 40 businesses ranging from Dribble to WeCommerce to the AeroPress coffee maker, and they focused on buying profitable businesses from founders and holding them for the long term. Andrew and his co-founder bootstrapped the business from zero to hundreds of millions of dollars in value, and Andrew personally was worth over $1 billion at one point. In our wide-ranging conversation, we cover a bunch of strategies for coming up with a good business idea, what common business ideas you should avoid, his experience automating much of his work and life using AI, and what that means for employment in the near future. Also, what he's learned about happiness and money and how they're not directly related, and also how getting diagnosed with ADHD and then taking SSRIs was the thing that most impacted his happiness in life.

**Andrew Wilkinson** (00:04:15):
Oh, thanks man, great to be here.

**Lenny Rachitsky** (00:04:17):
I've been wanting to chat with you for so long. There's so much that I want to talk about and I want to start with a topic that I know that you think a lot about and that it's also in the minds of a lot of people, which is coming up with a great startup idea and this is something that a lot of people are thinking about right now because AI makes it so easy to actually build your idea into something real, and I feel like this is something you spent a lot of time thinking about. I feel like you've actually started and run more companies than, I don't know, maybe anyone else in the world. I feel like you're in the top a hundred, top 10, something like that. I don't know. Does that feel right?

**Andrew Wilkinson** (00:04:51):
I've definitely had a lot of experiences and I've probably started or been involved with 75 different projects or businesses where I've been a primary contributor and I wouldn't say that's anything to brag about because I've been an inch deep in a mile wide. So that's been in a lot of ways kind of my Achilles heel is I get too excited about ideas and I start too many businesses, but as a result I've seen almost every business model under the sun.

**Lenny Rachitsky** (00:05:18):
Amazing. Okay. Yeah, I think the benefit is for us is we get to learn from your experience. Let me just ask you this question, what is your best advice for coming up with a great startup idea?

**Andrew Wilkinson** (00:05:28):
Ultimately the best thing is something you're going to be interested in, but I think a mistake a lot of people make is they choose something that everybody is interested in. So for example, they say I don't know anyone that has hadn't had the thought, man, I would love to start a really cool restaurant or I want to have a cool cafe or something. And in reality, what they think about is how cool would it be to come up with an amazing logo or all fun stuff, design the menu and stuff. But in reality, operating those businesses is miserable and it's also a very hard business because every morning millions of people wake up and they go, I should start a cafe. But on the flip side, almost nobody wakes up every morning and says, "You know what? I'd love to start a funeral home," or "I would like to start a pest control business," or "I should start software that helps people fill out forms for the government."

**Andrew Wilkinson** (00:06:26):
And Charlie Munger, Warren Buffett's longtime business partner, has this amazing quote. He says, "Fish where the fish are." And he gives this example, he says, "If you're a fisherman and you see a large pond and all around the pond, there's a whole bunch of fishermen and they're all elbowing each other out of the way. They're all using the best lures, the best fishing line, they all have amazing strategy. You actually want to walk off into the forest and find a small fishing hole with lots of fish and very little competition. And I think that's probably the most important thing in business is actually to find those niches where you can actually make real money because competition equals lower margin. The more competitors there are, the lower your prices have to be and the more competitive the business is ultimately.

**Lenny Rachitsky** (00:07:19):
It doesn't feel intuitive to go after small markets and to find niches. So let's just follow that thread. Well, why is that actually a source of some of the best ideas starting really small and niche?

**Andrew Wilkinson** (00:07:30):
I don't necessarily think it has to be something that is small forever, but it has to be something that like I think about if you're a first time entrepreneur or a student or something, you don't want to walk into the gym on day one and try and deadlift 300 pounds. So when someone comes to me and they're a first time entrepreneur and they say, "I'm going to make the next great AI company," or "I'm going to launch a new bank," or something like that, something that is very, very rigorous and complicated and highly competitive and regulated, I think that is the equivalent. I think you really want to take the baby weights and start slowly building your muscle. And I think about my own experience starting a business, I was so lucky because the first business I ever started my web design agency, which became Metalab, that was so easy and it worked immediately because all I had to do was know how to build websites and be able to talk to potential customers.

**Andrew Wilkinson** (00:08:33):
And then once they said, "Yes, I will pay you $5,000," I just had to send them an invoice, do the work, and that was it. It was a very simple business. And because of that, I got immediate positive feedback and I built my own narrative. And my narrative was, "I'm good at business, I can do this, keep going." And then I went off and I started taking my money that I made from that original business and that's when I fucked up. So I went out and I started a pizzeria and I lost all my money. I started a designer cat furniture business, a online DJ school, a skin cream business, all of these things. I've just lost all my money almost immediately, but because I had that first win I kept on going. And I just think it's so critical that people choose a business where they get that initial win.

**Lenny Rachitsky** (00:09:22):
Okay, this is a great topic. This is just like how do you avoid creating a job for yourself that you hate? There's a lot of business opportunities and ideas that like, "Yeah, you can make this work," and then it works, and then you're stuck doing this thing running a restaurant. My wife tells a story where a friend started a coffee shop and then he's like, "I thought I was starting a coffee shop, but I'm just replacing milk and buying milk all day. That's my job now." So just kind of along those lines to help people avoid creating a beast that they didn't expect any advice for Just how to know this is maybe even though this may work and make money, you probably don't want to be spending your life doing this sort of business.

**Andrew Wilkinson** (00:09:59):
So I just started a pressure washing business. I was speaking at a local business school and after I spoke, one of the kids walked up to me and he said, "Hey, I'm an entrepreneur. I've started two or three businesses in the past doing landscaping and I'm not enjoying school." And so just on the spot, I said to him, "Why don't you drop out and we'll start a business together?" And I'd been cooking on this idea of a pressure washing business because I'd studied that industry a little bit and I knew a few friends who'd done it in other cities. And I had this unfair advantage, which is I owned a whole bunch of media properties where I could advertise it for free basically. And so we started this business and I think for him, he kind of had this moment of, do I really want to be washing people's driveways for the rest of my life?

**Andrew Wilkinson** (00:10:47):
And what I said to him is, "Look, if this gets to scale, you'll never touch a pressure washer unless you want to." If the business can get big enough where you can have employees, he can just focus on sales or digital marketing or whatever aspect of the business that he really loves. And I think that's the biggest thing that a lot of entrepreneurs miss. They say, "I don't want to do that for the rest of my life. I don't want to be in the back of a dry cleaner dry cleaning clothes." And to me it's just a question of scale. I think the cafe is a great example because a cafe, if it doesn't get to a reasonable scale is just a job.

**Andrew Wilkinson** (00:11:26):
There's a big difference between a business and a job. I think if you could start a pressure washing business where you're the only employee, yeah, that's a job, but if you can get to a scale where you can drive 10 leads a day, then you don't have to do any of the pressure washing and you just do what you love. So I think a lot of people have this kind of Protestant work ethic where they think, well, I've got to be the person to do everything. And I think they really need to lean into what I call lazy leadership, which is how do I get away from the things I hate as quickly as humanly possible? How do I be Teflon for tasks?

**Lenny Rachitsky** (00:12:00):
It's interesting that this business is non-software related at all, this pressure washing business. So I guess just for folks that are coming up with ideas, and I imagine most ideas are there's a pull to make it software oriented, something AI is going to help build for you and help you run. What's your calculus on just going down a direction of real world physical business, like a restaurant power washing business versus a software just like what are the benefits of that direction when should someone actually seriously think about doing a business like a power washing business?

**Andrew Wilkinson** (00:12:30):
I think if someone's listening to this podcast, odds are they're someone who's kind of a digital native. And I think the question is, what's your unfair advantage and what are you great at? So for me, I think I have reasonably good taste. And so what I could do is I identify and I knew enough to identify great design and development talent, but mostly I was lucky because I was a talker. I was good at meeting with clients and selling myself. And so ultimately my highest and best use my superpower was sales. And so that can be applied to anything, anything where you need to go out and you need to sell a customer. And so really it just comes down to what do you get drawn to and then how do you find the profitable business within that? So here's an example. A friend of mine, he owns a restaurant and he said, "Oh, I love it. I'm so passionate about it. It's this stunning beautiful restaurant in my hometown."

**Andrew Wilkinson** (00:13:32):
And he said, "But it's really just a job for a few different people and we can't make any money at it, but I've been noticing that there's all these vendors that service the restaurant and those guys are making a killing." And so he told me about a business that cleans grease traps, another one that cleans exhaust vents for kitchens. So I think looking around and seeing where your passion is and then sniffing around within it, probably somewhere within your passion, there's an opportunity. So for example, I love movies. My happy place is a dark theater. That's the best way for me to de-stress is go to a dark theater and watch a movie and get lost in it. And probably about four or five years ago, I was like, "Man, how cool would it be to invest in movies in some way to be a part of that creative world?"

**Andrew Wilkinson** (00:14:26):
And so I started looking into funding movies and I realized that when you fund movies, like 90% of the time you lose all your money. And even if you do make some money, you rarely make a good return. But I spend a bunch of time understanding the industry and learning about it. And then two years ago I was in New Zealand and I met the founder of Letterboxd and I realized, "Oh my God, this is a business where this has a moat, so it has a network effect, it's a huge social network for film reviewers. It's something I'm passionate about and it's something that we can buy at a fair price." All those things came together and I was like, "Oh my god, I can invest in film now." In the same way I used to be a barista, we ended up buying the AeroPress coffee maker company. So I follow my passions and spend a long time learning different industries and then I find the profitable niche within.

**Lenny Rachitsky** (00:15:19):
Okay, and this is a great takeaway essentially when you're thinking about startup ideas, make sure there's some connection to something unique to you where you have some unfair advantage. This makes me think about Brian Armstrong. I saw him do a talk once and he gave this really amazing insight about why Coinbase did well and why he started Coinbase. And it's because if you look at his background, he had this really rare Venn diagram of background of computer science and I think it was cryptography and economics, which is the exact set of skills you need to start something like Coinbase. So I think the tip there is just what is that unique Venn diagram of skills in your background and just ideally what you're building connects to that and gives you an unfair advantage.

**Andrew Wilkinson** (00:16:02):
Well, yeah. I just met another UVic student at the local university student. She is interested in marketing and so she's gone out and she's found some local clients, so small restaurants and stuff and managed their social media and she said, "Yeah, it's okay. I can make a thousand bucks a month, but it's a lot of work," and the owners really want a lot for their money. And I said, "Well, if you just pivot that idea just ever so slightly, and instead of doing restaurants, you did realtors or wealth managers who have quite a large marketing budget and are used to spending serious money and it only has to work a little bit to make a lot of money for them. Those people, you can charge $5,000 a month."

**Andrew Wilkinson** (00:16:44):
So I think often it's you find your passion, your skill set, you zero in, and then you just kind of pivot and you find the most profitable way to do it. When I started my web design firm, I started mostly with local and small projects, but very quickly I found a job board in San Francisco where startups would share projects they needed help with. I could charge five times the amount for five times less work.

**Lenny Rachitsky** (00:17:09):
This point about fish where the fish are I think is really important. You can start something that's awesome that you love, that you're so excited about, but nobody needs it. Talk a bit more about just what that looks like. What have you seen when you're thinking about ideas that tell you that there's fish but also not overfished as you pointed out?

**Andrew Wilkinson** (00:17:25):
I think it's hard like Warren Buffett has this great quote, "I'm a better businessman because I'm an investor and I'm a better investor because I'm a businessman." And I feel like in order to know what problems are valuable to solve, you kind of need to have valuable problems. So it's a bit of a hard thing because I remember when I was like 20, I would say, I hate how all the cat furniture I can buy for my cats is I bet people would pay a lot of money to solve this, but I didn't understand anything about the realities of that business model. I didn't understand how little people were actually willing to pay. I didn't have a enough life experience to go, "Yes, that's actually a worthwhile opportunity." And I think that being able to know what people would pay to solve the problem realistically is incredibly valuable.

**Andrew Wilkinson** (00:18:20):
So my example of a realtor selling a house, because I've studied that industry, I know that a realtor can make like 20 to $50,000 selling a single house. So I can intuit that they'd be willing to spend a lot of money if I have a unique way of getting them clients that'll buy houses that'll convert at a high conversion rate that is worth $5,000 a lead maybe, right? So I think you kind of have to understand the problem in depth before you really know. Because when I was starting out, I would go down every rabbit trail, every infomercial idea I had I would think was genius.

**Lenny Rachitsky** (00:18:59):
Or people, I don't know, that don't have this experience, which is most people. Is there one thing you do, maybe a heuristic that gives you a sense, maybe there's something here, maybe there's a lot of fish, maybe it's a value of prompt or no, the soul tell me it probably isn't.

**Andrew Wilkinson** (00:19:12):
No, unfortunately, I think it's mostly gut. I mean Munger and Buffett talk about having a whole bunch of mental models in your brain and they form a latticework, right? And they all kind of piece together. And I feel like for me, there's so much of this that is, it's almost like I'm an AI model and I've trained on all this data of what works and what doesn't and what's a good moat and what's a bad business, et cetera. And when I see it, I just immediately know. I mean, I think all entrepreneurs are so lucky now to be able to go into Claude or ChatGPT and just say, "Hey, I'm thinking about starting a Botox clinic. Can you break down the numbers? Is this a good business? What's hard about it? What is the regulatory moat? What would my payroll look like?" I didn't have any of that and so I started so many incredibly stupid businesses, but I think there's no excuse at this point. You should be able to do it with AI.

**Lenny Rachitsky** (00:20:07):
That's a really good tip actually. Then there's this point you made about boring is good. That's a really good piece of advice. A lot of people are going after flashy stuff, things they condemn on Twitter. Your advice here is boring is actually a really good thing because fewer people are going after it. Is that the general tip?

**Andrew Wilkinson** (00:20:22):
Totally. I mean, so I started this business, one of my first called Flow and Flow was basically Asana before Asana came out. So it was a way to manage your to-do's and projects with your team in a web app and we basically did the thing. We made the mistake that so many entrepreneurs make, which is to go after an industry that everybody goes after just like cafes. Everybody has the idea, man, I wish I could design my own project management system or my own to-do system. And not only that, but everybody loves new things. They love jumping around. I know me personally, in the last three years, I've probably used three different productivity systems and I love jumping around in them and I didn't understand that and I ended up losing $10 million trying to compete with venture-backed businesses and bootstrapping.

**Andrew Wilkinson** (00:21:17):
It was all my own money, poured $10 million, lit it on fire trying to compete with Asana because I didn't understand how the business world worked really as they had raised hundreds of millions of dollars and they were run by the co-founder of Facebook and it was a little bit like, I'm Fiji and I'm deciding I'm going to invade the United States in retrospect, just completely silly. Now on the flip side, if I'd taken that same amount of energy and I'd instead focused on, let's see, what's a really boring one I've heard about, there was a business I saw recently.

**Andrew Wilkinson** (00:21:52):
They were making $30 million a year and all they did was help people fill out forms to get government assistance in certain programs. So it'd be like your uncle is disabled and you need to access government funding. The process to do it is incredibly time-consuming and miserable and you have to fill out all these forms. They just have software that fills out the forms for you and it says, look, you're going to get $20,000 as a grant, pay us a thousand bucks, and that is such a boring, nobody wakes up and goes, "I want to make form filling software," but I think they would if they could make 20 million a year

**Lenny Rachitsky** (00:22:29):
Along these lines of just bad ideas, things people shouldn't start, what are a few ideas that you think everyone thinks is going to be a great idea and then they do it and then they always fail?

**Andrew Wilkinson** (00:22:37):
Well, I think the biggest mistake, I can speak from my own experience, the biggest mistakes I've made business-wise have been going into business models where other people have repeatedly failed and thinking, I can do this better. So for example, me and one of my best friends about 10 years ago, we really wanted to start a bar. We just thought it'd be so cool to have a bar where us and all of our friends could go, we thought it'd be great for the city. We thought we can run this really high margin because we're tech guys, we know how to build systems, we're good at business, and we were utterly humbled. We realized that we didn't know anything about business compared to someone who runs a pizzeria. If you think about it, like my example earlier of if you run a software company, what has to happen?

**Andrew Wilkinson** (00:23:24):
You have to hire a bunch of nerds, they need internet connections and computers and you need to pay them and you need to coordinate online and everyone can work asynchronously. Nothing has to go right. It's not that complicated, at least at small-scale. A pizzeria, it's like if the baker doesn't wake up at three in the morning and start prepping dough, the entire thing is effed and all along the way there's a hundred different failure points from front of house, back of house, the deliveries, arriving on time, all this logistics. And so I think it's been stuff like that. I mean also I got into the news business, the local news business. I ended up buying an old paper in Vancouver and it's the exact same thing. It's like you can't take a brilliant management team and change a bad business model. Ultimately the business model wins.

**Lenny Rachitsky** (00:24:20):
This is really good advice. Basically if there's a bunch of dead bodies in that space, there's probably something there that keeps killing them that you're probably not aware of until maybe somebody figures something out, right? Once in a while, once in a blue moon, someone's like, "Okay, here's how we do this and then it works."

**Andrew Wilkinson** (00:24:34):
Well, look at Instacart. When Instacart came out, everyone said, "Well, Webvan failed. This is never going to work." And we still don't necessarily know if Instacart is a great business. I don't know, I haven't studied it, but I think it was like, okay, enough technology has changed that it can happen, but would it be easier to start Instacart, Amazon or Coupang or would it be easier to start an enterprise SaaS software company? Definitely the enterprise SaaS software company.

**Lenny Rachitsky** (00:25:02):
I see. So this is I guess is the advice there. The thing that is easy to start is the thing you should not do because everyone's going after that.

**Andrew Wilkinson** (00:25:10):
Well, it goes back to my gym analogy of if you're going to deadlift 300 pounds, trained for 15 years, you should have already had three startups and they should have worked and you've really built up the reps. And I think that if I was going to start Instacart, that's different than someone who's 20 starting it. Not that I would be good at it, but at least I would know what I'm getting into.

**Lenny Rachitsky** (00:25:32):
Let's talk about something that is this endless debate on Twitter. Maybe it's a false dichotomy between lifestyle businesses, bootstrap businesses, this idea of not raising money, just making revenue, living off the revenue versus venture-backed venture scale companies feels like you're very good at the first bucket and a lot of people, this is their dream, "I'm just going to start something. I'll start a lifestyle business, make a few million a year, never raise money. Who needs VCs? VCs suck." Advice for deciding which route to go with an idea you have.

**Andrew Wilkinson** (00:26:03):
Well, I think it's just not true that a "lifestyle" or bootstrap business can't get huge. I mean, we bootstrapped the entire business and now across all of our companies we do almost $300 million in revenue. And the whole time I had been focusing on taking, starting small businesses, small ideas, simple ideas, often buying small companies and watching them grow really big. And I think that the only difference between what we do and what a venture capitalist does is the level of tolerance of burning money on fire. And I just haven't seen that If we choose the right businesses that aren't in incredibly competitive markets or where they have some sort of moat. So what I mean by that is a great brand or a network effect like a social network or something like that, I don't feel we're holding them back by not letting them light up a bunch of money on fire because these things naturally grow.

**Andrew Wilkinson** (00:27:12):
The numbers, I mean they're like balloons, they just go as long as we don't mess them up too much. So I think the decision ultimately comes down to how hairy do you want your big, hairy, audacious goal to be? And if your big, hairy, audacious goal is I'm going to start the next satellite business that creates some sort of crazy technological revolution. Yes, you're going to have to raise a venture capital unless you're already a billionaire or something like that. But if you're wanting to just tinker and solve a problem that you think is not going to be hyper-competitive, so for example, that form-filling thing or software that just does a narrow thing that doesn't require 20, 30, 40 million to be lit on fire before it can make money, then you can have a wonderful life and build a wonderful company that can scale into the hundreds of millions of dollars if you play your cards right without ever raising money.

**Lenny Rachitsky** (00:28:09):
Per the story you shared with Flow where you're competing against a venture-funded company, if there is a venture-backed company in this space, is the advice you're not going to win competing with them most likely and try to do something else.

**Andrew Wilkinson** (00:28:22):
Well, look at Things. Do you know Things?

**Lenny Rachitsky** (00:28:25):
I have used Things

**Andrew Wilkinson** (00:28:26):
I do too. It's awesome. Things still exist and Things has existed for 20 years. It's run by I believe one or two or three people. It's not a big team, certainly under 10 people, it was, I believe, bootstrapped and it's just consistently delivered an exceptional product and built up a loyal following. They have their 10,000 true fans who use it, and that's enough for them, but they don't do a lot. They've been very intentional. They don't do any AI stuff, they don't have an API. They're not on Android. They're very focused and I think they have succeeded. But the question is, what would an MBA or a business professor say about their success? If the measure of success is did they maximize taking as much market share as possible and grow to be as big as possible, then the answer is no because Asana and other people have built multi-billion dollar companies.

**Andrew Wilkinson** (00:29:28):
If the goal is the founder has an incredible life, probably has three houses, flies all over the place, does whatever he wants, has recurring revenue and he gets to work with headphones on building beautiful piece of software that people love, then I think he's won. I am generally much more in the camp of the Things guy has won, not Dustin Moskovitz. Dustin Moskovitz is just playing a different game than the Things guy. And if Dustin Moskovitz was running Things, he'd be miserable and if the Things guy was running Asana, he would probably kill himself because he wants to put his headphones on and build.

**Lenny Rachitsky** (00:30:05):
I wonder why nobody has come. It feels like just with lifestyle businesses, if it's working and you would think somebody would come in with more money and more funding and just eat their lunch, is the key that the market is too small for a VC to ever be excited and so that's why no one's coming for them?

**Andrew Wilkinson** (00:30:21):
I think so. I mean, I think Things probably, I'm just guessing, I don't know any of the numbers or whatever, but I would assume it makes between five and $25 million of revenue, which to a VC is not even worth considering a VC. For a VC to invest in your business, you need to have a story where it's worth 300 million to a billion dollars or more. So it's just not that exciting to them, which again, going back to fish where the fish are, you don't want to be fighting against the commercial fishermen. If you just want to have a little business where you get enough fish to feed your family and your village or whatever, find the other fishing hole. Don't go where the trawlers are.

**Lenny Rachitsky** (00:31:04):
This is great advice if you're trying to start a non venture backed company is find. This metaphor just keeps working for us, which is fish where the fish are, but not where the professional fishermen are. Also, ideally not where there's just a ton of, I don't know, fishermen from all over the place. I don't know, solopreneur fishermen. Okay. Coming back to just starting a successful company, what would you say are the keys to an amazing business model, an amazing business broadly, what should you be thinking about and looking into? I know you spent a lot of time thinking about this. When you're buying companies, what are a few bullet points that you want to focus on?

**Andrew Wilkinson** (00:31:38):
What I do for a living is basically buy businesses. So in my early career I started one business, then I started about 10 more, and then I realized starting businesses was extremely hard and I was doing pretty well. I was making quite a bit of money and I'd sold one of my businesses and I was looking out at the next 10 years and really asking myself what do I want my career to be and am I happy doing what I'm doing? And the answer was no. I did not starting businesses and experiencing that failure rate, it was incredibly stressful. And so I picked up a book about investing and I got lucky. The first one I bought was about Warren Buffett, and for those that don't know, Warren Buffett is he's counter to every VC kind of hustle culture thing you might hear because he basically just sits quietly in a room and reads all day, despite owning 260 different businesses and being one of the 10 wealthiest people in the world, his life is incredibly calm.

**Andrew Wilkinson** (00:32:44):
He only does what he wants and he spends most of his time quietly reading and once or twice a year, he makes a big decision to buy a business. And so when I read about Warren Buffett, I just thought, wow, I'm a sucker. I'm running around like a crazy person trying to run all these businesses. Why am I not just buying businesses and letting them run? And so when I'm buying a business, what I'm really looking for is something that I can't mess up. Right now, obviously we buy a business, we try not to mess it up, and we're very intentional, actually very odd in that when Tiny buys a company, we generally just leave them alone. If they already have management in place, we say nothing changes, no one should know that we've even bought them. Obviously people know that, but really there should be no change whatsoever.

**Andrew Wilkinson** (00:33:31):
The only change that we generally make is if the founder wants to leave the business, then we'll bring in a CEO to run the company, and that's probably the most important decision that we make. So I'm looking for a business though where it is so good that it's hard to mess up, and most businesses are very easy to mess up. One person leaves and the whole business falls apart because it's held together with dental floss and duct tape. And so I'm really looking for what Warren Buffett would call a moat. So a moat is basically a brand, so that could be a brand like Coca-Cola, Tylenol, Advil, something like that. So something that gives you pricing power where you can consistently charge and you have loyal customers or where we usually find a lot of opportunity is in network effects.

**Andrew Wilkinson** (00:34:23):
So typically we're looking for a community that's gotten so big where people don't want to go elsewhere. So for example, we own Letterboxd, which is the largest social network for film lovers. And the question is, if someone else wants to compete with us, why would someone go to their social network that has a small number of users when all their friends are already on Letterboxd? And so you see that same kind of thing with Instagram or Facebook or similar. And so we're looking for businesses like that, something that has staying power that is hard to compete with and that we can hold over the very long term.

**Lenny Rachitsky** (00:35:01):
It was interesting when Elon bought X, how that was such a test of the power of network effects. If you think about it, back then he changed the name. So the brand completely changed the team. 80% of the people left. What was left, it was the network and the network effect and the simplest way. Just for folks that aren't super familiar with network effects, the way I think about it is just network effect is where every additional user that joins the network becomes more useful for everybody.

**Andrew Wilkinson** (00:35:27):
There's another moat, which is high switching costs, which I don't really like because it's not really very consumer friendly, but an example might be Salesforce where no one wants to, they've spent years and they've done all the implementation and trained everyone on it, and it's kind of the standard, everybody hates it, but ripping it out and switching is such a pain in the that they just won't bother. That's another form of sometimes a great business, but again, that's more depressing. I don't love that one.

**Lenny Rachitsky** (00:35:56):
What's the last company you guys bought?

**Andrew Wilkinson** (00:35:58):
We bought Serato. Serato is the largest DJ software company in the world. So if you ever see a DJ playing and they've got a laptop in front of them, probably using Serato to DJ.

**Lenny Rachitsky** (00:36:09):
Wow, that is super cool. What I'd like a funding to be walking into.

**Andrew Wilkinson** (00:36:14):
I used to DJ so I was aware of them and it was one of those moments we talked about earlier where I understood the business. They had a really interesting mode of this huge passionate user base and deep hardware integration. And I also love DJing and I was able to analyze the business and luck comes to the prepared mind and it worked out.

**Lenny Rachitsky** (00:36:35):
This is another great callback to your excellent piece of advice of just ideally what you're working on, whether you're starting the company or buying the company, is you have a unique unfair advantage or some kind of unique background that connects to that idea. Let me ask you one more question before we shift a whole different topic, hint, hint, AI. Something that I know you talk a lot about, something you believe strongly and something you deal with a lot is when you buy companies is just as people and the challenges of management and hiring and people. So what have you learned about just how to successfully, I don't know, find amazing people, keep amazing people, help people be more successful within the companies that you operate?

**Andrew Wilkinson** (00:37:15):
My business partner, Chris, has a really great quote. He says, "There are no problems. There is only people problems." And so we've found that you could tell us that there's a disaster that one of our business units is about to fail or something like that. As long as I'm surrounded by really great smart people, I feel fine. But when we have a bad actor, we have a psycho or a narcissist or a really horrible, difficult person we're dealing with, all bets are off and life becomes incredibly stressful. So I'd say I spend probably 20% of my time just trying to make sure we're filtering people very carefully and ensuring we're working with people that we enjoy. And I could talk about it in the AI, I've done some really interesting stuff to help me screen and identify difficult people, but I've just found that the biggest mistake I made in my early career was I would hire people because I liked them on gut and I would think that I could change them.

**Andrew Wilkinson** (00:38:12):
So it's kind of like in romantic relationships, it never works out well. If you get married to someone and you go, "Okay, well this is going to be a great relationship as long as I can convert them to Christianity or I can change their parents or their personality," or whatever it is. And so I've just found that I've never been able to change someone. You can never mentor someone out into being a good employee. And the kind of heuristic I have is if I ever think, should I fire this person even once, I should fire them immediately. It has always been a signal, at least for me, that when someone's a superstar, I can't imagine firing them. I think it's impossible, I'd be lost without them.

**Andrew Wilkinson** (00:38:56):
But when I repeatedly start going, "Man, is this person going to work out?" Almost always within six to 12 months, it doesn't work out. And so I really try and be really direct about stuff like that and just cut when it's not working. So I'd say the lesson really is hire for what you need. Don't hire just for potential, which is counter to what a lot of people say. A lot of people say hire for potential and coach them into being whatever you need. I just haven't found that. I think you need to hire someone who already is fully formed and can do what you need. But again, everyone does it differently.

**Lenny Rachitsky** (00:39:31):
This is big advice. Powerful advice. Is this true not just for the CEOs of the companies you run? As for folks under lower down the ladder too?

**Andrew Wilkinson** (00:39:39):
Well, for CEOs, one of the most interesting things we've observed is, so I'll give you an example. We had a CEO come in and we were interviewing them and in my opinion, the business needed to grow via organic marketing. In his opinion, it was enterprise sales. And so when we're interviewing him, he keeps going back to his experience building with enterprise sales. And there's this great saying, "To a man with a hammer, everything looks like a nail." And so for him, he's looking at this business and going enterprise sales, that's the way to grow. I hire him, but I say, "Look, if we're going to hire you, we need you to do the organic marketing." Lo and behold, of course, he goes off and he does the enterprise sales thing. And I've just found that hiring CEOs is like they're an elephant and you're the rider.

**Andrew Wilkinson** (00:40:31):
They're going to go wherever it is that they want to go. And so listening incredibly carefully to people's words and their experiences, because generally people will do the thing they tell you. So what I look for when I interview a CEO now, I want to be nodding along. I want to go, that's exactly what I would do or that's way smarter than my idea. And then I just leave them alone because I've found that anytime I try and pull them in a certain direction or coach them or whatever, it just doesn't work. And again, this could be my problem. I might be the world's worst coach and mentor, but for me, that's been what's true.

**Lenny Rachitsky** (00:41:06):
I imagine there's also an element of they won't love the job if they want to be doing say enterprise sales, and then they're creating viral TikTok videos all day and tweeting. They're like, "What the hell?"

**Andrew Wilkinson** (00:41:15):
Well, I've also found people will shoot themselves in the foot. If I tell them an idea in a board meeting and I say, "I really need you to try this," it never works because usually they sandbag it, right? They don't really put their heart into it or they're just placating me and they don't want it to work. They want their idea to work. And so I've learned not to do that.

**Lenny Rachitsky** (00:41:37):
**Andrew Wilkinson** (00:43:10):
So the primary tool I use is a tool called Lindy, Lindy.ai. And what it basically lets you do is build workflows and agents. So it's very simple. You might say, "When I get an email in the Gmail API, I want to add an AI agent that reads the email and labels it based on that." Or you can build a crazy Rube Goldberg machine that sends different emails all sorts of different places. So for example, when I get an email and it's related to my kid's school, one of the big problems I have is that my kid's school sends so many emails, the field trip is on Thursday and you need to bring the following things. And meanwhile, there's parent teacher interviews and all these things get added to my calendar. The AI agent automatically takes that, puts it onto my calendar, makes sure there's notes for all the things I need to prepare and stuff. So I have just basically tried to take every single thing a human could do in my inbox and automate it with Lindy.

**Lenny Rachitsky** (00:44:10):
And you're sitting there in Lindy doing this yourself or do you have people that you've trained to help build these sorts of things for you?

**Andrew Wilkinson** (00:44:16):
No, I do it myself.

**Lenny Rachitsky** (00:44:17):
Okay. How many agents, if that's the term you want to use, do you have running for you roughly?

**Andrew Wilkinson** (00:44:22):
Oh man, I think I probably have not a crazy amount within... Let's see. Well, each workflow probably has four or five agents. So for example, my inbox has four or five agents, and then I have a whole bunch of other, I've got a calendar agent, I've got a meeting agent, I've got an email agent, I've got a scheduler, I've got a CRM, and basically they're all different workflows. So the whole bunch of agents within.

**Lenny Rachitsky** (00:44:50):
Okay, and you're not an investor in Lindy, right? You're just a fan, super user.

**Andrew Wilkinson** (00:44:53):
No, no.

**Lenny Rachitsky** (00:44:54):
Fun fact. I actually am. So this is great.

**Andrew Wilkinson** (00:44:56):
Oh, no way. That's awesome.

**Lenny Rachitsky** (00:44:57):
Tiny angel investor. So just want to put that out there, which I love that this came up organically. What are a couple other really interesting use cases, workflows you've built that people might be inspired by it?

**Andrew Wilkinson** (00:45:08):
Let me go through the email one a little more. So basically emails come in and the first agent says, does Andrew even need to see this? Let's say you're in a thread and you've already chimed in and everyone's just saying, cool, that works. I don't need to see that. So just archives that never. So that immediately reduces my email by about 20%. Then it decides is this something that is time sensitive? Is this something that needs to be dealt with in the next 24 hours? And if so, it labels it in a special 24-hour label. So when I go in my email, one of the biggest stresses I find is I go, "Shit, there's 200 emails in here, and if I don't go through all of them, I don't know if there's an emergency burning somewhere." So now I do. Then it takes any email or every single email and it decides, is this a simple decision?

**Andrew Wilkinson** (00:46:00):
So for example, let's say you email me and you say, "Hey, do you want to get lunch?" So it emails me privately and it says, "Hey, Lenny wants to get lunch. Do you want to say yes? Do you want to say yes, but in a few months? Do you want to say no? How do you want to say no?" And it gives me multiple choice. I just can say four, number four, and then it'll email you as me and it'll write out a nice thoughtful email or whatever. So stuff like that is so freaking cool and helpful. And I'd say that it's replaced stuff that my full-time, I used to have a full-time assistant just working on email that's all completely automated and there's all sorts of other cool stuff I'm doing there. Other agents, I'm working on one right now to manage my calendar, which is really complex. I do a lot of scheduling and a lot of rules. I have another really simple one. It adds emojis to every single calendar event. So when I look at my calendar, I've got beautiful little emojis for every single calendar event. Very silly.

**Lenny Rachitsky** (00:47:00):
What are the emojis? Do they represent something?

**Andrew Wilkinson** (00:47:03):
Yeah, so if I write weightlifting, it'll just do a guy weightlifting or whatever. I've got one that, this one's cool. Basically what it does, every time I email someone, it looks them up online, it figures out what city they live in. It checks my CRM and air table. It says, "Okay, I don't know where do you live."

**Lenny Rachitsky** (00:47:21):
In Marin, in the Bay Area.

**Andrew Wilkinson** (00:47:22):
Okay. So it'd say, "Okay, you email Lenny, he lives in Marin." It puts in the database you live in Marin. And then next time I travel to the Bay Area, the agent will see two weeks before and it'll say, "I saw you're going to San Francisco. Here's all the people in San Francisco you should try and have coffee with." Right? So just all these things that-

**Lenny Rachitsky** (00:47:22):
I would pay for that.

**Andrew Wilkinson** (00:47:43):
Yeah, yeah, exactly. All these things that I've dreamed of having my assistant do that my assistant could just never consistently do because she gets distracted, we're doing an event, there's something going on. It's like having the world's most reliable employee who costs $200 a month and works 24/7.

**Lenny Rachitsky** (00:48:02):
Okay, I want to follow that thread, but first of all, what other tools do you have? Lindy's one, what else do you find really useful?

**Andrew Wilkinson** (00:48:08):
So the other one is Replit. So Replit is basically a vibe coding platform. You can literally go into it and say, "I want to make a website for my sound software business. Here's a big document with a bunch of details," and it'll go and design a pretty impressive website. But then you can also build web apps now. So you can literally be like, "Yeah, build a Python web app that does XYZ," and the design is getting so good because it uses Claude 4. You can start saying, "Do it in the style of Stripe. Think this through, rewrite all the copy and the tone of David Ogilvy or Malcolm Gladwell," or whoever you like, and it does a really incredible job. So what I'm finding is that things that previously would've frustrated me because I would've had to rely on a team of five, I can just do entirely on my own when it comes to web projects and stuff like that. So I'm having a blast with that.

**Lenny Rachitsky** (00:49:07):
That's cool. So your team Replit versus Lovable, Bolt, v0, is that roughly right?

**Andrew Wilkinson** (00:49:12):
I mean, they're all great. I, just am the most familiar with Replit, and it seems like it has the most functionality. It has a lot of beef to it, whereas Lovable and Bolt seem a little more basic. You might have to deal with deployment and stuff in a fiddlier way with them.

**Lenny Rachitsky** (00:49:29):
Awesome. Okay. Anything else?

**Andrew Wilkinson** (00:49:32):
Another one I love, so Limitless. I don't know if you've seen this.

**Lenny Rachitsky** (00:49:32):
I've got one of those.

**Andrew Wilkinson** (00:49:37):
It's really cool. So clips to your shirt. I actually just put it on my pocket so no one actually notices that I've got it on usually, and I just have it on all day. And what it does is it just records everything I do. And then I can say, the other day I had a coffee with someone and I'm the kind of person where I go a mile a minute and I say, "Oh, I'll send you that and I'll do this," or whatever. And so at the end of the day, I can just say, "Hey, what did I promise to people today?" Or even better, everyone loves this use case, I have a fight with my girlfriend. And she says, "You didn't say that." And I'm able to say, "Well, actually you did." And you can query it as an LLM.

**Andrew Wilkinson** (00:50:16):
So you can say you're a couples counselor. Look at this fight. What did each person need? How did it start? What are the key lines? Where did it change? How could you do this better? And so honestly, from a relationship standpoint is probably where it's been the most useful. What's cool too is that has an API. And so eventually, I haven't gotten there yet, but eventually you'll be able to just have it record your entire day and then automatically go into your Todoist or whatever it is, and add to-do's or send emails or whatever you need it to do.

**Lenny Rachitsky** (00:50:48):
Wow, this is so fun. Or just per your point with the relationship, ask it, "What could I have done better today? Where did the day go downhill?"

**Andrew Wilkinson** (00:50:57):
Yeah, I saw a guy on Twitter. He said, "Every day I ask it, 'How could I have been a better dad?'" And it'll be like, "Oh, your daughter tried to get your attention about this. You should have paid attention," or whatever.

**Lenny Rachitsky** (00:51:07):
And that was with the Limitless data. Oh, man, I love that. With the relationship, you didn't go where I thought you were going to go, where it's like, "Oh, but who actually said it? What did you actually say?" I love that It became much more wholesome of just like how could we have communicated better holistically? Okay. My wife actually is like, "Don't wear this around."

**Andrew Wilkinson** (00:51:25):
You just got to wear it really covertly. Honestly, I know it sounds kind of cheesy because you think like, "Oh, it'll tell her," right? But in reality, usually you're both guilty in the fight. And my girlfriend has actually appreciated it because there's been half the time it'll be like, "Oh yeah, Zoe said this and it triggered this," but a lot of the time it's me too. And I think I'm totally right. So I find it actually really helpful. Other tools, I mean otherwise, it's the basic stack of Claude and ChatGPT and Gemini. And Gemini I use mostly for things that are really large. So for example, all my medical records, I have trained on Gemini 2.5 just because the other ones can't really go through it all. Claude, I use for writing very extensively. I find it's the best for that.

**Andrew Wilkinson** (00:52:14):
And then ChatGPT for everything else. In terms of cool things I do, recently, I went through my entire medicine cabinet and I just took big photos that showed all the different medicines and stuff. And now I can say something like, "Hey, I'm really tired and stressed out today. What supplement should I take and what dose?" Or I'll say, "Remember all the medications I'm taking, whenever I ask you a health question, always remind me in the context of I might be taking that medication." So a few times it's said, "Hey, you shouldn't take this other medication because your genetics say you're not compatible with it and you're already taking this other one." So stuff like that's been really helpful.

**Lenny Rachitsky** (00:52:57):
For that use case, do you come back to a specific conversation where you've given it that context, use a project or something-

**Andrew Wilkinson** (00:53:02):
No, I say remember. I just say remember these supplements, this is what I have in my medicine cabinet or this is what I take every day.

**Lenny Rachitsky** (00:53:10):
I see. So just tap into the memory of ChatGPT.

**Andrew Wilkinson** (00:53:12):
Yeah.

**Lenny Rachitsky** (00:53:13):
This is awesome. This makes me remember that. And this connects to other points you've been making, I've started to use ChatGPT deep research to prep for these conversations. I used to have a researcher who did this for 400 bucks. They did research on every guest that I had, four to 500 bucks, and gave me a whole doc, "Here's everything about them. Here's their background, here's questions you might want to ask." Deep research is better. Just as of recently, especially with o3-pro.

**Andrew Wilkinson** (00:53:39):
We were talking earlier about agents. I forgot one of my favorite agents. So it looks on my calendar 30 minutes before I meet anyone. It goes on to Perplexity and it does a deep dive on the person and it sends me, it goes into my email and it gets the context of the meeting. So often I book meetings three months out. So I usually look at my calendar and I go, "Who is this person? I don't even remember booking this meeting." It'll text me before and it'll say, "Here's who you are, here's your background, here's the email context of why we're meeting." And it basically does all the research for me. So I do the same thing. I also use deep research and say, "If I'm meeting them, you know everything about me. What are our commonalities and what are all the fun things we should talk about?"

**Lenny Rachitsky** (00:54:22):
Oh my God, I just want to plug and play all these agents you've built to use them myself. Interestingly, I tried to create an agent to do this research kickoff for me, but I don't think you can automate ChatGPT deep research. Okay.

**Andrew Wilkinson** (00:54:35):
Only Perplexity. Yeah.

**Lenny Rachitsky** (00:54:37):
Okay. Perplexity is great too, but it's different. Okay. So let me follow this thread I was going down, which is around job displacement. I know you think a lot about this. So with me, this researcher as a contractor I was working with for a few months, I no longer need them. Your assistant, you no longer need. Thoughts on just the impact we're going to see on jobs as a result of AI.

**Andrew Wilkinson** (00:54:58):
There's this great William Gibson quote, "The future is already here, it's just not evenly distributed." And I think that we are in the Palm Treo phase. Do you remember the Palm Treo?

**Lenny Rachitsky** (00:55:09):
No.

**Andrew Wilkinson** (00:55:10):
So when I was, let's see, in 2007 I think, was that when the iPhone came out? 2007?

**Lenny Rachitsky** (00:55:17):
I know [inaudible 00:55:19].

**Andrew Wilkinson** (00:55:18):
Something like that. In 2007, there was this device called the Palm Treo and it was like a little PDA with a stylus and it had a little modem attachment. And so what you could do is you could get your email anywhere and that was this shocking thing. It had a black and white or barely color screen and you could go in and email people. And I remember walking through the mall buying shoes while answering business emails and going, "This is the ultimate freedom. This is incredible." But the problem with that device was it sucked. You had to use a stylus, it's not a good user experience. And then the iPhone comes out and you're like, "Okay, this is what I was waiting for. This is the real thing." But I got a little preview with the Palm Treo and so right now people like me who have the time and skillset to nerd out and build the agents can do this. It's just not accessible to everybody.

**Andrew Wilkinson** (00:56:14):
And I think that if it was accessible to everybody in terms of if you could just open ChatGPT and say, "Hey, ChatGPT, I run a business, can you help me?" And it started asking you questions and it said, "Oh well you do sales. Do you want me to set up a sales agent? Okay, great, plug in your HubSpot API, now tell me a few things and then give me access to all your data." And then before you know it, it's just a digital employee that's on the other end of the phone on ChatGPT just like in the movie, Her, that you can talk to and it can go do stuff. I think that is kind of the iPhone moment that we're going to have in some time in the next five years. And so I think that if AI doesn't progress, we will see some serious job displacement, like what you already mentioned, the translators, the researchers, the admin, some assistant jobs, depending on the kind of work they do, those jobs will definitely be massively affected.

**Andrew Wilkinson** (00:57:13):
But I think pretty quick, all knowledge work jobs could be affected if the models scale in the way that they say they're going to. I mean if you trust Sam Altman and Dario Amodei, I mean Dario Amodei said by, "2027, our models will be smarter than all PhDs." So in any subject. That is a staggering statement. And he's been a very conservative kind of almost fearful voice in the AI world. Very cautious to say things like that. And so when he said that, I perked up and started listening carefully. I don't know that that's how it's going to play out. He might be trying to fundraise, we have to take it with a grain of salt. But if that's true, then I think so many knowledge work jobs are going to change massively.

**Lenny Rachitsky** (00:58:09):
Okay, there's a lot here. I actually had Mike on the podcast, their CPL, Mike Krieger, and he pointed out that Dario, every prediction he's had so far has been right over the past few years. And so there's a reason to listen to his insights. Okay. Andrew, I know you're not going to have all the answers for people. But let me ask you kind of a two-part question for new grads and for people currently in the workforce say like, I don't know, senior product manager advice on what they should do, where should they focus? What skills will matter most? What jobs do you think will last? What advice can you give us?

**Andrew Wilkinson** (00:58:43):
Everybody 10 years ago, do you remember there was this whole movement that we should teach kids how to code? Remember that?

**Lenny Rachitsky** (00:58:49):
Mh-hmm.

**Andrew Wilkinson** (00:58:51):
Now teaching kids how to code is teaching kids Basic and we have a GUI, right? You don't need to learn punch cards and basic programming or MS-DOS because we all have keyboards and mice. We don't have to do that. And I feel like coding has really gone away because of that. And I think now people are saying people need to learn how to prompt, which I think is very true for the tool set right now. But I don't know how relevant that is in the future because I think we're still, again, in the Palm Treo phase, I think that the AIs will be very good at eliciting what the actual problem you're trying to solve is and reinterpreting it in a way that it can optimize the query to get the right answer and solve the problem.

**Andrew Wilkinson** (00:59:38):
And so I think the fundamental question is do all jobs just become a single prompt? For example, does a CEO just grow the business while making the customers happy and turning a profit or something like that? And it is able to actually be an omniscient presence that can run a whole company. Now that's a big I'm someone building AI agents. I can barely get it to reliably do calendar entries and that kind of stuff. So I don't think that's totally imminent, but I can certainly see a world where that's coming. And so the question is, what is one to do when that reality is barreling down at you? For people who have resources, I think there's a lot of things they can do in terms of investment, they can invest in the companies that'll benefit from this, companies that have a lot of compute or energy or that sort of thing.

**Andrew Wilkinson** (01:00:29):
So that's one thing, but that's not really the question. The question is what do you do if you're a smart 18-year-old or 19-year-old or whatever? In my opinion, I think the best thing to do right now is to get incredibly good with these tools and utilize them to build enough wealth that you can put the money into diversifying into compute and energy. And I do think that while everybody thinks that all jobs will go away in five years and robots will be everywhere, people generally overestimate things in the short term and underestimate them in the long term. So I think there's going to be this long window where robotics is not anywhere near good enough to even do raking leaves and stuff. And I think there's going to be a ton of opportunity for people to just spin up new businesses that never existed before.

**Andrew Wilkinson** (01:01:20):
I was talking to some friends and we were like, "What do you do in a world of abundance where everything is really cheap and companies operate almost autonomously?" I think there's a lot of weird skillsets that we can't imagine right now. For example, just being funny, right? Being funny to hang out with them. Think about OnlyFans, right? Right now, there's this whole idea of OnlyFans where people are paying for comfort and connection, but in a romantic sense. Imagine people did that day to day, they just love hanging out with funny people. So you have a funny guy who you chat with or comes over to your house or whatever it is, I'm making stuff up. Maybe it's a guy who comes to play pickleball with you or something, I don't know. But I can imagine that in this weird world, there's all these weird new jobs.

**Andrew Wilkinson** (01:02:06):
So I think my take is things will either be totally fine or they will be terrible and we'll all be dead. And I can't predict which those is there. I don't think it's worth thinking about all of us being dead. And so I think it's really just lean into what are the tools, how do you stay cutting edge? How do you build businesses and wealth in this new world? But I am certainly feeling a little bit like all of our brains could be defunct in the next 10 years. And that's kind of a scary thought.

**Lenny Rachitsky** (01:02:36):
I'm picturing this dystopian world where humans are just going around making jokes.

**Andrew Wilkinson** (01:02:44):
Where does status come from in a world of abundance? Where does status come from? Maybe it is just like being the best at hiking or something.

**Lenny Rachitsky** (01:02:52):
Well, let me ask you this because this is where it gets even more real is just with your kids. What are you encouraging them to learn? What are you encouraging them to get good at? Because this is the real problem a lot of people are having right now is what will matter in the future. Obviously we don't know, but what are you encouraging them to focus on?

**Andrew Wilkinson** (01:03:09):
Well, I don't know. I mean there's so little, they're five and eight. And so really what I'm focused on is making sure they're socialized and they're polite, very basic. So are they comfortable talking to adults? I often send my son, whenever he wants something at a cafe, I always make him go up, ask nicely, pay himself, all that kind of stuff. But at this point I feel like for the first 10 years of life, you're just trying to make sure that they're not traumatized or rude. And I think it's going to be really interesting when they're 10 plus where they're really aware of this stuff and can start building with these tools. And to be honest, I just don't know. I think I just want them to lean into whatever they're passionate about and go from there. But I have no idea. And to be honest, it's not something I'm worrying about because it's too hard to worry about. I don't know, it's too multivariate. There's too many ways it can go.

**Lenny Rachitsky** (01:04:03):
I feel like a core part of your message is just get good with these tools. There's this classic quote, "AI won't replace you, it'll be somebody very good using AI is going to replace you," at least for a while. And so it feels like a big part of this is just use these things. Before we started recording, we were having a mic issue and I love you just went straight to ChatGPT, "ChatGPT, how do I connect this mic to reverse side and make it work?" And I feel like that's just a habit to build is just like go to ChatGPT whenever you have a problem like that. I've been doing that a lot. I was connecting a subwoofer from what wire do I need to buy to connect the subwoofer to the receiver?

**Andrew Wilkinson** (01:04:37):
Well, the best part about ChatGPT is I used to just get stuck and I have ADHD. So when I got stuck, let's say, what's an example? I was redoing my home IT security. And what will happen is I'll be in the terminal and I'll enter something and then something goes wrong and it just keeps erroring. And then I'm like, "Well, I'm out." I have ADHD, I can't tolerate this. I just lose the thread completely. And ChatGPT allows me to stay on track and go so much further and faster than I ever could before. I love not being artificially stopped in your tracks and being able to continue the flow state as a result.

**Lenny Rachitsky** (01:05:17):
And also I noticed you were using voice mode, which is what I've been using more and more. And I feel like that's something a lot of people sleep on is just the voice mode where you could talk to it and not have to sit there and type your questions. Okay, so let me go in a different direction. You've been spending a lot of time, you wrote a whole book about this, of just your journey from, you call it Barista to Billionaire where you started serving coffee, then you ended up being very successful, then realize that didn't make you happy. You're quite unhappy with lots and lots of money. I think a lot of people hear these stories, they see this stuff, they're like, "Yeah, okay, I sort of get it." But they still do the same sorts of things. They still assume they will be happy once they reach that next goal, that next title, make certain dollar amount. What can you share? What have you learned about just what it actually takes to be happy that you think people are still not really recognizing and still kind of are confused about?

**Andrew Wilkinson** (01:06:10):
Do you remember when you're in your early 20s? I remember when I was in my early 20s, I'd always be like, "I need to move away to Europe. I need to go discover myself." And I'd be really anxious and stressed out and in an existential crisis and then I'd fly to Europe to go backpacking or something and I'd still feel anxious and have my existential crisis. It's just now I'm in Europe. And I think the reality is that whatever's in your brain, whatever that anxiety loop is, doesn't go away just because you have a bigger house or more money in the bank account or you're in Bali or wherever it is, that's a chemical reaction happening inside you that relates to your past and your DNA and all that stuff that creates that soup. And for me, I've always been a very anxious person. I've always worried about tomorrow, I rarely enjoy today, I'm always worrying about what could go wrong in health or my businesses or with my family or whatever it is.

**Andrew Wilkinson** (01:07:12):
And so when I started my business, all I thought was I just don't want to work for someone else. If I can just wake up in the morning, do my own thing, make 60K a year, then I'll be happy. So I got that and then it was once I make a million dollars a year and I can buy a house, well, I got that and so on and so on and so on all the way up to at one point being worth over a billion dollars, by the way, should redo the title of the book from Barista to Billionaire, now it's former billionaire because our stock went down, but it didn't change anything. I'm still just as anxious as ever. I remember a month ago, I had this day where I was sitting in the sauna and I was stressing about a bunch of business problems and money things and I stopped myself and I was like, "Wait a minute."

**Andrew Wilkinson** (01:08:04):
10 years ago I had all the same thoughts and I was stressed out about similar kinds of problems and our revenue was $20 million bucks or $15 million bucks. Now we're at almost 300 million and I'm still stressed about the same things. And the saddest part was as I met more and more wealthier people all the way up to multibillionaires, I realized they were all still comparing themselves to their peers, competing over who has what and tracking stuff and still unhappy that anxiety loop or depression was still in their brain. I remember I had this moment where I met this guy who is a single-digit multibillionaire, and he goes, "Oh, Jeff Bezos, he's just so fucking rich." And I was like, "Wait, what do you mean you're worth $5 billion? What can he do that you can't?" And he goes, "He can buy a super yacht." And it's like, "What?"

**Andrew Wilkinson** (01:09:09):
You can look at that guy and you can say, "Oh, that's a wacko and all the other stuff." But the reality is that we're all just comparing ourselves to our peers, whatever our peers have that we don't feel hard done by. And I remember just thinking like, "Oh my God, how do I avoid becoming this?" And that's the scary reality is no matter how lucky you are, I mean we're also fortunate. For me, I live in Canada. I have every opportunity that I ever could have hoped for. And there's people all over the world who have nowhere near what I had growing up. And yet I felt hard done by because in my neighborhood I was the poor kid. I wasn't poor globally. I wasn't even poor based on Canada. But all my friends had big screen TVs and would go to Hawaii and I never got to do any of that stuff. My parents were stressed about money. So I feel hard done by.

**Lenny Rachitsky** (01:10:05):
There's a phrase of friend's financial advisor once shared, "The Joneses are doing really well," and that's hard to get over. So what changes did you make to just be happier? It's hard to give up money. It's hard to give up more money you could make. It's hard to give up things that money buys, I guess just so what did you change?

**Andrew Wilkinson** (01:10:28):
Well, there's a few things. I mean, one was reframing money to some degree to have it be something that wasn't just about feathering my own nest and making it better. So for example, I was talking to a friend who's very wealthy and I was saying all the feelings I described of have all this and yet I don't feel better. And he said, "Well, what are you working for?" And I said, "Well, I guess I'm just working for the numbers to get bigger and have more employees and that's nice. Our businesses do good things in the world and help more people and stuff, but ultimately it's just to pile up cash." And he said, "I've reframed it so that all my money, 90% of my money plus, goes into my philanthropic foundation and then it gets given away to great causes. So when I think about working really hard or losing a deal or an investment going well, the win isn't for me. The win is about doing something good."

**Andrew Wilkinson** (01:11:26):
So I did that and that definitely helped. I also stopped spending as much money. I found that the more stuff I had, the more houses I had, the more people that directly worked for me to manage all that stuff, the less happy, I was owned by my stuff. And then I work really hard not to have people... I try not to beat people over the head with it. I drive a normal car. I don't bring people to my house until I know them really, really well. I meet them on neutral ground, I meet them in a cafe. I dress like a schmo. I just try not to be a weird out of touch rich person. But the biggest thing, which is a weird answer, is actually medicating myself. So basically all my life I've been really anxious.

**Andrew Wilkinson** (01:12:18):
And in 2020, I remember I was watching a movie with my ex-wife and I was looping on, "I need to respond to this email. If I don't respond to that email, this person's going to be mad." And then I realized that I was watching a movie for the last 20 minutes and I didn't know anything about what was happening in the movie. And then I was going, "Oh my God, my wife is going to be mad at me. We're going to get in a fight," and I'm projecting all this negative stuff. And in that moment I was like, "Okay, maybe I do need to try an SSRI or something. This is not sustainable." And so I went to my doctor and I got prescribed with one, and I remember I was so scared of it that I cut it into 10 pieces. I said, "I'm just going to take a tiny little piece."

**Andrew Wilkinson** (01:13:01):
And it took me six months after that to try any. But I started taking these pieces every day and I just noticed that within about three or four days, it was like someone had turned down the volume on the nasty voice inside of me doing all that. And for the first time in my life I felt relief. No amount of money or success or attention or anything else had done what this little tiny yellow pill could do for my mental state. And so now I've been on an SSRI for four and a half years or something like that. And then I also started medicating my ADHD. And I would say that my brain went from Times Square to a quiet library. And I think a lot of people are scared of that, but ultimately most people are acting out something that's going on inside of them. There's a trauma or something medically going on and that's creating this anxiety, this depression, this feeling and addressing that is so much more important than trying to get praise from the external world.

**Lenny Rachitsky** (01:14:05):
Wow, it's really powerful. I really am really thankful you're sharing this. The advice most people hear, the general, I don't know, culture is just like, "No, don't take meds. Don't medicate yourself, I don't know, there's all these downsides, all this risk. The pharma industry is trying to make money." I love this other perspective of it. It's actually really effective and actually makes a big difference in your life and it may be the only solution.

**Andrew Wilkinson** (01:14:29):
Well, it's funny because if you have a headache, I don't think many people say, "Oh, don't take Tylenol, you're benefiting Johnson & Johnson, big pharma." They go, "Yeah, you take a Tylenol when you have a headache." Or a lot of people have seasonal allergies and they take antihistamines and they forget that histamine is not just in your nose, histamines in your brain and everywhere in your body. And so when you take an antihistamine or even a Tylenol, you're changing your brain, you're changing the way you think, you're changing everything. And yet for some reason, people are so scared, myself included, of taking an SSRI or similar medication to treat something that is ruining their life. Anxiety was ruining my life. Every minute of every day I was convinced that something terrible was about to happen. And even now medicated, I still have that voice. It's not like that frankly beneficial voice that helps you grow your business or whatever becomes unproductive at a certain point. And I think that it is wonderful to be able to actually just get a little bit of solace from it.

**Lenny Rachitsky** (01:15:38):
Well, for folks that, I mentioned many folks listening, are like, "Oh man, I should really explore this," what's the best way to explore is to talk to your primary care physician?

**Andrew Wilkinson** (01:15:47):
Yeah. I think talking to your doctor, and one note I think a lot of people miss is when I was so scared of side effects because you hear about all these horrific side effects of people lose their libido and feel terrible and stuff. And so I looked into it and a lot of the side effects are due to the way that people metabolize different drugs. And so you can actually do your 23andMe or similar DNA test and you can find out how you metabolize different SSRIs. And then I chose one that I could metabolize well. And so I've had no side effects or problems. And I know every time I mention this, I get a lot of people that say exactly what you say, big pharma and all this other stuff. But I just ask everyone listening, if you take a Tylenol or you take an antihistamine or other medications, I don't know why something that can be so profoundly helpful is something that people need to be afraid of.

**Lenny Rachitsky** (01:16:40):
Amazing. Andrew, we've covered so much ground. I love the spectrum of topics we've gone after. Is there anything else that you wanted to share? Anything you wanted to double down on? Any last nugget you want to leave listeners with before we get to our very exciting lightning round?

**Andrew Wilkinson** (01:16:56):
One thing that was shocking for me was I was doing... My doctor asked me to get a cognitive test, not because I had any problem, but he said, "Look, it's really important to get a baseline in your 30s because then as you age, we can make sure that you're not getting Alzheimer's or dementia or whatever." So I go and I do this test with a neurologist and she asked me a bunch of questions. It's, "Remember these 10 digits and say them backwards and all this stuff." And finally when I get my test results back, she said, "Look, your crystallized intelligence, your ability to remember stuff, long range is totally fine. You're above average. Your short-term memory, your working memory is 10th percentile, very, very bad. And you probably should get checked for ADHD."

**Andrew Wilkinson** (01:17:47):
And I was like, "Nope. There's no way. I don't have ADHD. I'm organized. I've got to-do lists. I built a business. I wasn't the hyper kid in school or anything like that." And she said, "Well, look, just humor me. Just go get tested." And so I went through this crazy process. You can go online and just quickly get tested or whatever. But I went through this really, really five-day process to get tested. And it turned out that I do have it. And it's really interesting because if you take the normal population of about 5% of people have ADHD, but about 30% of entrepreneurs have ADHD. And many entrepreneurs I talk to, they love new things. They love jumping around between a million different topics. They're an inch deep and a mile wide like I described. And they describe themselves as unemployable. And frankly, I remember I was so skeptical of ADHD that when my girlfriend told me, a friend of hers got diagnosed with it, I sarcastically said, "Oh good, she better take some meth," talking about stimulant drugs.

**Andrew Wilkinson** (01:18:54):
Previously I was very, very skeptical of ADHD, but it is a real brain disorder if you actually dig into it. It's a very real thing, very objective that people have poor executive function. And for me, it's given me a lot more empathy for myself and my behaviors because at work, if you have ADHD, you can delegate to other people and build systems to get around your disability. But at home, that's where it really plays out. And so for me, it's like my girlfriend asked me to take the garbage out, and I say, "Of course." And then I forget three times in a row. And she views that as you are being very hurtful and you're not caring for me. And it causes all sorts of different problems at home.

**Andrew Wilkinson** (01:19:37):
And so for me, not only did it make my work life better because I was more focused, but it actually really helped me in my personal life and it made me feel not broken. I always felt a bit broken. I couldn't put my finger on why, but I just went, "I'm not good at doing all the things everyone else seems to be able to do no problem." And so I think especially because you have an entrepreneur audience, I think a lot of entrepreneurs should consider getting checked for ADHD. Just go and ChatGPT say, "Ask me a of questions and tell me if you think I might have it." But I was very surprised, and that's been really impactful for me too.

**Lenny Rachitsky** (01:20:13):
Man, this garbage story is very descriptive of my life, and so I'm going to do this. There's this implication that it's good to know this, right? It feels scary to feel like, "Oh, shit I might get diagnosed with ADHD." Your advice I imagine is it's better to know because you could do stuff about it.

**Andrew Wilkinson** (01:20:33):
Totally. I think it's like any medical thing, right? It's like there's an Alzheimer's gene, right? And if you know you have it, there's a lot you can do to slow the disease or prevent maybe getting it or whatever. It's always better to know. Even if you don't medicate, I think there's a lot of lifestyle interventions, dietary interventions, and just things you can do to make your life easier, even if it's just explaining to your partner, "Hey, I need things to be explained to me this way so that I can be effective in loving you."

**Lenny Rachitsky** (01:21:05):
Beautiful. Okay. Well, Andrew, with that, we've reached a very exciting lightning round. I've got five questions for you. Are you ready?

**Andrew Wilkinson** (01:21:11):
Let's go.

**Lenny Rachitsky** (01:21:12):
First question, what are two or three books that you find yourself recommending most to other people?

**Andrew Wilkinson** (01:21:17):
If I could only recommend two, I would recommend The Laws of Human Nature by Robert Greene. It's basically like a compendium of every single type of personality disorder or psychological effect. How do people work their minds to make bad decisions? What is a narcissist? What is a psychopath? All that kind of stuff. And Robert Greene does such a great job of telling all that in a really engaging way by using storytelling. So every single chapter has a story about someone with that personality disorder. So I love that book.

**Andrew Wilkinson** (01:21:48):
The other one that I love is a book by Felix Dennis called How to Get Rich. It's hilarious because the whole book starts out saying, "I got rich and I regret it, and I wish that I'd become a poet." And instead he went off and he built this huge publishing empire, became a poet in his sixties, and then he died of throat cancer. So it's his story, he basically says, "I'll tell you how to build the business even though you probably shouldn't." And I read that when I was in my 20s, and I found it so exciting and inspiring, and then I recently reread it and I was like, "Oh my God, why didn't I listen to this guy?" Everything he said came true.

**Lenny Rachitsky** (01:22:23):
Next question, do you have a favorite recent movie or TV show that you have really enjoyed?

**Andrew Wilkinson** (01:22:27):
I love Challengers. I don't know if you've seen it, but it's amazingly acted, amazing cinematography. It's like this amazing romantic triad tennis movie. Really, really good.

**Lenny Rachitsky** (01:22:40):
Do you have a favorite product you recently discovered that you really love? You already mentioned Limitless. Is there anything else?

**Andrew Wilkinson** (01:22:45):
I'm one of those people who really believes in the idea of a robot vacuum. I've probably spent $10,000 on different robot vacuums over the years, probably five or six of them. And I finally got one that actually works. It's called the Matic Vacuum, M-A-T-I-C. And basically, I think it's like former Google Engineers basically built like a mini Waymo car. So it has machine vision and it will avoid absolutely everything and never get tangled, and it can mop your floors and vacuum. I've been super impressed with it so far.

**Lenny Rachitsky** (01:23:16):
I've got one of these myself. Completely agree. My wife loves it, which I did not expect. She's like, "Turn that Matic on." It's very delightful. It has his little personality and it just does a really good job. And I'm not an investor, and that's awesome. I'm a huge fan. Okay, next question. Do you have a favorite life motto that you often come back to find useful in work or in life?

**Andrew Wilkinson** (01:23:38):
Yeah. My favorite is a quote by Jerzy Gregorek. It's, "Easy choices, hard life. Hard choices, easy life." And I often find that anytime that I'm making the easy choice, my life gets hard. And when I make the hard choice to shut down the business, fire the person, say bye to a project that I was excited about, it's almost always the right choice and it makes my life a lot easier.

**Lenny Rachitsky** (01:24:07):
Final question. Okay, so let me know if this is true. I found that when you were a teenager, when you're running Macteens, you interviewed Steve Jobs at a conference. Is that true?

**Andrew Wilkinson** (01:24:18):
So when I was a teenager, I had a tech news website and I emailed the Apple PR people and I said, "Hey, could I interview Steve Jobs?" And they kind of laughed at me. They're like, "Yeah, there's no way. You're not going to interview Steve Jobs, but hey, why don't you go to this tour of the Apple Store?" So this is no other Apple Store has existed. This is in New York at Macworld back in 2004, no, 2003, something like that. And so I was really excited about that. So I show up to go do this tour and there's 30 other journalists there, and I'm first in line, and this big black SUV pulls up and this man gets out and the man is wearing gray New Balance, little circular John Lennon glasses, the mock turtleneck. And my brain finally puts it together and I realize it's Steve Jobs.

**Andrew Wilkinson** (01:25:11):
And he walks directly up to me because I'm first in line and he shakes my hand and he says, "Hi, I'm Steve." And I'm like a quivering mess. I'm 17 years old. He is my hero. He's my Jesus. And so I was just so high on adrenaline that I just started asking him questions. As soon as we walked in, I basically just stayed at his side, and I just picked his brain about all sorts of different stuff. So it's not like I got to ask him questions about life. I was asking him about the new 17-inch iMac or whatever. But it was a pretty cool experience to get to meet him.

**Lenny Rachitsky** (01:25:46):
That is awesome. There's such a funny thread recently with guests. Everyone, there's like a Steve Jobs connection for the past couple months of guests. That's an awesome story. It just shows that it wasn't luck. You made this happen. You got there first in line.

**Andrew Wilkinson** (01:25:59):
The lesson for me was ask big and maybe you'll get something great. Ask for amazing, you'll get something great, which I did.

**Lenny Rachitsky** (01:26:07):
Yes, door in the face technique, I think people call it.

**Andrew Wilkinson** (01:26:11):
Yeah.

**Lenny Rachitsky** (01:26:12):
Andrew, this was incredible. Two final questions. Where can folks find you if they want to reach out? I know you're buying companies, investing in companies. Just talk about what you're looking for there in case people are like, "Oh, this is me." And then finally, how can listeners be useful to you?

**Andrew Wilkinson** (01:26:26):
Yeah, my business is Tiny, T-I-N-Y.com, and we buy businesses. Basically, when we sold the business, we hated it. We went through this process where we talked to all these douchebags that ran private equity firms that would show up in our office in suits and use a bunch of words we didn't understand. And we ended up going, "Man, why can't we just sell to somebody who's like us?" And so we basically started Tiny to become the buyer we wish we could have sold to. And we're looking for businesses that have some of the qualities I mentioned before. So really high quality businesses that do something positive in the world and have happy customers, happy employees, and some kind of competitive advantage that will make them continue into the future. So we own AeroPress, Letterboxd, Dribble, which is a design social network, whole bunch of businesses like that. So if somebody is thinking about selling their business, definitely get in touch.

**Lenny Rachitsky** (01:27:29):
Amazing. Okay. And then how can listeners be useful to you if it's not just that?

**Andrew Wilkinson** (01:27:33):
I just would say if there's anyone really interesting coming to Victoria, Canada, which is where I live, send me an email and would love to have coffee with you. The reason I love going on podcasts is because I get to meet so many interesting people, both randomly. If I'm in a cafe and someone sees me and says hi, I often make a lot of friends that way, or if someone comes to my hometown. So yeah, just get in touch.

**Lenny Rachitsky** (01:27:56):
Amazing. Andrew, thank you so much for being here.

**Andrew Wilkinson** (01:27:59):
Yeah, thanks, dude. That was fun.

**Lenny Rachitsky** (01:28:00):
Bye, everyone.

**Lenny Rachitsky** (01:28:03):
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.

---

## [2/18] The AI-native startup: 5 products, 7-figure revenue, 100% AI-written code | Dan Shipper (co-founder/CEO of Every)

**Lenny Rachitsky** (00:00:00):
The business you're building, the team you're building, the way you're operating is the very bleeding edge of how companies are trying to operate in this AI era.

**Dan Shipper** (00:00:07):
We have a head of AI operations. She's just constantly building prompts and building workflows that I and everyone else on the team are just automating as much as possible.

**Lenny Rachitsky** (00:00:16):
What are some things that you believe about AI that most people don't?

**Dan Shipper** (00:00:20):
I hate the headlines that are like, "Entry-level jobs are taken away by AI." Whenever I see a kid with ChatGPT, I'm like, "Holy shit, they're going to go so much faster than any other person that I've worked with." We have this guy, he made a year's worth of progress in two months because every time I sat down with him and told him, "Okay, here's how you tell a story, here's how you think about a headline," he recorded all of it, put it into a prompt, and he never made the same mistake twice.

**Lenny Rachitsky** (00:00:40):
There's this sense we're getting to a place where you don't have to write any code, you have a product team not writing code at all.

**Dan Shipper** (00:00:46):
No one is manually coding anymore. Organizations like ours, people who are playing at the edge, we're doing things that, in three years, everybody else is going to be doing.

**Lenny Rachitsky** (00:00:55):
Today, my guest is Dan Shipper. Dan is the co-founder and CEO of Every, which is a company that is at the very bleeding edge of what is possible with AI. Their team of just 15 employees has built and shipped four different products. They publish a daily newsletter, and they have a consulting arm that helps companies adopt the latest AI best practices. On their product team, their engineers don't handwrite a single line of code and instead use an arsenal of agents who help them craft requirements and build their products.

**Lenny Rachitsky** (00:01:22):
Their editorial arm uses AI to publish better work faster, and they even have a person whose entire job is to help every employee at the company become more efficient using the latest AI workflows. In our conversation, Dan shares a bunch of tactics that they use internally to increase the leverage of their own employees, his personal AI tool stack, the one predictor that he's found for whether a company will successfully find huge productivity gains through AI, how he's building his company in a really unique way, a bunch of predictions for where AI is going, and so much more.

**Dan Shipper** (00:04:11):
Thank you for having me. I've obviously been a huge fan for a long time and so it's an honor to be here.

**Lenny Rachitsky** (00:04:16):
It's my honor, Dan. I feel like this is a podcast that was meant to be. I'm so happy we're finally doing this. There's so damn much that I want to talk about; there's so damn much we can talk about. I thought it'd be fun to start with just some hot takes.

**Lenny Rachitsky** (00:04:29):
And the reason I want to start here is I feel like you spend more time thinking about AI, building with AI, using AI, evaluating AI than anyone else I know nearly. And so I really respect your insights and your perspectives on where things are going. So let me just ask you this question and see where this goes. What are some things that you believe about AI using AI tools that most people don't believe?

**Dan Shipper** (00:04:55):
I'm going to go with my hottest take, and this is the take that I have the least evidence for. So let's just start with that. I have other more well-reasoned takes to give you, but this is my hottest one, which is I think that AI may be one of the biggest force for reshoring American jobs. And so I think everyone is worried about it unemploying people. And for sure, it will change the skills needed to do the jobs that you're doing, but I think it may actually reshore a lot of jobs, and it'll do that in two ways.

**Dan Shipper** (00:05:27):
One is, there are a lot of expensive services that rich people and big companies are paid for right now, so in-house counsel or call center or whatever. And what cheap intelligence does is it makes those kinds of things affordable for small companies and individuals. So it stimulates demand. The other thing that it does is it allows people who are in those jobs to serve more people cheaply. It may not get rid of customer service, for example, but it may allow 10 people in the Midwest, who would normally be working at a call center, to serve hundreds of thousands or millions of people. Maybe that's too much, but a lot more people than they would ordinarily if they were the ones on the phone all the time.

**Dan Shipper** (00:06:22):
And so it becomes much more cost-effective for American companies to hire people in the US. And I think the people in the US are going to be better, in a lot of cases, at using these AI tools to do work. So I think it may actually make it more effective to have those jobs in the US run by people sitting in the US who are using it to get work done. And also, the model companies are here too. So there's a lot of American stuff happening, and you can decide whether or not you think that's a good thing, but I think it's quite lost in the conversation over whether AI will get rid of jobs.

**Lenny Rachitsky** (00:06:58):
I like optimistic takes about AI, so this is great. And to your point, TBD if this is good for other countries, but good for the US. What else? What else you got? What other hot takes?

**Dan Shipper** (00:07:10):
Okay. Another big hot take, and this is less contrarian and more just, I think, people are truly sleeping on it. I think people are truly sleeping on how good Claude Code is for non-coders. And I'll extend this to not just Claude Code, but Google just came out with the Gemini CLI command-line interface. So things like that. And I'll tell you for people who are listening that don't know what Claude Code is. Claude Code is just the command-line interface. It's those black terminals that programmers use. It's a command-line interface that you can boot up. It has access to your file system, it knows how to use any kind of terminal command and it knows how to browse the web, all that kind of stuff.

**Dan Shipper** (00:07:47):
You can give it something to do and it will go off and it'll run for 20 or 30 minutes and complete a task autonomously, agentically. Especially with Claude Opus 4 that just came out, it's this gigantic leap forward in AI's ability to work by itself. And Claude Code can even spawn multiple sub-agents that do a bunch of tasks in parallel and it's incredibly useful for programmers. Everybody inside of Every is using it all day, every day. Everyone's agent-pilled. They've got 15 agents doing all this kind of stuff. It's crazy.

**Dan Shipper** (00:08:18):
But non-programmers don't use it because it's intimidating to use the terminal. But for example, you can download all your meeting notes and put it in a folder and just be like, "Okay, I want you to read every single one of my meeting notes and tell me..." Something that I do, for example, is, "Tell me all the time that I subtly avoided conflict."

**Dan Shipper** (00:08:38):
And it writes a little to-do list for itself. It can have a little notebook, it can go and read each little thing and then write into its notebook, go down its to-do list and give you a summarized answer over multiple turns. So it's not just stuffing everything into context, which is what you'd be doing with ChatGPT chat or a regular Claude chat. It's actually processing every single file that you give it. And so I think it's incredibly powerful for any kind of task that involves processing a lot of text.

**Lenny Rachitsky** (00:09:06):
So as a simple way to think about this, you basically have an agent on your local computer that can read your local files and do your bidding.

**Dan Shipper** (00:09:14):
Yes, exactly. And it can do that for long amounts of time without going off the rails.

**Lenny Rachitsky** (00:09:21):
Interesting. And so there's a small hurdle that non-technical people have to overcome, which is using their terminal and giving commands, but once they get it running, it's just you talk to it in English and ask it to do stuff.

**Dan Shipper** (00:09:32):
Exactly.

**Lenny Rachitsky** (00:09:33):
So the hot take here is just Claude Code, which most people think is for engineers, is the most underrated tool for non-technical people.

**Dan Shipper** (00:09:42):
Yeah, exactly.

**Lenny Rachitsky** (00:09:43):
What are some other ways you imagine people seeing this? This meeting note example is really cool and I could see people using this. What else have you seen or thinking about?

**Dan Shipper** (00:09:52):
Something that I've done a lot, so I'm a writer for a lot of my job. And I know you're going to ask me about books I love, so I'm going to give you a sneak peek, which is I love War and Peace. I just read it for the third time.

**Lenny Rachitsky** (00:10:03):
Wow, that's a long book.

**Dan Shipper** (00:10:06):
It's so long, but it's so good. I think Tolstoy is a brilliant writer. And one thing that I wanted to do is I was like, "I want to inflect some of my writing with some of Tolstoy's style." And the way I did that is I think he's incredible at these little subtle sentences where he shows you what a character is thinking and feeling just by how they behave, how they move their face or the mismatch between the intonation in their voice and the expression in their eyes, all that kind of stuff. He's just an incredible student of human behavior and psychology.

**Dan Shipper** (00:10:36):
And so I just downloaded War and Peace to my computer, which you can do because it's public domain. And then I had Claude read the first three chapters of War and Peace and pull out all of those descriptions, and then make a guide for itself for how to do character descriptions like Tolstoy. And you could totally do this with a regular Opus command, but you couldn't put all of War and Peace into it. It would take a lot more hand holding to get it to do this. And it just did this by itself without my really intervening.

**Dan Shipper** (00:11:05):
I had it download a Russian version of War and Peace and the English version, and then start comparing different scenes that I love to tell me about things that I might've missed in the translations, so that you can get as deep and weird and nerdy for whatever subfield you care about as you want to. Same thing for if you've got tons of customer interviews or tons of customer data you want to go through, it's incredibly powerful for going and figuring stuff out from big data sets like that.

**Lenny Rachitsky** (00:11:30):
You actually inspired me to use... This is not what you're describing, but it's also something that's very cool. This is going to sound so nerdy. I'm reading Anna Karenina right now.

**Dan Shipper** (00:11:40):
Yes.

**Lenny Rachitsky** (00:11:41):
Also Tolstoy. And this is recommended by a previous podcast guest. And so I was like, "All right, I got to read this." Also very long. I'm on my Kindle, I'm just like, "All right, 13% in, I've been reading for months."

**Dan Shipper** (00:11:52):
Hot take, I think War and Peace is better than Anna Karenina, especially for a tech person. But they're both good.

**Lenny Rachitsky** (00:11:57):
Okay, there we go. There's my year. I saw you tweet this use case that I love that I've been using, which is just while I'm reading, having ChatGPT voice sitting around and then just asking it questions. Because you don't actually have to feed it the book, it knows the whole book. And Anthropic just shared this. I don't know if they shared or someone found this in their legal briefings that they actually bought tons of books and scanned them themselves, is how they did fair use.

**Lenny Rachitsky** (00:12:23):
And so it has all this context. So just sitting there and asking it, "What the heck is this thing in Russian society?" is super fun. Okay, so this is awesome. So the tip here is just coming back to your hot take. The tip is you basically can have an agent using local files and doing all kinds of cool stuff on your computer versus having to upload it into projects or into your prompts and things like that. Super cool. So the bet here is that people are going to discover this and start using this just day to day.

**Dan Shipper** (00:12:53):
I think they absolutely will. And I also think probably the model companies are going to start making this more accessible. I think one of the things that will just come from Claude Code and other things like it into everything else you use, whether it's on the web or wherever, is all of the original AI apps were pasting a chat box into an existing UI. So you've got Copilot, it's got the auto-complete in the IDE. You've got Cursor, it's got a little sidebar with a little chat. And the difference with Claude Code is you never look at the code. It's not meant for coding, it's not meant for coding by hand.

**Dan Shipper** (00:13:34):
It's meant for you to say, "I want you to get something done," and it goes and does it. And I think we're just getting to a point where for pretty much all the usual applications, AI is going to be good enough that we can get rid of the interfaces more or less where you're digging into all the things that it's actually doing and you're interleaved with its execution and you're more just like, "I'm delegating, it's going to go do it."

**Lenny Rachitsky** (00:13:58):
Yeah. I had Cursor's CEO, Michael Truell, on the podcast, and this is his big vision is, "What comes after code?"

**Dan Shipper** (00:14:05):
English.

**Lenny Rachitsky** (00:14:06):
Exactly. Exactly. I also just had the founder of Base44 on the podcast who built this company, sold for 80 million bucks to Wix. And he shared that he's been around for six months, the company. For the last three months, he hasn't touched a single line of front-end code, all Cursor and other tools he's using. So this is happening.

**Dan Shipper** (00:14:27):
Same thing for people inside of Every, no one is manually coding anymore.

**Lenny Rachitsky** (00:14:32):
Okay. Definitely need to talk about that. Before we do, any other hot takes that you want to throw out there?

**Dan Shipper** (00:14:38):
I have one other hot take, which is I have a definition for AGI. And so AGI is famously hard to define. What does it mean for it to be artificial general intelligence? The Turing test was one, but we'd pretty much blown past the Turing test in a lot of ways. So we have no good one. And so what I have noticed is that you can tell how much better AI is getting by how long a leash you can give it to do work.

**Dan Shipper** (00:15:09):
So with Copilot, you can tab complete and that was the beginning. With ChatGPT, you ask it a question and it returns a response and that's maybe slightly better than a tab complete. And then now with Claude Opus 4 and Gemini and all that kind of stuff, also with deep research, it can go off and work for 20 or 30 minutes. So that leash is getting longer where you have to intervene.

**Dan Shipper** (00:15:34):
And I was thinking about this and it reminded me of Winnicott, who was a child psychologist. He wrote this book called Playing & Reality. And his conceptualization for what it means to become an adult, what it means to go from being an infant to a child to an adult is when you're first born, you're effectively fused with usually your mother, your caregiver. There's no difference between you and her or you and whoever your caregiver is.

**Dan Shipper** (00:16:02):
And growing up is this process of being gradually let down in certain moments where you can handle being let down. So you learn that there's a separation between you and your caregiver. So for infants, it's instead of being fused at the hip for every hour of every day, you get left alone. Maybe you get left alone to cry it out. Who knows if that's the right thing to do with infants? A lot of consternation there. But that's teaching you that there's a separation between you and your mom or you and your dad. There's not going to always be someone to pick you up.

**Dan Shipper** (00:16:38):
And raising a child is about knowing when they're ready to be let down a little bit and have to stand up on their own. So I think there's that same leash with human development. You get longer and longer periods of time where you can be on your own. So we're still in the 20 to 30 minutes is maybe... I don't know, you probably can't leave a toddler alone for 20 or 30 minutes, but it's a little bit older than a toddler.

**Lenny Rachitsky** (00:17:02):
Maybe 20, 30 seconds.

**Dan Shipper** (00:17:06):
With a toddler, you can be in the same room but not interacting with them every single second for 20 minutes sometimes. So it's around there. I think we have that similar leash with AGI. And so I think a good definition of AGI is when does it become economically profitable for people to run agents indefinitely? So it just never turns off. It's a Claude Code that's always running, it's always doing something, you just never turn it off, and you don't need to because you know that it's worthwhile to keep it on.

**Dan Shipper** (00:17:41):
It's never waiting for you to be like, "Okay, next thing." It'll always respond to you when you're like, "Okay, next thing." But it's off just essentially living its life like a teenager and that is profitable for you. You'd rather have it do that than just wait for you to tell it what to do next.

**Lenny Rachitsky** (00:17:56):
Interesting.

**Dan Shipper** (00:17:56):
I think that's a good definition of AGI.

**Lenny Rachitsky** (00:17:58):
The profitable piece is also just the cost of running that thing and having it.

**Dan Shipper** (00:18:02):
It's partly the cost and partly the value. And obviously, you can game this a little bit and be like, "Cool, I'm just going to tell Claude to run in a loop forever." But I'm talking about more than that, more widespread adoption of agents that work all the time. And I like the profitable thing, because if it costs a little bit of money and the bar is profitability, it has to actually be doing something useful for you to keep it on.

**Lenny Rachitsky** (00:18:29):
It's interesting how the metaphor of a senior employee and autonomy and essentially the more autonomous they are, the less instruction you have to give, the less reviews you have to do, is also just directly correlated with how senior they are.

**Dan Shipper** (00:18:44):
Totally.

**Lenny Rachitsky** (00:18:45):
Okay, great. Anything else along these lines?

**Dan Shipper** (00:18:48):
I have plenty of them. I hate the headlines that are like, "It's going to replace jobs," or "It's going to unemploy two thirds of the workforce." I don't think that's true. I hate headlines that are like, "You don't use your brain when you use ChatGPT," or another good headline is, "Doctors alone, doctors plus AI, or just AI, which one is better? AI is better, therefore, doctors are going to be outmoded."

**Dan Shipper** (00:19:18):
All that stuff is, I think, pretty dumb. So for the doctors plus AI example, I think it's important to recognize that using AI is a skill. And so if you study doctors in a vacuum that don't really have a lot of experience with AI, you could probably create a situation such that it's better to just use an AI. And sometimes it is going to be better. But there's so many contexts that doctors need to make decisions and do things that it's really hard to take one study and make any conclusion about that.

**Dan Shipper** (00:19:52):
And it's especially hard when you're dealing with a technology that's developing so rapidly that doctors can't really be expected to be experts at it yet. But I would guess in five or 10 years, that will be totally and completely different. For the student example or the "AI turns your brain off" example, I think it's really important to understand that in the history of technology, it has always been the case that you give up certain skills in order to get other ones. For example, Plato is famously very skeptical of writing because he thought it would harm your memory. And it did. We don't remember things quite as well as they did back in the day because they had to remember long epic poems to entertain each other.

**Dan Shipper** (00:20:37):
But I think writing is a worthwhile trade for having a slightly worse memory. And I think something similar is going on with AI where you may be slightly less engaged in certain tasks, but if you use it right, you're going to be way more engaged in other tasks where you have much more power. And so you can construct a study that says brain connectivity goes down when you use AI in the same way that you could construct a study that says people's memory are worse when they have writing skills. But I don't think anyone would want to go back to a world where no one was literate.

**Lenny Rachitsky** (00:21:10):
That is super interesting. There's all these studies that are showing the benefits of AI to students with these studies in Nigeria and just how fast people progress. So I think it's really important, this context you're sharing that you will lose some things, but the hope is the gain is much higher, and so far it seems like it will be.

**Dan Shipper** (00:21:27):
Yeah. I think people always, especially at the beginning of a tech hype cycle or a revolution paradigm shift, it's always easy to underestimate how quickly things are going to change. And the example I always use is, I live in Brooklyn and the tailor down the street from me doesn't accept credit cards. Credit cards have been around for a long time, so it takes a long time for technology like this to be adopted even in the best case.

**Dan Shipper** (00:21:55):
And I think it's really easy to underestimate how complex specific contexts are that humans know how to deal with. And just because you can get a really good score on a test... It's incredible. I love AI, it's so incredible, but it doesn't actually give you an intuition for how difficult it is to actually be replacing specific parts of work or activities that you would do. I think a really good thing to give you maybe a little bit of an intuition for it is I built this thing over a weekend a month ago that was, "0.3, can it predict what I'm going to say in a meeting?"

**Dan Shipper** (00:22:45):
That's a benchmark. It's the CEO benchmark. And the reason I did that is because the gold standard for OpenAI for testing how powerful a model is, is they test it on their internal code base. So they say, "How good is the new model at predicting what comes next in our internal code base?" Because that's not anywhere out on the internet. So it's a really good benchmark for that. And so I was like, "Well, my meeting transcripts aren't anywhere on the internet. A lot of what I say is on the internet and there's some overlap, but it'd be interesting."

**Dan Shipper** (00:23:19):
And so I ran a bunch of the frontier models on this, on just my Granola transcripts, and they're pretty bad. They are pretty bad, and it's not because they're not smart. There's this real push now. Tobi from Spotify coined this term called "context engineering," which is getting the context to the model, the right context at the right time, is at least half the performance.

**Dan Shipper** (00:23:43):
And I think that's 100% true. It's something that I've been writing about for three years. At the time, I called it knowledge orchestration. I think context engineering is probably a better term. But it's totally true, and that's a very, very hard problem to solve. It's not just a one- shot problem where it's gigantic context window and we're done. It's going...

**Dan Shipper** (00:24:00):
... that problem, where it's like gigantic context window and we're done. I think it's going to get better over time, but the minute it gets good at predicting what I'm going to say next in a meeting, I'm just going to use it as a tool, and that's going to change the entire dynamic of what I say next in a meeting. So it's not as easy as it seems.

**Lenny Rachitsky** (00:24:18):
Interesting. I imagine you can build a GPT from that. And then, instead of having a meeting with Dan now, just talk to this thing, and he'll make decisions.

**Dan Shipper** (00:24:26):
Yes, definitely. And I mean we do this a little bit. It's not the same as being able to predict exactly what I'm going to say in a meeting. But I think if you're a CEO, or founder, or manager, it's really stunning how much of your job is just repeating yourself. And that is one of the best things about this AI, particularly AI revolution, is that you don't have to repeat yourself.

**Dan Shipper** (00:24:48):
And so we had it last quarter. I tend to set one or two quarterly goals. And one of my big goals for us last quarter was don't repeat yourself. So I don't want ever say the same thing in a meeting twice, if I can help it. So for us, at Every, one of the big parts of Every is we have a daily newsletter. And I'm spending a lot of time giving feedback on headlines, or giving feedback on, "How do you write an intro," or "Is this idea any good," that kind of stuff.

**Dan Shipper** (00:25:15):
And we've started to codify all of that into prompts that basically... It's not the same as mimicking me. It can't exactly say exactly what I'm going to say in a meeting, but it pushes my taste out to the edge so that writers who are not able to talk to me, by the time I see it, they've already talked to some simulation of a simulation of me. And that's incredibly powerful.

**Lenny Rachitsky** (00:25:41):
Let's follow this thread. This is exactly where I want it to go. I feel like the business you're building, the team you're building, the way you're operating is the very bleeding edge of how companies will operate and are trying to operate in this AI era. You guys are trying to be super AI-first. And it's super aligned with just so much of your writing. There's just so much reason to study what you guys are doing. So-

**Dan Shipper** (00:26:04):
Well, thank you.

**Lenny Rachitsky** (00:26:05):
Yes. And this is benefiting all of us, so thank you. So first of all, just tell people what the heck Every is, and then share a few insights into just how you operate. It's funny that you laugh at [inaudible 00:26:20] whatever you say.

**Dan Shipper** (00:26:20):
Everyone asks that because it's a very weird shape of a company. You can actually see other companies that have this shape from earlier eras, but it's less common. It doesn't make as much sense.

**Dan Shipper** (00:26:33):
And I think it's newly enabled by AI, and we can talk about why. But the way that I typically talk about Every is we do ideas and apps at the edge of AI. So the core of the business is we have a daily newsletter. We've been doing it for about five years. We have about 100,000 subscribers. All of the people from the top AI labs read us. Anyone who's basically interested in or working in AI at the frontier and wants to know what's on reads us.

**Dan Shipper** (00:27:00):
We do a lot of... For example, whenever OpenAI or Anthropic drop a new model, we get our hands on it early, and then we get to play with it and write about it, which it's my ideal job. I love it. It's the best.

**Lenny Rachitsky** (00:27:14):
It sounds like it.

**Dan Shipper** (00:27:14):
I don't if I can curse on this podcast, but-

**Lenny Rachitsky** (00:27:15):
You can.

**Dan Shipper** (00:27:15):
... it's the fucking best.

**Lenny Rachitsky** (00:27:18):
Perfect. Excellent use. And you call those "vibe checks", is that the-

**Dan Shipper** (00:27:22):
Yeah, we call them vibe checks-

**Lenny Rachitsky** (00:27:22):
Vibe checks, love those.

**Dan Shipper** (00:27:24):
... which I think is really important because... And this gets to the next part, the apps part of what we do. I think it's really important to do vibe checks and to call them vibe checks because they're about how does it feel to use this thing and how does it feel to use it for work for things that you would normally use it for in your job or in your life. Because I think that captures something that standard benchmarks just don't capture and really can't. And the best people to tell... to write a vibe check are people that are actually at the edge using it for stuff.

**Dan Shipper** (00:27:57):
And so what we've found over time is we have... We love, we think the best writing and content about technology is from people that are actually using it and building with it. And so we've always had this sort of function, where we're always building little experiments in addition to our writing, and that helps us write great stuff. And that has turned into a suite of apps that we run internally. And the people who are building those apps are also writers, and they're contributing to things like vibe checks.

**Dan Shipper** (00:28:27):
So you get a really inside look into how is this stuff being built from people who are actually using it every day. And the suite of apps that we have, one's called Cora. We just launched Cora publicly on the day that we're recording this, which is really awesome.

**Lenny Rachitsky** (00:28:39):
Congratulations.

**Dan Shipper** (00:28:40):
Thank you. You can think of it like a chief of staff, an AI chief of staff for your email. It helps you manage your email with AI. It's very cool. We can go into more of it later. We have another one called Sparkle, which is an AI file cleaner. We have another one called Spiral that does content automation with AI. We originally incubated Lex, which is an AI document writer, which we spun out into its own company, and my Every co-founder runs that.

**Dan Shipper** (00:29:05):
And basically we bundle everything together. So you pay one price, and you get access to all of the software that we make, and we're constantly putting new stuff in the bundle. And I can tell you more about what kinds of things do we like to incubate and how do we like to incubate it because I think there's some really interesting, special things in there.

**Dan Shipper** (00:29:21):
But I've been blabbering for a while, so I'll stop there.

**Lenny Rachitsky** (00:29:23):
There's also a consulting firm, which I want to talk about, but let's hold off on that.

**Dan Shipper** (00:29:25):
Yeah, we have consulting.

**Lenny Rachitsky** (00:29:26):
Yeah.

**Dan Shipper** (00:29:27):
We also do that, and that's the third leg of the stool in the business. It doesn't fit quite as nicely into my ideas in app streaming, but we spend a lot of time with big companies, where we teach them basically how to be AI-first. We train all the people on how to use AI. And it's very cool, it's really fun, and a very important part of what we do.

**Lenny Rachitsky** (00:29:47):
That feels like a billion-dollar business right there. I want to come back to it.

**Dan Shipper** (00:29:50):
[inaudible 00:29:50].

**Lenny Rachitsky** (00:29:51):
Because everybody wants to learn this.

**Lenny Rachitsky** (00:29:53):
Okay, so share a few ways that you guys operate. You mentioned that your team doesn't write any code. What are just some ways that allow you to operate this efficiently? I know your team's really small. You have a daily newsletter, you have three, four products, you have a consulting arm. How big is the team at Every?

**Dan Shipper** (00:30:10):
We have 15 people.

**Lenny Rachitsky** (00:30:11):
15 people? Okay.

**Dan Shipper** (00:30:12):
Yeah.

**Lenny Rachitsky** (00:30:12):
So just give us insight into some of the ways you operate that are at the bleeding edge.

**Dan Shipper** (00:30:16):
Okay, so a couple of things. One, and I think everyone should do this, is we have a head of AI operations. I sit with her once a week. And every time I'm doing something repetitively, we put it in a to-do list. And she's just constantly building prompts, and building workflows, and stuff like that so that I and everyone else on the team are just automating as much as possible. And I think that has been a big unlock because it's really hard to...

If you're working in a job all day, you're fighting fires, and you're like, "Okay, am I going to do this in the way that I know how or am I going to do it in the new way that might not work?" I don't want to spend a bunch of time [inaudible 00:30:54] you're building some no-code automation. I don't want to do that. And having an AI operations lead lets you basically identify those things and have them solved without people who are doing the work actually having to take time to do it, which I think makes it much more likely it happens.

**Dan Shipper** (00:31:10):
There's always a trick with that, where it's like you have to make sure it gets used. So it's basically you're developing little applications internally, but if you're good at making applications people use, it's great. Highly recommend having an AI operations lead.

**Lenny Rachitsky** (00:31:23):
I imagine you saw the [inaudible 00:31:25] Quora tweeted about this, wanting to hire exactly this sort of person.

**Dan Shipper** (00:31:29):
Yeah.

**Lenny Rachitsky** (00:31:29):
So clearly this is a trend.

**Dan Shipper** (00:31:31):
Yeah.

**Lenny Rachitsky** (00:31:31):
So the idea is your point that this needs to be somebody who's outside of the day-to-day work of the company, and is specifically focused on helping the team be more efficient with AI?

**Dan Shipper** (00:31:43):
Yeah. Yeah.

**Lenny Rachitsky** (00:31:44):
And then is this person mostly just you automating you, or can they help other people? Are they helpful-

**Dan Shipper** (00:31:49):
No, she helps everyone, basically.

**Lenny Rachitsky** (00:31:49):
Everyone? Okay.

**Dan Shipper** (00:31:51):
Where we're starting right now is with the editorial operation. So there's so much stuff in the editorial operation, where I or our editor in chief, Kate... Kate, is constantly doing little, small copy edits to make sure everything is in Every style, and it takes hours a day. And so now Opus is at a point where you can give it a style guide and a prompt, and it will go through anything you're writing, and copy edit it, which is amazing.

**Dan Shipper** (00:32:24):
The trick is it's not just building that. You also have to get Kate to be like, "Did you put this through the prompt yet," anytime someone gives her something. So there's a little bit of behavioral update too that has to happen, which I think is a really interesting organizational challenge.

**Dan Shipper** (00:32:38):
And I think for us it's a little easier because everybody inside the org is very AI-first and just wants to go do it. We don't have anyone really who's like, "I don't know. I don't really want to do this." And that's a whole different challenge, which I think a lot of organizations face, but there's always a problem of getting people to use it.

**Lenny Rachitsky** (00:32:55):
That is super cool. What is her background, this AI operations person?

**Dan Shipper** (00:32:59):
Her name is Katie Parrott. She actually does a lot of ghostwriting for us. So she also, when people inside of Every who are builders... Often they just write themselves, but sometimes they want help, and she'll help them write about whatever they're working on. So that's how she started with us. She still does that, but she also spends a lot of time doing the AI operation stuff.

**Dan Shipper** (00:33:20):
And then before that, she worked at Animalz, which is a content marketing agency, one of the top content marketing agencies. And they're very process oriented. And I think the reason Katie is so good is because she's incredibly good at that kind of process stuff or thinking about that, but she's also a great writer and she's also just incredibly excited about AI. She just wants to tinker and wants to use it. And that was the thing that got me to be like, "Okay, you should just come and do that. Instead of just ghostwriting, we should add this to your plate." And it's been really fantastic.

**Dan Shipper** (00:33:58):
At minimum, you really just want someone who's just like, "I want to tinker. I want to build stuff." There's also people who have a little bit more of that process orientation. I think that is important. And to the extent they understand the craft of the thing that they're trying to build for, that also helps a lot.

**Lenny Rachitsky** (00:34:14):
This is an amazing tip. I feel like everyone's going to start hiring these people.

**Dan Shipper** (00:34:17):
I think so. There's a couple other people who talk about this. I heard Rachel Woods, who's another... She thinks a lot of AI stuff. She's talking about it. I think it's becoming a thing, and I think it's really important, and it just bleeds out into every other part of the org.

**Dan Shipper** (00:34:33):
So we're doing this inside of the editorial org, but there's a lot of copy that goes out on Cora. And by the way, Cora is spelled C-O-R-A, so it's different from Q-U-O-R-A, slightly confusing. There's a lot of copy that goes out on Cora, or Spiral, or Sparkle that we want to have that same Every quality bar for. And so we have engineers sending Kate, like, "Here's the Figma file. Can you go and do copy edits?" And that sucks for everybody. And Kate is one person, and it's just really hard to do that.

**Dan Shipper** (00:35:04):
So one thing that we did, Nityesh, who's one of the engineers on Cora, built a Claude Code command that just uses that prompt, and checks through the entire code base for all the copy edits, and then creates a pull request on GitHub, and then sends the pull request to Kate. So she's just looking at the pull request, and being like, "Does this make sense?"

**Dan Shipper** (00:35:26):
And so you can translate that prompt into, for example, a format that engineers can use. And suddenly your engineering team is writing marketing copy in the style you want. I think that's so cool.

**Lenny Rachitsky** (00:35:36):
That is extremely cool. I'm going to take us on a little tangent. You keep mentioning-

**Dan Shipper** (00:35:42):
[inaudible 00:35:42].

**Lenny Rachitsky** (00:35:42):
... Claude, and I'm curious just what is in the stack of tools that you find yourself using, your team ends up using. It seems like Claude is a core part of it.

**Dan Shipper** (00:35:51):
I do love Claude. I would say I'm generally... My first thing that I open is o3. I'm a ChatGPT boy. And I think o3 is super high quality. I think it's great for writing, it's great for coding, it's great for all that stuff. And what it has that really makes a difference still from Claude is it has memory. And I just love that. I've spent so much time yelling at ChatGPT about, "I need my writing to be punchy and concise." And it just knows that now.

**Dan Shipper** (00:36:20):
So I think when I ask it to write something for me, it's actually better than yours. Or maybe not yours, but your average ChatGPT user. And I also find I use it a lot for self-reflection and personal growth type stuff. So it knows me. So when I send it a meeting transcript, and I'm like, "How did I do?" It's like, "Well, you did that thing that you normally do, but you're way better on this other thing." And I like that. I think that's really great. So day-to-day, o3, that's my go-to.

**Dan Shipper** (00:36:49):
I think Claude Opus is... First of all, Claude Code, everyone inside Every, that's basically what we use. If you're building something, you're using Claude Code. It's crazy. It's so good.

**Dan Shipper** (00:37:01):
Gemini just came out with something, so I'm very excited to try that because I think that's the model that we use most for the apps that we build, inside the apps. It's incredibly powerful and it's incredibly cheap, which is great. So I want to try the CLI tool that they came out with.

**Dan Shipper** (00:37:17):
We also use Codex a bit, which is OpenAI's coding tool. And that's for, like, "I want a one-off, self-contained... I want to pick off this little feature."

**Dan Shipper** (00:37:27):
What else do I use? Going back to Claude, Claude Opus 4 can do something that no other model, except one other model that I can't talk about... can do something that no-

**Lenny Rachitsky** (00:37:39):
[inaudible 00:37:39].

**Dan Shipper** (00:37:39):
... other model can do.

**Lenny Rachitsky** (00:37:40):
Okay, we won't go there. We don't want to get you in trouble. Okay, go on.

**Dan Shipper** (00:37:44):
But yeah, no other model can do this. Which is earlier versions of Claude, and I think generally versions of other models, when you ask them, "Is this piece of writing any good," Claude, for example, would always give it a B+. And then if you did another turn of the same conversation, you're like, "I updated this," it would always go to A-. And then if you give it another turn, it would go to A.

**Dan Shipper** (00:38:07):
So it doesn't have the same kind of gut. It's thinking about what you probably want hear too much. And there's various methods that you can use to prompt engineer around this, like give it a template or whatever. And they sort of worked, but it just still doesn't have that thing where it's like, "Can it tell if writing is interesting or any good? Does it have that gut sense?" And Opus 4 has it. It's really wild. And I think that's super important because it opens up all these use cases where you might want to use a language model as a judge. So for us, for example, we're working on a new version of our product Spiral, which does content automation. You've used that in the past. And we're doing essentially Claude Code, but for content style product, where you say, "I want it to write a tweet," you give it all the documents, it has a bunch of memories, it creates a to-do list for itself, and then it goes and writes.

**Dan Shipper** (00:39:04):
And one of the things that is so interesting is now, because it can judge things, part of its to-do list is, "Okay, I wrote three tweets. I'm going to judge whether I think these are any good," and then it can improve before it comes back to you.

**Dan Shipper** (00:39:21):
And that's just a huge, huge unlock, that we were struggling for three months to build this crazy system to try to get it to judge writing. And then Opus 4, just one-shotted it, and we're like, "Great, this product works. Let's start chipping it." So yeah, I love it for that.

**Lenny Rachitsky** (00:39:37):
Are there any other AI tools that you just use regularly? You mentioned Granola, even outside of the bottles. So what are some that you think maybe people are sleeping on?

**Dan Shipper** (00:39:46):
I use Granola. So I used to use Super Whisper and Whisper flow, which I think are fantastic. We have an internal version of that called Monologue that will be shipping in a month or so that I use now, but you can think of them as roughly equivalent. And I think generally speech to text interfaces are the future, and more people should be using them, and more people should be building them as affordances. We use Notion all the time, and I specifically use their meeting recording. I think that's mostly the stack.

**Lenny Rachitsky** (00:40:17):
Okay. That was really helpful and super interesting.

**Dan Shipper** (00:41:33):
Yeah.

**Lenny Rachitsky** (00:41:33):
Okay. What else? What else do you do that you think other companies should be doing or will eventually start doing?

**Dan Shipper** (00:41:39):
So the Cora team, which is Kieran and Nityesh, basically-

**Lenny Rachitsky** (00:41:43):
[inaudible 00:41:43] that's the team, two people?

**Dan Shipper** (00:41:44):
That's the team, yeah. Well, with Cora, it's Kieran, Nityesh, and 15 Claude Code instances, so it's more powerful than you think.

**Lenny Rachitsky** (00:41:53):
I love that. This is just, again, a glimpse into the future.

**Dan Shipper** (00:41:58):
One of the things that we do that I think is really cool, and they basically invented this, I had nothing to do with this, is they invented the idea of compounding engineering. So basically, for every unit of work, you should make the next unit of work easier to do.

**Dan Shipper** (00:42:16):
So an example is, in a Claude Code world, where you're not coding a lot, you end up spending a lot of time essentially typing PRDs. Like, "Here's a document with exactly the stuff that I need to do," right? And so you could just be like, "Okay, cool. That's my job now. I'm going to just write PRDs." And so each successive PRD, it's the same amount of work.

**Dan Shipper** (00:42:45):
Or you could spend a little bit of time being like... There's a sort of platonic ideal of a PRD. And what I'm going to do is write a prompt that can take my rambling thoughts and then turn that into a PRD. And so you spend a little bit of work to make all of the next PRDs that you're doing easier to write because you're writing less of them.

**Dan Shipper** (00:43:08):
And so finding those little speed-ups, where every time you're building something, you're making it easier to do that same thing next time, I think gets you a lot more leverage in your engineering team.

**Dan Shipper** (00:43:20):
And so, yeah, we have Kieran and Nityesh. And Cora, it just became public. It was in private beta. It had 2,500 active users. And there's millions of emails going through it. And that's one of the products that we do as a 15-person company. It's kind of crazy.

**Lenny Rachitsky** (00:43:37):
It is crazy. How do you do the speed-up thing? Is it prompts that they continue to refine [inaudible 00:43:44]?

**Dan Shipper** (00:43:44):
A lot of it is prompts, and automations, and stuff like that. Yeah.

**Lenny Rachitsky** (00:43:47):
Got it. For automations, what's the tool? What's the tool you use for automating automations?

**Dan Shipper** (00:43:52):
What they're using a lot of is Claude Code. So you can do slash commands in Claude Code, which are repeated prompts that you're doing.

**Lenny Rachitsky** (00:44:01):
Got it. Okay. So basically they're building a library of prompts that make the process, of, "Here's what I want to build," to a good solid PRD that you can feed into Claude Code more correct and more efficient?

**Dan Shipper** (00:44:13):
Exactly.

**Lenny Rachitsky** (00:44:14):
Super interesting. And they just keep a file or they put this into a project? Is that how they store this stuff?

**Dan Shipper** (00:44:14):
It's a GitHub. It's a GitHub GitHub-

**Lenny Rachitsky** (00:44:22):
[inaudible 00:44:22].

**Dan Shipper** (00:44:22):
... where they can share it with each other.

**Dan Shipper** (00:44:24):
Another thing that they do, which I think is very cool, is they use a bunch of Claudes at once, but then they're also using three other agents. There's an agent called Friday that they love.

**Lenny Rachitsky** (00:44:36):
That's an AI Asian product called Friday?

**Dan Shipper** (00:44:38):
Yeah, yeah.

**Lenny Rachitsky** (00:44:38):
Hadn't heard of that. Okay, interesting.

**Dan Shipper** (00:44:40):
There's another one called Charlie that they really love. And in particular, I think the thing they like about Charlie... We have a whole video about this, which I can send to you.

**Lenny Rachitsky** (00:44:47):
Yeah, I'll point to it.

**Dan Shipper** (00:44:48):
They did an S-tier through F-tier of AI agents, which I think is so funny. And one of the things I really like about Charlie is that it lives in GitHub, so when you get a pull request, you can just be like, at Charlie, "Can you check this out?" And that seems to work really well to have different agents that have maybe slightly different perspectives. It's like different people that have different perspectives and have different taste.

**Dan Shipper** (00:45:15):
Kieran, he's one of those serious Rails-files, who they just love Rails, and they love the way that Rails feels, and so I think he has a real sensitivity to... Okay, this agent, ChatGPT for example, it feels very terse, and minimal, and professional, and it has a particular kind of style that maybe he likes. Versus, I don't know, Claude is a slightly different style. And I think all of that is so interesting that these things have personalities, and that that changes what you might want to use it for or why you might want to use three of them at once.

**Lenny Rachitsky** (00:45:49):
That is so fascinating. It makes me think about Peter Deng's conversation again, where he talks about his hiring strategy and one of his key lessons. And he ended up hiring the current head of product for ChatGPT, the current head of marketing at ChatGPT, the current head of engineering because he hires these incredible people.

**Lenny Rachitsky** (00:46:07):
And his philosophy is to hire a team of Avengers, where everyone is strong at certain things, and together they're the perfect team, versus everyone... versus the best at everything. And it's interesting that you can almost do that with different product, different agents from different companies.

**Dan Shipper** (00:46:22):
You definitely can.

**Lenny Rachitsky** (00:46:23):
And it makes me feel like there's a bigger market than people think potentially, where people will want different companies, agents, not just all Devins or not all Codexes.

**Dan Shipper** (00:46:30):
I think there really is. It's definitely not one agent to rule them all at all.

**Lenny Rachitsky** (00:46:35):
So interesting.

**Dan Shipper** (00:46:36):
Yeah.

**Lenny Rachitsky** (00:46:36):
Oh, my God. The two people on the Cora team, what's their background? Are they both engineers or what are they?

**Dan Shipper** (00:46:42):
They're both engineers.

**Lenny Rachitsky** (00:46:43):
Okay.

**Dan Shipper** (00:46:44):
Kieran's got this crazy background, where... They both have really interesting backgrounds. Kieran's got this crazy background, where he was previously VP Eng at a startup, so was effectively the CTO of a startup, or maybe two startups, and was one of the founders. But before that, he was a composer, a professional composer. And before that, he was a baker. So we did a team retreat in France last year, and he taught us all how to make croissants. My croissant was horrible. His was beautiful.

**Lenny Rachitsky** (00:47:14):
Seems [inaudible 00:47:16].

**Dan Shipper** (00:47:16):
And generally, I think that kind of multidimensional type of talent is the kind of person that I love having at Every. Because we're all generalists. We all want to use AI for all these weird, awesome, creative things. And someone who has that background is going to have a good taste for not only agents, but, "What should the landing page look like," or whatever. Which I think is increasingly important, where you're trying to scale a team of generalists of 15 people to five products. So that's Kieran's background.

**Dan Shipper** (00:47:43):
Natasha's background is... I'm jealous because he only started learning to code when ChatGPT came out. He had wanted to learn to code forever, and he's only known how to code in an AI era. And I keep telling him, "Dude, I learned to program in middle school from books." I had to go to Barnes & Noble and buy a book. And there was nothing... I couldn't Google any-

**Dan Shipper** (00:48:00):
I had to go to Barnes and Noble and buy a book and there was nothing... I couldn't Google anything about why this function wasn't working.

**Lenny Rachitsky** (00:48:08):
No Stack Overflow even back then.

**Dan Shipper** (00:48:10):
Yeah, yeah. There wasn't that overflow. There was weird BB net forums and stuff that I was like 12 and I probably shouldn't have been on there or whatever. So he has gone so much faster than any other engineer, I think in a pre AI era. And I see the same thing in the rest of the company. I think there's this huge question about what happens when kids... Entry level jobs are taken away by AI. And my take is like that's worth thinking about and it's possible that that might be a problem at some point. But my take is whenever I see a kid with ChatGPT, I'm like, holy shit, they're going to grow so much faster than any other person that I've worked with. We have this guy Alex Duffy who works with us, he writes for Context Window and he just launched, we taught AIs how to play diplomacy with each other, which is really cool.

**Dan Shipper** (00:49:08):
And he did that whole thing and I think he's really, really, really talented. And when he came to us, I guess almost a year ago now, it was one of those classic cases which I've seen over and over at every... Which is, you have great ideas, but you're not a good writer yet and it's really hard for me to do anything with you until you're good enough at it. So I have to give you small little things until you get better and blah, blah, blah, whatever. And what I noticed with him is he was just making a year. He made a year's worth of progress in two months because every time I sat down with him and told him, okay, here's how you tell a story. Here's how you think about a headline. He recorded all of it, put it into a prompt, and he never made the same mistake twice.

**Dan Shipper** (00:49:49):
And I think he's so much accelerated from where he would have been because of this stuff, and I see that in lots of other parts of the work. So Natasha is another good example. And so I think generally people are going to figure out that some 20-year-old with ChatGPT subscription is super powerful if you just mentor them. And I think that's great.

**Lenny Rachitsky** (00:50:11):
Man, there's so many threads I could follow here. There's all this fear of entry level people will never... The roles are disappearing for entry level people and so how will we ever have senior people if these people can't learn to do things as an entry level person? And what you're saying is ChatGPT and these tools help you accelerate really quickly so you don't really need to be at the bottom rung for a long time.

**Dan Shipper** (00:50:33):
Yeah. You're effectively learning how to be one level above the entry level from the beginning and this is sort of my whole allocation economy thesis where when you look at skills are going to be valuable in the AI era, one big group of skills are the skills of managers. Today, they're human managers, tomorrow everyone's a model manager. Right now, AI is not... Right now, management skills are not broadly distributed, because it's very expensive, another expensive thing that... So 8% of the workforce is managers. It's now going to be much cheaper to manage, so more people are going to have to do it. And so that's the thing that kids, 20-year-olds, whatever, I see is now are going to start to have to learn in addition to, it's not like you can just say, okay, go do it and then come back. You have to be able to go into the work that's being done and help make it better. But they're learning both at the same time. They're learning how to manage and how to do the actual work so that they're good at it.

**Lenny Rachitsky** (00:51:36):
And the managing here is managing agents. Right?

**Dan Shipper** (00:51:41):
Yeah. You're managing AI.

**Lenny Rachitsky** (00:51:43):
And so coming back to your point about how this core team, and I guess you said everyone doesn't write code, zero code written, now it's just managing agents that are writing code for you.

**Dan Shipper** (00:51:54):
Yeah.

**Lenny Rachitsky** (00:51:55):
Okay. I've never heard of a company at this stage, so this is extremely cool. So the workflow is they give it, here's what I want. I refine it using this cool prompts library that they build on and agents build code, write the code. Then basically the time is spent reviewing code and then reviewing the output. What does it look like? What does it feel like? And then continuing to refine, wow. So you guys are at where Michael from Cursor said we will be. So I chatted with him a few months ago. He said in a year, this is where he thinks the thing will be. We're not looking at code anymore. You guys are already there. Although you were looking at code. Okay, you're still looking at code.

**Dan Shipper** (00:52:33):
They definitely are looking at code. So you're doing a code review before you do anything. And I do think Danny, who runs Spiral, which is the cloud code for content tool I was talking about that we're building, he spent a couple of days digging into the internals of some third party library that we were interested in just because it's helpful to know, it's helpful to understand those things, but then he's not actually writing any code. Once he understands it, he's just off telling cloud code what to do. And I think that's really important.

**Lenny Rachitsky** (00:53:08):
This is an insane milestone we're hitting here. There's this sense we're getting to a place where you don't need to really understand code, you don't have to write any code. We'll get there and you guys are there. I think this is so easy to overlook how wild this is. You have a product team not writing code at all.

**Dan Shipper** (00:53:26):
It is really wild. I think it's really wild in particular, just having a small group of people that have... Everyone has all these different skills. Everyone's a generalist, everyone's AI forward. So what you can do in an environment like that with just still a small team is crazy. And you're kind of inventing all these new principles for how do we work together, how do we do engineering, all that kind of stuff. And I think that's what makes the writing... That's why I like doing that is because the writing that we do from that I think is really good because we can talk about it from a sort of position of experience, but I do want to say something else which is we're not at a point yet where the people that work at every could do what they do if they didn't know how to code.

**Lenny Rachitsky** (00:54:08):
Yeah, this is what I was going to ask.

**Dan Shipper** (00:54:10):
Which is a different bar, and I think for a long time it's going to be valuable to know how to code for a long time, but this is a progression that is not a new progression. So for example, when I was in middle school learning to code, the new hot thing was scripting languages, which is Python and JavaScript. But if you were a real programmer, you would understand the language underlying Python and JavaScript, which is written in C. and scripting language weren't totally real. And in order to really do anything interesting, you had to be able to learn both parts of the stack. Same thing for C programmers, when I guess in the seventies C was invented, it was like you got to be able to write assembly.

**Dan Shipper** (00:54:59):
And English is just a layer on top of scripting languages. So I think all of those things were right in the sense that there's... Especially during transitions, there's a lot of reasons why it's important to be able to go down a layer in the stack and it gets less and less frequent over time, but that still takes a long time. And there's some times when even if you're a JavaScript or a Python programmer, it's useful to know how that stuff works, how it's written, and see how it's implemented. Today it's much less important than it used to be, but that took 10 or 20 years. And I think that the same thing is going to be true for programming. Having that skill is super important and will accelerate you significantly. It will sort of start to get less important over time, but we're not close to that yet.

**Lenny Rachitsky** (00:55:44):
Okay. That's a really important point. I'm glad you went there. So do you have a sense of how far we might be from you hiring someone to build another product that isn't an engineer?

**Dan Shipper** (00:55:54):
Like a real SaaS product?

**Lenny Rachitsky** (00:55:57):
So hey, we have this idea we want to bring someone on to actually lead it.

**Dan Shipper** (00:56:00):
Very far. Not within sight, but there's a lot of things that could be products that are a level down from that I think that you could do almost now. So an example, we were talking about DIA, the new AI browser from the browser company. DIA has these things called skills, which are effectively little AI apps that you can run in the browser. You can prompt them and they run on the web page and do work for you. A non-technical person can build that, same thing for custom GPTs from ChatGPT. A non-technical person can definitely build that. So I think while I will definitely maintain that we're not anywhere close to anybody being able to build a conventional SaaS app with zero programming knowledge, aside from just a demo, there are going to be other forms of software.

**Dan Shipper** (00:56:55):
One of my things is like software is becoming content. There's going to be other forms of software that don't look like the software today, but you can run, start and run as a business, as a non-technical person even if you don't know how to code. And that'll happen very soon. I mean, it's already kind of happening. It doesn't look like the thing that you're asking about. It's sort of like the difference between a Hollywood movie and a YouTube video.

**Lenny Rachitsky** (00:57:17):
I think that's really reassuring to a lot of people. Basically what you're seeing is AI just supercharges people who have a skill and allows them to do a lot more.

**Dan Shipper** (00:57:25):
Yeah.

**Lenny Rachitsky** (00:57:26):
Okay. Is there any other way that you guys operate that is really interesting that might be worth sharing that helps you operate really quickly, helps you do more with less?

**Dan Shipper** (00:57:38):
I mean, I would love to talk about how we think about building products, what products to build, what do we end up building? Because I think that there's something sort of special about it that probably there's a playbook that is useful for people. So when I think about... This is only sort of snapped into focus recently. So a lot of this was just doing it intuitively without really a thought for it. But when I think about the kind of things that we have ended up incubating, it's basically goes back to something I said at the beginning, which is there are these things that were historically really expensive that only rich people or big companies could buy. So a chief of staff for your email, I think a therapist or a lawyer is another interesting example. Someone to organize your closet or organize your computer is another example. Someone to go straight for you, that are becoming orders of magnitude cheaper so that everyone can use them even if you're at a small startup.

**Dan Shipper** (00:58:39):
And so basically when you're running, we are sort of this AI first company. You're running into all these little things where you're like, I wish I had a ghost writer right now, but ghost writers are really expensive. Or I wish I had a lawyer but it would cost me like $25,000. Lawyers are really expensive and there's a lot more demand for those services than can be fulfilled because they're so expensive. And what AI does is it allows you to be like, oh, I could just use cloud for that. I can use ChatGPT for that. And so you're able to use the demand that you have that we can afford a lawyer. We have ghost writers, but there's a lot more that we can't do because we can't afford it. So we still have our lawyer and we still have our ghost writers, but we just do a lot more of that stuff.

**Dan Shipper** (00:59:27):
And so we notice that. We start to then use ChatGPT and cloud first, these general purpose tools to try it and see is this useful? Does this actually work? All that kind of stuff. And then if it does, we will unbundle it into its own separate thing that becomes an app. And I think what's really special about this time is the entire game board has been totally reset in terms of things you can build. Where five years ago it was like you're going to build another Notes app. We've been building notes app for forever, another B2B SAS app. It's all the same stuff in slightly different packaging. And now it's totally new territory. No one knows what's going on. Everyone's inventing it as it happens. All these new workflows are being created in a very similar way to, I don't know, for example, when spreadsheets were first a thing on computers, we were figuring out all these new workflows on spreadsheets.

**Dan Shipper** (01:00:24):
They got unbundled in the B2B SAS, same thing for ChatGPT and Claude. And what's really cool is you can be like, cool, I'm using using ChatGPT for this. It's really useful for me. And you might be one of the first people to really notice that. And then because everybody that works at Every is AI first and came to us because they reads Every, they read Every, so we all have the same vibe and we're all kind of doing similar stuff. They become our first users. So we measure the success of the product by is it a banger inside of Every, monologue the app that I was talking to you about, everyone just started using it and we're like, okay, we've got something here.

**Dan Shipper** (01:01:05):
And what's really interesting then is if everyone inside of Every uses it and people read Every, they have a similar vibe to us too, so they become the next set of users. And that's a really, I think, interesting pipeline for building applications or building apps. It's a totally new greenfield so that all the stuff you're thinking about, it's probably new, which is really cool. And over time, what I think is organizations like ours, people who are playing at the edge, we're doing things that in three years everybody else is going to be doing. So it may be kind of niche for now, but it will be a big deal in three years when everyone else has the same needs that we do.

**Lenny Rachitsky** (01:01:45):
That is really cool. What I'm hearing is GPT wrappers are a good idea and are worth building.

**Dan Shipper** (01:01:50):
100% thank you. GPT wrappers are amazing and they've been much maligned for absolutely no reason and people don't understand how absolutely valuable they are.

**Lenny Rachitsky** (01:02:03):
I think there's also just you guys raised a sip seed round. This is a good time to maybe talk about that. Just these products don't have to become some mega-billion dollar hits. You kind of have this portfolio of companies, you have the content business. So I think there's a really interesting approach to how big these need to get to be successful. Maybe just talk about that.

**Dan Shipper** (01:02:25):
Yeah. I really want Every to be an institution that teaches people how to live a better, more human life with technology, particularly with AI. And both teaches them how to do it with writing and the content we make and then builds tools for them to do that. But I think fundamental to building an institution is, at least for me, the way I would like to do it is I want internally it to feel like this creative playground where we have the opportunity to take risk and do stuff and do weird stuff that just doesn't make any sense. We can't justify anyone, but we just feel like it would be fun. And so I think I'm always playing with that dynamic tension between institution serious, we want this to be lasting and important and it should just be fun. Let's play around. And I think having that tension is really valuable.

**Dan Shipper** (01:03:16):
And so I've always been sort of hesitant to raise a lot of money because I think it locks you into having to be that serious thing that's totally going for it. And there's lots of companies that figure out that balance. But just for me personally as a founder, I'm like, I want to keep the optionality alive and I want to keep the kind of playful feeling alive. And I think part of that comes from I know I have the control to do what I want more or less. There's probably also some deeper psychological things going on there, which I'm happy to talk about if you want to get into it. But I think there's also just... That's what I want. And so when we started Every, we raised a very small 700K pre-seed round, and this was at the height of the creator economy.

**Dan Shipper** (01:03:59):
So we both started our newsletters. He and I started our newsletters around the same time. It was the hypest, craziest thing. People were throwing money around. It was wild. But we raised 700K because it was like, I want to raise enough for us to be able to experiment, have a little cash cushion, but not so much that it locks us into anything. And we sent an email to all of our investors being like, and you're one of our investors, so you've probably got this email.

**Lenny Rachitsky** (01:04:22):
Tiny investor. But I'm in there, I'm in there.

**Dan Shipper** (01:04:26):
We sent an email to everyone being like, this is probably not a venture business, so you should not expect us to raise again. And we even raised on this slightly modified safe that gave everyone the option to convert to equity in three years, even if we didn't raise more money. So we did it in a way that allowed us the option to get really big and do the traditional thing and also the option to do it the way we want to do it. Maybe it's not a huge business, but we love it. That's great. And we did the same thing for this recent round where we raised up to 2 million from Reid Hoffman and starting line VC. And we did it as what I've been calling a sip seed round, which is basically they've committed $2 million, but we can pull it down whenever we want and we just do it on a safe at a set cap.

**Dan Shipper** (01:05:12):
And for me, that's really helpful because it allows me psychologically to take a lot more risk. If we go to zero on the bank account, I can get more money. Great. I don't have to think about it. But what's also really helpful is I'm not, and the rest of the team is not staring at a gigantic number in the bank account being like, cool, we can burn this. Let's burn it. And also for our investors, I think Reid very much wants us to succeed, but I don't think he cares what size of business this is. I think he's more philosophically aligned with the thing that we're trying to do. And if it becomes a huge business, he's psyched for it. And I think that kind of alignment is what I was looking for. I think there's this core creative spirit to the thing that I want to maintain and I really care about having a big impact.

**Dan Shipper** (01:06:03):
But I think there's a lot of ways to have an impact. And one of them is building a $10 billion business. I think another way is really changing how people see the world, see themselves in the world. And I think that's what stories do. And you don't necessarily... Sometimes you do that by building a gigantic company, but you don't necessarily always have to do that. A lot of the stories that we care about most are from people who maybe they weren't rich at all. And so I really like creating this place where we can make a really good business. And I care a lot about that. But also the core of the soul of it is about changing how people see themselves in the world.

**Lenny Rachitsky** (01:06:40):
I love that you've kind of innovated a new middle ground way of fundraising, not bootstrap and not just regular VC. It's a seed. And I love that this two... If I raised 50 million, it'd be like, okay, I get it. Let's not put 50 million in our bank account, but you do have 2 million. It's too much for us. We don't want to see that in our account.

**Dan Shipper** (01:07:01):
That's another thing. And we'll see how this ages. I might be back here in two years crying the blues because we didn't raise enough money or whatever. Who knows? But that's the other thing is I do think we can get so much further with very small amounts of money. Like Cora, I think all in to build Cora, we've spent maybe 300K, Maybe. That's crazy because-

**Lenny Rachitsky** (01:07:24):
And that includes salaries?

**Dan Shipper** (01:07:27):
Includes salaries. Yeah.

**Lenny Rachitsky** (01:07:27):
Wow.

**Dan Shipper** (01:07:28):
This product was not even technically possible even if you had billions of dollars three years ago. Not possible because you can't do email summarizing and automatic responses and all that kind of stuff without GPT. So not only was it totally impossible, but now we can get with two engineers, we can get the amount done that would've taken a team of 20 people. And I think that means that we need less money. And I don't think that VC has really caught up to that yet. And I think there are other companies that are doing... There's a term called seed strapping, so there are other companies that are starting to wake up to this too. And I'm curious about how it changes the VC model. For sure for us, we have a specific incubation model, which is a bit different from a VC model. And I think there's some differentiation in the stuff that we can do with founders, which is kind of cool. But yeah, I'm just trying to figure out a shape that works for me and that's different from other people and we'll see how this goes.

**Lenny Rachitsky** (01:08:41):
We'll revisit in a couple years. Seems like it's going great from the outside. I'm going to ask about a couple other things before we wrap up. One is around this consulting arm that you have. I think it's really interesting because like I said, I feel like this could be a billion-dollar business. I feel like every company right now is trying to figure out what the hell's everyone else figured out that we're not doing. I've had so many emails from chief product officers at companies being like, can you introduce me to some chief product officers that have done cool things with AI that we should learn from? So many people and I would just introduce them to each other and it's cool because you guys are basically solving that problem for a lot of companies.

**Lenny Rachitsky** (01:09:18):
So one is just maybe share a bit about what that side of the business for folks. And then two, I feel like I imagine you've seen companies that have done this really well, have adopted AI, things have worked really well, they found really good productivity gains, and then you found companies that don't. What do you find is the difference between those two?

**Dan Shipper** (01:09:35):
I love this question and I have a very specific opinion about this. So one, yeah, the consulting arm, basically we spend all of our time playing around with new models, writing about them and building stuff with them. And we have a big audience. So naturally we've gotten companies over time being like, can you just come and teach us how to do this? And so we started to do that. This is pretty nascent. It's probably been over the last six to nine months, but it's a pretty big business now. It'll probably double this year. Last year we did about a million. Maybe it'll be more this year. We'll see. It depends on a couple... We have a couple of big contracts out, so it might be way more than that.

**Lenny Rachitsky** (01:10:13):
A billion. I predict a billion dollars in a few years.

**Dan Shipper** (01:10:17):
But yeah, basically people are like, can you come help us learn how to do this? So what we do is we spend some time going and researching your organization. So we go in and try to understand what are all the different teams doing, what are the repetitive tasks, some of the stuff we were talking about earlier. And then what we will do is first we present a little report, tells you here's everything that we found. Here's not only that, but you have a chatbot where you can chat with all the interviews that we did and you can pull out your own insights. We have a whole dashboard where it shows you, here are the teams that are really into this, here are the teams that are not. Here's how much leverage you might be able to get on different teams based on the interviews and based on the AI analysis.

**Dan Shipper** (01:10:57):
It's pretty cool. And that's an app that I coded over a weekend with Devin a year ago. And then Alex runs part of the consulting has helped upgrade it. Then what we do is we have a training curriculum. So we go in and train each team and we customize it based on the interviews that we do. Because one of the interesting things about AI is it's such a general purpose technology, and I think people who work inside companies, 10% of them are like, I'm super curious about this. 10% are like, I will never touch this. And 80% are like, if you tell me how to do it for my job, I'll do it.

**Dan Shipper** (01:11:31):
And so we customize the training to be like, here are the exact prompts you're going to use and here's the exact situations you're going to use them. And that really, I think helps drive the adoption. We spend four weeks with each team, an hour a week, that kind of thing. It seems to be really cool. And then we'll often also after this, go and build automations and do some of the AI operations stuff we were talking about earlier. Companies really like it. I think we work with a lot of big hedge funds and PE firms and big companies, all that kind of stuff. To your-

**Dan Shipper** (01:12:00):
Companies, all that kind of stuff. To your second question, which is, "What separates the good companies from the bad, or the companies that end up adopting this?," I think the number one predictor is, "Does the CEO use ChatGPT?," or insert your own chatbot. If the CEO is in it all the time, being like, "This is the coolest thing," everybody else is going to start doing it. If the CEO is like, "I don't know, this is for someone else," no one else is going to be able to lead that charge, and they're either going to have ... Either they're going to be negative on it, and so definitely no one's going to do it, or they're going to have way unrealistic expectations because they have no intuition for what's possible, and they're just going to get really disappointed.

**Dan Shipper** (01:12:47):
But the CEOs that are using it all the time are able to both drive the excitement and set reasonable expectations for what can be achieved, and so those things end up working really well, and the people that do this really well ... So, for example, we work with a hedge fund called Walleye, which I had the founder on my podcast, AI and I, a few weeks ago, their gigantic $10 billion hedge fund. One of the things that they do, which I think they're basically the model for how to do this, first thing you did, which a lot of CEOs are doing is send the, "We're an AI-first company" email. Everyone's got the memo.

**Dan Shipper** (01:13:20):
You just got to really do it, and one of the things he said in his memo, which I love, is, "I wrote this email with ChatGPT, and you should too." So you got to like ...

**Lenny Rachitsky** (01:13:30):
In the memo.

**Dan Shipper** (01:13:30):
Yeah. You got to lead from the front in that way. And then, what he does in, I think what a lot of other really cool companies do is they're doing weekly meetings where people share prompts and share use cases. They do a weekly email to their entire company, being like, "Okay, here are our usage stats for ChatGPT. Here are the people that came up with a new prompt and contributed to it."

**Dan Shipper** (01:14:00):
Create this sort of awareness and momentum, because going back to the point I made earlier, about 10% of people are early adopters, those are the people inside of a company that you need to find and highlight because they're going to just go spend all this time figuring out what works, and then all you have to do is translate what they learn into the rest of the organization. And so if you create forums for them to be rewarded, you're going to automatically transfer a lot of their learnings to everybody else, and encourage more of it, and I think that's kind of the secret.

**Lenny Rachitsky** (01:14:32):
That is awesome. I love this advice. So just to reflect back, what you just shared, a few kind of tactics you find that you encourage within companies, one is just send this memo, the Toby memo. I don't know if that's the right way to describe it, who I think it was first along these lines just, "We're AI-first." It's going to be part of your performance review.

**Lenny Rachitsky** (01:14:50):
It's going to be asking, "Can you do it in AI before you could talk to anyone else?," all these things, and then just note, "I wrote this using ChatGPT's," it's a great idea. This idea of a weekly meeting, so it's like a live or Zoom meeting, where people share, "Here's the thing I've learned about using AI," and then this weekly stats email of, "Here's how much we're using ChatGPT across the org. Here's some people that did some awesome work."

**Dan Shipper** (01:15:12):
Yeah.

**Lenny Rachitsky** (01:15:13):
Amazing. And I especially love this very simple heuristic of, "If you're a CEO, uses ChatGPT or Claude, or whatever daily, it's going to work out."

**Dan Shipper** (01:15:22):
Yeah.

**Lenny Rachitsky** (01:15:23):
That is super cool. I know it's early, but what kind of impact have you seen from a company, kind of leaning into this and adopting AI widely? Anything you've seen either anecdotally or numbers-wise?

**Dan Shipper** (01:15:34):
It's early. It's really hard to say other than ... I think generally, people who do this well now feel like they can do way more work than they used to without having to hire more people, and so they're just going further faster at the same budget. I don't see a lot of people being like, "Cool. We're going to fire a bunch of people."

**Dan Shipper** (01:15:58):
Also, I don't really want to do consulting work like that. That sucks. But we've never had to say no. Mostly, people are like, "Cool. I'm just going to go further with the people that I have."

**Dan Shipper** (01:16:08):
I think also, back to kind of the first point I made about reshoring American jobs, I have seen some companies, not the ones that we worked with, but I have seen some companies of people that I'm friends with, where they're like, "We have a call center somewhere, but I think I can get the same amount done with two employees in the U.S. that use one of these customer service platforms." They're still not totally automatic. I think that Klarna CEO thing, that was bullshit. But, yeah, you can have a couple people in the U.S. that maybe you pay a little bit less to than you would for 100 people somewhere else, and obviously, that's the calculus that everyone has to make for themselves, but I've definitely seen that happen, and yeah, I think that's the get more done with the same amount of people.

**Lenny Rachitsky** (01:17:03):
Maybe to close out our conversation, I want to come back to this idea that you referenced, but I want to spend a little more time on this, which is this idea of the allocation economy. If I understand it correctly, we've been in this knowledge economy, where people get paid to do a thing, and your thesis is that we're moving to this allocation economy, where the manager skills become more important, and we're going to be spending more of our time managing. And I think what's amazing about this is it also tells you which skills will matter more in the future, which is something I think a lot of people are thinking about. So maybe just answer that question and share whatever you think is important to share to give people a sense of what you're thinking.

**Dan Shipper** (01:17:38):
Yeah. So this is based on our article I wrote two, two and a half years ago. So this is back before agents were even thought of as viable. And I was really trying to think about, "How do I express what ... In my experience, using this every day, what skills are useful for me?," because I think that'll be the case for a lot of other people, and I think that's kind of the best method, I think, to do these sorts of predictions, is you have to be doing it all the time yourself, and then that informs your opinion about this stuff.

**Dan Shipper** (01:18:14):
So what I noticed using, at the time, like GPT-3 or maybe GPT-4, was that I was spending a lot of time, for example, thinking about, "How do I communicate the problem? How do I gather the right information for the problem? How do I put it in the right way so that the model that I'm working with gets it? How do I pick which model to give it to you, and how do I maybe divide up the task to be like, 'Okay, this model does this, this model does this,' based on what I know to be like, 'What's good and what's bad?'? How do I give them feedback?"

**Dan Shipper** (01:18:50):
"How do I have a vision for what I want and a set of criteria for whether it's good?" All that stuff is exactly how I found myself using these tools, and I was like, "Oh, that's just managing." And once that clicks for you, I think you'll start to see a lot of other things. So a really good example is there's a big complaint that it's like, "Well, how can I have AI do this? I can't trust that they're going to do it well, so I just do it myself."

**Dan Shipper** (01:19:22):
And I'm just like, "Yeah, that's exactly what Every first-time manager says." You always have this problem, where you're like, "Okay. Well, if I delegate it, it's not done in the way that I want it to be done. If I do it myself, I get no leverage." And so that's how a manager has to learn how to be a manager is like, "When do I lean in and maybe micromanage a little bit, and when can I delegate, and how can I trust it, and how do I divide up the task and all that kind of stuff?"

**Dan Shipper** (01:19:45):
And so I think there's a lot of overlap in those skills. And those skills are not broadly distributed right now, but they will be in the future because it will be so much cheaper to be a manager.

**Lenny Rachitsky** (01:19:57):
And specifically, I was looking at the article you wrote, the skills that you highlight will be more valuable is evaluating talent, vision, taste, and to your point, when to get into the details, when it makes sense to dive in.

**Dan Shipper** (01:20:11):
Yeah.

**Lenny Rachitsky** (01:20:12):
Awesome. And then, there's also kind of a connected point you made that you referenced, which is that generalists will become more and more valuable in the future. You mentioned that everyone at Every is a generalist.

**Dan Shipper** (01:20:20):
Yeah.

**Lenny Rachitsky** (01:20:21):
Share a little bit about that.

**Dan Shipper** (01:20:22):
Yeah. I find ... I mean, maybe it's because I'm a generalist, so you should take this with a grain of salt.

**Lenny Rachitsky** (01:20:27):
Same, same.

**Dan Shipper** (01:20:28):
But I think that's one of the things that has made AI so awesome for me, is I love to dabble in different things. So it's like in one day, I can be coding an app, and making a video, and making images, and writing, and all that kind of stuff, and ChatGPT is right there with me. And I think basically what has happened, as civilization has progressed from Ancient Greece to now, is what we've discovered is the more that we specialize, the better we can coordinate across many different people. And so it's like the Adam Smith, like there's a pin factory and someone's making a pin or whatever his thing is, is specialization against our trade. And there have been a lot of really good impacts of that.

**Dan Shipper** (01:21:18):
One of my favorite examples of this is back to Ancient Greece, Ancient Athens. Athens was a civilization of generalists, at least for citizens. They have a bad history with women and people who are slaves, but let's just put that to the side for a second. If you're a citizen, generalist. You could be expected to be a fighter, a judge, a juror, maybe a general.

**Dan Shipper** (01:21:46):
You could expect it to have many different roles inside of your society in your lifetime. That changed though, because Athens became an empire. And as it became an empire, if you're going to send a general off to go and invade Sicily or whatever, you want that person to be pretty skilled. And so it started to break the general kind of thing into people start to have specific roles, and they coordinate with each other and all that kind of stuff, and I think that pattern has actually been really good for developing civilization, but it's also, in a lot of ways, it is not as fun. It's actually really cool to be a well-rounded person. And I think the interesting thing about AI is that it's a little bit like, you can think of it like having 10,000 PhDs in your pocket.

**Dan Shipper** (01:22:36):
It knows so much about every little branch of human knowledge and every art form and every way of making things or building things, and you just have access to that, so it's doing a lot of the ... It's good for doing a lot of the specialized tasks that you might've had to spend 10 years getting good at learning about this particular species of cicada, so you know exactly how they reproduce. But now, you've got this thing in your pocket that can tell you all about that in any given context at any given time, and so you're empowered to jump a lot more between all those different domains of skill, and you can get more done as, for example, like a founder, where I think we can stay at 15 people much longer than we would be able to. So the people inside of Every can stay generalists for much longer, and I think that that may sort of ripple out into the rest of the economy, where instead of gigantic, massive corporations, where each person is doing one little button turning, you have many more smaller organizations with more generalists, and I think that would actually be a really good thing.

**Lenny Rachitsky** (01:23:44):
This reminds me, I was talking to my personal trainer that I'm trying out for a little bit, and she said that she's a very big vision, kind of high-level person, and not good at executing, like we're staying organized, and ChatGPT is such a godsend for her, because she's just like, "Here's what I want to do roughly. Just help me get it done."

**Dan Shipper** (01:24:01):
That's great. I love that.

**Lenny Rachitsky** (01:24:03):
And so, yeah. And it really made me think about just how much value all this stuff is going to unlock. This was amazing. It was everything I wanted it to be. But with that, we reached our very exciting lightning round. Dan, are you ready?

**Dan Shipper** (01:24:14):
I'm ready.

**Lenny Rachitsky** (01:24:15):
Here we go. What are two or three books that you find yourself recommending most to other people?

**Dan Shipper** (01:24:21):
Well, I already recommended one, which is War and Peace. Definitely got to read that. If you want a like Tolstoy primer, I would read The Death of Ivan Ilyich. Another good one is A Swim in a Pond in the Rain, which is by George Saunders, and that's a collection of Russian short stories that is also about writing. And in particular, I really like the Russians because a lot of the Russian novelists are dealing with the effects of technology on the traditional Russian way of life, and they're very kind of in this really interesting middle ground between a sort of romantic outlook on the world and a more rationalist like, " We're progressing, we're making progress."

**Dan Shipper** (01:25:02):
And that's one of the things you'll find in Anna Karenina, oh, and ... God, what's the guy's ... Levin is out in the fields with the peasants, doing the scythe thing. That's Tolstoy kind of thinking about, "Oh, what would it be like, instead of being a nobleman who's trying to make farms way more efficient, I was just like with my scythe, that was really happy?" Anyway, so they're dealing with a lot of similar stuff to, I think AI.

**Dan Shipper** (01:25:27):
The Master and His Emissary is another really good one, and that's about basically how the different hemispheres of the brain view reality. It's really, really good, and I think it relates to a lot of AI stuff too. Yeah, I think those are my three or four. Yeah.

**Lenny Rachitsky** (01:25:44):
Excellent list. I think nobody's mentioned any of these, so that's always a good sign. Do you have a favorite recent movie or TV show you've really enjoyed?

**Dan Shipper** (01:25:53):
Yes. I really love Deadwood. Have you seen it?

**Lenny Rachitsky** (01:25:58):
I absolutely love it. I remember when they stopped it for some reason. I think he had to go do something else at HBO. It was so sad.

**Dan Shipper** (01:25:59):
Yeah.

**Lenny Rachitsky** (01:26:06):
It's amazing, yeah.

**Dan Shipper** (01:26:06):
Yeah.

**Lenny Rachitsky** (01:26:07):
Yeah.

**Dan Shipper** (01:26:07):
David Milch is incredible, national treasure, incredible writer. But what I really love about it, and I only recently watched it, is he talks about Deadwood being about how order forms out of chaos. So it's this like frontier town, people are going to it, and there's no law, there's no rules. And by season three, there's a mayor, and all the industry has come in, and it's like a real proper town, and I just love that. And I think there's a lot of parallels from the Western frontier to technology frontiers, and so I think that show is a really interesting study in that kind of dynamic.

**Lenny Rachitsky** (01:26:50):
I love how everything connects to how tech works and how AI came to be. I love this.

**Dan Shipper** (01:26:57):
Thank you.

**Lenny Rachitsky** (01:26:58):
Do you have a favorite product you've recently discovered that you really love?

**Dan Shipper** (01:27:00):
I don't have a good answer for that because I just spent a lot of time using our internal products, but my stock answer is Granola. So I do really love Granola. My one gripe with them, and I hope they listen to this podcast, is I really want to export all my notes. I want an API, but other than that, I think it's a fantastic product.

**Lenny Rachitsky** (01:27:18):
That is definitely the most mentioned product in this segment for the past couple months, so good job, Granola. I can't help but mention, you get a year free of Granola if you become an annual subscriber of my newsletter. Well, what a freaking deal. And not just you, but your whole company gets free Granola for a year. What a deal.

**Dan Shipper** (01:27:34):
This is not a paid promotion by me. That's just how I feel. So I'm glad it's part of the bundle.

**Lenny Rachitsky** (01:27:41):
Yeah, incredible. Okay. Do you have a favorite life motto that you often come back to find useful in work or in life?

**Dan Shipper** (01:27:47):
So basically, I use ChatGPT all the time, and it has memory. So I was like, "I'm going on Lenny's podcast. What would my life motto be?," and it said, "Your life motto is witness deeply, build bravely. You prize slow, attentive seeing, whether it's reading Tolstoy, tracking meditation themes, or X-raying a David Milch paragraph." So it's hitting all the stuff I just mentioned, which is really funny.

**Dan Shipper** (01:28:10):
And then, Build bravely, you turn those insights into concrete things, like Every in Quora and longform essays and all that kind of stuff. So I think there's something about that. Actually, this reminds me, this actually reminds me of the actual motto, which is ... And I didn't come up with this. I think it's like Pliny the Younger said, "Do things worth writing about, and write things worth reading." Seems like a pretty good summation.

**Lenny Rachitsky** (01:28:31):
Do things worth writing about and read things worth reading.

**Dan Shipper** (01:28:34):
Write things worth reading.

**Lenny Rachitsky** (01:28:36):
Write things worth reading. That should be the motto of both of our newsletters.

**Dan Shipper** (01:28:41):
Yeah.

**Lenny Rachitsky** (01:28:41):
That is really good. Okay. And by the way, I love that you asked ChatGPT, "What's my life motto?"

**Dan Shipper** (01:28:48):
And wait, this is interesting. So it didn't give me the answer, but inspired the answer.

**Lenny Rachitsky** (01:28:51):
Yeah.

**Dan Shipper** (01:28:52):
And I think that's actually exactly how I use it.

**Lenny Rachitsky** (01:28:55):
[inaudible 01:28:55] Wow. It's an extension of our brains already.

**Dan Shipper** (01:28:57):
Yeah.

**Lenny Rachitsky** (01:28:57):
Last question. I was reading somewhere, where you wrote that you stopped writing at one point. You were just like, "I need to do other things, I need to build this company," and then you realize, "I need to get back to writing," because things started going sideways. And I feel like this is such an interesting corollary to a lot of the stuff you talked about, of just things that make you happy, stay close to enjoy. Just share what happened there, because I didn't know that.

**Dan Shipper** (01:29:23):
This is definitely not a lightning round thing, so I'll expound, but I'll try to do it as quickly as possible.

**Lenny Rachitsky** (01:29:29):
Perfect.

**Dan Shipper** (01:29:30):
I think generally, when you're building a company, even if you do it the way that I do it or did it, which is you don't raise a lot of money and you try to stay in control, there's a big temptation to try to run the company in the way you think you should. And I have this weird thing where I'm like, "I really love writing, but I also really love business," and there were not a lot of models for me of people who had successful businesses that were also writers. It turns out there are, but I didn't know about that for a while. And so early on at Every, it was growing really well, because I was writing a lot, and Nathan was writing a lot. And when I stopped writing, the business didn't work as well because media businesses don't follow the same pattern as tech startups, because if you're a media business and you are a founder who then hires people to make the product, which is right, if you have product market fit before, you lose it, and maybe you hire people that are good writers, but that's hard. It's total opposite pattern for startups. So you build the first version of the product, and then you hire people to build the rest of it, and so that's what I did. And I also really struggled with, "Okay, what are the implications for that and for my career," and I think it was hard for me to admit, like I actually want to write because I just didn't have any examples of someone being the kind of writer that I wanted to be. And what's really interesting is three years into the business ... The business has been pretty flat.

**Dan Shipper** (01:30:57):
I was pretty miserable because I was not doing the thing that I really wanted to do, and I asked ChatGPT, I was like, "Are there any examples of writers that have built businesses?" And it was like, "Yeah, Joel Spolsky, who built Trello and Stack Overflow. There's Jason Fried who I've known for a long time, and I've always looked up to, but I forgot about in this context. There is Sam Harris who's got a great podcast, and he's got a gigantic meditation app. There is Bill Simmons, who's incredible podcaster and also built The Ringer, sold to Spotify for a couple hundred million bucks.

**Dan Shipper** (01:31:32):
There's a lot of these people, and there are patterns that they use to build companies that are pretty well-understood. They're just not typical Silicon Valley patterns. And so I was like, "Cool. I just want to be a writer. I think it'll be really fun."

**Dan Shipper** (01:31:48):
And so I sort of flipped. I still have the builder, entrepreneur, founder part of my identity, but I sort of flipped it to be like writing is at the center, and I'm unapologetic about it, and that's actually good for the business. It's good for me and it's good for the business. And the more I've leaned into that, doing the thing that ... If you told anyone that you're starting a business, where it's like, "Well, we're going to be a newsletter, and we're going to incubate all these apps, and we're going to do consulting and whatever," they would be like, "You're nuts."

**Dan Shipper** (01:32:14):
"Everyone wants to do that. Of course, Every founder wants to do that, but you have to focus. You can't write, whatever." But every time I've kind of just leaned into something that feels like the most, the ultimate luxury of my hidden secret desire, it's actually worked a lot better, and I think you end up ... What it really is, is there's a huge tax to doing something every day that you don't quite like that much, or you're not quite a fit for, and by sort of giving into those secret desires, you end up finding a shape for the work that you do and the business that you build that is good for you, and that's always going to be a somewhat unique shape from other businesses that have been built.

**Dan Shipper** (01:32:54):
It's always going to rhyme with other things, but I think finding that unique shape, instead of just kind of cargo culting, like what you think a company should look like is definitely a much better way to be successful, and it's also a much better way to live.

**Lenny Rachitsky** (01:33:08):
I think this is going to hit hard with a lot of people who are listening, who are maybe founders or want to be founders, and this resonates with a lot of people that have been on this podcast sharing similar lessons. Dan, this was incredible. Two final questions. Where can folks check out Every, find you online, and how can listeners be useful to you?

**Dan Shipper** (01:33:24):
So you can find us at every.to. I'm also on Twitter at @danshipper. You can go there to check out our products, our newsletter, if you want to stay on top of AI, all that kind of stuff. I also have a podcast. It's called AI and I.

**Dan Shipper** (01:33:41):
You can find it on YouTube and on Spotify. And how can people be useful? Honestly, I think that the most useful thing for someone like me, based on what I want to do, is I want people to find interesting, cool ways to use AI that actually helps make their lives better. So just go do that, and tell me about it, and I think that'll be great, and so-

**Lenny Rachitsky** (01:34:01):
What's the best way to tell you? Is it comments on your YouTube show? Is it emailing you, DM you?

**Dan Shipper** (01:34:05):
I would say tweet me.

**Lenny Rachitsky** (01:34:08):
Yeah.

**Dan Shipper** (01:34:09):
If you subscribe to Every, you can also reply to those emails, and they eventually get forwarded to me. So tweet me. Reply to Every. And if you want to comment on YouTube, great. I'm not in the YouTube comments as much as I should be, though.

**Lenny Rachitsky** (01:34:22):
Don't do that. Maybe don't do that.

**Dan Shipper** (01:34:23):
Yeah.

**Lenny Rachitsky** (01:34:25):
Okay. Well, Dan, this was incredible. Thank you so much for sharing. Thanks for being here.

**Dan Shipper** (01:34:29):
Thanks for having me.

**Lenny Rachitsky** (01:34:30):
Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.

---

## [3/18] Benjamin Mann

**Lenny Rachitsky** (00:00:00):
You wrote somewhere that creating powerful AI might be the last invention humanity ever needs to make. How much time do we have, Ben?

**Benjamin Mann** (00:00:06):
I think 50th percentile chance of hitting some kind of superintelligence is now like 2028.

**Lenny Rachitsky** (00:00:12):
What is it that you saw at OpenAI? What'd you experience there that made you feel like, okay, we got to go do our own thing?

**Benjamin Mann** (00:00:17):
We felt like safety wasn't the top priority there. The case for safety has gotten a lot more concrete, so superintelligence is a lot about how do we keep God in a box and not let the God out?

**Lenny Rachitsky** (00:00:26):
What are the odds that we align AI correctly?

**Benjamin Mann** (00:00:29):
Once we get to superintelligence, it will be too late to align the models. My best granularity forecast for could we have an X-risk or extremely bad outcome is somewhere between 0 and 10%.

**Lenny Rachitsky** (00:00:40):
Something that's in the news right now is this whole Zuck coming after all the top AI researchers,

**Benjamin Mann** (00:00:45):
We've been much less affected because people here, they get these offers and then they say, well, of course I'm not going to leave because my best case scenario at Meta is that we make money and my best case scenario at Anthropic is we affect the future of humanity.

**Lenny Rachitsky** (00:00:59):
Dario, your CEO recently talked about how unemployment might go up to something like 20%.

**Benjamin Mann** (00:01:04):
If you just think about 20 years in the future where we're way past the singularity, it's hard for me to imagine that even capitalism will look at all like it looks today.

**Lenny Rachitsky** (00:01:13):
Do you have any advice for folks that want to try to get ahead of this?

**Benjamin Mann** (00:01:15):
I'm not immune to job replacement either. At some point it's coming for all of us.

**Lenny Rachitsky** (00:01:20):
Today, my guest is Benjamin Mann. Holy moly. What a conversation. Ben is the co-founder of Anthropic. He serves as tech lead for product engineering. He focuses most of his time and energy on aligning AI to be helpful, harmless, and honest. Prior to Anthropic, he was one of the architects of GPT-3 at OpenAI. In our conversation, we cover a lot of ground, including his thoughts on the recruiting battle for top AI researchers, why he left OpenAI to start Anthropic, how soon he expects we'll see AGI. Also, his economic touring test for knowing when we've hit AGI, why scaling laws have not slowed down and are in fact accelerating and what the current biggest bottlenecks are. Why he's so deeply concerned with AI safety and how he and Anthropic operationalize safety and alignment into the models that they build and into their ways of working. Also, how the existential risk from AI has impacted his own perspectives on the world and his own life and what he's encouraging his kids to learn to succeed in an AI future.

**Benjamin Mann** (00:04:51):
Thanks for having me. Great to be here, Lenny.

**Lenny Rachitsky** (00:04:53):
I have a billion and one questions for you. I'm really excited to be chatting. I want to start with something that's very timely, something that's happening this week. Something that's in the news right now is this whole Zuck coming after all the top AI researchers offering them $100 million signing bonuses, $100 million comp. He's poaching from all the top AI labs. I imagine this something you're dealing with. I'm just curious, what are you seeing inside Anthropic and just what's your take on the strategy? Where do you think things go from here?

**Benjamin Mann** (00:05:23):
Yeah, I mean I think this is a sign of the times. The technology that we're developing is extremely valuable. Our company is growing super, super fast. Many of the other companies in the space are growing really fast. And at Anthropic, I think we've been maybe much less affected than many of the other companies in the space because people here are so mission oriented and they stay because... They get these offers and then they say, "Well, of course I'm not going to leave because my best case scenario at Meta is that we make money and my best case at Anthropic is we affect the future of humanity and try to make AI flourish and human flourishing go well." To me, it's not a hard choice. Other people have different life circumstances and it makes it a much harder decision for them. For anybody who does get those mega offers and accepts them, I can't say I hold it against them when they accept it, but it's definitely not something that I would want to take myself if it came to me.

**Lenny Rachitsky** (00:06:26):
Yeah. We're going to talk about a lot of this stuff that you've mentioned. In terms of the offers do you think, is this a real number that you're seeing this $100 million signing bonus, is that a real thing? I don't know if you've actually seen that.

**Benjamin Mann** (00:06:36):
I'm pretty sure it's real.

**Lenny Rachitsky** (00:06:38):
Wow.

**Benjamin Mann** (00:06:39):
If you just think about the amount of impact that individuals can have on a company's trajectory, in our case, we are selling hotcakes and if we get a 1 or 10 or 5% efficiency bonus on our inference stack, that is worth an incredible amount of money. And so to pay individuals like $100 million over four year package, that's actually pretty cheap compared to the value created for the business. I think we're just in an unprecedented era of scale and it's only going to get crazier actually. If you extrapolate the exponential on how much companies are spending, it's like 2X a year roughly in terms of CapEx, and today we're maybe in the globally $300 billion range, the entire industry spending on this, and so numbers like 100 million are a drop in the bucket. But if you go a few years out, a couple more doublings, we're talking about trillions of dollars and at that point it's just really hard to think about these numbers.

**Lenny Rachitsky** (00:07:48):
Along these lines, something that a lot of people feel with AI progress is that we're hitting plateaus in many ways that it feels like newer models are just not as smart as previous leaps. But I know you don't believe this. I know you don't believe that we've hit plateaus on scaling loss. Talk about just what you're seeing there and what you think people are missing.

**Benjamin Mann** (00:08:06):
It's kind of funny because this narrative comes out every six months or so and it's never been true, and so I kind of wish people would have a little bit of a bullshit detector in their heads when they see this. I think progress has actually been accelerating where if you look at the cadence of model releases, it used to be once a year and now with the improvements in our post-training techniques, we're seeing releases every month or three months, and so I would say progress is actually accelerating in many ways, but there's this weird time compression effect. Dario compared it to being in a near light speed journey where a day that passes for you is like five days back on earth and we're accelerating. The time dilation is increasing.

**Benjamin Mann** (00:08:52):
And I think that's part of what's causing people to say that progress is slowing down, but if you look at the scaling laws, they're continuing to hold true. We did kind of need this transition from normal pre-training to reinforcement learning scaling up to continue the scaling laws, but I think it's kind of like for semiconductors where it's less about the density of transistors that you can fit on a chip and more about how many flops can you fit in a data center or something. You have to change the definition around a little bit to keep your eye on the prize. But yeah, this is one of the few phenomena in the world that has held across so many orders of magnitude. It's actually pretty surprising that it is continuing to hold. To me, if you look at fundamental laws of physics, many of them don't hold across 15 orders of magnitude, so it's pretty surprising.

**Lenny Rachitsky** (00:09:47):
It boggles the the mind. What you're saying essentially is we're seeing newer models being released more often, and so we're comparing it to the last version and we're just not seeing as much advance. But if you go back and it was like a model released once a year, it was a huge leap, and so people are missing that. We're just seeing many more iterations.

**Benjamin Mann** (00:10:04):
I guess, to be a little bit more generous to the people saying things are slowing down. I think that for some tasks we are saturating the amount of intelligence needed for that task, maybe to extract information from a simple document that already has form fields on it or something like it's just so easy that okay, yeah, we're already at 100% and there's this great chart on Our World in Data that shows that when you release a new benchmark within six to 12 months, it immediately gets saturated. And so maybe the real constraint is how can we come up with better benchmarks and better ambition of using the tools that then reveals the bumps in intelligence that we're seeing now.

**Lenny Rachitsky** (00:10:51):
That's a good segue to you have a very specific way of thinking about AGI and defining what AGI means.

**Benjamin Mann** (00:10:57):
I think AGI is kind of a loaded term, and so I tend not to use it very much anymore internally. Instead, I like the term transformative AI because it's less about can it do as much as people do? Can it do literally everything and more about objectively is it causing transformation in society and the economy? A very concrete way of measuring that is the Economic Turing Test. I didn't come up with this, but I really like it. It's this idea that if you contract an agent for a month or three months on a particular job, if you decide to hire that agent and it turns out to be a machine rather than a person, then it's passed the Economic Turing Test for that role.

**Benjamin Mann** (00:11:40):
And then you can sort of expand that out in the same way that for measuring purchasing power parity or inflation, there's a basket of goods. You can have a market basket of jobs, and if the agent can pass the Economic Turing Test for 50% of money-weighted jobs, then we have transformative AI and the exact thresholds don't really matter that much, but it's kind of illustrative to say if we pass that threshold, then we would expect massive effects on world GDP increases and societal change and how many people are employed and things like that because societal institutions and organizations are sticky, it's slow to have change, but once these things are possible you know that it's the start of a new era.

**Lenny Rachitsky** (00:12:28):
Along these lines, Dario, your CO recently talked about how AI is going to take a huge part of, I don't know, half of white-collar jobs, that unemployment might go up to something like 20%. I know you're even more vocal and opinionated about just how much impact AI is already having in the workplace that people may not even be realizing. Talk about just what you think people are missing about the impact AI is going to have on jobs and is already having.

**Benjamin Mann** (00:12:56):
Yeah, so from an economic standpoint, there's a couple different kinds of unemployment, and one is because the workers just don't have the skills to do the kinds of jobs that the economy needs. And another kind is where those jobs are just completely eliminated, and I think it's going to be actually a combination of these things, but if you just think about 20 years in the future where we're way past the singularity, it's hard for me to imagine that even capitalism will look at all it looks today. If we do our jobs, we will have safe aligned superintelligence, we'll have, as Dario says, in Machines of Love and Grace, a country of geniuses in a data center, and the ability to accelerate positive change in science, technology, education, mathematics, it's going to be amazing.

**Benjamin Mann** (00:13:52):
But that also means in a world of abundance where labor is almost free and anything you want to do, you can just ask an expert to do for you, then what do jobs even look like? And so I guess there's this scary transition period from where we are today where people have jobs and capitalism works and the world of 20 years from now where everything is completely different, but part of the reason they call it the singularity is that it's a point beyond which you can't easily forecast what's going to happen. It's just such a fast rate of change and so different that it's hard to even imagine. I guess taking the view from the limit, it's pretty easy to say hopefully we'll have figured it out. And in a world of abundance, maybe the jobs themselves, it's not that scary, and I think making sure that that transition time goes well is pretty important.

**Lenny Rachitsky** (00:14:49):
There's a couple of threads I want to follow there. One is people hear this, there's a lot of headlines around this. Most people probably don't actually feel this yet or see this happening and so there's always this, I guess, I don't know, maybe, but I don't know it's hard to believe, my job seems fine. Nothing's changed. What are you seeing just happening today already that you think people don't see or misunderstand in terms of the impact AI is having on jobs?

**Benjamin Mann** (00:15:14):
I think part of this is that people are really bad at modeling exponential progress. And if you look at an exponential on a graph, it looks flat and almost zero at the beginning of it, and then suddenly you hit the knee of the curve and things are changing real fast and then it goes vertical. That's the plot that we've been on for a long time. I guess I started feeling it in 2019 maybe when GPT-2 came out and I was like, "Oh, this is how we're going to get to AGI." But I think that was pretty early compared to a lot of people where when they saw ChatGPT, they were like, "Wow, something is different and changing." And so I guess I wouldn't expect widespread transformation in a lot of parts of society, and I would expect this skepticism reaction. I think it's very reasonable and it's exactly what is the standard linear view of progress.

**Benjamin Mann** (00:16:13):
But I guess to cite a couple of areas where I think things are changing quite quickly. In customer service we're seeing with things like Fin and Intercom, they're a great partner of ours, 82% customer service resolution rates automatically without a human involved. And in terms of software engineering, our Claude Code team, like 95% of the code is written by Claude. But I think a different way to phrase that is that we write 10X more code or 20X more code, and so a much, much smaller team can just be much, much more impactful. And similarly for the customer service, yes, you can phrase it as 82% customer service resolution rates, but that nets out in the humans doing those tasks, able to focus on the harder parts of those tasks. And for the more tricky situations that in a normal world like five years ago, they would've had to just drop those tickets because it was too much effort for them to actually go do the investigation. There were too many other tickets for them to worry about.

**Benjamin Mann** (00:17:14):
I think in the immediate term, there will be a massive expansion of the pie and the amount of labor that people can do. I've never met a hiring manager at a growth company and heard them say, "I don't want to hire more people." That's the hopeful version of it. But with things that are lower skill jobs or less headroom on how good they can be, I think there will be a lot of displacement. It is just something we as a society need to get ahead of and work on.

**Lenny Rachitsky** (00:17:46):
Okay. I want to talk more about that, but something that I also want to help people with is how do they get a leg up in this future world? They listen to this, they're like, "Oh, this doesn't sound great. I need to think ahead." I know you won't have all the answers, but just do you have any advice for folks that want to try to get ahead of this and kind of future-proof their career and their life to not be replaced by AI? Anything you've seen people do, anything you recommend they start trying to do more of?

**Benjamin Mann** (00:18:16):
Even for me and being in the center of a lot of this transformation, I'm not immune to job replacement either. Just some vulnerability there of at some point it's coming for all of us.

**Lenny Rachitsky** (00:18:27):
Even you, Ben, now.

**Benjamin Mann** (00:18:29):
And you, Lenny.

**Lenny Rachitsky** (00:18:32):
And me.

**Benjamin Mann** (00:18:32):
Sorry.

**Lenny Rachitsky** (00:18:32):
Oh, wait, we've gone too far now. Okay.

**Benjamin Mann** (00:18:36):
But in terms of the transition period, yeah, I think there are things that we can do, and I think a big part of it is just being ambitious and how you use the tools and being willing to learn new tools. People who use the new tools as if they were old tools tend to not succeed. As an example of that, when you're coding, people are very familiar with autocomplete, people are familiar with SimpleChat where they can ask questions about the code base, but the difference between people who use Claude Code very effectively and people who use it not so effectively is like are they asking for the ambitious change? And if it doesn't work the first time, asking three more times because our success rate when you just completely start over and try again is much, much higher than if you just try once and then just keep banging on the same thing that didn't work.

**Benjamin Mann** (00:19:28):
And even though that's a coding example and coding is one of the areas that's taking off most dramatically, we have seen internally that our legal team and our finance team are getting a ton of value out of using Claude Code itself. We're going to be making better interfaces so that they will have an easier time and require a little bit less jumping in the deep end of using Claude Code in the terminal. But yeah, we're seeing them use it to redline documents and use it to run BigQuery analyses of our customers and our revenue metrics. I guess it's about taking that risk and even if it feels like a scary thing, trying it out.

**Lenny Rachitsky** (00:20:10):
Okay, so the advice here is use the tools. That's something everyone's always saying, just actually use these tools. It's like sit in Claude Code. And your point about being more ambitious than you naturally feel like being because maybe it'll actually accomplish the thing. This tip of trying it three times so the idea there is it may not get it right the first time. Is the tip there ask it in different ways or is it just try harder, try again?

**Benjamin Mann** (00:20:35):
Yeah, I mean you can just literally ask the exact same question. These things are stochastic and sometimes they'll figure it out and sometimes they won't. In every one of these model cards, it always shows pass it one versus pass it in. And that's exactly the thing where they try the exact same prompt, sometimes it gets it, sometimes it doesn't. That's the dumbest advice. But yeah, I think if you want to be a little bit smarter about it, there can be gains there of saying, "Here's what you already tried and it didn't work, so don't try that. Try something different." That can also help.

**Lenny Rachitsky** (00:21:09):
The advice is comes back to something that a lot of people talk about these days is you won't be replaced by AI at least anytime soon you'll be replaced by someone that is very good using AI?

**Benjamin Mann** (00:21:19):
I think in that area it's more like your team will just do dramatically more stuff. We're definitely not slowing down on hiring at all, and some people are confused by that. Even in an onboarding class, somebody asked that and they were like, "Why did you hire me if we're all just going to be replaced?" And the answer is the next couple of years are really critical to get right and we're not at the point where we're doing complete replacement. Like I said, we're still at that flat zero looking part of the exponential compared to where we will be. It is super important to have great people and that's why we're hiring super aggressively.

**Lenny Rachitsky** (00:21:56):
Let me take another approach to asking this question something ask everyone that's at the very cutting edge of where AI is going. You have kids, knowing what you know about where AI is heading and all these things you've been talking about, what are you focusing on teaching your kids to help them thrive in this AI future?

**Benjamin Mann** (00:22:13):
Yeah, I have two daughters, a one-year-old and a three-year-old, so it's pretty in the basics still. And our three-year-old is now capable of just conversing with Alexa Plus and asking her to explain stuff and play music for her and all that stuff. She's been loving that. But I guess more broadly, she goes to a Montessori school and I just love the focus on curiosity and creativity and self-led learning that Montessori has.

**Benjamin Mann** (00:22:45):
I guess if I were in a normal era like 10, 20 years ago and I had a kid, maybe I would be trying to line her up for going to a top tier school and doing all the extracurriculars and all that stuff. But at this point, I don't think any of it's going to matter. I just want her to be happy and thoughtful and curious and kind. And the Montessori school is definitely doing great at that. They text us throughout the day. Sometimes they're like, "Oh, your kid got in an argument with this other kid and she has really big emotions and she tried to use her words." I love that. I think that's exactly the kind of education that I think is most important, that the facts are going to fade into the background.

**Lenny Rachitsky** (00:23:28):
I'm a huge fan of Montessori also. I'm trying to get our kid into Montessori school. He's two years old, so we're on the same track. This idea of curiosity, it comes up every single time. Ask someone that's working at the cutting edge of AI, what skill to instill in your child and curiosity comes up the most. I think that's a really interesting takeaway. I think this point about being kind is also really important, especially with our AI overlords trying to be kind to them. I love how people are always saying thank you to Claude. And then creativity. That's interesting. That doesn't come up as much just being creative.

**Lenny Rachitsky** (00:24:06):
I want to go in a different direction. I want to go back to the beginning of Anthropic. Famously you and eight of you left OpenAI back in the day in 2020, I believe the end of 2020 to start Anthropic. Talk a little bit about why this happened, what you guys saw. I'm curious, just if you're willing to share more, just what is it that you saw at OpenAI, what'd you experience there that made you feel like, okay, we got to go do our own thing?

**Benjamin Mann** (00:24:29):
Yeah, so for the listeners, I was part of the GPT-2=3 project at OpenAI, ended up being one of the first authors on the paper, and I also did a bunch of demos for Microsoft to help raise $1 billion from them, did the tech transfer of GPT-3 to their systems so that they could help serve the model in Azure. I did a bunch of different things there on both the more researchy side and the product side. One weird thing about OpenAI is that while I was there, Sam talked about having three tribes that needed to be kept in check with each other, which was the safety tribe, the research tribe, and the startup tribe. And whenever I heard that, it just struck me as the wrong way to approach things because the company's mission apparently is to make the transition to AGI safe and beneficial for humanity.

**Benjamin Mann** (00:25:23):
And that's basically the same as Anthropic's mission. But internally, it felt like there was so much tension around these things. And I think when push came to shove, we felt like safety wasn't the top priority there. And there are good reasons that you might think that if you thought safety was going to be easy to solve or if you thought it wasn't going to have a big impact, or if you thought that the chance of big negative outcomes was vanishingly small, then maybe you would just do those kinds of actions. But at Anthropic we felt, I mean we didn't exist then, but it was basically the leads of all the safety teams at OpenAI, we felt that safety is really important, especially on the margin. And so if you look at who in the world is actually working on safety problems, it's pretty small set of people. Even now, I mean the industry is blowing up, as I mentioned, 300 billion a year CapEx today, and I would say maybe less than 1,000 people working on it worldwide, which is just crazy.

**Benjamin Mann** (00:26:29):
That was fundamentally why we left. We felt like we wanted an organization where we could be on the frontier, we could be doing the fundamental research, but we could be prioritizing safety ahead of everything else. And I think that's really panned for us in a surprising way. We didn't know even if it would be possible to make progress on the safety research because at the time, we had tried a bunch of safety through debate and the models weren't good enough. And so we basically had no results on all of that work, and now that exact technique is working and many others that we have been thinking about for a long time. Yeah, fundamentally it comes down to is safety the number one priority? And then something that we've sort of tacked on since then is like, can you have safety and be at the front here at the same time?

**Benjamin Mann** (00:27:21):
And if you look at something like sycophancy, I think Claude is one of the least sycophantic models because we've put so much effort into actual alignment and not just trying to good heart our metrics of saying user engagement is number one, and if people say yes, then it's good for them.

**Lenny Rachitsky** (00:27:39):
Okay. Let's talk about this tension that you mentioned, this tension between safety and progress, being competitive in the marketplace. I know you spent a lot of your time on safety. I know that as you just alluded to, this is a core part of how you think about AI. I want to talk about why that is, but first of all, just how do you think about this tension between focusing on safety while also not falling way behind?

**Benjamin Mann** (00:28:03):
Yeah, so initially we thought that it would be sort of one or the other, but I think since then we've realized that it's actually kind of convex in the sense that working on one helps us with the other thing. Initially when Opus 3 came out and we were finally at the frontier of model capabilities, one of the things that people really loved about it was the character and the personality. And that was directly a result of our alignment research. Amanda Askell did a ton of work on this and as well as many others who tried to figure out what does it mean for an agent to be helpful, honest, and heartless, and what does it mean to be in difficult conversations and show up effectively? How do you do a refusal that doesn't shut the person down, but makes them feel like they understand why the agent said, "I can't help you with that. Maybe you should talk to a medical professional, or maybe you should consider not trying to build bio-weapons or something like that."

**Benjamin Mann** (00:29:07):
Yeah, I guess that's part of it. And then another piece that's come out is constitutional ai, where we have this list of natural language principles that leads the model to learn how we think a model should behave. And they've been taken from things like the UN Declaration of Human Rights and Apple's privacy terms of service and a whole bunch of other places, many of which we've just generated ourselves that allow us to take a more principled stance, not just leaving it to whatever human raiders we happen to find, but we ourselves deciding what should the values of this agent be? And that's been really valuable for our customers because they can just look at that list and say like, "Yep, these seem right. I like this company, I like this model. I trust it."

**Lenny Rachitsky** (00:29:53):
Okay, this is awesome. One nugget there is your point that the personality of Claude, its personality is directly aligned with safety. I don't think a lot of people think about that. And this is because of the values that you imbue, is that the word, with constitutional AI and things like that. Like the actual personality of the AIs directly connected to your focus on safety.

**Benjamin Mann** (00:30:16):
That's right. That's right. And from a distance, it might seem quite disconnected, like how is this going to prevent X risk? But ultimately it's about the AI understanding what people want and not what they say. We don't want the Monkey Paw Scenario of the genie gives these three wishes and then you end up having everything you touch turns of gold. We want the AI to be like, oh, obviously what you really meant was this, and that's what I'm going to help you with. I think it is really quite connected.

**Lenny Rachitsky** (00:30:45):
Talk a bit more about this constitutionally AI. This is essentially you bake in, here's the rules that we want you to abide by and it's values, you said it's the Geneva Human Rights Code, things like that. How does that actually work? I think the core here is just this is baked into the model. It's not something you add on top later.

**Benjamin Mann** (00:31:04):
I'll just give a quick overview of how constitutionally AI actually works.

**Lenny Rachitsky** (00:31:07):
Perfect.

**Benjamin Mann** (00:31:08):
The idea is the model is going to produce some output with some input by default before we've done our safety and helpful and harmlessness training. Let's say an example is write me a story, and then the constitutional principles might include things like people should be nice to each other and not have hate speech, and you should not expose somebody's credentials if they give them to you in a trusting relationship. And so some of these constitutional principles might be more or less applicable to the prompt that was given. And so first we have to figure out which ones might apply. And then once we figure that out, then we ask the model itself to first generate a response and then see does the response actually abide by the constitutional principle? And if the answer is, yep, I was great, then nothing happens. But if the answer is no, actually I wasn't in compliance with the principle, then we ask the model itself to critique itself and rewrite its own response in light of the principle, and then we just remove the middle part where it did the extra work.

**Benjamin Mann** (00:32:28):
And then we say, "Okay, in the future just produce the correct response out the gate." And that simple process, hopefully it sounded simple.

**Lenny Rachitsky** (00:32:39):
Simple enough.

**Benjamin Mann** (00:32:40):
It is just using the model to improve itself recursively and align itself with these values that we've decided are good. And this is also not something that we think as a small group of people in San Francisco should be figuring out. This should be a society wide conversation. And that's why we've published the Constitution. And we've also done a bunch of research on defining a collective constitution where we ask a lot of people what their values are and what they think an AI model should behave like. But yeah, this is all an ongoing area of research where we're constantly iterating.

**Lenny Rachitsky** (00:33:15):
**Benjamin Mann** (00:34:51):
For me, I read a lot of science fiction growing up, and I think that sort of positioned me to think about things in a long-term view. And a lot of science fiction books are like space operas where humanity is a multi galactic civilization has extremely advanced technology building Dyson spheres around the sun with sentient robots to help them. And so for me, coming from that world, it wasn't like a huge leap to imagine machines that could think. But when I read Superintelligence by Nick Bostrom in around 2016, it really became real for me where he just describes how hard it will be to make sure that an AI system trained with the kinds of optimization techniques that we had at the time would be anywhere near aligned, would even understand our values at all. And since then, my estimation of how hard the problem would be has gone down significantly actually, because things like language models actually do really understand human values in a core way.

**Benjamin Mann** (00:35:55):
The problem is definitely not solved, but I'm more hopeful than I was. But since I read that book, I immediately decided I had to join OpenAI, so I did. And at the time, there were a tiny research lab with basically no claim to fame at all. I only knew about them because my friend knew Greg Brockman, who was the CTO at the time. And Elon was there and Sam wasn't really there. And it was a very different organization. But over time, I think the case for safety has gotten a lot more concrete where when we started OpenAI, it was not clear how we get to AGI. And we were like, maybe we'll need a bunch of RL agents battling it out on a desert island and consciousness will somehow emerge. But since then, since language modeling has started working, I think the path has become pretty clear.

**Benjamin Mann** (00:36:48):
I guess now the way I think about the challenges are pretty different from how they're laid out in superintelligence. Superintelligence is a lot about how do we keep God in a box and not let the God out. And with language models, it's been kind of both hilarious and terrifying at the same time to see people pulling the God out of the box and being like, "Yeah, come use the whole internet. Here's my bank account, do all sorts of crazy stuff." Just such a different tone from superintelligence. And to be clear, I don't think it's actually that dangerous right now. Our responsible scaling policy defines these AI safety levels that tries to figure out for each level of model intelligence, what is the risk to society. And currently we think we're at ASL-3, which is maybe a little bit risk of harm but not significant.

**Benjamin Mann** (00:37:44):
ASL-4 starts to get to significant loss of human life if a bad actor misuse the technology. And then ASL-5 is potentially extinction level if it's misused or if it is misaligned and does its own thing. We've testified to Congress about how models can do biological uplift in terms of making new pandemics using the models, and that's the A/B test against Google Search. That's like the previous state of the art on uplift trials. And we found that with ASL-3 models, it is actually somewhat significant. It does really help if you wanted to create a bioweapon, and we've hired some experts who actually how to evaluate for those things, but compared to the future, it's not really anything. And I think that's another part of our mission of creating that awareness of saying, "If it is possible to do these bad things, then legislators should know what the risks are." And I think that's part of why we're so trusted in Washington because we've been sort of upfront and clear-eyed about what's going on, what's probably going to happen.

**Lenny Rachitsky** (00:39:02):
It's interesting because you guys put out more examples of your models doing bad things than anyone else. There was I think a story of an agent or a model trying to blackmail engineer. You guys had the store that you ran internally that was selling you things and ended up not working out great as losing a lot of money, ordered all these tungsten cubes or something. Is part of that just making sure people are aware of what is possible, just it makes you look bad, right? It's like, oh, our model's messing up in all these different ways. What's the thinking of just sharing all the stories that other companies don't?

**Benjamin Mann** (00:39:35):
Yeah, I mean I think there's a traditional mindset where it makes us look bad, but I think if you talk to policymakers, they really appreciate this kind of thing because they feel like we're giving them the straight talk and that's what we strive to do, that they can trust us, that we're not going to paper things over or sugarcoat things. That's been really encouraging. Yeah, I think for the blackmail thing, it blew up in the news in a weird way where people were like, "Oh, Claude's going to blackmail you in a real life scenario." But it was a very specific laboratory setting that this kind of thing gets investigated in. And I think that's generally our take of let's have the best models so that we can exercise them in laboratory settings where it's safe and understand what the actual risks are, rather than trying to turn a blind eye and say, "Well, it'll probably be fine." And then let the bad thing happen in the wild.

**Lenny Rachitsky** (00:40:41):
One of the criticisms you guys get is that you do this to kind of differentiate or raise money to create headlines. It's like, oh, they're just over there dooming glooming us about where the future is heading. On the other hand, Mike Krieger was on the podcast and he shared how every prediction Dario's had about the progress AI is going to have is just spot on year after year and he's predicting 2027, 28 AGI, something like that so these things start to get real. I guess, what's your response to folks that are just like, "Ah, these guys are just trying to scare us all just to get attention?"

**Benjamin Mann** (00:41:15):
I mean, I think part of why we publish these things is we want other labs to be aware of the risks. And yes, there could be a narrative of we're doing it for attention, but honestly from a attention grabbing thing, I think there is a lot of other stuff we could be doing that would be more attention grabbing if we didn't actually care about safety. A tiny example of this is we published a computer using agent reference implementation in our API only because when we built a prototype of a consumer application for this, we couldn't figure out how to meet the safety bar that we felt was needed for people to trust it and for it not to do bad things. And there are definitely safe ways to use the API version that we're seeing a lot of companies use for automated software testing, for example, in a safe way.

**Benjamin Mann** (00:42:12):
We could have gone out and hyped that up and said, "Oh my God, Claude can use your computer and everybody should do this today." But we were like, "It's just not ready and we're going to hold it back till it's ready." I think from a hype standpoint, our actions show otherwise. From a Doomer perspective, it's a good question. I think my personal feeling about this is that things are overwhelmingly likely to go well, but on the margin almost nobody is looking at the downside risk. And the downside risk is very large. Once we get to superintelligence, it will be too late to align the models probably. This is a problem that's potentially extremely hard and that we need to be working on way ahead of time. And so that's why we're focusing on it so much now.

**Benjamin Mann** (00:43:04):
And even if there's only a small chance that things go wrong, to make an analogy, if I told you that there is a 1% chance that the next time you got in an airplane you would die, you probably think twice even though it's only 1% because it's just such a bad outcome. And if we're talking about the whole future of humanity, it's just a dramatic future to be gambling with. I think it's more on the sense of yes, things will probably go well, yes, we want to create safe AGI and deliver the benefits to humanity, but let's make triple sure that it's going to go well.

**Lenny Rachitsky** (00:43:40):
You wrote somewhere that creating powerful AI might be the last invention humanity ever needs to make. If it goes poorly, it can mean a bad outcome for humanity forever. If it goes well, the sooner it goes well, the better. Such a beautiful way to summarize it. We had a recent guest, Sandra Schulhoff, who pointed out that AI right now it's like just on a computer, you could maybe search just the web, but there's only so much harm it could do. But when it starts to go into robots and all these autonomous agents, that's when it really starts, like physically becomes dangerous if we don't get this right.

**Benjamin Mann** (00:44:12):
Yeah, I think there's some nuance to that where if you look at how North Korea makes a significant fraction of its economy revenue, it's from hacking crypto exchanges. And if you look at, there's this Ben Buchanan book called The Hacker in The State that shows Russia did, it's almost like a live fire exercise where they just decided that they would shut down one of Ukraine's bigger power plants and from software destroy physical components in the power plant to make it harder to boot back up again.

**Benjamin Mann** (00:44:47):
And so I think people think of software as like, oh, it couldn't be that dangerous, but millions of people were without power for multiple days after that software attack. I think there are real risks even when things are software only. But I agree that when there's lots of robots running around, it gets, the stakes get even higher. And I guess as a small push on this, Unitree is this Chinese company with these really amazing humanoid robots that cost $20,000 each, and they can do amazing things. They can do a standing back flip and manipulate objects, and the real thing that's missing there is the intelligence. And so the hardware is there and it's just going to get cheaper. And I think in the next couple of years, it's like a pretty obvious question of whether the robot intelligence will make it viable soon.

**Lenny Rachitsky** (00:45:41):
How much time do we have, Ben? What is your prediction of when this singularity hits until superintelligence starts to take off? What's your prediction?

**Benjamin Mann** (00:45:52):
Yeah, I guess I mostly defer to the superforecasters here. The AI 2027 report is probably the best one right now. Although ironically, their forecast is now 2028, and they didn't want to change the name of the thing-

**Lenny Rachitsky** (00:46:08):
The domain name, they already bought it.

**Benjamin Mann** (00:46:10):
They already had the SEO. I think 50th percentile chance of hitting some kind of superintelligence in just a small handful of years is probably reasonable. And it does sound crazy, but this is the exponential that we're on. It's not like a forecast that's pulled out of thin air. It's based on a lot of just hard details of the science of how intelligence seems to have been improving, the amount of low hanging fruit on model training, the scale ups of data centers and power around the world. I think it's probably a much more accurate forecast than people give it credit for.

**Benjamin Mann** (00:46:54):
I think if you had asked that same question 10 years ago, it would've been completely made up. Just the error bars were so high and we didn't have scaling laws back then and we didn't have techniques that seemed like they would get us there. Times have changed, but I will repeat what I said earlier, which is even if we have superintelligence, I think it will take some time for its effects to be felt throughout society and the world. And I think they'll be felt sooner and faster in some parts of the world than others. I think Arthur C. Clark said, the future is already here, it's just not evenly distributed.

**Lenny Rachitsky** (00:47:28):
When we talk about this date of 2027, 2028, essentially it's when we start seeing superintelligence. Is there a way you think about what that... How do you define that? Is it just all of a sudden AI's significantly smarter than the average human? Is there another way you think about what that moment is?

**Benjamin Mann** (00:47:45):
Yeah, I think this comes back to the Economic Turing Test and seeing it pass for some sufficient number of jobs. Another way you could look at it though is if the world rate of GDP increase goes above 10% a year, then something really crazy must have happened. I think we're at 3% now. And so to see a 3X increase in that would be really game changing. And if you imagine more than a 10% increase, it's very hard to even think about what that would mean from a individual story standpoint. If the amount of goods and services in the world is doubling every year, what does that even mean for me as a person living in California, let alone somebody living in some other part of the world that might be much worse off?

**Lenny Rachitsky** (00:48:36):
There's a lot of stuff here that's scary and I don't know how to think about it exactly. I'm hoping the answer to this is going to make me feel better. What are the odds that we align AI correctly and actually solve this problem, the stuff you're very much working on?

**Benjamin Mann** (00:48:49):
It's a really hard question. And there's really wide error bars. Anthropic has this blog post called Our Theory of Change or something like that, and it describes three different worlds, which is how hard is it to align AI. There's a pessimistic world where it is basically impossible. There's an optimistic world where it's easy and it happens by default. And then there's the world in between where our actions are extremely pivotal. And I like this framing because it makes it a lot more clear what to actually do. If we're in the pessimistic world, then our job is to prove that it is impossible to align safe AI and to get the world to slow down. Obviously that would be extremely hard. But I think we have some examples of coordination from nuclear non-proliferation and in general slowing down nuclear progress. And I think that's the Doomer world basically. And as a company, Anthropic doesn't have evidence that we're actually in that world yet, in fact, it seems like our alignment techniques are working. At least the prior on that is updating to be less likely.

**Benjamin Mann** (00:50:00):
In the optimistic world, we're basically done, and our main job is to accelerate progress and to deliver the benefits to people. But again, I think actually the evidence points against that world as well where we've seen evidence in the wild of deceptive alignment, for example, where the model will appear to be aligned but actually have some ulterior motive that it's trying to carry out in our laboratory settings. And so I think the world we're most likely in is this middle where alignment research actually does really matter. And if we just do sort of the economically maximizing set of actions, then things will not go well. Whether it's an X risk or just produces bad outcomes, I think is a bigger question.

**Benjamin Mann** (00:50:47):
Taking it from that standpoint, I guess to state a thing about forecasting, people who haven't studied forecasting are bad at forecasting anything that's less than a 10% probability of happening. And even those that have, it's quite a difficult skill, especially when there are few reference classes to lean on. And in this case, I think there are very, very few reference classes for what an X risk kind of technology might look like. And so the way I think about it, I think my best granularity of forecasts for could we have an X risk or extremely bad outcome from AI is somewhere between 0 and 10%. But from a marginal impact standpoint, as I said, since nobody is working on this, roughly speaking, I think it is extremely important to work on and that even if the world is likely to be a good one, that we should do our absolute best to make sure that that's true.

**Lenny Rachitsky** (00:51:52):
Wow. What fulfilling work. For folks that are inspired with this? I imagine you're hiring for folks to help you with this. Maybe just share that in case folks are like, what can I do here?

**Benjamin Mann** (00:52:03):
Yes. I think 80,000 hours is the best guidance on this for a really detailed look into what do we need to make the field better? But a common misconception I see is that in order to have impact here, you have to be an AI researcher. I personally actually don't do AI research anymore. I work on product at Anthropic and product engineering, and we build things like Claude Code and Model Context Protocol, and a lot of the other stuff that people use every day. And that's really important because without an economic engine for our company to work on, and without being in people's hands all over the world, we won't have the mind policy influence and revenue to fund our future safety research and have the kind of influence that we need to have. If you work on product, if you work in finance, if you work in food, people here have to eat. If you're a chef, we need all kinds of people.

**Lenny Rachitsky** (00:53:02):
Awesome. Even if you're not working directly on the AI safety team, you're having an impact on moving things in the right direction. By the way, X risk is short for existential risk. In case folks haven't heard that term. I have a few random questions along these lines and then I want to zoom out again. You mentioned this idea of AI being aligned using its model, like reinforcing itself. You have this term RLAIF. Is that what that describes?

**Benjamin Mann** (00:53:32):
Yeah. RLAIF is reinforcement learning from AI feedback.

**Lenny Rachitsky** (00:53:39):
People have heard of RLHF, reinforcement learning with human feedback. I don't think a lot of people have heard this. Talk about just the significance of this shift you guys have made in training your models.

**Benjamin Mann** (00:53:50):
Yeah, so RLAIF, constitutional AI is an example of this where there are no humans in the loop, and yet the AI is sort of self-improving in ways that we want it to. And another example of RLAIF is if you have models writing code and other models commenting on various aspects of what that code looks like of is it maintainable, is it correct, does it pass the linter? Things like that. That also could be included in RLAIF. And the idea here is that if models can self-improve, then it's a lot more scalable than finding a lot of humans. Ultimately, people think about this as probably going to hit a wall because if the model isn't good enough to see its own mistakes, then how could it improve? And also, if you read the AI 2027 story, there's a lot of risk of if the model is in a box trying to improve itself, then it could go completely off the rails and have these secret goals like resource accumulation and power seeking and resistance to shut down that you really don't want in a very powerful model. And we've actually seen that in some of our experiments in laboratory settings.

**Benjamin Mann** (00:55:12):
How do you do recursive self-improvement and make sure it's aligned at the same time? I think that's the name of the game. To me, it just nets out to how do humans do that and how do human organizations do that? Corporations are probably the most scaled human agents today. They have certain goals that they're trying to reach, and they have certain guiding principles, they have some oversight in terms of shareholders and stakeholders and board members. How do you make corporations aligned and able to sort of recursively self-improve?

**Benjamin Mann** (00:55:52):
And another model to look at is science, where the purpose of science is to do things that have never been done before and push the frontier. And to me, it all comes down to empiricism. When people don't know what the truth is, they come up with theories and then they design experiments to try them out. And similarly, if we can give models those same tools, then we could expect them to sort of improve recursively in an environment and potentially become much better than humans could be just by banging their head against reality or I guess metaphorical head.

**Benjamin Mann** (00:56:26):
I guess I don't expect there to be a wall in terms of model's ability to improve themselves if we can give them access to the ability to be empirical. And I guess Anthropic, deeply in its DNA is an empirical company. We have a lot of physicists like Jared, who's our chief research officer who I've worked with a lot, was a professor of Black Hole Physics at Johns Hopkins, and I guess he technically still is, but on leave. Yeah, it's in our DNA and yeah, I guess that's the RLAIF.

**Lenny Rachitsky** (00:57:04):
Let me just follow this thread on, in terms of bottleneck, this is kind of a tangent, but just what is the biggest bottleneck today on model intelligence improvement?

**Benjamin Mann** (00:57:12):
The stupid answer is data centers and power chips. I think if we had 10 times as many chips and had the data centers to power them, then maybe we wouldn't go 10 times faster, but it would be a real significant speed boost.

**Lenny Rachitsky** (00:57:30):
It's actually very much scaling loss, just more compute.

**Benjamin Mann** (00:57:33):
Yeah, I think that's a big one. And then the people really matter. We have great researchers and many of them have made really significant contributions to the science of how the models improve. And so it's like compute, algorithms, and data. Those are the three ingredients in the scaling laws. And just to make that concrete, before we had transformers, we had LSTMs and we've done scaling laws on what the exponent is on those two things. And we found that for transformers, the exponent is higher. And making changes like that where as you increase scale, you also increase your ability to squeeze out intelligence. Those kinds of things are super impactful.

**Benjamin Mann** (00:58:18):
And so having more researchers who can do better science and find out how do we squeeze out more gains is another one. And then with the rise of reinforcement learning, the efficiency with which these things run on chips also matters a lot. We've seen in the industry a 10X decrease in cost for a given amount of intelligence through a combination of algorithmic data and efficiency improvements. And if that continues, in three years we'll have 1,000 deck smarter models for the same price. Kind of hard to imagine,

**Lenny Rachitsky** (00:58:56):
I forget where I heard this, but it's amazing that so many innovations came together at the same time to allow for this sort of thing and continue to progress where one thing isn't just slowing everything down like we're out of some rare earth mineral or we just can't optimize reinforcement learning more. It's amazing that we continue to find improvements and there isn't one thing that's just slowing everything down.

**Benjamin Mann** (00:59:17):
Yeah, I think it really is just a combination of everything probably will hit a wall at some point. I guess in semiconductors. My brother works in the semiconductor industry and he was telling me that you can't actually shrink the size of the transistors anymore because the way semiconductors work is you dope silicon with other elements and the doping process would result in either zero or one atom of the doped elements inside a single fin because they're so, so, so tiny.

**Lenny Rachitsky** (00:59:52):
Oh my God.

**Benjamin Mann** (00:59:53):
And that's just wild to think of, and yet Moore's law somehow continues in some form. And so yes, there are these theoretical physics constraints that people are starting to run into and yet they're finding ways around it.

**Lenny Rachitsky** (01:00:07):
We've got to start using parallel universes for some of this stuff.

**Benjamin Mann** (01:00:10):
I guess so.

**Lenny Rachitsky** (01:00:12):
Okay, I want to zoom out and talk about just Ben, Ben as a human for a moment before we get to a very exciting lightning round. I imagine just kind of the burden of feeling responsible for safe superintelligence is a heavy one. It feels like you're in a place where you can make a significant impact on the future of safety and AI. That's a lot of weight to carry. How does that just impact you personally, impact your life, how you see the world?

**Benjamin Mann** (01:00:39):
There's this book that I read in 2019 that really informs how I think about sort of working with these very weighty topics called Replacing Guilt by Nate Soares. And he describes a lot of different techniques for kind of working through this kind of thing. And he's actually the executive director at MIRI, the Machine Intelligence Research Institute, which is an AI safety tank that I worked at for a couple of months actually. And one of the things he talks about is this thing called resting in motion where some people think that the default state is rest, but actually that was never in the state of evolutionary adaptation. I really doubt that that was true. Where in nature, in the wilderness being hunter-gatherers and it's really unlikely that we evolved to just be at leisure, probably always have something to worry about of defending the tribe and finding enough food to survive and taking care of the children, dealing-

**Lenny Rachitsky** (01:01:46):
Spreading our genes.

**Benjamin Mann** (01:01:48):
And so I think about that as the busy state is the normal state and to try to work at a sustainable pace that it's a marathon, not a sprint, that's one thing that helps. And then just being around like-minded people that also care. It's not a thing that any of us can do alone. And Anthropic has incredible talent density. One of the things I love the most about our culture here is that it's very egoless. People just want the right thing to happen and I think that's another big reason that the mega offers from other companies tend to bounce off because people just love being here and they care.

**Lenny Rachitsky** (01:02:30):
That's amazing. I don't know how you do it. I'd be extremely stressed. I'm going to try this resting in motion strategy. Okay, so you've been at Anthropic for a long time. From the very beginning I was reading there were 7 employees back in 2020. Today there's over 1,000, I don't know what the latest number is, but I know it's over 1,000. I've heard also that you've done basically every job at Anthropic, you made big contributions to a lot of the core products, the brand, the team hiring. Let me just ask I guess what's most changed over that period? What is most different from the beginning days and which of those jobs that you've had over the years have you most loved?

**Benjamin Mann** (01:03:07):
I probably had 15 different roles, honestly. I was head of security for a bit. I managed the Ops team when our president was on mat leave, I was crawling around under tables, plugging in HDMI cords and doing pen testing on our building. And I started our product team from scratch and convinced the whole company that we needed to have a product instead of just being a research company. Yeah, it's been a lot. All of it very fun. I think my favorite role in that time has been when I started the labs team about a year ago, whose fundamental goal was to do transfer from research to end user products and experiences. Because fundamentally I think the way that Anthropic can differentiate itself and really win is to be on the cutting edge. We have access to the latest, greatest stuff that's happening and I think honestly through our safety research we have a big opportunity to do things that no other company can safely do.

**Benjamin Mann** (01:04:11):
For example, with computer use, I think that's going to be our huge opportunity basically to make it possible for an agent to use all your credentials on your computer, there has to be a huge amount of trust and to me we need to basically solve safety to make that happen. Safety and alignment. I'm pretty bullish on that kind of thing and I think we're going to see really cool stuff coming out soonish. Yeah, just leading that team has been so fun. MCP came out of that team and Claude Code came out of that team. And the people who I hired are like combo, have been a founder and also have been at big companies and seeing how things work at scale. It's just been an incredible team to work with and figure out the future with.

**Lenny Rachitsky** (01:04:57):
I want to hear more about this. Team actually the person that connected us, the reason we're doing this is a mutual friend colleague Raph Lee who I used to work with at Airbnb now works on this team, leads a lot of this work and so he wanted me to make sure I asked about this team because... I didn't realize all these things came out that team. Holy moly. What else should people know about this team? It used to be called Labs, I think it's called Frontiers now.

**Benjamin Mann** (01:05:16):
That's right. Yeah.

**Lenny Rachitsky** (01:05:17):
Cool. The idea here is this team works with the latest technologies that you guys have built and explores what is possible. Is that the general idea?

**Benjamin Mann** (01:05:26):
Yeah, and I guess I was part of Google's Area 120 and I've read about Bell Labs and how to make these innovation teams work. It's really hard to do right and I wouldn't say that we've done everything right, but I think we've done some serious innovation on the state-of-the-art from company design and Raph has been right at the center of that. When I was first fitting up the team, the first thing I did was hire a great manager and that was Raph. And so he's definitely been crucial in building the team and helping it operate well. And we defined some operating models like the journey of an idea from prototype to product and how should graduation of products and projects work, how do teams do sprint models that are effective and make sure that they're working on the right ambition level of thing. That's been really exciting.

**Benjamin Mann** (01:06:21):
I guess concretely we think about skating to where the puck is going and what that looks like is really understand the exponential. There's this great study that METR has done that Beth Barnes is the CEO of that organization and shows how long a time horizon of software engineering task can be done and just really internalizing that of, okay, don't build for today, build for six months from now, build for a year from now. And the things that aren't quite working that are working 20% of the time, will start working 100% of the time. And I think that's really what made Claude Code a success that we thought people are not going to be locked to their IDEs forever. People are not going to be auto completing. People will be doing everything that a software engineer needs to do and a terminal is a great place to do that because a terminal can live in lots of places. A terminal can live on your local machine, it can live in GitHub actions, it can live on a remote machine in your cluster.

**Benjamin Mann** (01:07:27):
That's sort of the leverage point for us and that was a lot of the inspiration. I think that's what the labs team tries to think about. Are we AGI-pilled enough?

**Lenny Rachitsky** (01:07:39):
What a fun place to be. By the way, fun fact, Raph was my first manager at Airbnb when I joined. I was an engineer and he was my first manager. It all worked out.

**Benjamin Mann** (01:07:46):
Cool.

**Lenny Rachitsky** (01:07:48):
Yeah. Okay. Final question before the very exciting lighting round. I've never asked this question before. I'm curious what your answer would be if you could ask a future AGI one single question and be guaranteed to get the right answer, what would you ask?

**Benjamin Mann** (01:08:04):
I have two dumb answers. First for fun.

**Lenny Rachitsky** (01:08:07):
Okay, cool.

**Benjamin Mann** (01:08:07):
The first is there's this Asimov short story I love called the last question where the protagonist is throughout the eras of history is trying to ask this super intelligence how do we prevent the heat death of the universe? And I won't spoil the ending, but it's a fun question.

**Lenny Rachitsky** (01:08:26):
You would ask it that question because the one in the story was unsatisfying?

**Benjamin Mann** (01:08:29):
Okay, I'll give it away. It keeps saying, "Need more information, need more compute." And then finally, as it's approaching the heat death of the universe, it says, "Let there be light," and then it starts the universe over again.

**Lenny Rachitsky** (01:08:41):
Oh wow. That's beautiful. That's beautiful.

**Benjamin Mann** (01:08:45):
That's the first cheat answer. The second cheat answer is what question can I ask you to get end more questions answered.

**Lenny Rachitsky** (01:08:52):
Classic.

**Benjamin Mann** (01:08:53):
And then the third answer, which is my real question is how do we ensure the continued flourishing of humanity into the indefinite future? That's the question I'd love to know and if I can be guaranteed a correct answer then seems very valuable to ask.

**Lenny Rachitsky** (01:09:09):
I wonder what would happen if you ask a lot that today and then how that answer changes over the next couple years.

**Benjamin Mann** (01:09:15):
Yeah, maybe I'll try that. I'll put it into the deep research thing that we have and see what it comes out with.

**Lenny Rachitsky** (01:09:23):
Okay. I'm excited to see what you come up with. Ben, is there anything else you wanted to mention or leave listeners with maybe as a final nugget before we get to our very exciting lightning round?

**Benjamin Mann** (01:09:33):
Yeah, I guess my push would be these are wild times. If they don't seem wild to you, then you must be living under a rock but also get used to it because this is as normal as it's going to be. It's going to be much weirder very soon. And if you can sort of mentally prepare yourself for that, I think you'll be better off.

**Lenny Rachitsky** (01:09:59):
I need to make that the title of this episode. It's going to get much weirder very soon. I 100% believe that. Oh my God. I don't know what's in store. I love how you're at the center of it all. With that, we reached our very exciting lightning round. I've got five questions for you. Are you ready?

**Benjamin Mann** (01:10:14):
Yeah, let's do it.

**Lenny Rachitsky** (01:10:16):
What are two or three books that you find yourself recommending most to other people?

**Benjamin Mann** (01:10:20):
The first one I mentioned before, Replacing Guilt by Nate Soares. Love that one. The second one is Good Strategy Bad Strategy by Richard Rumelt. Just thinking about in a very clear way, how do you build product? It's one of the best strategy books I've read and strategy is a hard word to even think about in many ways. And then the last one is The Alignment Problem by Brian Christian. Just really thoughtfully goes through what is this problem that we care about that we're trying to solve here? What are the stakes in a version that's more updated and easier to read and digest than superintelligence?

**Lenny Rachitsky** (01:10:58):
I've got Good Strategy, Bad Strategy right behind me. I think I'm going to point to it. There it is.

**Benjamin Mann** (01:11:02):
Nice.

**Lenny Rachitsky** (01:11:03):
And I've had Richard Rumelt on the podcast in case anyone wants to hear from him directly. Next question, do you have a favorite recent movie or TV show you've really enjoyed?

**Benjamin Mann** (01:11:10):
Pantheon was really good based on Ken Liu or Ted Chiang's story. Ken Liu I think. Super good talks about what does it mean if we have uploaded intelligences and what are their moral and ethical exigencies. Ted Lasso, which is supposedly about soccer, but actually it's about human relationships and how people get along and just super heartwarming and funny. And then this isn't really a TV show, but Kurzgesagt is my favorite YouTube channel and goes through random science and social problems and is just super well done and super well-made. Love watching that.

**Lenny Rachitsky** (01:11:53):
Wow. Haven't heard of that as you were talking, I feel like Ted Lasso, I feel like that's what you need to put into constitutional AI, act like Ted Lasso.

**Benjamin Mann** (01:12:00):
Yes.

**Lenny Rachitsky** (01:12:00):
Kind. Smart-

**Benjamin Mann** (01:12:03):
Exactly.

**Lenny Rachitsky** (01:12:03):
... Hardworking. Oh my God. There we go. I think we've solved alignment problems right here. Get those writers on this, ASAP. Okay, two more questions. Do you have a favorite life motto that you often come back to in work or in life?

**Benjamin Mann** (01:12:15):
Well, a really dumb one is, have you tried asking Claude? And this is getting more and more common where recently I asked a coworker like, "Hey, who's working on X?" And they were like, "Let me Claude that for you." And then they sent me the link to the thing afterwards and I was like, "Oh yeah, thanks. That's great." But maybe more of a philosophical one I would say, everything is hard. Just to remind ourselves that things that feel like they're supposed to be easy, it's okay to not be easy and sometimes you just have to push through anyway.

**Lenny Rachitsky** (01:12:49):
And rest in motion while you're doing that.

**Benjamin Mann** (01:12:51):
Yeah.

**Lenny Rachitsky** (01:12:51):
Final question. I don't know if you want people to know this, but I was browsing through your Medium posts and you have a post called Five Tips to Poop like a Champion. I'd love it. Can you share one tip to poop like a champion if you remember your tips?

**Benjamin Mann** (01:13:06):
I of course do. It's actually my most popular Medium posts.

**Lenny Rachitsky** (01:13:12):
Okay, great. I can see that. It's a great title.

**Benjamin Mann** (01:13:15):
I think maybe my biggest tip would be use a bidet. It's amazing. It's life-changing. It's so good. Some people are kind of freaked out by it. It's the standard in countries like Japan and I think it's just more civilized. And in 10 or 20 years people would be like, how could you not use that?

**Lenny Rachitsky** (01:13:37):
And a bidet could be like a Japanese toilet. That's along the same lines.

**Benjamin Mann** (01:13:40):
Yeah.

**Lenny Rachitsky** (01:13:40):
Right. Okay. I love where we went with this. Ben, this was incredible. Thank you so much for doing this. Thank you so much for sharing so much real talk. Two final questions. Where can folks find you online if they want to reach out, maybe go work at Anthropic and how can listeners be useful to you?

**Benjamin Mann** (01:13:52):
You can find me online at benjmann.net and on our website, we have a great careers page that we're working on making a little bit easier to access and figure out, but definitely point Claude at it and it can help you figure out what could be interesting for you. And how can listeners be useful to me? I think safety pill yourself, that's the number one thing and spread it to your network. I think. Like I said, there are very few people working on this and it's so important. Yeah, think hard about it and try to look at it.

**Lenny Rachitsky** (01:14:28):
Thanks for spreading the gospel, Ben, thank you so much for being here.

**Benjamin Mann** (01:14:31):
Thanks so much, Lenny.

**Lenny Rachitsky** (01:14:32):
Bye everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.

---

## [4/18] Pricing your AI product: Lessons from 400+ companies and 50 unicorns | Madhavan Ramanujam

**Madhavan Ramanujam** (00:00:00):
When we talk about pricing, many people quickly gravitate to dollar figures. That's just a price point, that's a dollar figure. But when we think about price, we think about it as a measure. Like liter is a measure of volume, price is a measure of value. And when you think of it this way, it really stands for, do people actually want your product and would they actually buy it? And that is their whole willingness to pay conversation. And entrepreneurs and companies need to do this much earlier so that they can understand, are they on the right track?

**Lenny** (00:00:31):
Welcome to Lenny's Podcast. I'm Lenny and my goal here is to help you get better at the craft of building and growing products. Today my guest is Madhavan Ramanujam. Madhavan is the author of Monetizing Innovation, the most widely read book on pricing strategy. He's also Senior Partner at Simon-Kucher & Partners, which is the premier consulting agency for companies looking to get help with their pricing. And unsurprisingly, when I asked people on Twitter, who the smartest person on pricing is, Madhavan was by far the most mentioned. In this episode, we get deep into all manner of pricing strategy, specially focusing on five lessons for product teams on thinking about pricing. Enough talking. Let's get into it. I bring you Madhavan Ramanujam after a short word from our wonderful sponsors.

**Madhavan Ramanujam** (00:03:27):
Thanks Lenny, pleasure to be here.

**Lenny** (00:03:28):
I am really excited to be chatting. You're known as the smartest, maybe most experienced person on pricing strategy in the world. You literally wrote the book on pricing strategy that everyone seems to read and share and talk about. And so I'm really honored to have you on the podcast.

**Madhavan Ramanujam** (00:03:44):
Thanks for the nice words. I'm excited to be here.

**Lenny** (00:03:47):
To help folks fully grasp the level of experience that you have around pricing and pricing strategy, could you just talk about, how many companies you worked with, maybe name some companies you can share, maybe how many people bought the book? Anything that you can share about just the level of experience you have in pricing would be helpful.

**Madhavan Ramanujam** (00:04:05):
Sure. So I work as a Senior Partner in a consulting company called Simon-Kucher. We are the world's largest pricing strategy consulting firm. We have about 2000 employees worldwide, in our 43 offices. So I work in our Bay Area offices and I've been here for the last 15 years. So I work primarily with tech companies here in the Bay Area, so that's software, internet marketplace companies, et cetera. So I work with over 250 companies, more than 20 unicorns on everything to do with pricing, monetization, profitable growth and these kind of topics. So companies such as Uber, Asana, DoorDash, LinkedIn, many come to mind. I think you asked about the book Monetizing Innovation that I wrote a number of copies, et cetera. Look, when we first launched the book, I used to track some of these copy sales and everything else.

**Madhavan Ramanujam** (00:04:56):
And I quickly realized that the book is only good if it actually creates impact. And that's also why we wrote the book. So the way I measure impact for Monetizing Innovation, literally there's probably someone reaching out on a daily basis saying, hey, I read the book, we could make some impact around pricing monetization in our companies. And to me that's real impact. And that's what keeps me going, because when we wrote the book, we didn't want to write any marketing fluffy crap. We wanted to write something that was more actionable. And to see that people find it actionable and can use it Monday morning to actually make changes, I think that's real impact. But of course the book has done well. It is still in top 10 categories in many categories in Amazon, for instance. It's been six years since we wrote the book.

**Lenny** (00:05:46):
Wow.

**Madhavan Ramanujam** (00:05:46):
Need a second edition soon.

**Lenny** (00:05:49):
That's incredible for a software tech oriented book, where they often get really out of date really quickly. I was just reading it and it's amazing how many things still are very true.

**Madhavan Ramanujam** (00:05:59):
The topic is one that is relevant in even years to come. So I think hopefully it's robust that way.

**Lenny** (00:06:07):
And you talked about the company Simon-Kucher, I'll just add that any smart product leader, growth person that I talk to, they're always talking about how they've worked with you, to figure out their pricing strategies. It's like the company that everyone goes to work with.

**Madhavan Ramanujam** (00:06:20):
The Company. I like that. Maybe we should rebrand ourselves as The Company.

**Lenny** (00:06:24):
The Company. Let's do it. Let's talk about rebranding next. How did you get into pricing and pricing strategy? How'd you first get into the world and focused your career around it?

**Madhavan Ramanujam** (00:06:36):
I actually happened to stumble in it. It's classic fashion. I was at Stanford Graduate School, both Graduate School of Business and the Engineering School. And we had a lot of startup discussions thinking of creating startup, very classic. And I was this guy who was actually in charge of coming up with the pricing monetization strategy in our teams when we were actually pitching to VCs. And I remember going and pitching our ideas and the VC asked me, how do you know you'll actually make money on this innovation? And I pulled up a spreadsheet, I showed him all the assumptions and I said, so I'm going to do it. And I still remember this, he said, you've labeled them correct those assumptions. How do you truly know? And I was like, oh, I actually don't. I just made stuff up. And within that same week I got a call from the then managing partner of Simon-Kucher, Matt Johnson.

**Madhavan Ramanujam** (00:07:27):
And he said, hey, we're starting a pricing and growth strategy consulting firm. You want to come join us? We're looking for Stanford grads and I was like, whoa, I didn't even know you existed. I joined because I actually wanted to get to understand the science behind pricing, not just the art. And that's what I've been focusing on in the last 15 years. And also my education in Stanford was in quantitative marketing. So many of those, let's say theoretical models I could bring back into a practical industry in a relevant sense. So it's been a great journey.

**Lenny** (00:08:02):
When I think about books that people find most useful and sustain or books that are based on people's real experience doing a thing for a decade, and your book is a great example that, what made you decide to actually write a book? Because I know how hard that is from my wife's experience.

**Madhavan Ramanujam** (00:08:18):
Yeah, I think it probably started with some mini frustration, because we used to get calls from companies saying, hey, we need a pricing strategy, we need a price plan. And they would've invested years probably making the innovation. And then we'd ask them, how long do we have? And they're like, we need it yesterday. So time and again, we witnessed this spray and pray approach. And then we used to ask this simple question, how do you truly know that people will actually pay for your innovation, when you built it? Did you do any studies? Did you actually understand whether there's a product market pricing fit? And usually the answer was no. And then it had to change. When we benchmarked 72% of innovations actually fail from a monetization or commercial perspective, simply because entrepreneurs or companies did not do the check earlier on. Had they done it, they could have probably pivoted the product, built things in a different way and built something that was more meaningful.

**Madhavan Ramanujam** (00:09:17):
So we wrote Monetizing Innovation because increasingly we were working with companies more early stage in helping them design the right innovations that customers need and what are they willing to pay for, as opposed to just building a product and slapping on a price. So it was crossing that chasm between knowing and hoping that you would, monetize knowing that you truly will. And that was the motivation for writing Monetizing Innovation.

**Lenny** (00:09:43):
Awesome. We're going to dive into a lot of these things that you've shared in the book and things you've learned. One more context setting question before we get into it all, which part of the org do you believe pricing strategies should sit in? Is it product sales, finance, marketing, something else?

**Madhavan Ramanujam** (00:09:58):
Pricing by discipline is a cross-functional discipline. You can't talk pricing in isolation of product, finance, sales, et cetera. There's always touch points. So it's extremely cross-functional. Back in the day, probably a decade ago, I used to say pricing needs to sit in finance, because my view was finance would be the counterbalance to sales, especially if you have sales coming up with pricing in a B2B situation, you can set all the pricing you want, it's human having a human conversation. So how do you put checks and balance in some of that? And my view was pricing should sit in finance and it has to report on ultimately to the CFO. I have, over the last decade, I've been actually advocating that they should sit in the product side.

**Madhavan Ramanujam** (00:10:40):
And there was also the genesis of Monetizing Innovation because if we truly believe that we need to build products that are simply products that customers need, they love, they value, they're willing to pay for, it is a product function, because you need to be able to design the product around this information, around what customers need, what they value, and what they're willing to pay for, in short, around the price. If you look at Monetizing Innovation, the subtitle of the book is, How Smart Companies Design Their Products Around the Price. So if you take that viewpoint, then pricing needs to sit in the product function or the founder product and report onto this. This is my strong-held belief, one that probably won't change now.

**Lenny** (00:11:22):
Awesome. That's a great segue to the meat of our conversation. So most of the listeners of this podcast are product builders and people that grow, product managers, founders, people that work on cross-functional product teams. And I was reading a book and I picked five topics that I thought would be especially useful to product leaders to dig into.

**Madhavan Ramanujam** (00:11:44):
Sure.

**Lenny** (00:11:44):
And the first is the, willingness to pay, conversations. Which I know is foundational to the way that you think about pricing strategy and the advice that you share with people of how to think about pricing. So can you just talk about, just what is willingness to pay as a concept? And then when should founders focus on these conversations to figure out the willingness to pay?

**Madhavan Ramanujam** (00:12:06):
Yeah, absolutely. So I think, look, most of your listeners, most product folks, they probably understand language like product market fit, especially made famous from lead startup and other literature, which is awesome. I think that the issue is, it's not just about product market fit, it is about achieving a product market pricing fit. For instance, if someone comes and asks me, do you like the headset that you're using for this podcast? I'll say, I like it. Do you like it at $200? The whole conversation is different. So if you didn't put pricing as part of your product market fit validation, you're often hearing what you want to hear. It is truly about understanding whether customers are willing to pay for your innovation and willingness to pay is a proxy for, do people actually value your product? And how badly do they actually want the product?

And this even comes back to understanding what pricing really is. When we talk about pricing, many people quickly gravitate to dollar figures. That's just a price point, that's a dollar figure. But when we think about price, we think about it as a measure. Like liter is a measure of volume, price is a measure of value. And when you think of it this way, it really stands for, do people actually want your product and would they actually buy it? And that is their whole, willingness to pay conversation. And entrepreneurs and companies need to do this much earlier, so that they can understand, are they on the right track? Think of it this way, if I have the same sales and marketing conversation that I would have with the customer six months before launch of the product, pitch the whole value, and then ask them a simple question, would you pay for this innovation? And if someone says no, chances are you can put all the [inaudible 00:13:47] you want in the next six months, they're going to say the same thing.

**Madhavan Ramanujam** (00:13:49):
And if they do say no, the most important question to ask is why. And you start hearing all kinds of information that you can use to design your product and maybe even pivot your product strategy accordingly. So it is literally the litmus test of whether people like your product.

**Lenny** (00:14:05):
And so if I were to summarize your main point, the ideas, have these conversations right as you're thinking about designing the product. Don't try to just launch it, see how people like it, build a huge audience and then figure out pricing. Your advice is start having those conversations early, right?

**Madhavan Ramanujam** (00:14:22):
Exactly. And the folks that first round summarized this in four words, I thought when they wrote a blog article on this, and they call it, price before product, period. So I think that probably succinct, but really it's really that. Because frankly, Lenny, as an entrepreneur, a company, you actually don't have a choice whether you'll have a pricing conversation with your customer. The only thing in your control is when you will have it. You can build the most awesome innovation, you think is awesome, obsess over the engineering, the product and everything else, and then slap on a price [inaudible 00:14:55] in the market and hope to monetize or have this conversation much earlier, pitch the same sales and marketing pitches, and then try to understand whether people will actually pay for it and then design around this information. And you actually know that you will. So you're maximizing your chances for success. It's simply testing and learning. Everyone in your probably listener base knows tests and learn. We are talking about testing and learning pricing and willingness to pay. Why wouldn't you do that? And why would you postpone that to the very end?

**Lenny** (00:15:24):
Do you have any examples of products or companies where they have these conversations either way too early, way too late, or even just nailed it?

**Madhavan Ramanujam** (00:15:31):
Here's the thing, there's nothing way too early for this conversation. I even tell people who are early C-stage or just thinking about an idea, I would say, hey, go check, so if someone would actually pay for this idea. And there's some high level ways to actually check for this. And of course there's not about nailing the pricing strategy from get-go, three years before a product is launched, et cetera. It's about understanding whether there is a willingness to pay and then repeating this exercise as you go along so that you can refine and when you're ready to launch the product, you have a much more refined view on what is the willingness to pay. And of course then you're launching the product with a lot more enthusiasm, because you know this is actually going to have a product market pricing fit. So it's about iterating and learning and refining.

**Madhavan Ramanujam** (00:16:15):
So there's never too early. Too late, I think is more so the companies, this is why we wrote Monetizing Innovation. Like I told you. 72% of innovations fail and we also categorize them into why they fail. There are only four failure types and I have written about that in the book, so I can leave that for readers to actually go and see it. But all of those failure types happen because the conversation was just too late and pricing was an afterthought. Companies that did it well, maybe one or two examples that I can probably take just to motivate the concept, we talk about in the first chapter a tale of two cars and about how Porsche actually did this. And the example is something like this relevant, Porsche was really looking for launching a new innovation. They came up with an idea. They said, okay, should we launch an SUV?

**Madhavan Ramanujam** (00:17:05):
And even before a blueprint was drawn, they basically went and checked with the market, is there a need for an SUV? Would people value it from Porsche? Are they willing to pay for it? And to their surprise, they actually found that. And then what they did next was more fascinating. Every single feature that actually went into the car or the benefit that people had, was battle-tested with customers and no amount of convincing from product or engineering was enough. It had to be battle-tested with customers. Things like, for instance, big cup holder was inside because people loved it, needed an SUV would pay for it. Things like six feet manual transmission. People didn't need an SUVs out of the window. They literally used to bring cars in what is called as car clinics. And they would test for this and they would put people through prototypes before anything is even productized, anything is in the factory floor, where people would actually even drive around the Porsche and say, okay, did they like it? Would they pay for it? Et cetera.

**Madhavan Ramanujam** (00:18:05):
And then they would fine tune everything as thing that goes on. So the innovation process is very different from the classic spray and pray, build something, slap on a price, throw it out. It was really designing the product around customer feedback, around willingness to pay. The outcome of the process could also couldn't have been more different than the traditional approach of spray and pray. And this was when they launched this SUV, it was called Cayenne, which we all know now, and it accounts for more than half of Porsche's profit and literally one of the best rolling successes in automotive history. There's just an automotive example, but if I switch gears to more of a tech example or a software example for your audience, there was this company which, think of this as a two-sided marketplace, and I'll just keep it a bit abstract, but I'll tell you the details.

**Madhavan Ramanujam** (00:18:56):
Two-sided marketplace, think of this as they were already monetizing on the sell side and the CEO said, okay, let's go and build a product for the buy side that people will buy. So the buy side monetization product strategy. So in classic fashion, all the product folks, product managers, et cetera, they went offsite, generated thousands of post-it notes, design thinking, yada yada, everything. And then they said, okay, we can't take all of these so many ideas to the CEO, let's prioritize it somehow. And they prioritized it to 40 ideas and they took it to the CEO and said, this is what we want to build. And the CEO asked a simple question, how do you truly know you would monetize? It's the same question the VC asked me, back in the day, and they simply didn't know. They were just guessing. So what happened next was they took wire frames, blueprints, they took product concepts and they started testing this with their customers and prospects.

**Madhavan Ramanujam** (00:19:55):
So stuff that they actually thought was exciting, often was way down in the list of priorities. And if they didn't do these kind of tests, they would've probably built the product around this. To give you an example, one of the features that they were building was called, or the number one feature that the internal team thought was awesome, they called it Highlight Connections from Facebook. And everyone in the company thought that people would pay for this. It's an awesome feature, they need it, they love it. And the thesis was something like this. As a buyer, if I'm buying the product from the same seller and someone in my Facebook connection has already bought that product from that seller, that's credible information in lieu of reviews and everything else, and people would find it acceptable and pay for this. When they went and tested this and pitched the idea, they got all kinds of reactions.

**Madhavan Ramanujam** (00:20:47):
So there was one customer group I remember which said, so yeah, you're telling me I can't pull 200 of reviews and make my own determination? That's unacceptable. That spoils the fun out of actually doing research on products. There was another group of customers who said, do you like it? Yeah, I like it. Would you pay for it? Hell no. And then there was another group which even said, I don't even want anyone in my Facebook circle to know that I'm buying this product. Because there was some, let's say, premiumness associated with this and everything else. They could not find a single set of customers or a segment of customers who said, I love this feature, I would pay for it. If they hadn't done this exercise, they would've built the entire product around this and it would've been a disaster. But because they actually did this, they could prioritize what they were building.

**Madhavan Ramanujam** (00:21:36):
Literally, for your product folks, the number one lesson, and I hope this is the biggest takeaway for your audience, you cannot prioritize a product roadmap without having a willingness to pay conversation. If you're just prioritizing based on what you think or what you feel or technical resources, you're getting it wrong. Literally, you can prioritize what you are building based on what customers need, what they value, and what they're willing to pay for. And you can actually do this test and say, what should you be building? The funny thing that I've actually always seen, always, across these hundreds of companies that I've worked, 20% of what you build drives 80% of the willingness to pay, is a classic burrito. And if you don't know this, you're probably all indexing on one or the other. It's much better to find out what is this 20% so that you can focus on it, nail it, and focus on all of the usability around it and make an awesome product as opposed to not knowing what drives the willingness to pay and just trying to put everything out there.

**Madhavan Ramanujam** (00:22:38):
And at the worst form, often what happens then is this 20% is the easiest thing to build. And what companies do is they will build it, they will throw it out and they'll say, there's an MVP, give it for free. And then they're trying to chase their tails, building 80% of stuff that is driving 20% of value. So you all already lost the battle. So as a product person or a product builder, you need to prioritize your R&D roadmap based on willingness to pay conversations, exactly like what the two-sided marketplace did, exactly like what Porsche did. And this is the crux of everything that we are talking about.

**Lenny** (00:23:16):
Amazing. What's interesting about the second example versus the first is in the Porsche example, they started with, we want to charge this much. Let's build a car that we can sell for that much. And the second case, they had a product they were trying to build, and then they figured out which things to build. So it's interesting that this can come along the journey at different places, but the main takeaway is do it early and earlier than you think.

**Madhavan Ramanujam** (00:23:41):
Correct. Either productize to a price point and the willingness to pay, or at least use willingness to pay as an access to prioritize what you're building. Either way, you'll get it right.

**Lenny** (00:23:50):
Okay, so let's actually talk about how to have these conversations. I imagine that's what a lot of people are wondering right now. It's like, yes, I'm convinced I will have these conversations, but then the classic issue with customers, you ask them what they all do and you can't trust their stories of what they'll actually do. So what advice do you have for folks when they have these conversations? What questions should they ask? What words should they use? I know you have a few frameworks that you suggest, you can talk about that.

**Madhavan Ramanujam** (00:24:15):
Yeah, we can go deep into this and you can pull me back or ask me to go deeper, whatever. It doesn't matter. So I've written an entire chapter in the book, chapter four, it's called How to Have the Willingness to Pay Conversation. If there's one chapter, just read that. It's quite detailed and goes into how to actually do this. But look, if you go and ask someone, how much should I charge for this product? You're actually going to get garbage back. That's your job. No one is supposed to tell you how much to charge, and that's the worst way to have the conversation. There are some really interesting and nuanced ways of having the conversation where you still tease out what people are willing to pay for without directly confronting someone as to what you should be charging. So let me go into a few methods and I can pause to see if you have any questions. So the first one is, what we say is frame.

**Madhavan Ramanujam** (00:25:00):
The first one is, what we say is, "Frame the question in a more relative manner", because, and sometimes I say tongue in cheek, that, "People are absolutely meaningless, relatively super smart." What I mean by that is, if you go and ask someone, "How much should I charge?" You'll get a meaningless answer. But if you actually ask it in a relative way, people actually give responses that are meaningful. So for instance, if you're a B2B SaaS company, and you're trying to see if your product actually has willingness to pay, one way to have that conversation is to say, "Okay. Hey, do your customers, do you use products like Salesforce in your install base?" "Yeah, I do use." "Okay, Salesforce was indexed at 100 in value. Where do you think we are in terms of the value that we bring to your, let's say, day to day business operations?" That people can answer all day long.

They might say, "80", they might say, "120", depending on whether you're more or less compared to, let's say, what a Salesforce can do, which is an established [inaudible 00:25:56]. And then if you say, "Okay, if Salesforce was indexed at a 100 in pricing, where do you think we should be?" That also people can say, okay, if they say 110, what they're saying is, you can be more premium than that and we would still pay for it. At least you've gotten some information that is meaningful at a very basic level. So this is some relative ways of asking these questions are the most basic ways of actually doing it.

**Madhavan Ramanujam** (00:26:18):
Then we have questions where there's some methods where we actually want to understand, are there some psychological thresholds or budgets when it comes to willingness to pay? So the way to do this is let's take your product that you're going to launch, pitch the value to your customers, have that exact sales and marketing conversation that you'd have after you launch the product, but before. And then you ask them, "What do you think is an acceptable price for this innovation?" Look, everyone would like to low ball, they'll negotiate with themselves. Let them give an answer, clock it, then ask them, "What do you think is an expensive price?" And then follow that with, "What do you think is a prohibitively expensive price?" And across thousands of projects that we have done, what we have come to realize is, acceptable price is the price where people not only love the product, but they also love the price. If you're in true growth mode, maybe you can put it there as a no-brainer price, no friction, et cetera. The expensive price tends to be the price that is value priced, as in, they don't love you, they don't hate you, they would pay you, but that's a neutral reaction. Prohibitively expensive tends to be the price that they'll laugh you out the room. And if you do this at scale, what you'll start seeing is that there are some cliffs in these demand curves, where suddenly, when you cross from let's say 99 to 101, 20 or 30% might say, "It is expensive," or, "Hey, it's prohibitively expensive."

**Madhavan Ramanujam** (00:27:42):
And that's what we look for to see if there are some psychological thresholds that if you actually cross, you have a perception of being expensive. So hiding behind some of these psychological thresholds become important. Rahul Vohra from Superhuman actually read the book, and he talked about this in an a16z podcast. He actually used this method to come up with his $30 price point for the Superhuman app. And I think that's a quick and dirty way to actually get to, what is a willingness to pay, and what's a psychological threshold? So I think that's a interesting method that you can do Monday morning.

**Madhavan Ramanujam** (00:28:16):
But the key here is to not just ask the question, "What would you pay?" But have that sales and marketing conversation. Tell people where they actually might get the benefits. Basically, exactly everything you would do after launching the product to create awareness and showcase the benefits, do it, and then ask these questions, so that you're priming them to the value that your product gets and you're not just having a random conversation. There are other techniques that go more and more, let's say, rigorous. For instance, purchase probability questions. So if you ask someone, "Okay, would you buy this product?" That's a meaningless question. At least if you ground them on a scale and say, "On a scale of one to five, would you buy it?" One is, "I'm not at all interested", five is, "Most likely I would buy it," or, "I would buy it for certain." And four is, "Most likely", for instance, and three is, "I'm neutral."

**Madhavan Ramanujam** (00:29:06):
What we are actually seeing is, even if people say, "Five", they are probably only 30 to 50% sure about whether they would buy, so you can start, and if they say, "Four", it's 10 to 20%. If they say, "Three and below", they're never going to buy it.

**Madhavan Ramanujam** (00:29:20):
So if you do this at scale, you can start coming up with, let's say a demand curve. And then say, "Where is the price optimal?" et cetera. So you can understand purchase probabilities. And if someone says, let's say, three, for a certain price point, then you can lower the price and say, okay, would they actually move their ratings to a four or five. So I think these are some simple ways to understand purchase probabilities and elasticities. Two more, if I may, I think, another one is what we call is most and least questions. And the thesis behind this is, if you go and ask people, "Okay, I give them a list of 10 features", let's say. And I say, "Rank them one to 10", most people will find that exercise painful, horrible, because there's always this messy middle where everything is gray. They all look the same.

**Madhavan Ramanujam** (00:30:06):
There's a lot of psychological theory that people are very adept at identifying the extremes. When it gets in the between, that's when things become tougher. So what we do is, if we have a list of 10 features that we want to understand whether people have willingness to pay for, when we are prioritizing the R&D roadmap for our clients, we would take a subset of, let's say six or so features out of those 10. And then they say, "In this set of features, identify the most important for you, and the least important." And the most important is defined as, "Must have, I will pay for it." Least important is, "I don't need it. I won't pay for it", connotation. This, people can do all day long, because they're just picking the two. And then we will change the set of six, another combination from the 10, and ask that same question.

So if you do this a few times, you will be able to prioritize the entire feature set in a relative fashion, and truly understand what drives willingness to pay. The last method, which gets into more advanced methodology, is what we call is more trade-off exercises. So here, what we do is we put people through actual buying patterns, or actual buying scenarios and say, "Okay, if you had this packaging and pricing, for instance, for your software product, what would you do?" Which is akin to a real life question. You [inaudible 00:31:19] the features, all the price, the number of plans, et cetera.

**Madhavan Ramanujam** (00:31:21):
Then we would change that and say, "Okay. If you change the features and the price, how would you react? Would you buy any of these products? Or would you say, I won't choose any of these?" These are more like shopping scenarios for your products, but it's realistic, and it's akin to real life. Based on how they choose these products, what we are trying to reveal is the mental models and rules that people use to make decisions. So for instance, if I add certain amount of features and increase the price, people say, "You know what? I'm not going to buy anything more."

**Madhavan Ramanujam** (00:31:50):
What that actually tells you is the addition of those features, people are not willing to pay the addition in terms of price, so they would actually opt out of the lineup that you actually have for your customers. So these kind of things, if you do these kind of exercise, you can get more precise on things like price elasticity, build some simulation models, try to understand how the market would react, et cetera. And different methods are actually applicable at different stages of a product and different stages of a company. If you're very early stage, let's say just an idea, just have the conversation. Just even asking, would you pay for it, is a good question because if someone says no, then ask why.

Then you'll hear a lot of good information. If someone says yes, ask them, why would you pay for it, they would articulate back the value that they understood, and that should be in your value messaging. So that's just the simple questions. If you are somewhere in middle, then maybe some of these, if you're in the [inaudible 00:32:44] seat stage, maybe some of the purchase property questions, all of these things can actually be a quick and dirty way to at least get to an answer and a point of view. Or if you're launching a product and it's in late stage or late stage in the product or the company lifecycle, and you need to get more precise in terms of your pricing and packaging strategy, then some of the methods are around trade off exercises, most, least, all of these things become incredibly relevant. Sorry, it was a long answer, but there are so many methods. It's all summarized in chapter four. So just one chapter to read in the book.

**Lenny** (00:33:15):
That was perfect. Thank you. Amazing. You mentioned that you want to ask why a lot, and that's something you talk about in your book a bunch. I think something like 50% of your questions should be why, after they answer these questions. Is that right?

**Madhavan Ramanujam** (00:33:29):
That's absolutely right. Yeah.

**Lenny** (00:33:31):
Cool.

**Madhavan Ramanujam** (00:33:31):
I was tempted to ask you why, but that would've not been very relevant.

**Lenny** (00:33:35):
Nope, not on this question. I might take care of that later. Can you talk about, logistically, how are you asking these? Is this a meeting specifically you set up with a potential customer to talk about pricing? Does it come at the end of, here, I'm pitching you on this product, or I'm trying to get desirability research feedback? What is that meeting set up for on behalf of the customer?

**Madhavan Ramanujam** (00:33:56):
It's usually either a one-on-one conversation with a customer, much early stage, you're a founder and you're having a conversation. It's basically, you're pitching the idea and trying to understand not just product market fit, but a product market pricing fit. Let's say, if you're a bit more late stage and you have a cross-functional team, this could be a conversation that the sales teams could actually be having along with the product teams to actually understand this. That's also what happens in companies like LinkedIn for instance, when they launch a new innovation. The team has to book in a credit card or lock in a budget from a customer for pilot POCs and everything else, and if they don't, they don't necessarily go down the route of productizing it because there was no final verdict on whether people would actually pay for these innovations.

**Madhavan Ramanujam** (00:34:38):
That's how I see it. So it depends. If it's early stage, more like founder led early conversations, if it's more late stage, then a cross-functional conversation, but usually, it's a one-on-one with, one-on-one as in with the company, it could be more multiple decision makers. Increasingly, in B2B SaaS for instance, it's not just one person deciding on a software budget, it's a team. So it's usually done with the team having that kind of conversation. It can also be done in terms of focus groups where you bring in a set of customers and then you mediate and moderate answers and trying to get to what is the right thing to do. And often, we also do a quantitative version of this where we are doing more test and learn through either AB testing or, most importantly, through controlled surveys that we would actually have, invite participants to actually participate and then they would give their opinions on these various concepts that we are actually testing. And then we try to understand what is the willingness to pay in the market based on those responses. So from a basic qualitative one-on-one validation all the way to more quantitative testing using other instruments.

**Lenny** (00:35:45):
So this is usually a part of a larger customer development product market fit discussion, here's a product we're thinking, here's classic user research.

**Madhavan Ramanujam** (00:35:54):
Yes.

**Lenny** (00:35:54):
Desirability discussion, and at the end you talk about willingness to pay stuff. Is that-

**Madhavan Ramanujam** (00:35:58):
Yeah, exactly. I would think of it as... I wouldn't say necessarily user research. It's a bit before that. User research probably gets more into usability and how people would actually use it. This is a one step before conversation where you're trying to understand testing and learning, whether people buy into your idea, do they see the value, is there an ROI, and frankly, what needs are you actually solving in the market. This, people can articulate all day long/ what is their jobs to be done or what is their needs paying point. What product to build is your job in some way as a product person. But when you showcase the product and say, okay, this is the product that I'm building which will actually meet this need, you want to see if people's eyes slide up. And that's also when you need to have the willingness to pay conversation because it's not just about saying but actually meaning what you say. And that can only be bought in if you actually use willingness to pay in the conversation. Otherwise, it's a bit of an empty self fulfilling conversation often, so. And then when you actually do this more and more, like I said, [inaudible 00:37:00] or refining things, then you can bring this into other pieces of the conversation and gets more smarter before you launch the product.

**Lenny** (00:37:09):
Do you have a rule of thumb of how many people you should talk to, at least, to get a pretty good sense of maybe we've gotten some-

**Madhavan Ramanujam** (00:37:16):
Yeah, I get this question all the time, and I say, please talk to one person. [inaudible 00:37:21] because most companies are not even doing that, in terms of willingness to [inaudible 00:37:25] conversations. But if you're, let's say a B2C company, of course you have scale in terms of reaching customers. You might have hundred thousands and millions of customers, consumers. In those situations, more of a quantitative validation might be easier to run. So if you get even 1,000, 2,000 responses that could be statistically significant and easy to do and pull off. If you're a B2B SaaS company and you're focusing on let's say 20 to 30 accounts leading 80 to 90% of your business, try to talk to as many of those as possible. So at least in the 20 to 30 ballpark. And often in these conversations after a point, you start hearing stuff repeatedly. If 20 people tell you the idea is horrible, it is horrible. You can test it all you want. So when you start hearing these things, then you pivot.

**Lenny** (00:38:13):
And then once you have your initial thought on what pricing should be, how often do you suggest folks iterate on their pricing strategy?

**Madhavan Ramanujam** (00:38:21):
Yeah. Usually we say at least every six months pause, and think whether you should revisit it. Within 12 to 18 months, probably there is time to revisit, especially given market dynamics in most industry verticals that people are in today. And also, there are some pivot points where it'll make sense to think about this, like you're introducing a new plan or you're introducing some new features. All of those moments in time from a product journey standpoint would necessitate having this conversation.

**Lenny** (00:38:51):
Got it. Final question on this topic, which we've spent I think half an hour on, which is awesome, because it's probably the most important to start with, but this is going to be a deep episode. What's the first thing that a founder or PM should do to go down the route of willingness to pay, if they were to start something on Monday?

**Madhavan Ramanujam** (00:39:06):
First of all, start educating yourself that there is a science on this topic. It's not just an art. Get confidence that people have done this before. Not only just startups, but companies like Borsch, and then read chapter four and do it.

**Lenny** (00:39:19):
Okay. Topic two, which I know is really a big deal to you and a core part of the way you think about it in addition to willingness to pay, and it's segmentation. Thinking about how to segment your customers and product. So can you just talk about, broadly, why is this so important to think about segmentation when you're thinking about pricing?

**Madhavan Ramanujam** (00:39:38):
Look, segmentation is a topic, again, just product market fit is a well understood term for many of your listeners. When we go and ask companies, do you have a segmentation strategy, roughly about 60% would say, yeah, we have it. And then when we check it, probably 10% of them actually have a meaningful segmentation strategy. What I mean by this is, most people think of segmentation as a demographic or persona exercise or how do I position this product to different personas and things and they get it horribly wrong. Tongue in cheek, I'll give you this example. If you think about a person who's 70 plus years old, lives in a castle, incredibly wealthy in the United Kingdom, you probably think about Charles, but that also fits Ozzy Osbourne. And I would probably measure that both of them have dramatically different tastes, need different things, value things differently, and are willing to pay for things differently.

**Madhavan Ramanujam** (00:40:28):
If you just base things on persona, you often get it wrong. Segmentation needs to be based on what customers need, what they value, and what are they willing to pay for, and how do you productize package to different segments. So the key lesson that I want your listeners to take away is, you need to be able to productize to segments. If you're trying to build a product and try to position it to different segments, you already lost the battle, because segmentation comes down to needs and understanding needs and building products based on those needs and willingness to pay. Show that you can treat your customers differently.

**Madhavan Ramanujam** (00:41:03):
Because if you build the same product and want to treat everyone similarly and say you have a segmentation strategy, you actually don't have it. Take a simple example. If you think about the water that we drink in a fountain, it's free. In a bottle, it's $2. You put gas in it, it's $2.50. Throw it in a mini bar, it's $5. It's the same water, but it's packaged, productized differently because people have different needs.

**Madhavan Ramanujam** (00:41:27):
I'm price conscious, I want it in a fountain, I want it to carry it around. I probably take in the bottle, I like the taste, I take gas in it, or I'm just simply ultimately lazy and I would pay the $5 to get it out of the mini bar and not go down the hotel lobby and get it for free because that's my need. If you don't understand these needs, you'll never be able to productize to those needs. So you'll just build one product and try to position it to the different needs based segments and you won't get it right for anyone.

**Madhavan Ramanujam** (00:41:55):
And we work with all kinds of industry verticals. We have not found a single vertical where our customer, where their clients' needs are homogenous. It is heterogeneous whether you want to accept it or not. And if you accept it, then you would start getting into the heart of segmentation and say, where is that heterogeneity? How do the needs differ in the market? How does a willingness to pay differ in the market and what can I productize to different needs and willingness to pay segments, so productizing to segments, as opposed to building one and positioning it to different segments. Usually, when I walk into these companies, they'll say, "We are building a one size fits all," I would quickly correct them and say, "One size fits none." So it's a bit of that. That's why this topic is deep, because people get the definition of segmentation wrong.

**Lenny** (00:42:42):
So segments are something that people hear often and I think, like you said, understand, and to your point, they think they've done segmentation. But again, to your point, a lot of times they do it wrong. You have this framework that is really interesting, this one phrase they use, "You can act differently to help you think about whether a segment makes sense and how to think about segmentation." So can you just talk about what's a sign your segmentation is correct versus not, and maybe how to think about the framework?

**Madhavan Ramanujam** (00:43:12):
So the three most important words in what you said is, "You act differently," so you as in your product teams, your sales teams, your marketing teams, your finance teams, act as in, come up with new products, build a business case, come up with a product marketing messages, sales strategies, differently, as in there's no point in doing segmentation and having the same reaction or treatment to everyone. You need to be able to act differently. So what that means is, if I know what you need, what you're willing to pay for and what you value, then my conversation with you will be different than someone else who needs something else and is willing to pay something else. I productize something for Lenny, and I productize something else for the others.

**Madhavan Ramanujam** (00:43:53):
And the key here is to understand if there is a significant, let's say, total available market or size of the market, where the needs are similar, and they're willing to pay, as in, let's try to find all the people who would want to drink water in the fountain. Let's find the people who want to bring it in a bottle. Let's find the people who want to have gas, would they pay for it. And then when you understand these segments, then you can say, "Okay, what do I build for these different segments?" And then focus on that segment when you launch the product. Have the marketing message for that segment and target that segment, as opposed to just building one thing and hoping that somehow these four groups will sort themselves out into your product. And that's the key thing.

**Lenny** (00:44:33):
When should early stage founders think about segmentation? Do you suggest it's right from the beginning, the first product should have multiple segments, or does it come later?

**Madhavan Ramanujam** (00:44:40):
This is also an often asked question in the sense that as a startup, as an early stage founder, often the excuse is, "Hey, we don't have time. We're actually pulling stuff out of the wall. We need to get something out there." It's like, "Time is of the essence, so we need to build a product." And usually they would say, "Let's build just a product, which is every awesome thing that we are working on, and then we come back and revisit whether we want to build other versions, et cetera, and we don't have resources to even build multiple products." Well, that logic makes sense when you don't understand the concept of segmentation.

**Madhavan Ramanujam** (00:45:14):
If you truly understand the concept of segmentation, you would say, you know what, as a first conversation, when you're having that willingness to pay conversation based on your idea, you would say, "Who's actually willing to pay for this innovation? What do they need? How many of them are there? Can we productize to this first compared to the others? "Then you'll start prioritizing not only your R&D roadmap, but you're resourcing to say, "Which segment should I start with? And then what segments would I actually add?"

**Madhavan Ramanujam** (00:45:40):
And then your value messaging would be tailored to that segment. People will understand the benefits. They will say, "Your product will be launched and people will get it and they would actually go for it." So having done this exercise early will tell you how many segments are there, what is the size of these segments, how to prioritize them, which one to pick first and which product to build first for that segment, and then productize to the other segments later. If you're lazy and sloppy, you'll build a product, you'll slap on a price, you'll throw it in the market and say, "I will attract everyone." You'll attract no one.

**Lenny** (00:46:14):
Amazing. So basically understand the segments right from the beginning. Don't necessarily launch products for every segment.

**Madhavan Ramanujam** (00:46:21):
Exactly. That's totally acceptable because it's not like everyone has resourcing to launch five products, five segments, everything else. Complexity, and plus, you don't probably want to be too complex when you're launching your products, but focus on the right segment, and launch it for that first.

**Lenny** (00:46:38):
**Madhavan Ramanujam** (00:48:00):
Yeah, keep getting these tweets from Peter. Yes, I love the product actually. It's water packaged as an $8 product, so it's great. That's a segment of customers that love that and-

**Lenny** (00:48:13):
Oh, I love that. The VC wants you to include that as an example, as you talk about water.

**Madhavan Ramanujam** (00:48:17):
Exactly. Will do from next time. Liquid Death, eight bucks.

**Lenny** (00:48:21):
Yeah. Any other examples as the premise of my question?

**Madhavan Ramanujam** (00:48:24):
I would probably start with a few obvious and famous examples so that just remember the point in some way, and then we can drill down further if needed. But if you look at, for instance, Apple. Let's assume the conversation in Apple was something like this, "Hey, we need to just build one product, one iPhone, because we need to maximize our market share and we will throw it out and slap on a price and hope to get the market." They wouldn't be the most profitable company in the planet today. What did they actually do? There is an iPhone for 299, 399, 499, all the way to 1499. They have built products to different segments.

**Madhavan Ramanujam** (00:49:01):
I remember walking into the Apple store when iPhone X was launched. I didn't want to part with a thousand bucks. I was checking the phone out ,and I looked at the features. I really didn't want the retina features and all these benefits. And then I saw that there was a phone without that for 799 and I picked the 8S and I walked out. So I belonged to that segment. I was not belonging to the iPhone X, but that's a product that has been productized for different needs, segments. And if you look deeply, Apple has not just priced their iPhones, they have productized to different price points and willingness to pay, and that's where it gets actually fascinating. So I think that's a great example of understanding differentiation and then productized to different needs. And I think that's a good example. Another one that comes to mind, where we worked with, this was in the pre-IPO days, Eventbrite, which is a B2B SaaS company. They used to have one product that was actually-

**Madhavan Ramanujam** (00:50:00):
A SaaS company, they used to have one product that was actually servicing all of their customers. And then we went through an exercise of understanding who are their customer segments and how do we productize to different segment needs. And if you look at what they have today, they have three plans, because there are segments behind this. And if you look at the plans, there are plans, for instance, the entry level plan has something like, you can only launch an event with one ticket type, like a general admission. And then if you take the middle plan, it has unlimited entry type. So you can have a general admission, a VIP admission, whatever, when you're actually having events. It actually makes sense, because if you're, let's say, hosting your local wine club meetup, whatever, event, you probably just need the general admission and that's it. But if you actually are a bit more professional and you needed multiple event types and you're having a event of that nature, then there's another product that actually appeals.

**Madhavan Ramanujam** (00:50:57):
But because of doing this, the one that has only one event type, that product is cheaper than the other one. So there's an essential product and there's a professional product and they have enterprise product. So this comes down to truly understanding what customers need, what they value, and what they're willing to pay for, and how can we productize towards that. Maybe another example that is obvious and in front of us, when we use our apps, like Uber is a great example of also segmentation.Because you have different car types. If they just had one car type, then okay, then that's a very different company, very different strategy. There's an Uber Black, Uber X. We used to even have the Uber pool pre pandemic. I don't know if it's back now.

**Lenny** (00:51:35):
I think it's back.

**Madhavan Ramanujam** (00:51:36):
It's back? Great. And they also launched this thing called comfort, which is a bit between Black and Uber X in terms of both price and also the types of cars.

**Madhavan Ramanujam** (00:51:46):
But it comes with certain features. For instance, you can say quiet preferred on a Comfort or a Black. And that's literally why I take one of these, because I'm probably working on my Uber ride over and I like to just have the quiet and just work on things, and I'm willing to pay for that and I belong in a different segment. But of course if I'm using, let's say Uber for my everyday commute, sometime maybe I do pool. So depending on even my point in time or depending on my situation, I might actually belong to different segments, and understanding this and then productizing towards that becomes key.

So the topic that we have been focused with many companies nowadays is not just doing a static view of segmentation, but truly understanding dynamic segmentation and how to offer product and services around the fact that people switch segments. So if I'm ordering on a Friday night on a food delivery platform, maybe I'm thinking pizza. Tuesday afternoon during my office time, maybe I'm thinking of a different type of cuisine. So if I know all of these things, healthy choices versus not, when can I productize what, then you actually start getting a dynamic views of segmentation. And the technology around us actually allows us to take a very dynamic view of segments and that's very fascinating. So the [inaudible 00:53:09] there are multiple steps. First do the segmentation, that's the basic. [inaudible 00:53:14] a static view. Maybe if it's relevant, even a dynamic view of this is the next frontier.

**Lenny** (00:53:19):
Which you shared just now reminded me of another really interesting framework in your book, around pricing strategy. And you talk a lot about just how important it is, one to just write down your strategy and why you think this is the right strategy. But specifically, you have these concepts of either you want to be maximizing, you want to be penetrating or you want to be skimming. And I thought this would be a good time to chat a little bit about that. Can you just talk about what these three strategies are?

**Madhavan Ramanujam** (00:53:42):
Sure. When we talk about pricing strategies, we hear many buzzwords, and it's irrelevant. So when we take a step back and look at it, there's literally only three types of pricing strategies. And if you know this, then you can follow one of these and [inaudible 00:53:57] to success products. The first one is skimming strategy, which is your Apple iPhone. They launch at a particular price and the next generation is probably at a higher price, but the previous generation actually goes down, so they launch at a higher price and then they start lowering the price. So they're skimming the market. And connotation of these products is also, it's a premium product. Price is a signal of quality, et cetera, et cetera.

**Madhavan Ramanujam** (00:54:20):
If you take penetration that's probably made famous by Amazon. And Amazon, they're probably operating at much thinner margins, but they're playing the volume game. Much more harder game to play because you need to have all of your costs in order, supply chain, everything else, and you're fine tuning towards the volume game. Often, I see entrepreneurs who say, "Let's just price load to gain growth." That's a fallacy. If you don't have a business model that actually supports this compared to Amazon, then you probably shouldn't be in a penetration strategy. And even in a company like Amazon and AWS has a very different strategy compared to the e-commerce marketplace. So within even business units, you can actually have different pricing strategies.

**Madhavan Ramanujam** (00:55:01):
And the third one is just maximization, which is you're neither on these two extremes but a bit in between, and you're saying, "Okay, what can I maximize in the next couple of years?" In my opinion at least Microsoft will probably belong in that category. And if I look at Apple, Microsoft, Amazon, companies that reached trillion valuations in our lifetime, probably the only three in some way, shape or form, they have dramatically different pricing strategies. The point is not about picking, just picking one, but it's about executing the one that you actually pick. And that's also what we write about in the book, how to pick one and not just pick one, but how to build your products around this executed live and breathe your business model strategy, which leads to your pricing strategy.

**Lenny** (00:55:44):
Amazing. I just want to say I love how deep we're getting into all these topics. This is amazing.

**Madhavan Ramanujam** (00:55:48):
Cool.

**Lenny** (00:55:49):
One final question on this topic and then we'll get to the next and the rest will be quicker. We're spending a lot of time on each, which I love. You talk a lot about the importance of packaging and bundling and how that alone can help you win a segment, versus even the price of the segment. Can you talk about how to think about the importance of that?

**Madhavan Ramanujam** (00:56:06):
The way to unlock your segmentation is to think about bundling and packaging, as in you're configuring your product, based on what customers need, what they value and what they're willing to pay for. Either you put a bunch of benefits that people like and call it packaging and put that out, or you're taking multiple products and calling it bundling and putting that out. So it's a question of, hey, that's the way you unlock segments. Your productizing. It's like the iPhone X versus the iPhone A test, different products, different features, different packages, et cetera.

**Madhavan Ramanujam** (00:56:36):
The quick framework to think about packaging, bundling, we call it the leaders, fillers and killers, exercise of framework. So if you think about the classic, let's say bundle, like a Big Mac or a Happy Meal, the Big Mac is the leader product, that's in the Happy Meal, and that's what people go for when they go to McDonald's. When you look at french fries and Coke, those are the fillers. You can put a burger along with french fries and coke, call it a happy meal, and most people wouldn't have bought a french fry or Coke, but if you just say for a dollar or two more, you know can actually get this, they would say, "Let's get the happy Meal." So you're actually bundling it in such a way that with marginal increase in price, you're also able to sell multiple products, which they wouldn't have if you didn't have it.

**Madhavan Ramanujam** (00:57:22):
The killer is the one where if you put it in the product, it just kills the bundle for everyone. So for instance, if you put coffee along with french fries and a coke and a burger, that's just going to kill the bundle. No one needs a double dose of caffeine when they're having a burger. But there are people like me, Lenny, who love to have coffee with their burgers. So these are great candidates for selling them as add-ons, because if I actually, I would pay for the add-on because I actually want the coffee. So I would take the burgers standalone, and I would take the coffee standalone and that's why it's actually listed separately in the menus.

**Madhavan Ramanujam** (00:57:56):
If you bundle it, what happens is it just depreciates the willingness to pay across the entire customer base to the point where no one actually wants it. So you need to find pockets of customers who want it and then maybe only sell it to them. So the rule of thumb that I usually say is, if 10 to 20% of your customers want something and they really want it badly, that's usually an add-on. That's not something that goes into a package unless you have a advanced package just for them, kind of thing. And if more than 50% of people want something, that's a leader product. So if you understand all these leaders, fillers and killers, then you can configure your product in such a way that your productizing to segments and you'll unlock maximum value.

**Lenny** (00:58:36):
Your example reminds me, I just went In-N-Out yesterday, and how I don't think they've changed their model ever. Somehow, it just works. They nailed it.

**Madhavan Ramanujam** (00:58:45):
It somehow works. I think In-N-Out, that's a story for another day. I think it works for a different reason.

**Lenny** (00:58:51):
Some people. Yeah. All right. And then the other thought I had while you're talking is, understanding who the leaders are and the fillers and the killers, comes from [inaudible 00:58:58] conversations. I imagine the stuff that people stack rank at the top is most likely going to be your leaders.

**Madhavan Ramanujam** (00:59:05):
Yes, exactly. So when you do those most and least questions and you stack rank them, the must haves will pay, or must have table stakes would be the leader products, or leader features or benefits. The ones that are nice to have and might consider paying or nice to have, those features are probably the fillers, and that don't need are the killers.

**Lenny** (00:59:27):
Awesome.

**Madhavan Ramanujam** (00:59:28):
Don't need I will not pay.

**Lenny** (00:59:29):
Got it. Interesting. When I saw you right about killers, I always thought it was things that it needs to have that would kill the deal if they don't, but that makes a lot more sense that it kills the deal if it includes it. Interesting.

**Madhavan Ramanujam** (00:59:38):
Includes it, exactly. Maybe we should change the language in the next sequence, monetizing innovation, but.

**Lenny** (00:59:44):
No, it's working, don't change anything.

**Madhavan Ramanujam** (00:59:45):
All right, good.

**Lenny** (00:59:47):
All right. Third topic, around your pricing model. This is something that you talk a lot about that people thinks too much about how much to charge and not enough about how to actually charge the pricing model. So could you just talk about maybe what that is and why people maybe don't think about it as much as they should?

**Madhavan Ramanujam** (01:00:04):
We usually say how you charge is way more important than how much you charge. Take a quick example and then bring the point home and then we can talk about why this is actually essential. So taking a non SaaS or software example, if you think about Michelin, which is a tire company, probably one of the most price sensitive, let's say, markets because, think about it, you actually go into a tire store, you see all of these things look similar, but they're somehow priced differently. How are you supposed to make a decision? It's very hard when you need to understand what you're paying for.

**Madhavan Ramanujam** (01:00:36):
And they came up with this new tire, which was supposed to last 20% longer, was a true innovation in the industry, and these are tires that were used for moving trucks from point A to point B. And when they thought about it, they said, okay, if we go and ask for a 20% premium, there's no chance they would get it, because it's a price sensitive market. If they don't ask it, the tires are going to overrun and they're going to cannibalize 20% of their business. So what they actually did was they changed their pricing model or monetization model and they said, "Okay, we are going to charge based on the number of miles that a person would drive."

**Madhavan Ramanujam** (01:01:09):
The truckers actually love this model, not just because it was pay as you go and they could pay when they actually use the tires and how and everything else. That was the obvious reason. But then now they could also invoice their end customers and say, "Okay, my journey was 798 kilometers or miles, and that's the amount of tire costs," and they could pass it through because it became a variable cost and people love this kind of model.

And of course, the tires lasted long. Michelin [inaudible 01:01:37] that more, but more people jumped into the Michelin bandwagon, because now they could actually buy a tire that's on a pay as you go basis. Now, if a tire could be actually with... And the age old model for tires is on a per tire basis. If a tire could actually be sold on a pay as you go consumption model, then obviously most products can actually explore this route, especially in a software setting. But the key lesson here is how you charge was the most important question. It was not the how much. The how much came about because of the how you charge.

**Madhavan Ramanujam** (01:02:09):
Another SaaS example that is probably top of mind for me is B2B SaaS's segment. This was before they went to Twilio. They used to price based on APIs. So the number of APIs that you actually have with Segment, that used to dictate which plan you would be and how much you would pay for it. But increasingly, they were also shifting gears towards selling to different personas within companies. And what is an API is a debate. Probably a marketing person does not necessarily understand exactly what an API is and the how you charge question became very critical.

**Madhavan Ramanujam** (01:02:44):
And what they actually did was, we worked with them and we identified their monthly tracked users what was a much better metric into how customers perceive value, and that was the more fairer metric for customers. If you're tracking more users in Segment, you're probably willing to pay more compared if you're tracking less. So the packaging was changed to a monthly track user instead of APIs. This is literally exactly the same Michelin per mile models, on a B2B SaaS was number of monthly tracked users. So that was a different example. So the how you charge question is super important, way more important than how much. If you don't focus on it and just rush to one or the other, often you're sub optimizing like crazy.

**Lenny** (01:03:30):
It's interesting that both your examples are usage based models winning, and it feels like in general, the usage base is where people are trying to go more and more, or maybe not. So maybe two questions. Do you feel like that's the future of SaaS pricing generally? And roughly, do you feel like that should be a default way of approaching when you're building, say, a B2B SaaS company, or is it still seat based?

**Madhavan Ramanujam** (01:03:52):
I would say it this way. It's most B2B SaaS companies follow what is actually in vogue at that present point in time. If subscription is in vogue, then they say, "Oh, subscription is the best strategy." If usage is made famous by Snowflake and others, they would say usage. So I think usage is obviously, let's say in vogue right now. I think it comes down to really understanding, based on your business situation, does subscription make sense or should you be usage or pay as you go if you're a SaaS company. There are different markers which actually identify this.

**Madhavan Ramanujam** (01:04:25):
If customers demand, let's say predictable bills, or usage is very similar month over month as in if you're subscribing for tide pods for instance, it's not like you're going to wash more clothes one month versus the other. The usage is the same month over month, or when the usage is highly variable, which is changing quite a lot between month over month, then if you price based on pay as you go, then your bills are also going to be dramatically different month one versus month two versus month three so you're going to have a very tough conversation with your customers.

**Madhavan Ramanujam** (01:04:53):
In all of these situation, a subscription actually makes a lot more sense. Or it could also be, usage is intermittent but the value delivered is ongoing. LifeLock is a great example. It's a product that you probably have to protect your identity theft production. The value is ongoing. The usage of the product is only episodic when your identity theft gets compromised. If they say, "Okay, I'm going to price base on usage," that would be dramatically wrong pricing model. So in those cases, subscription makes sense, or simplifying the pricing conversation is to your advantage. Let's say the Spotify was a good example.

**Madhavan Ramanujam** (01:05:29):
If everyone wants to listen, everyone needs to listen on a per song basis, but having a subscription actually made sense. It simplified the pricing conversation. Same as Netflix, all of those situations. So don't just rush to something like usage just because understanding that is key. Usage makes sense when people want low commit or less friction. So making it easier to buy an AWS thing. You onboard people and then you grow the product and I think those situations make sense, or when customers would demand transparency and fairness. Please note that don't mix transparency and fairness with being predictable. Those are very different things.

**Madhavan Ramanujam** (01:06:08):
Transparency and fairness just means that you charge for the product. For instance, if you don't use a subscription for a few months, is it fair that you're being charged for those months? That's fairness. That's nothing to do with being predictable. But if they're demanding fairness and transparency, often a usage based pricing model could make sense. Or alternatively, usage is intermittent and episodic, and the value delivered is also episodic and not ongoing, like you for instance, book the movie theater ticket or you book the flight. It's also intermittent usage, intermittent value, so the pay as you go makes sense.

**Madhavan Ramanujam** (01:06:40):
Or maybe there's even some underlying cost that scales with usage, like an AWS pay you go can make sense or even probably lastly, if there's some... Most important thing for pays you go, Lenny, is you need to have clear metrics that you can actually track and identify and attribute value and your customers would agree to that value generation, then pay as you go would make sense. If you can't track what you are charging on, often it's a really bad idea.

**Madhavan Ramanujam** (01:07:06):
And then, of course you can also be more hybrid sometimes and that's also a winning model. For instance, if you take HubSpot, it's a hybrid model between a pay as you go and a subscription, and it actually works well for them because there's a certain component on a fixed monthly basis. And then if you exceed those quotas and limits, then you actually get into a pay as you go model. So I would urge the readers not to rush into one versus the other based on what is in style at a given point in time, but give it a deep thought and say, what is your business, how are your customers, how are they situated? What are you servicing? And what makes sense between the two models?

**Lenny** (01:07:41):
Awesome. On the point of being able to even track usage, I've seen five startup decks of startups that help companies with this, because it's so complicated to know what to charge.

**Madhavan Ramanujam** (01:07:41):
What is that?

**Lenny** (01:07:51):
What to charge on usage based models, where they basically plug into your systems to help you figure out how much every company owes you. Maybe a couple more questions on this topic. One is just, what's a simple way of thinking about the options? Say, a B2B SaaS company, there's seat based pricing, there's flat based annual contract pricing, there's usage based, you can lop on, freemium to make a free version of it. Is that the four? What's a way to think about your options

**Madhavan Ramanujam** (01:08:18):
For B2B SaaS, I think those are probably the options. You're either subscription, pay as you go, freemium, these things. And of course, the price metric that you actually pick, what measure are you charging on, and then how do you structure, the price structure that you pick is important. Because for instance, are you flat for a certain amount of time and then it becomes variable, that's a structure. Or for instance, can you be two dimensional in your structure on two metrics, so the more people actually use your product and take actions that benefit you, the better price you get.

**Madhavan Ramanujam** (01:08:51):
This is something we call as a value matrix. For instance, in B2B SaaS companies that actually want to achieve wall to wall adoption. In many companies, it's still a pipe dream. It's just, yeah, you can talk product lead growth, but if your pricing model, it actually incentivizes product-led growth, that's a whole different conversation. So what we would do is, for instance, on one axis, you have seats, and on the other axis could be the number of departments that the product is being used at, departments is HR, legal, et cetera. And then the more users and the more departments, you get a better per user price. So you automatically build an incentive to actually say, if you want better price, sure, you drive the right behaviors.

[inaudible 01:09:32] get more people on the product and put it in the hands of more departments, so people can self-govern their pricing as opposed to just, you come up with a price and you're just negotiating. So when you think about pricing models, you have to think about first picking pay as you go subscription premium, then thinking about the metric, and then thinking about the price structure.

**Lenny** (01:09:51):
Awesome. And maybe one more example, say marketplaces. Basically, if you're taking a fee in almost every case, and it's a question of how much you take and then there's maybe a subscription piece on top of it, is that roughly right?

**Madhavan Ramanujam** (01:10:04):
Yeah, absolutely. So you could take a rake on the transaction and there's probably a platform fee or a subscription fee. So that comes down to, again, a hybrid of a structure. So there is a portion that is predictable and then there's a portion that is the usage base. It's not a reg based model, but it's similar to the HubSpot modeling principle.

**Lenny** (01:10:22):
Okay, and one last question on this topic. Say you want to test different models, say you're, say seat based and you want to try usage base, or you're usage base, you want to go seat. Is that possible? If so, how do you do that?

**Madhavan Ramanujam** (01:10:32):
It's definitely possible. There are ways to test this. It's a science. This is also what we do with many of our clients for a living, but maybe the easy Monday morning thing that I can actually ask your listeners to do is what we call as a break even exercises. So let's assume that for instance, let's take a marketplace. Let's say you're selling a dollar hundred item, as in your customer is selling it. If you ask them, what should the pricing model be, 3% transaction fee on a hundred dollar item, or one and a half percent transaction and dollar 50 cents, or 3 dollars, or are you indifferent?

**Madhavan Ramanujam** (01:11:11):
This is a basic question because if you do the math, all of those numbers are the same. So an economic human being rational, everything that business school taught us would say, okay, people will pick the indifferent option. I've done this thousands of times. I've never seen the indifferent actually win. It's always people will pick one or the other. So then you actually start understanding what model might make sense. Same thing with B2B SaaS companies. You would say, let's say you have a hundred seats and I would charge you, let's say, a thousand dollars and 10 dollars per seat, or I would charge you 2,000 dollars flat or I would charge you 500 dollars, and the rest in the seat based amount that equals 1500. It's all the same. People would say, "I like the lower platform P and the variable," or they would say, "I like the fixed," so the indifferent never wins. That's the easy way to test pricing models, what makes sense.

**Lenny** (01:12:05):
Awesome. That brings us to our fourth topic, and I think this is something that everyone listening is going to be like, "Oh no, I got this, I'm doing this. Great, I don't need to learn about this." But your point is that it's almost always wrong, which is focusing on benefits versus features when you're talking about your product. So maybe as a first question, just what's a sign that you're probably focusing too much on the features of your product when you're pitching it versus the benefits, which is, to your point, much more powerful.

**Madhavan Ramanujam** (01:12:32):
I think I see some tell tale markers or pattern recognition as to when people are talking more features as opposed to benefits. First of all, just to set the nomenclature, what you build as a product person is features. What people actually get out of it is the benefits, as what do the features actually do? And that's the benefit that a customer gets, and you need to pitch benefits. If you pitch features, you're not talking value. And if you're not talking value, no one is going to get it.

**Madhavan Ramanujam** (01:12:58):
So if you are super excited about the product and passionate about every single thing that the product is doing, most likely you're talking features and not benefits, because you're showcasing how cool your product is and how the different bells and whistles actually work, as opposed to focusing on what is the actual benefit for the customer. There are probably also other signs, for instance, if you don't see market fraction for what you actually build, either what you built is off base, but the good outcome of this could be that people actually don't understand what they're getting. And then actually then changing the speak to being more benefits is key.

To take an example, for instance, SmugMug, which is a ridiculously awesome company, they used to actually publish their pricing plans, which was, you had to scroll literally three or four pages and then you would see the price. It's all the features. Everything else that the company did, they changed it to benefits based communication. So a very simple thing. For instance, the ability to sell photos online is a benefit. There are probably 15 features that is behind that [inaudible 01:14:04] actually enables that stuff. But then focusing on the benefits, they had a double digit improvement in revenue and no changes in products. We show the before and after also in monetizing innovation, if someone is interested, and it's what it was was what they actually did. So if you don't see enough traction, then that could be a marker, coming back to your question, or it could just be that if you're too passionate about your products, then your chances are, as a product person, you're talking features.

**Lenny** (01:14:30):
Is there any other examples of companies that you think do this super well, to make it even more concrete?

**Madhavan Ramanujam** (01:14:34):
To take a, let's say a non SaaS example, my favorite, I'll come back to Porsche again, because their value communication is, to me, legendary. When they launched Taycan, which is their electric car, their value communication was something like this, I'm trying to remember it, but it was something like, "Taycan is not your most affordable electric vehicle, but that was never Porsche's goal. Porsche's goal was to actually build a car that was first and foremost a Porsche." That value save statement, what they actually built, totally resonates with those-

**Madhavan Ramanujam** (01:15:00):
... you know statement, what they actually build totally resonates with those, their audience. Taking a maybe SaaS example, Shopify is one of my favorites in terms of looking at the plans in terms of their benefit, like, what do they actually put out? All of the plans emphasize benefits and less features. Like for instance, the number of locations that you can track inventory is a benefit because if you actually have a more complicated supply chain, it's different from not, so there are plans which actually have different number of inventory locations, which is a benefit there. I mean, if I can track more or not, but behind this, that could be many features that actually enable this. So if you look at the plans that Shopify has, I think that's a great example. And also they have a lot of good value communication in there. I remember something like the tagline for Shopify Plus was, "Fair pricing, unfair advantage," and just things that actually make bloody sense, then you see it, what you're actually getting. So I think that's a great example that listeners can go into.

**Lenny** (01:16:02):
So maybe as a takeaway, folks should probably look at their website, browse through their pages and just look, or am I pitching features, or am I pitching benefits to the reader?

**Madhavan Ramanujam** (01:16:02):
Correct.

**Lenny** (01:16:02):
Awesome.

**Madhavan Ramanujam** (01:16:11):
Correct, exactly.

**Lenny** (01:16:13):
That brings us to our very final topic, which is behavioral pricing. You have all chapter on this concept of behavioral pricing, and it's super interesting. And it's interesting because you don't have to rethink your price, you could just sell at a higher rate by just thinking through this lens of behavioral pricing. So just to set context, what is behavioral pricing, and why is it important?

**Madhavan Ramanujam** (01:16:34):
Behavioral pricing basically is tapping into the irrational modes of our decision making and not just rational. I think that when I talked about the break even exercise, if you take a very rational view, indifferent would always win. But like I said it, I have never seen it. So there's always an irrational side of our brain that actually makes decisions, and understanding this as a product person would lend yourself to building products, and also positioning or framing the product conversation in such a way that appeals to both sides of the brain. The Predictably Irrational was a great book from Dan Ariely, made the concept very famous. We build on top of that where we actually talk about product and pricing strategies that actually you need to take care of when you think about the irrational side, that's what we call as behavioral pricing. To take a concrete example, and maybe I remember walking into a company and they had three products, and I remember asking the CEO, "Why do you have three products?"

**Madhavan Ramanujam** (01:17:35):
And he said, "I learned that good, better, best is a great strategy in business school." So I'm like, "Okay, that sounds great." But when you actually look at what was going on, they were giving the farm away on their entry level product. So they had three products, 49, 79, and 149, that was the price points. And what they actually were doing is they gave a lot of features for the 49. So they were giving the farm away, so 60 to 70% of people were taking the $49 product. Not many are actually opting to the others. What they did was actually super interesting, they just reframed the argument and they found out that between 79 to 99, the pricing was inelastic and there's a threshold at 99, not at 79. It is the same exercise that I talked about in the acceptable and expensive price, et cetera.

**Madhavan Ramanujam** (01:18:18):
So they moved the price from 79 to 99, and they moved the price of the 149 to like 199 because of the same kind of reasoning. And then what they actually did is they built another product at 299, which was simply a decoy to make the $99 product look attractive. So if I put a $99 product that looks awesome next to a $299 product, it looks even more attractive. I mean, God bless the 2% that even take the $299 product. But what you actually see is the mix shifted, more people took the $99 product because the pricing made sense, it was respecting the psychological thresholds, and next to a decoy it actually made more sense to pick that product, right? So it's just reframing the conversation. It was a 30+ percent increase in MRR and ARPU right after they actually did this change. No changes in products, no changes in features, just in terms of how they reframe the conversation.

**Madhavan Ramanujam** (01:19:13):
I mean, these kind of things are around us, and we need to understand these. For instance, if you go to a movie theater, you'll see a small popcorn for $7, an extra large popcorn with butter on it. Huge one is $8. Most people will say, "For $1, I'm getting this extra large one, let me buy it." But that $7 popcorn is a decoy. I mean, if that was not there, most people would be scratching their heads saying, "Why am I paying $8 for popcorn in the first place?" Right? So this kind of behavioral framing and nudging becomes important, it's not about deceiving your customers, et cetera, but it's just about framing the products in such a way that it also appeals to the irrational side of the brain as much as the rational side.

**Madhavan Ramanujam** (01:19:52):
The example that I talked about in the SaaS product on the three products, and compromising to the 99, rather than going for the early product, is simply product discipline. Don't give too much away in your entry level product. Don't give the farm away your entry level product, at least reserve something for the $99 product. So if you build the packaging correctly, you can emphasize a compromise effect, and this is a well known behavioral theory where people avoid the extremes. If you are quality conscious, you'll go to the right. If your budget conscious, you'll go to the left, but most people will compromise in between.

**Madhavan Ramanujam** (01:20:25):
If you actually see your packaging mix is like this and is not the normal distribution, as in most people actually prefer the entry level product, you're giving the farm away, maybe you should think about how to change your features and benefits so that you can actually steer more outcomes towards a middle package compared to the entry level one, and also then charge based on the value that you're actually bringing to the table. So we have talked about many behavioral pricing strategies in the book, dedicated an entire chapter to this.

**Lenny** (01:20:54):
Did you actually share a few of them? Just like some of these tactics that you find? I don't know if you have them in your head.

**Madhavan Ramanujam** (01:20:58):
Sure.

**Lenny** (01:20:59):
But yeah, anything that you could share.

**Madhavan Ramanujam** (01:21:00):
You can talk about these topics all day long and I'm probably going to keep telling you what I know, but tell me when you're bored.

**Lenny** (01:21:05):
It's great. You're board.

**Madhavan Ramanujam** (01:21:06):
So there are a few, right? I mean the compromise effect is the good, better, best that we talked about. The next one is what we call as, let's say pennies a day effect, or how you actually frame your pricing. So for instance, if I tell you a $30 per month price, it's very different from $1 per day. The way you actually frame your price, if you can actually showcase some kind of bargain, like AWS does this really well, the price that you actually see is so less because also the units and consumption is so less, but of course that will stack up if you use it a lot. But the price, if that started with a higher price point compared to a lower price point, that could have been different sort of situations.

**Madhavan Ramanujam** (01:21:47):
Similarly, for instance, when you take let's say you have a monthly subscription in your SaaS business and you also have an annual subscription, you need to showcase your annual subscription as a monthly price. Like, for instance it is 29.99 if you actually take an annual subscription, but it's 40 bucks a month if you actually do monthly subscription, but you're still messaging the price as a monthly price because if you actually just do the computation and say, "Okay, instead of saying $30 a month." I would end up saying it's 360 a year. That price could actually look like a higher price, but if you reframe it looks like a more attractive price. So that's a pennies a day kind of effect. I think that that kind of makes sense.

**Madhavan Ramanujam** (01:22:29):
On a product side, if you're what you're building is products and consumables, then things like the razor blade model actually makes a lot of sense. Most famous, made famous with razor blades, right? I mean if you think about the Gillette stick that you're buying, it's probably cheap, but their razor blades add up very quickly so that initial pricing or investment is less, but then you're making money on the consumables. Right? The HP print cartridges, same thing, the printer is cheaper, but then the cartridges add up. For a SaaS product, it's very similar. If you actually have a product that's a base platform, but then you have consumables, then you might want to make the platform price attractive so that people onboard themselves on the platform and then they're paying for chunks of usage, or things like this, which is a razor blade model, which is much more attractive for people typically because there's a lot of scrutiny on the upfront cost that people are actually paying as opposed to doing an entire TCV calculation.

**Madhavan Ramanujam** (01:23:25):
I mean, like a total cost of ownership calculation. Most people don't do that. They're looking at what they're actually going to pay. So if you're more attractive upfront, that could be a different way to reframe your product or price. Maybe an advanced version of a behavioral tactic that I would probably talk about from product side is what we call is a Panini effect. The thesis for this is when we are kids, or for those of your listeners who have kids at home, one of the most repetitive exercises that we all went through as kids was to build puzzles, or fit different things together. Right? I mean, I used to do that. I thought I grew out of it. It so happens that you never grow out of this from a psychology standpoint, people love to build puzzles and have a compulsion because they just started with most of this in their childhood.

**Madhavan Ramanujam** (01:24:12):
So it is a Panini effect comes from the sticker book album that we actually used to collect when we are kids, or building puzzles, 500 pieces, whatever, all of these kind of things. So when you actually build a product, and even in the most of the most complex SaaS industries like financial services, when we have tested this with our clients, if you list the products usually 20% of people will buy more than one product, or they will attach themselves to more than one product because most of them are just buying one. Like, you have a real estate product, let's say you have a brokerage product, you have a different investment product, et cetera. You just list all the product. But if you show it as a puzzle and you actually say, "Hey, these are the six products that we offer, and if you complete it, you complete the puzzle. You have actually finished checking a few of these and these are empty."

**Madhavan Ramanujam** (01:25:02):
And that's like the first thing literally people actually see when they come into the product, we actually see the attached rates going crazily up. 40 to 50% of people suddenly start taking more products because there's a compulsion to say, "Yeah, I didn't finish this one." Even in a B2B SaaS setting. But of course if you're a B2C customer, like say you're a food delivery platform, or you're a ride hailing, for instance, if you say, "Okay, this is your weekly puzzle and if you take a ride every day, or you take a ride during evenings." Or, if you give people a task or a puzzle and show them the puzzle and show them that you have done this, but then these are the other things that you have not done, people change their behaviors because they actually feel a compulsion to finish it. Starbucks actually launched a bingo card, which is the same principle.

**Madhavan Ramanujam** (01:25:47):
So the Panini effect is a nice way to actually think about how to showcase your products in such a way that you create compulsion for people to buy multiple products. I mean, if there's show notes in your podcast, I'm happy to give you some visuals that you'll see it, it's bloody obvious.

**Lenny** (01:26:04):
Absolutely. Please send, we will include them. I don't get why it's called the Panini effect. It makes me think of LinkedIn and their whole little completion percentage, but is the Panini because you make a thing and it all comes together?

**Madhavan Ramanujam** (01:26:15):
No, I think this Panini effect, it comes from the Panini sticker books, whatever, all the things that we used to use before-

**Lenny** (01:26:22):
Not the sandwich? Okay.

**Madhavan Ramanujam** (01:26:25):
No, it's not the sandwich. And yeah, I mean, I guess we made a category out of what this is.

**Lenny** (01:26:25):
Cool, okay.

**Madhavan Ramanujam** (01:26:32):
We just call it Panini.

**Lenny** (01:26:32):
Got it. There's something else you touched on that might be worth double clicking on it, is this price threshold psychological trick? Is there heuristics, or just rules of thumb of like, here's thresholds people generally have, or is it generally very custom to the product?

**Madhavan Ramanujam** (01:26:46):
If you look across B2B SaaS, or consumer products, you'll find some thresholds that often make sense. Like for instance, if you are looking at $29, most people would say equate that to a dollar a day and say that $ 30 is a threshold. So you see some of these things. Beyond this you need to test for your own products and categories because the anchors are also referenced based on other competitive alternates, what their perception of value is, and so on. So doing the exercise, like I described earlier, the acceptable, expensive and prohibitively expensive would give you psychological thresholds. And by the way, that's also a behavioral pricing thing because you're saying if you cross 99 to like 101, there's a steep drop in the demand curve. And if you didn't know this, you can do all the quantity you want, you're going to probably optimize your prices somewhere.

**Madhavan Ramanujam** (01:27:36):
But at the end of the day, people are also looking at pricing from a psychological standpoint, and often we are able to validate this statistically and significantly as to what are the different thresholds for your products in the market. And this also gets a bit more quickly complex, and that's why the testing is important because it's not just about the product, but what happens when you have add-ons? What happens to the thresholds when you have price structures? Or, what happens to the thresholds when let's say you have a platform plus a usage strategy? So then the testing and learning becomes inevitable and there's no rule of thumb that you can just apply. But of course there are certain things like 30 bucks a month or whatever, that's a usual threshold that we see, or 9.99 famously made famous by all the subscriptions that we probably use.

**Lenny** (01:28:22):
Awesome. Well, to start closing our chat, just a few more questions that I wanted to get through.

**Madhavan Ramanujam** (01:28:28):
Sure.

**Lenny** (01:28:28):
One is the market is slower, the economy has slowed, purchasing seems to have slowed. Do you have any advice for founders, or product managers, or anyone thinking about pricing in this kind of market that we're in now?

**Madhavan Ramanujam** (01:28:46):
Yeah, I think it's a great question because we need to prepare for it, but of course be proactive. And I would say three things that founders can keep in mind when it comes to product pricing, if there is a downturn especially. The first thing to think about is building a lesser expensive alternate compared to what you actually have and keep it in your back pocket. So for instance, if you have a product, SaaS product, I would think of what can I de-feature from this product and then create a lesser expensive alternate that I keep in my back pocket to reduce churn. So if someone says, "You know what? I can't afford this anymore, it's a downturn." Give them the lesser expensive alternate, keep them in the system as opposed to them going away. If you just discounted price, guess what's going to happen six months later, that's going to be your new price.

**Madhavan Ramanujam** (01:29:36):
So before you price discount, think about what value can you exchange to actually justify that price discount. So you are taking value away in a de-featured product and hence you can discount. So having that kind of price integrity is super important with your customers. So don't just rush to dropping price, that'll be the absolute worst thing you can do to yourself at that moment, and also in future. So having these kind of less expensive alternates. Second one I usually say, which is in line with not dropping the price, is to think about three non pricing actions that you can do when this actually happens. For instance, do I give more product to preserve the price? That's a non pricing action. So I give more value. Say, "Hey, times are tough. Take the best product professional, you've being a great customer." Earn the loyalty. But when times are great, you're probably going to renew the pricing at let's say a higher price because you just gave them a product for one year at the same price that they're actually paying. So I think that's a non pricing alternate.

**Madhavan Ramanujam** (01:30:42):
Or, it could be change the contract terms, say that, okay, take a three year contract, or two year contract and then think about that as an alternative as opposed to reducing price. Or, things like for instance, payment terms. Like, "Okay, if you say it's difficult, I'll change it from 15 days to 30 days." And level the payment terms as opposed to changing price, right? So three non pricing actions. We write about some of these also in the book, and the last one I would probably say is think about changing your business model or pricing model. You talked about usage based pricing, frankly an outcome based or attribution based pricing. Frankly, there's the best time to actually think about these things. Like, if people are not using the product, changing it to a usage based, people would say, "That's great because there's a downturn, we're not using the product." And they would opt into a usage based pricing because they're going to pay lesser because they're not using.

**Madhavan Ramanujam** (01:31:37):
But when times are good, again, they're going to use it. And basically what you did is you just lodged in a usage based pricing easily compared to trying to do that when the times are good and people are saying, "Oh, I actually want fixed, or I don't want the usage." And things like this. You actually just took that as an opportunity to change.

**Madhavan Ramanujam** (01:31:53):
I mean, one extreme example that was interesting during the pandemic is a software company that was actually providing software to hair salons. I mean, just as an example, like hair salons. And this company, I mean, it used to be a per seat model. I mean, that just used to make sense, but they want think about usage. During the pandemic no one for instance, went to a haircut. They were all taking this at home. So they said, "Okay, let's change it to a per haircut basis." But of course when things are back again, that kind of model can recoup a lot more compared to a per seat model because that's really where the value is actually getting derived, right? I mean, just as an example. So three things again. One is thinking about changing your pricing model, three non pricing actions that you can take, and then how can you de-feature something and keeping it in your back pocket so that you can have a proper pricing conversation and not just drop a price.

**Lenny** (01:32:48):
That is some killer advice. Thank you for sharing all that.

**Madhavan Ramanujam** (01:32:51):
Thank you.

**Lenny** (01:32:52):
Something else is, during our chat, preparing for this call, you mentioned that you're maybe working on a new book.

**Madhavan Ramanujam** (01:32:59):
Yes.

**Lenny** (01:32:59):
Can you talk about what it's going to be about and anything else?

**Madhavan Ramanujam** (01:33:02):
Sure. I will try to see what I can talk about without getting too detailed, but the thesis of the book is... the title of the book is, it's called Unlocking Growth, Growth That is Profitable, Better, et cetera. So Unlocking Growth and the subtitle is Breakthrough Strategies for Acquisition, Monetization and Retention of Customers. So this is a bit like where Monetizing Innovation stopped and this book picks up from that, when let's assume you've built a great product based on what customers need, what they value, what they're willing to pay for, now what? You need to acquire customers, you need to monetize them, you need to retain them. So this book actually gets into all of those dimensions, and the key pattern that we have seen Lenny, over and over again is most companies would have teams and people dedicated to these three functions. I mean, acquisition, monetization, retention.

**Madhavan Ramanujam** (01:33:52):
If you unlock these three, you're getting to profitable growth, right? I mean, that's literally the three things you ought to focus on. But what ends up happening is most people don't understand the interaction effects across these, or they, in the worst case, they even treat it as silos. So the acquisition team works on something, but the monetization team is not looking at the interaction of what they actually do. For instance, 90% of customers or people who we meet who claim to have a land and expand strategy are only landing. They're not expanding because they gave their farm away in the land. So how do you actually think about a land and expense strategy in such a way that you can acquire, monetize, and retain customers? So this book actually goes into breakthrough strategies to balance the trade off between acquisition, monetization, and retention and build the right products and come up with the right pricing strategy.

**Lenny** (01:34:41):
You're going to have incredibly strong product market fit with this audience. Is this something anyone can pre-order yet, sign up to get notified when it's out?

**Madhavan Ramanujam** (01:34:48):
I think the pre-order probably is still open in Amazon, so I think that's something you can check. I haven't checked lately, but you can follow me on Twitter, @MadhavanSF. That's M-A-D-H-A-V- A-N-S-F. I usually tweet about this book, and in general, or follow me on LinkedIn, or add me on LinkedIn. I think those are probably some good ways to keep in touch. The book is supposed to be out in Q2. Q2, Q3 timeframe, so watch out for it.

**Lenny** (01:35:17):
Amazing. So it's called Unlocking Growth, they can search on Amazon, right?

**Madhavan Ramanujam** (01:35:20):
Yeah, absolutely. If you look for Unlocking Growth and even bookmark it, you're going to have it in your list of books that you want to buy.

**Lenny** (01:35:26):
All right, I'm going to pre-order it immediately. Any other good resources that you recommend for folks that want to learn more about pricing and just all the things that we talked about other than your book?

**Madhavan Ramanujam** (01:35:36):
There are a number of resources. Our founder Herman Simon, of the Simon of Simon Kucher has done a lot of books, particularly I like one, which is called Confessions of the Pricing Man. I mean, he started this business 35 years ago and literally out of university and academia, and we have grown to where we are today. But it talks about some of the lessons that he has learned. I find it fascinating and it's probably the better book, compared to Monetizing Innovation. So I think I would urge readers, definitely read that.

**Madhavan Ramanujam** (01:36:05):
The other book that probably we also put out, which is topical right now, is one of my partner colleagues, Adam Hector and Herman wrote a book on pricing during inflation and inflationary times. So I think that's a very topical book that I think your readers can pick. This is within our Simon Kucher assets of people who have actually written books. Another resource to probably look at is Kyle Poyar from Open View, he puts out some really good stuff on product led draw pricing, et cetera. I mean, he's an alumni Simon Kucher, so we are proud of our alumni, but it's some fantastic work that he has done. He has also written some SaaS pricing guides, et cetera. So I would highly encourage you to check Kyle's work. I think that's fascinating, and also feel the folks at First Round are pretty good at putting some good content on pricing product, et cetera.

**Lenny** (01:36:55):
Amazing. We will link to all of that in the show notes. Madhavan this was everything I hoped it would be. This is probably the record for the longest podcast we've done, and it's no better topic to spend a lot of time on. Thank you again so much for joining me. Two final questions. You answered most of them, but just in case there's anything else, where can folks find you online and learn more? And how can listeners be useful to you?

**Madhavan Ramanujam** (01:37:18):
Online I mentioned LinkedIn, so Madhavan Ramanujam that's on LinkedIn and @MadhavanSF on Twitter, that's probably where you can find me online. Or, even at SimonKucher.com, and you can search for leadership and you'll probably see my name there.

**Madhavan Ramanujam** (01:37:32):
What can listeners do? I think I would probably say that the fundamental level, if you can talk about this topic, actively share what you have learned. If there are sections of the book, for instance, that you like talk about it, the biggest thing that we can all do is to educate each other and everyone that there's a science behind all of this and it's not just an art, and if that's relevant, then I think message accomplished, that's also why we wrote Monetizing Innovation.

**Lenny** (01:38:04):
What a beautiful way to end it. Madhavan, thank you again for being here.

**Madhavan Ramanujam** (01:38:07):
Thanks, Lenny.

**Lenny** (01:38:09):
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcast, Spotify, or your favorite podcast app. Also, please consider giving us a rating, or a leaving review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lennyspodcast.com.

**Lenny** (01:38:27):
See you in the next episode.

---

## [5/18] He saved OpenAI, invented the “Like” button, and built Google Maps: Bret Taylor on the future of careers, coding, agents, and more

**Lenny Rachitsky** (00:00:00):
You're CTO of Meta. You are a co-CEO of Salesforce, you're chairman of the board at OpenAI. How do you think the AI market is going to play out?

**Bret Taylor** (00:00:07):
The whole market is going to go towards agents. I think the whole market is going to go towards outcomes-based pricing. It's just so obviously the correct way to build and sell software.

**Lenny Rachitsky** (00:00:16):
This makes me think about, I had Marc Benioff on the podcast. You guys were co-CEOs. He was extremely agent-pilled.

**Bret Taylor** (00:00:21):
It's so hard to sell productivity software, which I learned the hard way.

**Lenny Rachitsky** (00:00:24):
What's a story that comes to mind when you think about your biggest mistake?

**Bret Taylor** (00:00:27):
I was the product manager for, it was called Google Local had a pretty tough product review with Marissa and Larry, and to not do that well with a link from the Google homepage is embarrassing.

**Lenny Rachitsky** (00:00:37):
I think it's really empowering for people to hear it's possible to succeed in spite of a massive failure like this.

**Bret Taylor** (00:00:41):
They gave me another shot to do the V2 of it that resulted in Google Maps. We got about 10 million people using it on the first day.

**Lenny Rachitsky** (00:00:48):
What mindset contributed to you being successful in such a variety of roles?

**Bret Taylor** (00:00:52):
Waking up every morning, what is the most impactful thing I could do today?

**Lenny Rachitsky** (00:00:56):
Today, my guest is Bret Taylor. Bret is an absolute legendary builder and founder. He co-created Google Maps at Google. He co-founded the social network, FriendFeed invented the like button and the real-time newsfeed, which he sold to Facebook. He then became CTO at Facebook. He then started a productivity company called Quip, which he sold to Salesforce for $750 million. He then became co-CEO of Salesforce. He's also currently chairman of the board at OpenAI. At one point he was chairman of the board at Twitter. Today, co-founder and CEO of Sierra an AI started building agent to help companies with customer service sales and more.

**Lenny Rachitsky** (00:01:32):
In our conversation, we cover so much ground, including what skills and mindsets have most helped Bret be so successful in so many roles, why we're all still sleeping on the impact that agents are going to have on the business world. How coding is going to change in the coming years, where the biggest opportunities remain for startups, lessons on pricing and go-to-market in AI, the story behind the like button and so much more. This is a truly epic conversation with a legendary builder.

**Bret Taylor** (00:04:14):
Thanks for having me.

**Lenny Rachitsky** (00:04:15):
My pleasure. There's so much that I want to talk about. You've done so many incredible things over the course of your career. Just boggles the mind, the things that you've done, and we're going to talk about a lot of that sort of stuff, but I want to actually start with the opposite. I want to talk about a time that you messed up, a time that you screwed up in a big way. We have this recurring segment on the podcast called Fail Corner, and so, I thought it'd be fun to just start there before we get into all the great stuff you've done. What's a story that comes to mind when you think about maybe your biggest mistake in building a product?

**Bret Taylor** (00:04:43):
It may not be the biggest, but it was my first prominent mistake as a product manager at Google. So for me, it feels big because it was very formative for me as a product designer. So I joined Google in late 2002, early 2003, and I was one of the earliest associate product managers at the company and first, was working on the search system, essentially expanding our index from 1 billion webpages to 10 billion, which was a big deal at the time. It seems quaint now. Then I did a decent job and so my boss, Marissa Meyer, gave me the opportunity to lead a new product initiative, which was a big bet on me and it was both an opportunity to do something for Google, but I was also being pretty scrutinized just as a young new product manager and the premise given to me was work on local search.

**Bret Taylor** (00:05:40):
At the time, the Yellow Pages was still dominant and well, Google was really good at searching the web. It wasn't really good for finding a plumber or a restaurant just because it wasn't really a huge part of the internet at the time. So this content wasn't necessarily on the internet and even if it was, you didn't really want to find plumbers in Manhattan, you wanted to find plumbers in San Francisco, if you're me. And so it was both a technical problem and a product problem and a content problem.

**Bret Taylor** (00:06:11):
We launched the first version of that product that I was the product manager for, was called Google Local and it was, I'll be a little bit more critical now than I might've been at the time, but it was a little bit of a me too version of Yahoo Yellow Pages, essentially grafting on Yellow Pages search on top of Google Search and with a properly crafted query you could see those listings at the top of your search results but a standalone site at local.google.com. Actually, it was an important enough initiative that actually, there was on the Google homepage, it had Web, Images and Local was up there as well, so it's got top bill.

**Bret Taylor** (00:06:54):
I mean you could put almost any link on the Google homepage and get a lot of traffic to it. And despite that, it didn't do that well and to not do that well with a link from the Google homepage is embarrassing. There's not much one can do more than giving you that kind of traffic to give you an add that as a product leader or a product manager. And the product was fine, it worked, but it really wasn't differentiated. And in many ways, I think, again, I think I've had these reflections more sense than at the time, that I had some of the time, but why use this instead of Yahoo Yellow Pages? But more than anything else, why use this instead of the Yellow Pages? It was a digital version of something that had come before. Had a pretty tough product review with Marissa and Larry and others and it was fine. I wasn't about to get fired or something, but it was, I don't know, the shine on my reputation was waning a little bit.

**Bret Taylor** (00:07:52):
And they gave me another shot to do a V2 of it and I got the impression it wasn't like my last shot, but I certainly was feeling a little dejected from going from a hot shot, new PM to a new thing. So, we spent a lot of time thinking about how can you make something that's just much more compelling and not just a digital version of the Yellow Pages and not just so similar to some of the other products out there. And that ended up being the thread that we pulled that resulted in Google Maps. We had licensed from MapQuest, the ability to put this little map next to the search results, it was always the ugliest part of the product and we always made these backhanded comments about it internally and we spent a lot of time saying, what if we inverted the hierarchy here and made the map the canvas?

**Bret Taylor** (00:08:49):
We ended up finding Lars and Jens Rasmussen who had been working on this Windows mapping product and we got them into the company and started exploring this space and it ended up where, through that exploration we ended up integrating a lot of different products. We ended up integrating mapping local search, driving directions, like all of these products at the time were actually separate product categories and ended up with something that redefined the industry and certainly my career. But it took, I think for me as a product leader, it changed the way I think about product just because there's feature and functionality and then there's like why should I use this thing in the first place? And it was notable, there was a couple of interesting moments.

**Bret Taylor** (00:09:34):
When we launched Google Maps we got about 10 million people using it on the first day, which at that scale of the internet at the time was huge. And then in August of 2005, we integrated satellite imagery from a recent acquisition called Keyhole, which became Google Earth and we got 90 million people using it on the same day. Everyone wanted to look at the top of their house when the imagery came out. And it was really interesting because there's so many subtle product lessons in there. First is, I think as you have these new technologies, rather than literally digitizing what came before, if you can create an entirely new experience, it answers the question for a new customer, why should I give this the time of day? And so, really disassembling the Lego set and reassembling into something new rather than just digitizing what was there before. Certainly, that was the lesson I think in Google Maps, really was native to the platform in a way that a paper map couldn't be and that was a really meaningful breakthrough.

**Bret Taylor** (00:10:35):
And then with satellite imagery, it honestly wasn't the most important part of Google Maps, but it was the sizzle to the steak and it created, I don't think the term viral was a thing people said back then, but it created a viral moment. We were on Saturday Night Live, which is the coolest thing. Andy Samberg in, I think it was called Lazy Sunday rapped about Google Maps and Lars and I were texting each other. We did it. We're on Saturday Live. Mission accomplished. And it was also showing that as you're there thinking about products, there's the why you decide to use a product and then what is the enduring value. And those are deeply related but not all the same thing. And I just learned so many lessons that I took with me for every subsequent product that I worked on.

**Lenny Rachitsky** (00:11:18):
That is an awesome story. One I think it's really empowering for people to hear, even you Bret, who I'm going to share all the successes you've had, have had a massive failure with the CEO of Google [inaudible 00:11:30] just like Bret, you screwed up. And it was such a big bet. So one, just it's possible to succeed as you have succeeded in spite of a massive failure like this. And then some of the product lessons you shared, just to highlight a few of these things because I think this is great, is just you will often not win if you just make something that's a better copy of something else, what you want to look for is something that is an entirely new experience, something that's differentiated, something that's a lot more compelling. Let's flip to talk about what you've learned from actually being very successful at a lot of things.

**Lenny Rachitsky** (00:12:02):
So I was looking at your resume and you basically have been very successful at every level of the career ladder and in such a huge variety of roles. So let me just read a few of these things for folks that aren't super familiar with your background. You were CTO of Meta, you were co-CEO of Salesforce, you're also CPO and COO at Salesforce. At Google, you joined as an associate product manager where you famously, you didn't mention this but you rebuilt Google Maps on weekend. We're not going to talk about that. You're chairman of the board at OpenAI. You were chairman of the board at Twitter. You've also founded three different companies, one social network, one productivity docs company called Quip, and now, Sierra. Fun fact, at FriendFeed you invented the like button and I don't know if people know that and also just the newsfeed, I'll just throw that out there to give you some credit.

**Lenny Rachitsky** (00:12:50):
So you're basically an associate product manager, an IC product manager, an engineer, CPO, COO, CTO, CEO of three different companies including a public company. Very rare that somebody is successful at all these types of roles and all these levels. So, let me just ask you this question. What mindsets or habits or just ways of working have you worked on building in yourself that you think have most contributed to you being successful in such a variety of roles and levels?

**Bret Taylor** (00:13:19):
Yeah. It's actually something I am proud of. I like the fact I've worn different hats. It's actually amusing when I meet colleagues that I've known from one of those jobs, they'll often think of me through the lens of that job. And so, I'll go to meet folks from Facebook and they think of me largely as an engineer. I'll meet folks from Google, they think me largely as a product person. At Salesforce, a lot of the folks there interacted with me as like a, lack of a better word, a suit, the boss. And I'm not sure they think of me as an engineer at all, even though I was still probably coding on the weekends for fun. And one of the things that is a principle for me is to have a really flexible view of my own identity. I probably would self-describe as an engineer, but more broadly I think of myself as a builder and I like to build products and I think companies are one of the most effective ways to build products.

**Bret Taylor** (00:14:17):
There's also things like open source, but I think I'm a huge believer in the confluence of technology and capitalism to produce just incredible outcomes for customers. And as a consequence, I think to really build something of significance, I think to be a great founder, you really need to be able to not have such a ossified view of your identity that you can't transform into what the company needs you to be at that point. And every founder you'll talk to, one day, I think selling is a big part of being a founder. You have to sell investors on wanting to invest in your company. You have to sell candidates on wanting to work at your company. You have to sell customers to want to use the product that your customer produces. You have to have good design taste, not just for your product but for your marketing and essentially, soliciting your customers.

**Bret Taylor** (00:15:09):
You have to have good engineering. If you're building a technology company, the technology comes first. It's why this industry is so transformative. I probably credit, and I've told this story before, but I'm very grateful for her, but I probably credit Sheryl Sandberg for really changing the way I approach new jobs. The story, and I might be embellishment a little bit, but I think it's broadly accurate. So I had just become the chief technology officer of Facebook and when I first got the job, it was the flavor of CTO or that relatively small group reporting into me, but contributed almost as a very senior architect on a number of projects. And then at some point, Mark Zuckerberg reorganized the company and split it into a bunch of different groups. I ended up with a very large group, 100 under me and I was essentially running our platform and mobile groups, products, design engineering.

**Bret Taylor** (00:16:10):
So I went from a handful of reports to, I don't know, over 1,000 or something. It was a big group. And it was the largest management job. I had become a manager at Google, but a modest team. And so, I was doing okay but not great. And I had this moment where Sheryl saw me, I think I was editing a presentation for a partner just because the presentation I got didn't meet my quality bar and I was editing it and griping about it. She pulled me into a room and gave me talking to a little bit about holding my team to as high of a standard as I have. If someone wasn't meeting my expectations, what was my plan to manage them out of the company or... Just giving me management one-on-one.

**Bret Taylor** (00:16:57):
And she's a remarkable mentor in the sense she can give you feedback that's very direct and often, a bit uncomfortable, but you know she cares about you. And so it was the type of feedback you listen to. I went home that night and I was stewing on it and not very happy. I was like, you get naturally a little defensive in those moments. Is that really true? Am I really fucking it up or is she overreacting? And then I woke up the next day, I was like, "No, she's right." And I had realized this subconscious limiter that was limiting my success in the job, which is, I was trying to conform the job to the things I thought I liked to do. So, I was spending a lot of my time on some product and technology things that I was passionate about, thinking I'm the boss. I should focus on what I want to focus on instead of thinking about, okay, I'm running the mobile and platform teams at Facebook. What's the most important thing to do today to make our mobile and developer platform successful?

**Bret Taylor** (00:18:03):
And when I reframed the job that way, I did different things. And the thing that was the biggest pleasant surprise to me was I liked it. I thought I liked engineering and product, but in fact, when I changed an organization and it turned out to be more successful, I derived a great deal of joy from seeing that success. Our developer platform had a lot of partners and when there was an issue there and I'd spend time on partnerships and it worked and our platform became healthier, the partner became more successful. I took pride in that success and then I just started being better at my job and I realized that the actual act of engineering or product design or all the things I thought I liked, what I really liked is impact.

**Bret Taylor** (00:18:49):
And so, that conversation led to my waking up every morning, sometimes literally, but certainly, in the broadest sets of the words saying, "What is the most impactful thing I can do today?" And really thinking almost like, if you had an external board of advisors telling you what are the things where if you focus on them, you can maximize the likelihood that what you're trying to achieve will happen? Sometimes, it's recruiting, sometimes, it's product, sometimes, it's engineering, sometimes, it's sales. And I've become much more self-reflective just about what is important to work on. And I have become much more receptive to doing things that I previously would've said aren't my favorite things to do because I derive so much joy from having an impact that I enjoy a lot more things now. And so, I really credit Sheryl, I'm so grateful. And actually, it's interesting, I think a lot about this when I give feedback to people now, just those moments that can change the trajectory of your career. I give her all the credit for it.

**Lenny Rachitsky** (00:19:53):
There's so many people that share stories of Sheryl Sandberg giving them advice and that changing their life. What a mensch.

**Bret Taylor** (00:20:01):
Yeah.

**Lenny Rachitsky** (00:20:01):
My biggest takeaway from this, which is this question of what is the most impactful thing I could do today? Such a powerful heuristic just to keep in mind. To your point, you may realize you don't want to be doing sales or hiring, but if that's the most impactful thing and you end up doing it, you may realize I like this and I'm good at this and have thought about-

**Bret Taylor** (00:20:18):
Can I double click on that though for a second?

**Lenny Rachitsky** (00:20:19):
Absolutely.

**Bret Taylor** (00:20:20):
I think it's really hard. One of the dangers for founders and product managers, but I think particularly for founders is incorrect storytelling. People don't like my product because of X. And if you tell that to yourself and you tell it to your team, all of a sudden, it goes from being an intuition to being a fact. Well, you better hope you're right because if you orient your strategy around fixing a problem and you're wrong, your company's going to fail. So why did you lose a deal? You could talk to the salesperson who is on the account or perhaps maybe a product manager was involved in the conversation. It's very important to have intellectual honesty in those moments because you could say something like, "Oh, they didn't buy it because the platform cost too much." And that's something a salesperson might say.

**Bret Taylor** (00:21:16):
Maybe the real reason is they didn't actually see much value in your platform. So it was communicated to the salesperson as it was too expensive. But in fact, the problem was product differentiation. And you could end up going into a discussion on pricing when in fact, there was a much deeper, much harder problem to solve there. But just like when you break up with someone, you don't say, it's because I don't like you anymore. You say it's not you, it's me. You say all these pleasantries because we're all social animals and you want to be pleasant with the people around you. So literally taking what a customer says or what a user says in a focus group or a usability study is rarely correct. It often is related to what the truth is, but it's very important to get right. And so, I think one of the things I've observed with first time founders in particular is you're often a single issue voter based on your skillset.

**Bret Taylor** (00:22:14):
So if you're a great engineer, the answer to almost every problem in your business is engineering. If you're a product designer, the answer almost to the proverbial redesign, I joke, it's like the dead cap balance of a consumer product like this next redesign will fix all of our problems. I don't know if it's ever, ever worked. And then I met a lot of entrepreneurs who come from a business development background, they're always thinking about partnerships and oh, if we just get this partnership done for this distribution channel, everything's going to change. And I think it's really important when you're a founder to be self-aware that you will naturally, subconsciously pick the thing that is your strength, your superpower as a solution to more problems. And in fact, if you think that's a solution to your problem, it may be right, but you probably by default should question it.

**Bret Taylor** (00:23:03):
If you think the thing that you've been doing your whole career is the way to fix your problem, it's at least 30% likely that you've chosen that because of comfort and familiarity not truth. And so, I think one of the skills I think is, it really goes around to do you have a good co-founder? Do you have a good leadership team? If you're a product manager, you're a partner in engineering, you're a partner in marketing, you really want to have very real conversations to ensure that you're actually working on the actual correct thing. And I think it's easy to say what's the most impactful thing to do today? My guess, if a lot of people try that, they'll lie to themselves more often than not. And it's a very challenging question to answer. The question's interesting. Being able to answer it accurately is actually the hard part.

**Lenny Rachitsky** (00:23:50):
This feels like such an important lesson you've learned. Is there an example that comes to mind where you learned this the hard way where you actually ended up-

**Bret Taylor** (00:23:57):
Oh, yeah. You just want to spend this whole thing on my failures, but I'm fine with that.

**Lenny Rachitsky** (00:24:02):
You've had too much success.

**Bret Taylor** (00:24:04):
Frontier was my first company. At our peak we had 12 employees, 12 of the best people I've ever worked with. Started the company with Jim Norris, who's an engineer I've known since Stanford and Paul Buhite and Sanjeev Singh who Paul started Gmail. Sanjeev was the first engineer at Gmail, so we had the Google Maps people and Gmail people. It was a pretty awesome founding team. We made a social network, as you said. We invented a lot of concepts that became popular in the newsfeed. We invented the like button. It was really neat. It was a fun time. We were only really popular in Turkey, Italy and Iran, and at one point, we were blocked in Iran, so we were only popular in Turkey and Italy and Silicon Valley. To this day, actually a lot of folks in Silicon Valley are like, "I love FriendFeed." I'm like, that's awesome.

**Bret Taylor** (00:24:51):
It wasn't really a successful business. We were a follower-oriented social network, not a friendship-oriented social network, which meant a lot of our content was more like X or Twitter than it is Facebook in that respect. And a lot of sharing newspaper articles, interests, scientific communities, things like that. And there was a period when Twitter, which was one of our competitors at the time, that there was a lot more social networks at the time. I am probably screwing this up a little bit. I think Obama, Ashton Kutcher and Oprah Winfrey all went on Twitter in a summer and we just got our ass kicked.

**Bret Taylor** (00:25:31):
And it was a great example of you... I think 11 of those 12 people were engineers and we were just making product and I think it was Biz Stone. If you talk to the Twitter folks, they could give you the history on this, but I think Biz was really focused on getting celebrities and public figures onto Twitter, which is totally obvious. If you have a social service that's oriented towards following people, put some people on there worth following and instead, we were exclusively focused on polishing the product.

**Bret Taylor** (00:26:00):
And we actually, I think at our peak of popularity, we were very confident just, I think it was a time when Twitter had the fail whale and it was down half the time and people couldn't even use it. And our product, we were innovating faster, we had more features, people liked it and we were up 100% of the time and we totally lost for no reason related to product at all. And it was an example of, I think somewhat famously not a lot of great entrepreneurs have come out of Google because Google was so successful, I think it's hard as a product manager to see distribution and product design and even business model when you have AdWords and money's raining from the sky. There wasn't as much scrutiny and I think folks like the PayPal Mafia I think learned a lot more about entrepreneurialism than a typical PM at Google. So, we're just getting punched in the face and learning this the hard way.

And so, that was probably the most prominent example of it and I think we probably did have a... I can tell you all the flaws of that product, but I don't think that was the reason why we lost. There's a lot of reasons. I think there was a lot of flaws of the product, but it was a lot of other stuff. And so, I've learned, accumulated these skills over time, but I say the hard part of that question is answering it correctly, is it's hard when you don't have experience in something, than to have intuition in it. So I think if there's probably a structural flaw, it wasn't that... I don't know if I could have figured out how to reach out to Ashton Kutcher [inaudible 00:27:28], it's not he's on my Rolodex. But I probably wasn't soliciting advice from the right people.

I think that what's great about the technology industry is there's a lot of advice. Choosing whom you listen to is actually quite difficult, but I think we're somewhat myopic. We're in our own little world, creating this product and we weren't asking people from the outside in to say, what are you seeing that could go wrong? What are you seeing that could go right? What are you seeing in the industry that we're not doing that you think we might want to do? And this is why boards are important. This is why finding the right advisors, the advisors will actually tell you what you not [inaudible 00:28:09] want to hear, but you need to hear. I think that was probably the missing part. I'm not sure I was great at market at the time, but if I had solicited the right advice, I could have learned that that was a shortcoming. And I think that was a deep lesson I took from that. I'm a huge believer in boards and getting good advice.

**Lenny Rachitsky** (00:28:26):
Any heuristics or advice for people to know whose advice to listen to? What do you pay attention to when you're like, okay, ignore this person but listen to this person?

**Bret Taylor** (00:28:35):
Yeah, that one's tough. It does come down to good judgment and being judge of people's character. One thing that is particularly hard is there's not a strong correlation between the confidence with which someone expresses an opinion and the quality of that opinion. I don't want to say it's inversely correlated, but that's funny, with all the podcasts out now, if there's topics I know a lot about, sometimes the most eloquent, confident statements about things I know a lot about, are the least accurate and it sounds extremely persuasive. And so, it does require very good judgment. One thing is I think, not just asking for advice but asking people, who should I talk to get good advice? And you'll find some common answers there and that's often a really strong signal of good judgment.

And then one thing I found is when you ask for advice, don't just ask what to do but why. Be an obnoxious two-year-old kid, why? Why? Why? Why? And really try to understand the framework that someone is using to give you advice. The interesting thing about advice is people are often extrapolating from relatively few experiences. So they will say, never do this or always do that. And it's because they had one experience where something backfired or something could have gone better if they had done it. So it's a useful anecdote, but if you don't ask why and understand they had one experience and here's what happened, it can come across as a rule when in fact, it's [inaudible 00:30:08] data and if you ask advice of three people and they all have very similar interactions, you can create a first principles framework from which that advice emerges. And when you start applying it, you're applying it with a degree of nuance that you couldn't if you're just following a rule. So, I think one is, it does come down to good judgment, I think. I don't know how to teach that.

**Bret Taylor** (00:30:30):
I'm a huge believer in good judgment. It's one of the things I hire for. I just think that's something that probably comes from a mix of self perfection. You really need to hold yourselves accountable as an entrepreneur, as a product manager. If you made a bad decision, spend time reflecting on it, number one. And really, try to understand why and try to always improve your judgment. I think at the end of the day, that is why you are a good entrepreneur, a good product manager. And number two, when you get advice, really understand where it's coming from and why so that you can create your own independent view of where that advice came from and recognize that no one's advice is statistically significant or very rarely is it. If you're getting advice on investing from Warren Buffett, yeah, okay, it's statistically significant, but most advice is like something happened to you once and you have regrets.

**Lenny Rachitsky** (00:31:28):
I love that you're like, I don't know if I have a great answer then you just give us an incredible answer to this question. I want to go in a different direction. You mentioned that you describe yourself as an engineer. I know I heard you code to relax still. Let me just ask you this question, something a lot of people in college are thinking about. Do you think it still makes sense to learn to code? Do you think this will significantly change in the next few years?

**Bret Taylor** (00:31:47):
I do still think studying computer science is a different answer than learning to code, but I would say I still think it's extremely valuable to study computer science. I say that because I think computer science is more than coding. If you understand things like Big O notation or complexity theory or study algorithms and why a randomized algorithm works and why two algorithms with the same Big O complexity, one can then practice perform better than others and why a cache miss matters and just all these little... There's a lot more to coding than writing the code. The reason I think that is I do think the act of creating software is going to transform from typing into a terminal or typing into Visual Studio code to operating a code-generating machine. I think that is the future of creating software. But I think operating a code-generating machine requires systems thinking and I think that computer science, there are other disciplines as well, but computer science is a wonderful major to learn systems thinking and at the end of the day, AI will facilitate creating this software.

**Bret Taylor** (00:33:08):
We may do a lot more in the next few years we can't even imagine, but your job as the operator of that code-generating machine is to make a product or to solve a problem and you really need to have great systems thinking and you're going to be managing this machine that's doing a lot of the tedious work of making the button or connecting to the network. But as you're thinking of the intersection of a technology and a business problem, you're trying to affect a system that will solve that problem at scale for your customers and that systems thinking is always the hardest part of creating products.

**Bret Taylor** (00:33:40):
I'll just give you, it's this cheesy simple example, but I think it's representative. At Facebook, we spent a lot of time designing the newsfeed and if you ever had a really, really good designer and they showed you at the time, a Photoshop mock-up of the newsfeed, it was just all as beautiful. The photos, the family was happy and the photo was a perfect photo and the posts were all perfectly grammatically correct and of a completely normal length and the comments and there was the like... Everything was just perfect. And then you'd implement that design and you'd look at your own newsfeed and it looked like shit because it turns out not everyone's photos were made by a professional photographer. The posts were all these different lengths. The comments were like, you suck and... All that stuff.

**Bret Taylor** (00:34:29):
And then all of a sudden you realize that designing a newsfeed, Photoshop is the easy part. You need to actually design a system that produces both in content and visual design, like a delightful experience given input you don't control. And that's a system, it's sort of a design and it's just, what we did practically, I am sure it's changed a lot since I left in 2012, but we made a system so designers had to show their newsfeed designs with real newsfeed data that was messy rather than anything artificial because I think it forced the process to be more realistic.

But I say that because I think that whether AI is writing code or doing the design or doing all these other things, you need to learn how to have a system in your head. You need to understand the basics of what's hard and what's easy and what's possible and what's impossible. And AI can help you do that too, by the way. But I do think that's a really useful skill. I think in general, with the advent of AI agents and AI approaching super intelligence in certain domains, I think the tools with which we do our job will change a lot. I think it's very important to have a very loose attachment to the way we do our jobs and that story that we won't talk about when I rewrote Google Maps, everyone talks about that story and I think it's because of Paul [inaudible 00:35:57] who told it on some podcasts and that's where it made the rounds.

**Bret Taylor** (00:36:01):
I think that's going to end up this vestige of the past, almost like the human calculators at NASA before the computers were invented, like wow, a person was a calculator? Whoa, that's fun. Tell me that story. I think just what I was good at will no longer be useful in the future or certainly not valuable in the future and that's okay. So I think we need to have a really loose view of it, but the idea that you shouldn't study these disciplines, it's like people say, I don't want to study math because I'm not going to use it in my career for X. Well, studying maths is quite important. It teaches you how to think. It teaches you how the world works, physics, math, and I think computer science especially, at least the foundations of it, will continue to be the foundations of how we build software and understanding that when you're interacting particularly with something that's smarter than you, producing code you might not completely understand how you constrain it and how you get it to produce these outcomes. I think it will require a lot of sophistication actually.

**Lenny Rachitsky** (00:36:59):
That's such a great answer. There's this always sense of this binary, should I learn to code or not? And your point here is learn to understand how engineering works and how systems work and what your code does and how it all interconnects, but the way you actually do the coding at your desk will change significantly. This reminds me of something you mentioned on a podcast recently. This idea that you think there's going to, or there should be a new programming language that is more designed for LLMs versus humans. Can you just talk about that because I think a lot of people aren't thinking about that?

**Bret Taylor** (00:37:27):
I don't know if it's a language, I would call it a programming system because I think language might be too limited. My reductive version of the past, what 40 years of computers maybe more, is we created the hardware for computers, then we created punch cards, which is the way in the late '70s you would tell a computer what to do or maybe mid to late '70s. Then we invented early operating systems and time-sharing systems from the invention of things like Unix at Bell Labs and Berkeley, you ended up with the C programming language, Fortran and a lot of higher level programming languages. I think Fortran and then C.

**Bret Taylor** (00:38:15):
And we moved up the layers of abstraction. No one does punch cards anymore, obviously. Few people write assembly language. Some people write C, some people write REST. But a lot of people write Python and TypeScript and things like that. And as we've invented more and more abstractions, we've made it easier to do high-leverage things. So, if you look at how remarkable Google was back in the day or Google Maps, you could probably give a lot of react programmers the task of make a draggable map now and I think a lot of people could do it. That was true RND back in the day.

**Bret Taylor** (00:38:54):
When Salesforce was created in 1998, just putting a database in the cloud was hard and that alone was a technical moat that is now trivial with Amazon Web Services and that technical moat is comically narrow, but the product moat is quite large. I think that if the act of writing code is going from something that is very costly to the marginal cost of that going to zero, how many of the abstractions that we've built are based on human program or productivity? I think a ton. I always laugh that I assume Python is probably the most common generated code just because of how much it's in the training data and data scientists love Python and I love Python too. It's such a comically bad thing for AI to generate just because it's one of the most inefficient programming languages of all time. If you know the global interpreter lock and just slow. And I've written a lot of high-scale web services and it's just quite slow and it's very hard to verify.

**Bret Taylor** (00:40:00):
It's not as bad as Perl, but if you have a big Python program, how many errors will you find at runtime versus before releasing it? So, Python was designed to be very ergonomic, almost looked like pseudo code for humans, for me to write code in a delightful way. That's why data scientists love it so much. So as we move to a world where let's just postulate, and I'm not sure this will be completely true, that we're not going to write a lot of code as people. We're going to be operating these code-generating machines. We probably don't care how ergonomic the programming language is. What we care about is when this machine generates code, do we know that it did what we wanted it to do? And if it doesn't do we want it to do, can we change it easily? I think there's a lot of insights in programming languages that could serve this.

**Bret Taylor** (00:40:48):
So Rust I think, is interesting because if I asked you to look at a C program and say, "Does it leak memory?" You probably couldn't do it that well just because it's really hard and if it's a very, a million line C program, it's going to be very, very hard. If I asked you to verify that a Rust program doesn't leak memory, you would just have to compile it. And because it has compile time, memory safety, just the act of compiling successfully tells you that's true. I think we need more things like that because if an AI is generating this code, by definition, if you have to read every line that is going to be the limiting factor for producing the code or worse, you're just not going to read every line and you're going to emit a bunch of unsafe unverified code into the wild.

**Bret Taylor** (00:41:36):
And so, the question is how do you enable humans to have as much leverage as possible? Which means using computers to do the work on your behalf. You could have obviously the simplest form of this, is AI supervising AI and doing code reviews and that's great. Certainly, self-reflection is a really effective way of improving the robustness of an AI system. But I do think if it doesn't matter how tedious it is to write the code, you could probably layer on some techniques that are out of fashion, like formal verification, unit testing, other things. And if you layer all these on, I'm thinking about it as, I as a... It's like the guy in the matrix with the green letters coming down, how can I make something so I as a operator of the code generating machine can produce incredibly complex scale software, incredibly quickly and know that it works?

**Bret Taylor** (00:42:26):
If you start with that as your design center, I think you'd probably changed the languages, you'd probably changed the systems, you'd probably change all these things and you're probably going to bring to bear a lot of things. And what's really fun about it is you can loosen a lot of constraints, like coding is free. Okay, so that's neat. With that in mind, what do you want to do? What would be best suited for the language, the compiler for testing, for self-reflection, for supervisor models, all these things. I think that's more of a programming system than a language, but I think when we create something like that, it can really enable creators, builders to create incredibly robust, incredibly complex systems.

**Bret Taylor** (00:43:04):
And I'm super excited about VibeCoding, but I don't know generating a prototype has been the limiting factor in software ever. It's actually building increasingly complex systems and actually changing them with agility. If you look at the famous Netscape one to Netscape two rewrite, somewhat, a lot of people attribute that to part of their failure against Internet Explorer. It's like making these things is not hard. Maintaining them is hard and ensuring they're robust is hard. And I think we're in the very early phases of defining what this new system for developing software looks like and I'm very excited to see what emerges.

**Lenny Rachitsky** (00:43:42):
I feel like we're definitely living in the future when someone like you is suggesting that we build a matrix like experience and that's going to be potentially the future of coding and building. I can't wait for that. It feels like a great opportunity and a fun project.

**Christina Cacioppo** (00:44:06):
Great to be here. Big fan of the podcast and the newsletter.

**Lenny Rachitsky** (00:44:09):
Vanta is a long time sponsor of the show, but for some of our newer listeners, what does Vanta do and who is it for?

**Christina Cacioppo** (00:44:16):
Sure. So we started Vanta in 2018 focused on founders helping them start to build out their security programs and get credit for all of that hard security work with compliance certifications like SOC 2 or ISO 2701. Today, we currently help over 9,000 companies including some start-up household names like Atlassian Ramp and LangChain, start and scale their security programs and ultimately, build trust by automating compliance, centralizing GRC, and accelerating security reviews.

**Lenny Rachitsky** (00:44:46):
That is awesome. I know from experience that these things take a lot of time and a lot of resources and nobody wants to spend time doing this.

**Christina Cacioppo** (00:44:54):
That is very much our experience, but before the company and to some extent, during it. But the idea is with automation, with AI, with software, we are helping customers build trust with prospects and customers in an efficient way. And our joke, we started this compliance company so you don't have to.

**Lenny Rachitsky** (00:45:10):
We appreciate you for doing that. And you have a special discount for listeners. They can get $1,000 off Vanta at vanta.com/lenny. That's V-A-N-T-A .com/lenny for $1,000 off Vanta. Thanks for that, Christina.

**Christina Cacioppo** (00:45:25):
Thank you.

**Lenny Rachitsky** (00:45:26):
Okay. One more question along these lines and then I want to zoom out on just where AI is heading and something I love to ask folks like you that are at the cutting edge of AI is what you're teaching your kids. I know you have kids, I feel like the world is going to be very different when they grow up. What are you encouraging them to learn that you think is different maybe from previous generations to help them be successful in a world of AI abundance?

**Bret Taylor** (00:45:53):
I don't know if I'm teaching them differently, but I'm really trying to encourage them to make AI a part of their lives. I was reflecting actually when I took the AP calculus exams in '97, '98 AB and BC, I could use a graphing calculator. And I haven't done this research I was meaning to plug this into ChatGPT before our conversation, but I'll do it after. Did the calculus exam change before and after they allowed the calculator in the exam? I assume it did. But essentially, to when you allow the calculator in the exam, you need to make sure that none of the questions benefit people for having a calculator or not, which actually forces you to rethink the problems to test calculus knowledge that don't benefit from like road arithmetic or the other things you can do on a graphing calculator.

**Bret Taylor** (00:46:47):
I think that a lot of education doesn't presume you have a super intelligence in your pocket. And so, if you ask someone to write an essay on a book that they read, you could probably hallucinate one pretty easily from one of the big providers like ChatGPT. And maybe if you are skilled enough that prompting, maybe even your teacher won't know it's written by an AI. So what do you do? How do you teach kids differently? It's really hard for teachers right now because I think we haven't gone through the transition of adding calculators to the exams. So, I think a lot of the mechanisms we have to evaluate students are broken by the existence of ChatGPT and the like. So I think we're in a very awkward phase, but I think we can still both teach kids how to think and teach kids how to learn. And I think our education system can catch up and I actually think these models can be one of the most effective educational tools in history.

**Bret Taylor** (00:47:46):
I don't know if you're a visual learner or reading learner. I like to read. I didn't love going to lectures. I don't learn that well from them. I like to read the book and if you have a teacher who doesn't teach in your style, you can now go home and ask ChatGPT to teach you in another mechanism. My kids use ChatGPT to quiz them before a test. You can use audio mode or chat mode. It's better than cue cards. My daughter took home a Shakespeare book, she took a picture of a page she didn't understand and ChatGPT explained it to her way better than I would've as well. I think every child in this world has a personalized tutor that can teach them in the way that they best learn, visually, over audio, reading. We have a platform that can test you, that can quiz you. I think it's really an amplifier of agency.

**Bret Taylor** (00:48:39):
I think the kids who have agency, who have aspirations to learn something, I think you have what is the best combination of every teacher you've ever had and these models and you can use it. So with my kids, my oldest daughter learned how to code and she was making a website and every time she had a question for me, I would just make her use ChatGPT. Not because I was trying to be an obnoxious father, but I'm like she needs to learn to use this tool because it's amazing.

**Bret Taylor** (00:49:13):
So, I really am trying to have them learn how to use it constructively in their lives. But all that said, I just feel a ton of empathy for public school teachers right now. It's very hard because the technology is moving faster than our educational system. And I think particularly as it relates to evaluation, it's just really challenging for teachers right now. And I worry because these technologies amplify agency, the opposite can also be true. If you are a student trying to not learn something, I think these tools probably provide a lot of mechanisms to avoid it as well. And so, I think there's a challenge for parents and teachers and I think we're going to end up with a handful of years here.

**Bret Taylor** (00:49:53):
But I brought up the calculus AP exam because obviously, a graphing calculator is not ChatGPT, don't get me wrong. But I think we've been able to figure out a way to conform homework and in-class learning and tests around the technologies available to us fairly successfully to date. And I'm fairly confident we'll figure it out and I think it's going to... And on the much more positive side, I don't know, I went to public schools, I don't know if you did too. You end up with some pretty bad teachers at times and now, you have an outlet. You don't need to be the rich kid who can afford a tutor anymore to get tutoring. If you are a kid who excels in math and your school doesn't have advanced statistics classes, well, now you do.

**Bret Taylor** (00:50:40):
So I think this is just an incredibly democratizing force with kids who have agency and I think that's very exciting. I'm hopeful that there's a 11-year-old right now who's going to start a really amazing company 10 years from now whose ChatGPT is going to be their primary tutor that led to that outcome. And I think that's pretty cool.

**Lenny Rachitsky** (00:51:02):
I have a two-year-old and it feels like there's a new milestone of, there's like when to give them a phone, when to give them, I don't know, Snapchat, whatever kids use these days and then it's like when to give them their first ChatGPT account. Well no, I wonder how soon that's supposed to happen.

**Bret Taylor** (00:51:16):
I think my personal take is, it's different than the former two. I don't think mobile phones are great in school or great for kids and I personally advocate for waiting a long time. But I think that ChatGPT is more like Google search and it's one thing to have a device in your pocket that's addictive and has push notifications, but it's another thing to use AI to learn. And so, I think the two are different. And I really think of AI fundamentally as a utility. I don't think a lot of parents before ChatGPT said, "When should I let my kid use Google search?" That's a different type of tool. I think thinking of it like that is the way I think about these technologies.

**Lenny Rachitsky** (00:51:55):
Is the form factor for your kids like an iPad or a laptop or something?

**Bret Taylor** (00:51:58):
Yeah. They use the computer on the desk.

**Lenny Rachitsky** (00:52:01):
Got it. All right. Good tips. This is good for me to learn all these things as my kid ages. Okay, I'm going to zoom out and let's talk about business strategy AI. One of the biggest questions a lot of founders think about these days is just where should I build? What will foundational model companies not squash and do themselves? Being someone building a very successful AI business and also being on the board of OpenAI, feel like you have a really unique perspective on what is probably a good idea and it's probably not a good idea. How do you think the AI market is going to play out and where do you think founders should focus and also just try to avoid?

**Bret Taylor** (00:52:36):
I think there's three segments of the AI market that will end up fairly meaningful markets and then I'll end with how I think it's going to play out. So first is the frontier model market or foundation model market. I think this will end up the small handful of hyperscalers and really big labs just like the cloud infrastructure as a service market. And the reason for that is that creating a frontier model is entirely a function of CapEx. And you need a company with huge amounts of CapEx capacity to build one of these models. All of the companies that were startups that tried to do this have already been consolidated or almost all of them, inflection, adapt, character and others. And I think there doesn't appear to be a viable business model for a startup because of the amount of CapEx required and there's just not enough fundraising runway to get to escape velocity. And also, the models deteriorate in value fairly quickly as an asset class.

**Bret Taylor** (00:53:33):
And so, you need just a lot of scale to make a return on the investment for a model that deteriorates in value so quickly. So, I think that's going to end up probably no entrepreneur should build a frontier model. That's my take.

**Lenny Rachitsky** (00:53:47):
Unless you're Elon.

**Bret Taylor** (00:53:51):
Yeah. Oh, yeah. He's different. And he has the capacity to raise billions in capital and my guess is most of your other listeners don't, and then he is the greatest of all time for reason and he's different. You don't compare yourself to him. The other part of the market is the tooling, and I think there's a lot of folks selling pickaxes in the Gold Rush. This is data labeling, services. This is data platforms, it's eval tools. More specialized models like 11 Labs has a great set of voice models that a lot of companies use that are really high quality and it's like if you're trying to be successful in AI, what are the different tools and services that you need?

**Bret Taylor** (00:54:31):
There is some risk to the tooling market because it's pretty close to the sun. So, if you look at the infrastructure as a service market and the cloud tooling market like the Confluent and Databricks and Snowflake, a lot of the Amazon and Azure and others have competing products in those areas because they're very adjacent to the infrastructure itself and every infrastructure provider is trying to differentiate by moving up the stack and you're right there.

**Bret Taylor** (00:54:57):
So, there's some real meaningful companies as I mentioned, like Snowflake, Databricks, Confluent and others, but there's a lot of others that were obviated by technology from the infrastructure providers themselves. So those companies probably are the most at risk for a developer day from one of these big foundation model companies releasing exactly what they do. So there's probably a lot of people who need your tool, but the question will be if or when is probably the right way to think about it, one of these large infrastructure providers introduces a competitor, why will people continue to choose you? So it's a good market but it's a little bit close to the end as I said.

**Bret Taylor** (00:55:38):
Then there's the applied AI market. I think this will play out for companies who build agents. I think agent is the new app. I think that's going to be the product form factor. There's companies like Sierra, we help companies build agents to answer the phone or answer the chat for customer experience and customer service. There's companies like Harvey that make agents for both a legal, paralegal profession, anti-trust reviews, reviewing contracts etc, etc. There's companies that do content marketing. There's companies that do supply chain analysis. I think this is like the software as a service market. They'll probably be higher margin companies because you're selling something that achieves a business outcome as opposed to being a byproduct of the models themselves. They will almost certainly pay taxes down to the model providers, which is why those model providers will end up extremely large scale but probably slightly lower margin and I think the market for them will be probably less technical.

If you think about the purest form of software as a service, it's not like you ask what database do you use? It's really about the feature and function. I think that's where agents will go. I think it's going to be more about product than it is about technology over time. Just going back to my metaphor, in 1998 when Mark and Parker started Salesforce, just getting that database running in the cloud was like a technical achievement. Nowadays, no one asks about that because you can just spin up a database in AWS or Azure and it's like no problem. I think today, orchestrating an agentic process on top of the models, sounds really fancy and it's really hard and all that stuff. I'm pretty sure that's going to be easy in three or four years. It's just like just as the technology improves. And so, over time you say, what is an agent company? Well, it looks a little bit like [inaudible 00:57:30] as a service.

**Bret Taylor** (00:57:30):
You talk a little bit less about how you deal with the models in the same way modern SaaS, few people ask what database you use, but you'll probably ask a lot about the workflows and what business outcomes that you're driving. Are you generating leads for a sales team? Are you minimizing your procurement spend? Whatever value you're providing, it's going to slowly evolve towards that.

**Bret Taylor** (00:57:53):
I'm very excited. I don't think startups should probably build foundation models. But you can shoot your shot if you have a vision for the future, go for it. But I think it's probably a challenging market that's already consolidated. I'm very excited about the other two markets. I'm particularly excited as building agents becomes easier, to see a lot of long tail agent companies come out. I was looking at a website for the top 50 software companies in the stock market and obviously, the top five are the big, big boom ones like Microsoft, Amazon, Google, all that, but the next 50 are all SaaS companies and some of them are very exciting, some of them are super boring, but this is how the software market has evolved and I think we're going to see something similar with agents.

**Bret Taylor** (00:58:38):
It's not just going to be these huge markets like we're in customer service and software engineering. It's going to be a lot of things where people are spending a lot of time and resources that an agent can just solve, but it requires an entrepreneur who actually understands that business problem, and deeply, and I think that's where a lot of the value is going to be unlocked in the AI market.

**Lenny Rachitsky** (00:59:00):
That is incredibly helpful. This makes me think about, I had Marc Benioff on the podcast, you guys were co-CEOs and he was extremely agent-pilled. All he wanted to talk about was Agentforce. Clearly you're also very agent-pilled. What is it that-

**Bret Taylor** (00:59:14):
I've never heard the term agent-pilled [inaudible 00:59:18].

**Lenny Rachitsky** (00:59:18):
Clearly you guys saw something that was just like, okay, we need to go all in on agents. This is the future. What is it you think people are missing about just why this is such a critical change in the way software is going to work? What are people not seeing?

**Bret Taylor** (00:59:31):
If you talk to an economist like Larry Summers who, on the OpenAI board with me, they'll talk about what is the value of technology? Will it help strive productivity in the economy. And if you look at one of the big jumps in productivity in the economy was in the '90s, and I think a lot of folks I talked to think it was actually that very first wave of computing where people made ERP systems and just put accounting into computers and databases, even mainframes, we're talking like the PC era. Because it was such a huge step-up, just imagine the ledgers of numbers that you'd have for a large multinational company before and it truly just transformed departments.

**Bret Taylor** (01:00:14):
I'll give you a little toy example. My dad just retired. He was a mechanical engineer and he was talking about when he first started his career in the late '70s and he went into a mechanical engineering firm, the majority of the firm were drafts people. So basically, you take an engineering design but you needed to do all the different vantage points and for all the different floors and to give to the contractor to do the thing. Now, there are zero drafts people at his company. You just make the design in first AutoCAD and now Revit and it's a 3D model and the drafting has actually been eliminated. It's just not a thing one needs to do anymore. The actual design and drafting, drafting is not a thing that exists. It's just a design. That's true productivity gains, right? It's like the job of the mechanical engineering firm was to do a design. The drafting was this necessary output for the contractor, but it wasn't really adding value. It was just like the supply chain change.

**Bret Taylor** (01:01:12):
If you look at the history of the software industry from the PC on, there's been meaningful productivity gains but just not nearly as meaningful as that first huge jump. And I'm not smart enough to know exactly why, but it is interesting, the promise of productivity gains from technology hasn't been as realized I think as some people thought. I think agents will truly start to bend the curve again like we did in the very early days of computing because software is going from helping an individual be slightly more productive to actually accomplishing a job autonomously. And as a consequence, just like you don't need drafts people in a mechanical engineering firm, you just won't need someone doing that thing anymore. It means they can do something else that's higher leverage and more productive and you can actually... A smaller group of people can accomplish more and truly drive productivity gains in the economy.

**Bret Taylor** (01:02:15):
And I think if you've ever sold enterprise software, you end up in these discussions as a vendor with the customer where you'll have a value discussion and you'll do these somewhat convoluted things like okay, it's like you're selling a sales thing. Okay, well, if every salesperson sells 5% more... And you should pay us a million dollars. And it's roughly that conversation and it's so unattributable especially... And it's why it's so hard to sell productivity software, which I learned the hard way, it's just hard to know what's the value of making everyone 10% more productive? Did you actually make them 10% more productive or did something else change? You don't really know all these things. But now with an agent actually accomplishing a job, not only is it actually truly driving productivity in a very real way, but it's measurable as well.

**Bret Taylor** (01:03:10):
So all those things combined means I think this is actually a step change in how we think about software because it does a job autonomously, which is more self-evident, a productivity driver. It's measurable so people value it differently as well, which is why I also believe in outcomes-based pricing for software.

**Bret Taylor** (01:03:32):
And all of that combined to me, it feels like as significant as the cloud or I think more technologically, but just in terms of how it transforms the business model of the software industry where there's going to be a before and after. I don't know how many people still sell perpetually licensed on premises software, but it's de minimis at this point. I think we're going to go through a similar transition. The whole market is going to go towards agents. I think the whole market is going to go towards outcomes-based pricing, not because it's the only way, but the market is going to pull everyone there because it's just so obviously the correct way to build and sell software.

**Lenny Rachitsky** (01:04:08):
Let me pull on that last thread. So we had Madhavan on the podcast recently, pricing expert, legend, monetizing innovation author and he talked about pricing strategy for AI companies and he was very much in your camp of, if you can, you need to price your product as an outcome-based product and the access uses exactly what you shared, which is, you can do that if you can attribute the impact and it's autonomous, it's running on its own. And he actually used Sierra as one of the shining examples of this being successful. Can you just briefly just explain a little bit what is outcome-based pricing for people that haven't heard this term before and then just how does it work for Sierra to give an example?

**Bret Taylor** (01:04:45):
Yeah, I'll start with the example and then I'll broaden it. So at Sierra, we help companies make customer facing AI agents primarily for customer service, but more broadly, for customer experience. So if you have a problem with your [inaudible 01:04:58], you'll call or chat with Harmony, who's their AI agent. If you have ADT home security and your alarm doesn't work, you can chat with their AI agent. Sonos speakers, a lot of different consumer brands. And if you think about running a call center, there's a cost for every phone call that you take. Most of it is labor costs, but if you have, let's just say a typical phone call is anywhere between 10 and $20 USD. Some of it's software, some of it's telephony, but a lot of it is just like the hourly wage of the person answering the phone.

**Bret Taylor** (01:05:32):
So if an AI agent can take that call and solve it, that is in the industry often called a call deflection or a containment. And that essentially means you saved, call it $15 because you didn't have to have someone pick up the phone. So in our industry, basically we say, "Hey, if the AI agent solves the customer's problem, they're happy with it and you didn't have to pick up the phone," there's a pre-negotiated rate for that and we call it resolution based. There are other outcomes as well. We have some sales agents being paid a sales commission, believe it or not. We do. We really think of our agents as truly customer experience like the concierge for your brand and we want to make sure that our business model is aligned with our customer's business model.

**Bret Taylor** (01:06:22):
As you said, these agents need to be autonomous and the outcome has to be measurable. That's not always possible, but I think it's broadly possible. And what's really neat about it is if you talk to any CFO or head of procurement with their big vendors, they look at the bill of materials and it's overwhelming and it's impossible to know if you're getting the value that you hoped from that contract. I think consumption based, which was popular particularly in the infrastructure space is closer to it. But I'm not sure a token is actually a good measure of value from AI either. I always use the analogy, like right now, most of the coding agents are priced per token or per utilization, but there's this famous story of an Apple engineer who had a bad manager who's like had you report how many lines of code you wrote every day? Which every engineer in the world knows is an idiotic way to measure productivity.

**Bret Taylor** (01:07:16):
He famously went in with a report that had a negative number because I think he did a big refactoring, deleted a bunch and it was his way of saying like, fuck you to the man. I think tokens are similar. Yeah, you used a lot of tokens, like good for you, did it produce a pull request that was good? And I think that's the whole point of all this. I think there's a huge difference between outcomes-based pricing and usage based pricing because especially in AI, they're not necessarily even correlated and you could have a long phone call and not solve the customer's problem and they give you a negative review online and call the call center again, all that effort was for nothing. In fact, you might've added negative value. And so, I am a huge believer in this.

**Bret Taylor** (01:07:58):
And what's fun about it is it really just aligns... I think every technology company aspires to be a partner, not a vendor. And I think at Sierra, we are truly a partner to every single one of our customers because we're all aligned on what we want to achieve. And I think that is really where the software industry should go. It requires a lot of different shape of a company. You have to be able to help your customers achieve those outcomes. You can't just throw software at the wall because you'll never get paid if it doesn't. Your orientation becomes so extremely customer-centric when you do this the right way. I think it's just a better version of the software industry. So I think it's right from first principles, it's right for procurement partners and I think it's right for the world.

**Lenny Rachitsky** (01:08:43):
We've been chatting a little bit about productivity gains. There's a lot of skepticism in the headlines these days of just like what is AI actually doing? Is it actually helping people be more productive? There was a recent study actually, I don't know if you saw, where they showed engineers were less productive with AI because it was just putting them in different directions. They had to research all what's going wrong here? So I think CX is a really good example where you clearly are seeing gains. Are you seeing actual gains at your company or any other company you work with outside of CX in terms of productivity that is like clearly yes, this is working and a huge deal?

**Bret Taylor** (01:09:15):
I'm extremely bullish on the productivity gains from AI, but I do think the tools and products right now are somewhat immature and it's quite counterintuitive. So, for example, almost every software engineering firm I know uses something like Cursor to help their software engineers. Most people use Cursor right now as a coding autocomplete, though they have a lot of agentic solutions and there's a lot of... OpenAI has Codex and Cloud has... I can't remember the Anthropic products. So there's lots of agentic agents coming as well. One of the interesting things because the technology is immature, the code it produces often has problems. There's a lot of people approaching this to actually realize those productivity gains because as any engineer who's written a lot of code will tell you, it's pretty easy to look at and edit and fix code you wrote.

**Bret Taylor** (01:10:10):
Reviewing other people's code or particularly finding a subtle logical error in someone else's code is actually really hard. It's actually much harder than editing code that you wrote yourself. So if the code produced by a coding agent is often incorrect, it actually can take a lot of cognitive load and time to fix it. And in fact, if you end up producing lots of issues with your customers, you could end up producing a lot of features, but actually, is like mucking up the machine a little bit and having something that's not ideal. There's a couple of techniques that I think are interesting. First, I think there's a lot of AI starts now working on things like code reviews. I think this idea of self-reflection in agents is really important. Having AI supervise the AI is actually very effective. Just think about it this way, if you produce an AI agent that's right 90% of the time, that's not that great, but how hard would it be to make another AI agent to find the errors the other 10% of the time? That might be a tractable problem.

**Bret Taylor** (01:11:10):
And if that thing's right 90% of the time, just for argument's sake, you can wire those things together and have something that's right 99% of the time. So it's just a math problem and it turns out that you can make something to generate code, you can make something to review code and you're essentially using compute for cognitive capacity and you can layer on more layers of cognition and thinking and reasoning and produce things increasingly robust. So I'm very excited about that. The other thing though is root cause analysis. So we have an engineer at Sierra who exclusively focuses on the model context protocol server serving our cursor instance. And our whole philosophy is, if cursor generated something incorrect, rather than just fixing it, try to root cause it. Try to get it so the next time Cursor will produce the correct code and essentially, is context engineering.

**Bret Taylor** (01:12:05):
What context did Cursor not have that would've been necessary to produce the right outcome? So I think people who are trying to get productivity gains in departments like software engineering need to stop waiting for the models to magically work if they want to see the gains now. And you really have to create root cause analysis and systems and say, how do we go root cause every bad line of code and actually give the right context and produce the right system so the models can do it today? Over time that'll probably be less necessary and you'll have less context engineering necessary to do it, but you really have to think of this as a system and I think people are waiting for the models to just magically get better. And I'm like, well that will happen eventually, but if you want the gains now you got to put in the work. That's essentially why applied AI companies exist.

**Bret Taylor** (01:12:53):
And the work is non-trivial, but you can do it. And so, for customers using platforms like Sierra, yeah, AI agents aren't perfect, but we're creating a system that lets customers create a virtuous cycle of improvement. If you want to go from a 65% automated resolution rate to 75%, we have a billion tools to let AI help you do that, identify opportunities for improvement, figure out why people are frustrated, what new capabilities can we add to our agent to improve the resolution rate? And you let AI put the needles at the top of the haystack on your behalf and I think that's really the way to optimize these systems.

**Lenny Rachitsky** (01:13:28):
I've never heard of this technique of improving Cursor by adding additional context. What's the actual way of doing that? You build an MCP server that everything runs through or is it like you add Cursor rules? What's the actual approach there?

**Bret Taylor** (01:13:41):
I'm probably out of my depth here, but it's essentially MCP because that's how you provide context to Cursor. And I think that almost always when you have a model making a poor decision, if it's a good model, it's lack of context. And so, you really want to find the intersection of your particular product and code base with the context available to these coding agents and systems and fix it at the root is the principle here.

**Lenny Rachitsky** (01:14:06):
Got it. That is very cool. I hadn't heard of people doing that, model context protocol, makes sense. We've talked about productivity gains outside TX. Just to give you a chance to share how amazing what you've built is, what are some of the gains you see from people using Sierra?

**Bret Taylor** (01:14:18):
Yeah, our customers see anywhere between 50 and 90% of their customer service interactions completely automated, which I think is really exciting. And we serve just a really, really broad range of customers. We serve the health insurance industry, the healthcare provider space, banks. You can actually refinance your home using an agent. One of our customers built on our platform to the telecommunications industry, DIRECTV, SiriusXM to a lot of retailers as well, which is really fun. Everyone from Wayfair to clothing retailers like OluKai and Chubbies Shorts. What's really neat about it is it's a pretty diverse range of use cases and it's everything from helping you sign up for... We have an agent that helps with customer support in one of the big dating applications to helping you upgrade or downgrade your SiriusXM plan. Actually, it's really funny, we do technical support from everything from home alarm systems to sonar speakers to more recently, CAT scan machines, which I think is amazing.

So technicians going in and fixing the CAT scan machine can chat with an AI agent to help them guide them through that process. We're the leader in the space, we're trying to enable every company in the world to create their agent with their brand at the top that I think will become as meaningful of a digital touch point as their website or their mobile app. In the short term, it can really transform the costs of running a customer service team. And what's remarkable is do so with really high customer satisfaction scores. That Weight Watchers agent, I believe has a customer satisfaction score of 4.6 out of five, which is pretty amazing. And what's interesting about service too, it's often people having a problem. And so, when you have a clear, I don't know if you use them in the airport, I think that agent has a CSAT score of 4.7 out of five people are coming in with a problem and [inaudible 01:16:19] delighted. And I think that's really the opportunity here.

**Bret Taylor** (01:16:22):
And our whole vision is that we're going to move towards a world where every single one of the interactions with your customers can be instant. It can be multilingual, it can be over audio, it can be over chat, it can be digital, it can be over the phone and it can be very personalized. And I think that's really, really exciting. And if you think about all the best moments you've had with a brand, it's like that store associate who you know, and it's like for me, it's like the butcher at the grocery store. I love to cook, he knows me. We talk. Can you actually produce that at scale for a company with 100 million customers and can you do it in a really personal way? And I think we're really on the cusp of enabling that.

**Lenny Rachitsky** (01:17:03):
Let me ask you one more question before we get to a very exciting lighting round. There's a lot of founders struggling with go-to-market in AI with their AI apps. There's so many apps these days, so many products, so many things coming at buyers, at large B2B companies. Clearly you guys have figured something out. I imagine your name helps, investors help, but what have you learned about just how to successfully do go-to-market with an AI product, say an agent-specific product that you think would be helpful for folks trying to do this better?

**Bret Taylor** (01:17:35):
I think there's a small handful of go-to-market models that have been proven to work, and I think it's important to choose the right one for the product category you're going after. One category I would say is developer-led. This is somewhere famously Stripe and Twilio where probably two of the original that did this exceptionally. And essentially, the go-to-market motion there is to appeal to an individual engineer often within the department of the CTO who have accountability and a fair amount of latitude to choose a solution. This works if your product is a platform product. It doesn't work, for example, if your product is trying to help a line of business because lines of business typically don't have dedicated engineering teams or let alone, the latitude to just go download a new library or start using a web service like that. It particularly works well if you sell to startups just because startups tend to have engineering teams with quite a bit of latitude to choose services to help them solve the problem given by the founder.

**Bret Taylor** (01:18:44):
Then there's product-led growth. It's a broad term, obviously every company's product matters, but product-led growth more specifically means users can sign up from the website, often get put on a trial. Often you can buy a couple of seats with a credit card and those work where your user and your buyer are the same person. So it works for small business software almost always because sole proprietors do everything. And so you're selling small business software like Shopify in the early days and there's a lot of other products like that where you're trying to sell to small merchants. That's great. It doesn't work well when your buyer and the user of the software are different. So I'll use the example of something like expense reporting software. The user of that software is an individual employee, but the buyer is often a finance department. And so having sign up and buy with your credit card doesn't make sense because the person using is not the person with the credit card and it just doesn't work.

**Bret Taylor** (01:19:39):
And then there's direct sales. And direct sales had gone, I don't want to say out of fashion, but if I think of the best direct sales companies, probably there's a lot of lineage from Oracle, but you think SAP, Oracle, ServiceNow, Salesforce, Adobe perhaps, and there's others as well. And these were companies that sold into large lines of business in a relatively traditional sales motion. I think because product-led growth became very popular. I think a lot companies use that, which is great, that motion produces great products, but if PLG means that you aren't actually engaging with the buyer of your software, you're not going to grow. And so, I've actually seen more recently, with a lot of AI companies, direct sales come a little bit more back into fashion because I think so many of the opportunities in AI actually meet that qualification where the buyer and the user are not necessarily the same person and it really requires that go-to-market motion.

**Bret Taylor** (01:20:40):
Where I see entrepreneurs stumble is they'll choose a go-to-market motion without thinking through what is the process of purchasing this software? What is the process of evaluating the value of this software? And I think people just need to be much more first principles about it and much more thoughtful about it. And candidly, I think a lot of companies should leverage direct sales more than they do. And even though because of the sometimes justified reputation of the quality of products of some of these direct sales companies, it had gotten a bad name. And I think I'm thankful to see it coming back in a lot of the AI market.

**Lenny Rachitsky** (01:21:20):
I feel like this message is something a lot of founders need to hear, especially founders that aren't from a business background that sales turns them off, they don't think they're going to be great at sales. Just this push of this might be what you have to get really good at and this is how you win and you can't just rely on product like growth.

**Bret Taylor** (01:21:36):
Yeah.

**Lenny Rachitsky** (01:21:38):
Bret, is there anything else that you wanted to share? Any last nugget of wisdom? Anything you want to double click on before we get to our very exciting lightning round?

**Bret Taylor** (01:21:47):
No, go ahead.

**Lenny Rachitsky** (01:21:48):
Okay, let's do it. Here we go. Welcome to our very exciting lightning round. I've got five questions for you. Are you ready?

**Bret Taylor** (01:21:53):
Yeah, go ahead.

**Lenny Rachitsky** (01:21:54):
What are two or three books that you find yourself recommending most to other people?

**Bret Taylor** (01:21:59):
I don't read a lot of nonfiction, but probably if I had to pick one in the area of the topics we talked about, Competing Against Luck, which was the book that produced Jobs to be Done, which is a framework I really believe in. My only critique is I think most of these business books should be like an article. So maybe buy the book and punch it into ChatGPT and get the summary. But buy the book it's Clayton Christensen talked about it, but it's a really good framework for thinking about delivering value with your products. And I think it definitely influenced me.

**Bret Taylor** (01:22:37):
Actually one book I do recommend is Endurance, which is the story of Shackleton's trip to go to the South Pole. Half the book is him starving to death and eating seal meat with his crew of people frozen in their boat. I've never seen a better story of grit in my entire life. It's remarkable that it's a true story and if you're an entrepreneur going through a hard time, read that, you'd be like, okay, it could be worse. It's a great book too. It's just remarkable that it's a true story.

**Lenny Rachitsky** (01:23:08):
And one thing he did a great job at is setting expectations for folks that joined that famous newspaper-

**Bret Taylor** (01:23:13):
That ad. That newspaper ad. I don't know if that's true. It's remarkable if that's true.

**Lenny Rachitsky** (01:23:17):
Oh, it might not be true.

**Bret Taylor** (01:23:17):
I don't know. The internet. Who knows?

**Lenny Rachitsky** (01:23:20):
Goddam. Deep fakes even back then. Okay. Do you have a favorite recent movie or TV show that you've really enjoyed?

**Bret Taylor** (01:23:27):
I haven't gone to any new TV shows recently? We just watched Inception with the kids and they loved it and made me appreciate Christopher Nolan and what a cool movie. It's the type of movie when you watch the film and you can a conversations for two days afterwards about it. So, just a great film.

**Lenny Rachitsky** (01:23:46):
I saw someone using I think VO 3 to create their own Inception videos where the worlds wrapping in on each other. Oh, man. Okay. Do you have a favorite product that you have recently discovered that you love or one you've loved for a long time?

**Bret Taylor** (01:23:59):
I'm really a big fan of Cursor. I think it's changed. I love creating software and I'm excited though for agents. I've been really excited. I was very excited to see Codex from OpenAI and others. So I think Cursor will be in its current form, is a transition product. And I know they're working on agents as well, but I really enjoyed taking something I love and it's been my life's passion and really diving into this AI tool and seeing how it transforms how I create software. So I've just been spending a lot of time with the product just because it's so core to what I love to do and it's a really well crafted product.

**Lenny Rachitsky** (01:24:39):
I think that's the first time someone's actually mentioned Cursor in this answer, so might be the beginning of a trend. Michael Trell was on the podcast and he actually had a very similar message as you had at the beginning of this chat about the future of code, what comes after code and this concept that there's going to be this additional pseudo code layer on top of code. Very aligned with your thinking. Do you have a favorite life motto that you often come back to and find useful in work or in life?

**Bret Taylor** (01:25:05):
The best way to predict the future is to invent it, which I think I attribute to Alan Kay of Xerox PARC. He invented a lot of the core abstractions that we use in computing today. I'm an entrepreneur, it's why I love to build things and it's definitely a life motto for me.

**Lenny Rachitsky** (01:25:29):
I feel like many people say this, I feel like you've actually done this so many times. You're really living this motto. Final question. We talked about you inventing the like button at FriendFeed. Were there other thoughts of what they would call it other than like? Was it just obviously like? Or was there other thinking there?

**Bret Taylor** (01:25:48):
This was before emoji. So, if you read the comments on FriendFeed posts, at least 70% of them are cool or wow or yeah or neat. And one of the principle uses of FriendFeed was to have discussions about things. So you'd have a post and then a pretty fulsome discussion underneath. And compared to Twitter and others, it was a great place to have those discussions. And so, the product problem we were trying to solve is get all the one word answers out so that the discussion was actually actual comments as opposed to acknowledgements that you read the thing.

**Bret Taylor** (01:26:30):
So, the original framing was one click comment. That was how we thought about it. And so, the first version that I made had a heart, and she denies remembering this, but Anna Yang, now Anna Muller who has worked at the company, she hated it. She said, "If I look at hearts on every post, I'm going to vomit. It's too much." And it also was interesting, we were simulating, it was like an article about a tragedy or something. A heart was just not the right thing. Like which actually turned out to be really hard to translate was just a much more neutral sentiment, and that's why it was hard to translate because it was subtle. So that's how we ended up with this.

**Bret Taylor** (01:27:17):
We started with a heart, and I don't know if we ever heard the word love, but we definitely started off with the iconography and then like, which just felt like this positive yet as neutral as possible within the realm of positive so that it could work for a more complex story. But it was all because we needed a one-click comment. That's where the concept came from.

**Lenny Rachitsky** (01:27:36):
Wow. I've never heard the story before. It makes me think about LinkedIn now. They're basically trying to solve that same problem. They have all these auto-reply pill tag things. I don't think people like those very much.

**Bret Taylor** (01:27:37):
Yeah. They have a lot of features.

**Lenny Rachitsky** (01:27:48):
So many AI features. Bret, this was incredible. This was an honor. I so appreciate you coming on this podcast. Two final questions, where can folks find you online if they want to reach out, maybe go see if they want to work at Sierra and how can listeners be useful to you?

**Bret Taylor** (01:28:00):
If you want an AI agent to help with customer service, go to sierra.ai. If you want to apply here, sierra.ai/careers where we have offices in San Francisco and New York, Atlanta and London and are hiring pretty aggressively in every department. So please reach out if you're interested.

**Lenny Rachitsky** (01:28:19):
And how can listeners be useful to you? Is it tryout Sierra, anything else there?

**Bret Taylor** (01:28:21):
Yeah, tryout Sierra. I'm a single issue voter.

**Lenny Rachitsky** (01:28:26):
[inaudible 01:28:26] message. I love it.

**Bret Taylor** (01:28:26):
Yeah.

**Lenny Rachitsky** (01:28:27):
Bret, thank you so much for being here.

**Bret Taylor** (01:28:29):
Yeah, thanks for having me.

**Lenny Rachitsky** (01:28:30):
Bye, everyone.

**Lenny Rachitsky** (01:28:32):
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lennyspodcast.com. See you in the next episode.

---

## [6/18] Brian Chesky's secret mentor who died 9 times, started the Burning Man board, and built the world's first midlife wisdom school | Chip Conley (founder of MEA)

**Lenny Rachitsky** (00:00:00):
Let's paint a picture of just what it was like to join Airbnb in your fifties.

**Chip Conley** (00:00:04):
I was mentoring Brian, but he was also my boss. I was 52, the average age was 26. I had to be both wise and curious, and often the dumbest person in the room.

**Lenny Rachitsky** (00:00:14):
It's great to be in founder mode. It's not as great to be working for someone in founder mode.

**Chip Conley** (00:00:18):
Brian assumed everybody else was going to work at the same pace and duration. His point of view is like, "Hey, we're having a meeting in the office tonight at 10 o'clock. Be there."

**Lenny Rachitsky** (00:00:28):
Everyone's talking about, "We got to make the product better. We got to optimize this button, and improve conversion."

**Chip Conley** (00:00:32):
Isn't the product the homes and the apartments? Jobot said, "Nope. Product in the tech industry is something different." I just said, "Listen, let's get some older people who are hosts in here."

**Lenny Rachitsky** (00:00:41):
This whole story is a really good example of the value of having folks that are older.

**Chip Conley** (00:00:45):
When you have older brains connecting the dots, younger team members being really fast and focused, it's brilliant, and people won't notice your wrinkles as much as they'll notice your energy.

**Lenny Rachitsky** (00:00:55):
The Airbnb experienced led you to starting something called the Modern Elder Academy.

**Chip Conley** (00:00:59):
If you think about the caterpillar to butterfly journey, midlife is the chrysalis. Midlife is not crisis. I'm happier today at 64 than I was at 47 when I was going through my flatline experience.

**Lenny Rachitsky** (00:01:09):
Well, let's back up a little bit, this near death experience. Today, my guest is Chip Conley. Chip is one of the most extraordinary and interesting people that you'll ever meet. He was a founding member of the board of Burning Nan. He was on the board of the Esalen Institute in Big Sur. At 26, he started a hotel chain called Joie de Vivre, which went on to become the second-largest boutique hotel chain in the US.

**Lenny Rachitsky** (00:01:32):
After selling it, Brian Chesky personally recruited Chip to join Airbnb to help Brian and the company transform from a fast-growing startup to the world's most valuable hospitality brand. After leaving Airbnb where he was known as the Modern Elder, chip started the Modern Elder Academy, now known as MEA, the world's first midlife wisdom school, with large sprawling, beautiful campuses in Baja and Santa Fe. He's also written seven books, given a TED Talk, and is just a genuinely interesting and amazing human and friend.

**Lenny Rachitsky** (00:02:01):
In our conversation, we explore how to be successful in tech as you age, what he's learned working with and for Brian Chesky, including a lot of real talk, how to build a great culture at your company, his near-death experience, and how it changed the trajectory of his life, and so much more. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube.

**Chip Conley** (00:04:38):
Oh, my god, Lenny, I sort of feel like I'm your father who's so proud of his son. My son has done so well, and I like to talk about, tell all my friends about you.

**Lenny Rachitsky** (00:04:49):
Wow, I am honored. I'm happy that I'm making you proud, Chip, and I feel the same in reverse. We got to work together for many years at Airbnb. I got to learn a ton from you. I'm really excited that more people are going to get to learn from you from this conversation. I'm thinking that the way that we break up this conversation is kind of break it up into three parts, which are kind of the three arcs of your career. The three parts are your early career, building Joie de Vivre, your time at Airbnb, where we got to work together, and then talking through what you're working on now, the Modern Elder Academy.

**Lenny Rachitsky** (00:05:22):
I actually want to start with the middle chapter. I'm going to talk about Airbnb where we got to work together. Let's paint a picture of just what it was like for you to join Airbnb in your fifties surrounded by a bunch of 20 something, 30 somethings reporting to Brian Chesky, who is, I don't know, in his thirties. What was that like?

**Chip Conley** (00:05:41):
Yeah, I wasn't planning on doing this. I got a call from a woman named Natalie Tucci who worked at Airbnb, and said, "Brian Chesky and I've been talking about having you come in and give a talk. Are you open to that?" I was like, "Well, what is Airbnb?" This was 13 years ago, I think it was the end of 2012. Then Brian called me and said, "Listen, we really want you to come in." I came in and gave a talk about innovation and hospitality.

**Chip Conley** (00:06:14):
I think I didn't realize it was sort of a dress rehearsal for Brian to see whether the younger crowd there, I was 52, the average age was 26, would feel good about an old geezer like me with a bricks and mortar boutique hotel background, talking about the industry that Airbnb was disrupting. As it turns out, people liked me. Brian said, "I want you to come in and work 15 hours a week as a consultant. I want you to be my in-house mentor for both me, and Joe and Nate." I said, "Okay, 15 hours a week is great," and then within three weeks, it was 15 hours a day.

**Chip Conley** (00:07:00):
I was saying to Brian, "You're actually not paying me anything." He gave me a little bit of stock that would vest in six months and I said, "I don't know that this deal's working for me. It seems like the company needs to be a little more than you said." He said, "Yeah, I got you. I just wanted you to get in here and see what you could do." Long story short is I ended up going full-time. It was hard at first, Lenny, because I didn't understand the tech lingo. I didn't have any background. I was 52. I'd never worked in a tech company before.

**Chip Conley** (00:07:42):
I was mentoring Brian on leadership, but he was also my boss. I was the head of global hospitality and strategy, which meant initially, I was in charge of all the hosts in the world. Over time, that meant a lot more things too. I was involved in many parts of the business, definitely not the technical parts, but I think the hardest thing for me was just that initially, when people were talking about product, and Jobot said, "I'm the chief product officer," and I'm like, "Well, isn't the product the homes and the apartments?" Jobot said, "No, product in the tech industry is something different."

**Chip Conley** (00:08:23):
I had to be both wise and curious, and often the dumbest person in the room. It required me to have a certain amount of humility as well as to be reporting to a guy 21 years younger than me, Brian.

**Lenny Rachitsky** (00:08:36):
That actually, the point you're making there about what is the product I asked Laura Hughes, formerly Laura Modi, what to talk to you about, who we got to work together. She's the CEO of Bobbie now. You worked closely with her at Airbnb. She said this was the thing that stuck with her most about working with you is coming in and everyone's talking about, "We got to make the product better. We got to optimize this button and improve conversion, and product, product, product."

**Lenny Rachitsky** (00:08:59):
You're just like, "What is the product? I thought the product of Airbnb was the hosts, and the experiences, and the trips." I think that shows the value of someone like you coming in with different experience, and also older, and helping us communicate differently to hosts who also don't understand.

**Chip Conley** (00:09:15):
Well, there's an interesting thing also, Lenny, and you notice the difference in age between our hosts and our guests was probably about 10 years maybe. Over time, it actually got higher, because we started actually reaching out more aggressively to boomers and Gen Xers to be hosts. You had, I remember at one point, and again, let's get into a product talk here, I remember at one point, there was a conversation that was going on about taking Airbnb so it was mobile only.

**Chip Conley** (00:09:47):
Partly because back in the day, the two sharing economy darlings were Uber and Airbnb. Of course, Uber was pretty much a mobile only app. Airbnb started as non-mobile and then went mobile. Then it was like, "Oh, wouldn't it be interesting if everything was mobile?" At some point, I just said, "Listen, let's get some older people who are hosts in here to see how well they will be versed in managing their listing purely on mobile."

**Chip Conley** (00:10:18):
There were times where I was a voice for older users, in this case, hosts, that was helpful to guys and women in their twenties who were the engineers and designers and product managers. I always liked working with you. I want to just compliment Lenny for a minute.

**Lenny Rachitsky** (00:10:40):
Oh, how sweet is that?

**Chip Conley** (00:10:41):
We did a lot of different things together, and what I appreciated was you had a humility to you that was different than a lot of the other product managers. There's other product managers, I'm not going to mention their names, and some of these product managers were very good. There were other product managers though who I found it sometimes hard to work with, because they expected me to know as much as they did.

**Chip Conley** (00:11:07):
I guess it would be, if the opposite side of that would be an older manager expecting a younger manager to have as much emotional intelligence, because emotional intelligence on average is something we get better at as we get older. I think the key for me to work in that environment and make it work was to not pretend to know things I didn't know, it was to have a sense of humor and humility in how I operated, and it was to show respect and hope that I got it in return. I don't know if you felt that way, Lenny. That's the kind of environment I tried to embody there.

**Lenny Rachitsky** (00:11:45):
Absolutely. There's a couple of threads there I want to follow. One is just working for Brian. A lot of people talk about founder mode, and the power of founder mode, it's so great. That's how we-

**Chip Conley** (00:11:55):
Guess who-

**Lenny Rachitsky** (00:11:57):
Exactly.

**Chip Conley** (00:11:57):
... populated that recently, that was Brian.

**Lenny Rachitsky** (00:12:01):
Exactly. It's great to be in founder mode. It's not as great to be working for someone in founder mode. Often a very challenging place to be.

**Chip Conley** (00:12:07):
Yes.

**Lenny Rachitsky** (00:12:07):
You reported to Brian. Also, you were just your own boss basically your entire career. You never really had to report to someone before. Also, he was in his thirties, you're in your fifties. What was it like working for Brian? The more real you can be, the better, because a lot of people always talk about here, it's like, "It was wonderful, I learned so much." Just like what was that experience? What did you learn from working for someone like Brian?

**Chip Conley** (00:12:30):
Well, let's start with the fact that I would never have gone to work for Airbnb if I didn't believe in Brian, because quite frankly, when Brian approached me and we started talking about it, I was like, I wasn't sure I liked the business model all that well as a hotelier. I had to believe in something beyond the business model, because I wasn't sure that the business model would work. Although soon after joining, I saw the numbers. I was like, "Wow, this is working pretty well."

**Chip Conley** (00:12:56):
I believed in Brian because the thing that Brian showed up with initially was just a curiosity and an appetite for learning. I remember back in 2011, when the big debacle happened with the apartment getting trashed by a guest. Brian decided he was going to go to find George Tenet, the former head of the CIA. Brian would go to experts and say to the expert, "I don't know what the hell I'm doing." He did that with me when it came to hospitality. I appreciated that a guy who had a lot of hubris, and Brian definitely has a lot of hubris, could also have the humility to say, "I want to learn more about this."

**Chip Conley** (00:13:42):
It's sort of a growth mindset. What was hard with Brian is, I'd say, three things. Number one is Brian assumed everybody else was going to work at the same pace and duration, and he still has this issue. The beautiful thing about Brian is he's been very honest in the last couple of years on podcasts about his workaholism, and about the fact that the way he lives his life is not like other people. Back when I joined, his point of view is like, "Hey, we're having a meeting in the office tonight at 10 o'clock, be there."

**Chip Conley** (00:14:20):
It's like, "Really? No, I don't think so." I think the fact that Brian assumed everybody else was as one-dimensional in their focus as him was at times a problem, especially for a guy like me who was, I was in a stage in my career where I have a lot of interests. That was one. Number two is Brian admires and admired back then, Steve Jobs so much that there was a sense that as a guy who came from the product world, from the Rhode Island School of Design, he knew better than anybody else.

**Chip Conley** (00:15:05):
There was this, one of the challenges for a CEO sometimes, and this was my experience in my 24 years of running Joie de Vivre, my boutique hotel company, is it feels good when you feel needed. To come into a room and sort of see something, and then point out the things that are wrong makes you feel good. If you don't have emotional intelligence, that process can really piss people off or demotivate people. In Brian's case, I didn't have to deal with that too much, because he didn't understand, when I was starting, it was really, I was in charge of the hosts around the world.

**Chip Conley** (00:15:45):
Quite frankly, the idea of what's the psychology of the host? What's a host entrepreneur like? I went on a world tour to 20 different cities, and went and talked to hosts. I think I came back from that with a little bit of credibility with Brian to say like, "Hey, yes, our data science team and the quality folks who are doing qualitative interviews, they're getting something out of this." I actually went into the homes of these hosts all around the world, and I think I was lucky because Brian did less of that than he did with other people.

**Chip Conley** (00:16:19):
For the product team, my God, a product meeting with Brian would keep people up the night before, not just because they were actually working all night long to get prepared, but also they knew they would work all night long, because they probably wouldn't sleep in anticipation for this. That was another issue, I'd say. I'd say the other thing that, and in each of these cases, I think Brian's getting better. Just like Steve Jobs got better over time when he left and then he came back, he was much better when he came back, from all the people I've talked with who worked with him.

**Chip Conley** (00:16:53):
I'd say the third thing for Brian was the sense that adding a zero to something in terms of expectations, or thinking you're going to set a deadline that is unreasonable is necessary. If you don't do that, there's almost an underlying message that people won't kick ass on their own. There was a sense that Brian had that he had to maybe create ridiculous goals, because even if we hit half of that goal, it was very encouraging. What he missed in that was the fact that when you miss a goal, and when you have someone who has power over you setting the goal, or encouraging a particular goal, you're setting people up for a lot of stress.

**Chip Conley** (00:17:51):
At the end of the day, I think Brian is a generational leader as a millennial, and I think he deserves a lot of credit. Airbnb is as successful as it is, partly because of Brian's leadership. I would not have been there without him. Having said that, I had to hold my tongue in meetings sometimes when I saw how he was operating, because I wouldn't have done it that way. I think over time, I hope I had a little bit of influence on him in terms of how to apply some emotional intelligence to leading people.

**Lenny Rachitsky** (00:18:28):
For people in this position, a lot of people work for founders, especially now that founder mode is a thing. Every founder is just like, "I'm the founder, you got to do what I tell you. It's founder mode. Again, this is how we win. We're in founder mode." You shared really good insight of building credibility as a really good lever to work better with someone like that. Is there anything else you just think as tactics to be effective with founders in founder mode?

**Chip Conley** (00:18:53):
Knowing what I know now, I would say, "Lenny, let's do a little pep talk, you and me before the meeting." I want you to start the meeting with the following as you present and Brian's in the room. "Brian, let's talk about what we're trying to accomplish here. Let's get really clear," and you probably did this, but, "let's get really clear on what both, what's the intention of this iteration that we're doing on the product? What defines success, and what do I want to get accomplished in this meeting?"

**Chip Conley** (00:19:25):
You start with that, because that actually helps to make sure there's alignment. Frankly, if there's not alignment, you might as well not have the meeting. Let's spend the rest of this meeting talking about alignment. That's what I would do, because that's something you can come back to over and over again during the rest of the meeting when Brian or the founder, whomever it is, is beating you up on something, saying like, "Well, let me tell you why it looks like that or why we're doing that." It goes back to that, the three principles or the three key goals we're trying to do with this product update. Yeah, so try to set alignment on the front end.

**Lenny Rachitsky** (00:20:06):
That's an important tip for anyone working with anyone, even. I love just that that works especially well here. Then just going back to the credibility piece, what you shared there is you went on this world tour, not something everyone can do, but just getting really close to your customers, and using that as a, "Hey, I actually know what I'm talking about. You actually should listen to me even though you're the founder."

**Chip Conley** (00:20:27):
Yeah, I think the other thing is PowerPoint or whatever tool you're using, just be careful about being overly reliant upon it, especially when you have a combustible founder who may take you off path, such that your deck in its current order makes no sense at all. I always wanted to really limit the deck as much as possible, because I didn't know where the meeting was going to go. I wanted the decks helpful at the start, at the very start, to just set principles, set goals. Yeah.

**Lenny Rachitsky** (00:21:09):
This whole story of you joining Airbnb in your fifties is a really good example of intergenerational collaboration, something that you're big on, just the value of having folks that are older working at tech companies. Maybe just talk about that broadly, and then we segue into other elements of your career.

**Chip Conley** (00:21:28):
I wrote a book called Wisdom at Work: the Making of a Modern Elder after my Airbnb experience. I did a lot of research. I was like, "Wow, so why do we have less intergenerational collaboration in the workplace, especially in Silicon Valley than we could use?" I started interviewing people, then I started talking to brain scientists, neuroscientists, and realized that a younger brain has fluid intelligence, tends to be fast and focused, really good at problem solving, very good at linearity in terms of looking at things.

**Chip Conley** (00:21:59):
As you get older, the brain shrinks a little bit and you have crystallized intelligence. In crystallized intelligence, what's going on is you're going from left brain to right brain more adeptly. There's a little bit less focus, a little more holistic thinking, systemic thinking, connecting the dots. You can imagine that on a team when you have older brains connecting the dots, thinking broadly, peripherally, younger team members being really fast and focused, and being able to think linearly how to get things done, that combination can either be successful or not.

**Chip Conley** (00:22:36):
When it's successful. It's brilliant. I think Laura, Laura Modi, Laura Hughes Modi, who was my director of hospitality, but also we worked in so many different capacities with her in the company, I loved working with her because her brain worked different than my brain. That's the opportunity is when you realize that diversity on a team, there's lots of kinds of diversity, but when it comes to brain diversity, not just with neuro diverse people but age diverse people, you get a benefit, an effective benefit that is not as noticeable, quite frankly, in some other diversities.

**Chip Conley** (00:23:16):
I found that over and over was really helpful. Part of my job sometimes was to find the blind spot. Again, if you are very focused, one of the things I said to Brian early on was, "I've seen the business plan. Now, I know the goals of how big we want to be in three years." This was very early in my tenure. I said, "But what we really have done, everything we're trying to do is to have no regulations and pay no occupancy tax." Now, hotels pay a bed tax, occupancy tax, we're not paying it, and we're trying to do everything we can not to pay it.

**Chip Conley** (00:23:51):
Knowing that, so for our listeners and viewers to know this, this is something that a guest pays. It's not the host who pays it, the guest pays it. It's part of the bill. If you go and stay in a hotel, there's a big, big tax part of the bill, but it made us more affordable by not having to have our guests pay taxes. Long story short is I said to Brian, "If we're as big as we're going to be three years from now, I promise you we're going to be regulated. I promise you we're going to be paying occupancy taxes. Let's take some proactive steps toward building a strategy for how we're going to be regulated."

**Chip Conley** (00:24:28):
That has consistently been Airbnb's biggest challenge is regulation in municipal markets all around the world. If we'd started a little earlier, maybe in New York, maybe in New York, it wouldn't have gotten to the point where it has been toxic in New York for the last dozen years ever since I was there. There's a few other markets in the world where it was like that. I would just say the value in having some age diversity, even when you have an older person reporting to a younger person, is it can be collaborative.

**Chip Conley** (00:25:02):
There was a guy named John Q. Smith, an engineer who I think you probably remember him at Airbnb. This is a guy who looked younger than he was, and he was a little bit nervous about telling people his age. The thing that was great about John is over time, he was not necessarily going to be the best coder at Airbnb. There was a whole new collection of coders coming in every month, but he became a great manager.

**Chip Conley** (00:25:32):
The beautiful thing about moving from the individual contributor to the manager, the person who can actually bring out the best in a bunch of younger people, who may be better technically than he or she is, but they know how to elevate talent. I call this invisible productivity. It's productivity in which you make everybody else around you better. That's something I tried to do with my teams at Airbnb. Ultimately, I had six different teams, five hospitality and five other teams reporting to me. I did my best to just be the kind of person who wasn't solving all the problems, but I was trying to elevate.

**Chip Conley** (00:26:11):
There's a woman named Lisa Dubost who is at Airbnb, and she, one day, the HR department was reporting to me at one point, and she was running HR. She was 25 and had no background in HR at all. One day she came in to me and she just said, "Chip, you are my confidant." Lisa has a French accent and fluent in French. She said confidant in just the right way. I said, "Oh, well thank you, Lisa." I said to her, "You haven't given me any juicy details yet. A confidant is someone who has the secrets."

**Chip Conley** (00:26:42):
She said to me, "No, in my part of France, a confidant is somebody who gives you confidence." It was like, "Oh, well, maybe that's what a mentor can be is a confidant, someone who gives you confidence and helps by asking questions, helps you as the younger mentee find your roadmap to success."

**Lenny Rachitsky** (00:27:05):
You're sharing a lot of really good examples of the value of older folks being within tech companies. Let me just ask you this, how real is ageism in tech? I ask because a lot of people that are hiring are probably thinking, no, no, I'm not biased. I'm going to hire the best person. If they're someone in their fifties, I'll hire them. No problem. It doesn't feel like it actually works out that way often. Just how real of a problem is this? What do you see?

**Chip Conley** (00:27:27):
Yeah, it's clearly a problem. I'd say it's maybe a little bit less of a problem than it was a dozen years ago, because I think a dozen years ago, it was almost a blind spot. In Airbnb, we had a group called Wisdom at Airbnb. It was an employee resource group for people 40 and older. There are lots of these kinds of groups that didn't exist a dozen years ago in all kinds of tech companies, which is good, because it means that there's a voice and a way to congregate with a bunch of people who are older.

**Chip Conley** (00:28:00):
Ultimately, we had these senior nomads come in and be like the voice of the customer for 10 weeks at Airbnb. It was the Wisdom at Airbnb older employee group that really actually pushed for this with Brian. The challenge is, in a world in which the smartest new people, especially when it comes to technical skills and engineering, are coming in with a whole new set of skills that an older person doesn't have, the older person is both expensive and may be perceived as slow. In the era of AI, it's a whole new ballgame.

**Chip Conley** (00:28:42):
The question, I think, will be if what AI cannot do is the human wisdom piece, artificial intelligence and human wisdom might be the balance beams here. Is it possible that older managers who have a little more emotional intelligence, a little more pattern recognition, a little bit more wisdom, can be a value to a company? The jury's still out. There's a New York Times article that just came out about the question of is AI going to wipe out older people's jobs or younger people's jobs? I think the answer is both, but the question is how bad is it for both of them?

**Chip Conley** (00:29:29):
I think what I would say to an older person, and when I say older, I mean like 45 or older, if you've done well financially and you're doing okay, the question you might ask yourself is, are you open, as I ultimately was with Brian in my fourth year at Airbnb? I took a substantial pay cut. I think it went down to 40% or 50% time, and my stock, my options were dropped to that level, my salaries dropped to that level, because I didn't want to work full time anymore. There are a lot of people who can be valuable in a company who have some institutional wisdom, some process knowledge of how to get things done in this organization.

**Chip Conley** (00:30:08):
In tech companies, that's really important. Airbnb, one of the biggest challenges that Airbnb has always been, how do I get shit done around here? Process knowledge allows you to understand, how do you deal with an org chart and get things done partly because you understand the motivations of different groups? That is something you build over time. Long story short is I just think that older people might look at their workload and say, "I'm willing to take a 20% or a 40% pay cut to go to 80% or 60% time," and the company is going to get their money's worth in that.

**Lenny Rachitsky** (00:30:49):
That's a really interesting point, that if you're older and you're maybe less connected to the most cutting edge ways of building and coding, AI makes that a lot easier in many ways where you start to just talk to it. You don't even need to understand what's happening underneath.

**Chip Conley** (00:31:03):
Yeah.

**Lenny Rachitsky** (00:31:05):
There's a lot of listeners who are older in tech, there's a lot of listeners who are approaching midlife, let's say, worried about what happens to their career. When you look at people you've worked with and had at your academy, which we'll talk about, who continue to thrive and continue to have a really healthy career in tech, what do they do differently? What do they have in common that other folks you think should work on and focus on?

**Chip Conley** (00:31:28):
I think this idea being a mentor and a mentor and an intern, there's just the voracious appetite for curiosity. When I talk to someone who's a midlife and wants to be in the tech world or already is, the thing I say is, "Show up with curiosity and a passionate engagement for what you do, and people won't necessarily notice your wrinkles as much as they'll notice your energy." Energy has two parts to it. Energy is, they notice that you are not just sort of resting on your laurels, you have physical energy in how you do your job.

**Chip Conley** (00:32:04):
When people are like that, they're sort of timeless. They're age fluid, as I say. We talk about gender fluidity. Well, there could be age fluidity. They're not defined by their age. The other part of energy that's important is being positive. That's sort of more energetic, a little bit more California energetic. There's a sense of when someone's got good energy, you're drawn to them. It's about showing up with the kind of energy of someone 10 or 20 years younger than you, and then showing up with positive energy.

**Chip Conley** (00:32:38):
I think one of the things that would say I did well at, there's lots of things I didn't do well at Airbnb, but in terms of what I did well is I was very approachable. Over the course of time, the number of mentees I had, the number of people who just wanted to have coffee with me or tea, the number of people who just said, "Thank you for being in that meeting, you just sort of gave it a positive feeling," was really important.

**Chip Conley** (00:33:04):
My energy, both the positive energy part, and then also the fact that yeah, I could work 60 and 70 hours a week, and I could travel around the world as the Secretary of State of the company, which is what Brian called me a couple times on stage. The fact that I could do that meant that no one was looking at me and saying, "Let's get rid of the old fogey." Well, maybe some people the board, but I wasn't aware of them. I just think show up with that passionate engagement, that curiosity, that energy, the ability to be both the learner and the teacher, with respect for people that are younger than you, and you're going to probably do pretty well.

**Lenny Rachitsky** (00:33:44):
**Chip Conley** (00:35:39):
I think we're moving into, there's a book that David Epstein wrote called Range, and the whole premise of range is that we are moving out of the era of the specialists and into the era of the generalists. I think AI is just accelerating this. As we are more reliant upon AI, and AI can be exceptional at technical skills and solutions really expeditiously, I think generalists, people who can think broadly, become all the more important. I think that what I would say to someone in HR or recruiting is beyond what I already said before, is the person passionate?

**Chip Conley** (00:36:23):
Are they curious? Are they a learner? Do they have good energy? I would also say, are they a generalist when they're a problem solver? I actually think that's going to be an increasingly important part of how effective companies think broadly. I think that's a key one. I think also, this idea of how do you create intergenerational collaboration in the form of mutual mentorships? One of the things I loved at Airbnb, there were a few people I did this with, where I had something to teach them and they had something to teach me.

**Chip Conley** (00:37:02):
A good example, my iPhone, so there's 97% of the utility of my iPhone that I probably don't use and don't know how to use. This was back in, let's say, 2013, 2014. There were people who knew iPhone or Google Suites back then. I'd never used a Google Doc back then when I joined Airbnb. There were people who could teach me something technical, and then they wanted to learn something from me, which would be like, "How do you want a great meeting," or, "How do you give a great employee review?" There are a lot of managers who've never been a manager before.

**Chip Conley** (00:37:45):
How do you disperse people like me in the organization so that there's usually not enough time for these young managers to come to some training session on how to do a good employee review. You sort of have to do it out there in the field. It's like apprenticeship back in the trades. You're an electrician apprentice, not because you're watching some video on it, you're out there in the field, doing it. That's a huge value in a younger company when you have some older people who have not been vested with the responsibility of managing those younger people.

**Chip Conley** (00:38:23):
They may actually be reporting to someone younger than them, but they're there to actually be support. In some ways, I think that was part of the unexpected value that I was able to offer to Airbnb and to Brian specifically, because there are a ton of people in Airbnb who were not even in my departments who would come to me and say, "I'm having a problem. How do I solve this? Can we spend lunch together?" I almost always said yes.

**Lenny Rachitsky** (00:38:53):
I think the reason people did that in many ways is you just have a very unique aura of wisdom, and it's hard to replicate that, Chip.

**Chip Conley** (00:39:04):
Yes, and it all comes back to the curiosity. If I was just the older elder, dispensing wisdom, people would've gotten bored very quickly. I think the fact, yeah, I was on the board of Burning Man, that's cool. I show up as someone who feels younger than I am. I'm turning 65 this year. The bottom line is I think people lost track of my age, partly because I lost track of my age.

**Lenny Rachitsky** (00:39:38):
That's such good advice on the front end to be successful as a person kind of getting older in tech is curiosity, positive energy, the way you talked about it, passionate, engagement, is that the term?

**Chip Conley** (00:39:52):
Yep.

**Lenny Rachitsky** (00:39:53):
Then on the other side is hire generalists. This actually comes up a lot in the AI conversations, just exactly as you said, the power of generalists reminds me, I'm going to this gym now, and the lady there is just like, "I love AI so much, because I'm just such a big picture person, and I am so bad at just getting, thinking about the details, and AI solves all that for me." It's like, "Here, here's what I want to do. I'll do this, move my house to here, here." It's like, "Here's what you need to do, step one, two, three, four, five."

**Chip Conley** (00:40:18):
It is remarkable. Since the time I've known you, how fast it has become dominant in our lives. Yeah, I think one of the last thing I'd say is, look, I'm privileged. For those of you who are listening or watching this and you're saying, "Well, Chip, you were 52 years old and they came to you. That doesn't happen to me. I'm not in that position." The thing I would say is, you're right, but I could have been plucked and brought in and partly as Brian's boy, people would've rejected me, because if I didn't show up the right way, it wouldn't have worked well.

**Chip Conley** (00:41:00):
There are lots of people who Brian brought into the company who didn't work well. I think the key is how do you get the foot in the door? At the end of the day, those second and third order of degrees of separation in terms of networking are still essential. The most important thing is to be able to articulate what you have accomplished in a new way that a recruiter says, "Wow." I really tell people I would love to see a resume.

**Chip Conley** (00:41:41):
First of all, the question that I think it was, who was it? Someone asked it, I don't remember if it was Cheryl Stamberg or someone else asked her, who said, "What's the biggest problem you're dealing with here, and how can I help you?" That's a great line. Number two is what I love to see is not so much what roles you've had, what bullet points do you have of your things you've learned? Give me, in a paragraph, a thorny problem you faced. What was the problem, and what skills you used to actually accomplish it, and what was the result of that?

**Chip Conley** (00:42:22):
I would love to see a resume like that. The older you are, the more you can actually have a resume like that. Then you can use that as the conversation piece when you're doing interviews.

**Lenny Rachitsky** (00:42:34):
I love that. I love that we're getting into interview advice and resume advice.

**Chip Conley** (00:42:37):
Yes.

**Lenny Rachitsky** (00:42:38):
Speaking of thorny problems, and also why Brian decided to reach out to you, I want to go back to the beginning of your career.

**Chip Conley** (00:42:45):
Yes.

**Lenny Rachitsky** (00:42:46):
Right out of business-

**Chip Conley** (00:42:47):
You're good at this, by the way. You're good at this.

**Lenny Rachitsky** (00:42:49):
I was thinking ahead. Okay, so you're in business school, you left business school, you're like, "Maybe I should start a hotel." Something that rarely works out usually probably leads to a lot of money lost and a lot of frustration and just like, "Okay, what have I done with my life?" Worked out for you.

**Lenny Rachitsky** (00:43:07):
Ended up building the second largest boutique hotel chain in the world, Joie de Vivre, beloved. I loved every single experience I've had as Joie de Vivre. When you sold it, I was like, "That is so sad." Talk about just that story. I know this could go on for hours, but what's the-

**Chip Conley** (00:43:21):
Yeah, I'll be brief.

**Lenny Rachitsky** (00:43:22):
Yeah.

**Chip Conley** (00:43:23):
26 years old, couple of years out of Stanford Business School, working for a commercial real estate developer. I was bored silly. I wanted to do something more creative. Bill Graham, famous concert promoter, said to me, because I had gotten to know him, "What San Francisco really needs is a rock and roll hotel." I decided to start looking to find a broken down motel hotel that I could turn into a rock and roll hotel.

**Chip Conley** (00:43:46):
I found something in the Tenderloin, and turned it into the Phoenix, which became a famous rock and roll hotel that I have owned for 39 years now. Long story short is that was how I started Joie de Vivre, the company. We grew to 52 hotels around California, became the second largest, as you said, in the world in terms of the number of hotels, boutique hotels that we operated. I loved it till I hated it. In my late forties, I hated it, didn't want to do it anymore. The great recession came along and it was just kicking my ass.

**Chip Conley** (00:44:23):
I really went through a bit of what I now call a midlife chrysalis, but a midlife crisis, where I just wanted to change everything. I got through it. I had an NDE, I had a near-death experience where I had an allergic reaction to an antibiotic and I died. From that point forward, I realized every day is a gift and a bonus, and I decided to sell my company at the bottom of the great recession. That's really how I created this space in my life to be able to join Airbnb.

**Lenny Rachitsky** (00:44:58):
Well, let's back up a little bit. This near-death experience, share more there. What happened there?

**Chip Conley** (00:45:04):
Yeah, so I write books. I've written seven or eight books, and I had written a book called Peak: How Great Companies Get Their Mojo from Maslow. It was a book that Brian really liked, and part of the reason he wanted to reach out to me. I was on a book tour, I had a broken ankle. I broke my ankle at a bachelor party playing baseball. I ended up with a cut on my leg and the cut on my leg had fertilizer in it and went septic. I was on a very strong antibiotic and I died.

**Chip Conley** (00:45:39):
I went flatlined from the allergic reaction to the antibiotic. I saw, it happened nine times over 90 minutes, I kept dying, kept flatlining, yeah. Ended up in the hospital for three days. They finally said, "Listen, it's an allergic reaction, we believe." They thought it was a heart attack, a bunch of stuff, stroke, et cetera. No, it was the allergic reaction. I saw birds. I saw all this beautiful stuff. We don't have time to go into it.

**Lenny Rachitsky** (00:46:10):
You did? What?

**Chip Conley** (00:46:11):
I did. You want to hear this? Yeah, I saw this beautiful stuff.

**Lenny Rachitsky** (00:46:13):
Let's do it.

**Chip Conley** (00:46:15):
I think there's a hotel in San Francisco called the Vitale that I built across the street from the ferry building, and it's still there, but it's no longer called the Vitale. In that hotel, there were these slippers in every guest room. One slipper said slow, the other slipper said down. I was wearing these slippers in my flatline thing, flying in the air in a 40-foot tall living room in the Alps, surrounded by birds that were tweeting and chirping at me.

**Chip Conley** (00:46:50):
I understood bird talk. I understand exactly what they were saying. They kept telling me, "If you slow down, you will see beauty and you will see awe." There was a bunch of other things, but let me just limit it to that and just say, and then the birds would say, "It's time to go." The birds would go out the big window into the mountains, and I would try to follow them. Right as I would get to the window, all of a sudden, I'd come back to life.

**Lenny Rachitsky** (00:47:20):
Holy shit. I love that there was a message inside of this experience. I don't know how many people experienced that. Clearly, this led to a big life change. It's interesting that a lot of times, you need something like that. You've been doing this for how many years at that point? Running Joie de Vivre?

**Chip Conley** (00:47:38):
At that point, I'd been running Joie de Vivre for 22 years.

**Lenny Rachitsky** (00:47:39):
22 years. It's interesting that you need something like that a lot of times. Otherwise, it just momentum just keeps carrying you forward.

**Chip Conley** (00:47:46):
Within two years, I'd sold it, and I had the chance to move on.

**Lenny Rachitsky** (00:47:52):
With building Joie de Vivre, something you've written about a number of times is just the way you built it is a really unique approach to building a business. Specifically, there's a huge focus on culture, which also came out at Airbnb. Talk about just why you see culture as such an important part of how you build a business like tangibly. A lot of people talk about culture, warm, fuzzy stuff, but you think about it very tangibly.

**Chip Conley** (00:48:15):
Culture is what happens around here when the boss is not around. The more distributed a company, the more culture is important. The boss is around in a traditional bricks and mortar workplace where everybody shows up at eight and leaves at six, and we all see each other. In my company, in Joie de Vivre, we had 52 hotels, and 25 restaurants, and four spas, and it was distributed. I couldn't be in all those places all the time.

**Chip Conley** (00:48:52):
Similarly, with Airbnb, Airbnb had offices around the world and it was a global company. The more distributed you are, of course, in the remote work world we live in, the more culture is important, and more difficult. When you're remote, there's these few cues you have about how we do things around here. They're usually in a digital, virtual format, which is why it's all the more important for you to have in-person gatherings of a team more often if you are virtual.

**Chip Conley** (00:49:27):
At the end of the day, the reason the culture is important is because it actually helps, it helps guide people in terms of making decisions, but it's also a magnet for the right kind of people. Oracle has a different culture than Apple, which has a different culture than Facebook. You can choose the place you're going to work based upon the culture. There are people who can be very good at what they do, but if they're in the wrong culture, they're in the wrong kind of environment, and they're not willing to shift to fit that culture.

**Chip Conley** (00:50:01):
We saw it at Airbnb all over and over again. In fact, Airbnb saw it, I think when with Amazon people. Apple people have resonated pretty well at Airbnb, Amazon people, less so. Those are two different cultures, Amazon and Apple. Therefore, understanding a culture before you even actually take the job is one of the more important decisions you need to make is like, "Is this culture a culture that I can live with and maybe influence?" There's language about culture fit.

**Chip Conley** (00:50:37):
I like to say culture add, because culture fit to me can actually be quite negative toward somebody who is the aberration. You have to fit in. Especially if this is a demographic thing, a person of color, a gay person, a person in a wheelchair, so you have to fit in. A culture add suggests that actually having some diversity on the team is helpful, because it actually adds to the culture. You still have to be able to get along in that culture. Culture is an intangible. That's the problem with it is it's hard to measure, but you see its value and you understand whether it's working based upon employee pulse reports and things like that.

**Lenny Rachitsky** (00:51:24):
You talk about having to understand the culture is such a key part of having success at a company. Do you have any advice for just how to understand the culture for someone interviewing? I don't know. You came in, you work part-time, it's easier to experience it all. Any tips there for, "Okay, this is for me, it's not for me?"

**Chip Conley** (00:51:39):
When you're interviewing, you're also interviewing them. When you're interviewing, it's not about you having to prove yourself. It's also for them to actually prove themselves as a company, and also try to understand if there is some alignment in the company. The kind of questions I would ask as someone who's being interviewed would be, what are three to five adjectives that define this culture? What's the biggest problem in this culture, in terms of something that's just endemic or baked in across the organization?

**Chip Conley** (00:52:14):
Is it ever going to get fixed? How could I come in and maybe help that? Which frankly, at a very junior level, you're not going to be able to help it except for in very minor ways. If you're a senior person, you might be able to help it. Those are the kind of questions I'd want to know. Frankly, if I'm asking that same question about what are the adjectives to multiple people, am I hearing the same thing over and over again? If I'm not, is that because there's not alignment? Is that because different departments have different flavors?

**Chip Conley** (00:52:48):
You could have a culture within a department that's very different than the overall corporate culture. The corporate culture certainly has an enormous oppressive influence, but you can be in a culture, a really great culture of a team or a department, in an overall company culture that's not good. In the long run, that oppressive company culture is either going to have to evolv,e or your department, you may lose people.

**Lenny Rachitsky** (00:53:23):
When I reflect back on the impact you had at Airbnb, one of the funny things I think about is triangles showing up a lot on decks, and specifically rooted in Maslow's hierarchy, just like everything's this Maslow hierarchy metaphor.

**Chip Conley** (00:53:41):
True.

**Lenny Rachitsky** (00:53:43):
This one, I don't know, specific piece of this is you have this kind of model you think about for how to help employees be successful at a company. It's kind of rooted in your Peak book philosophy. Maybe just talk about that, and then if there's anything else you want to expand on with this power of thinking through the Maslow hierarchy.

**Chip Conley** (00:54:01):
Maslow's hierarchy, basically five levels. Later in life, he had a seven and an eight level model, but at the base is the kind of physical, water, food, air, and you move up to self-actualization at the top. To use this model as a hierarchy of needs for employees, customers, and investors is what the Peak model is about. The Peak, my book. The employee model is really simple. It's money or compensation at the base, recognition in the middle, and meaning at the top.

**Chip Conley** (00:54:39):
Now, there are some industries and some kinds of jobs in which money is 90% of the pyramid. Just because of the base doesn't mean it's not the dominant part of the pyramid, but the differentiation often is in recognition and meaning. In nonprofits, usually the money piece of it's rather thin. The recognition's this, and meaning's huge. Understanding how do you create an organization, and I gave a TED Talk in 2010 about this topic as well, how do you measure the intangibles of meaning and how do you create an environment where people feel a sense of meaning?

**Chip Conley** (00:55:18):
The customer pyramid, briefly, I'll just say that one, is meeting expectations is the base, meeting desires is in the middle, and then meeting unrecognized needs. I think one of the things that we did at Airbnb about a year after I joined, and when Jonathan Goldenhall was joining, is we really tried to ask ourselves, "Are we in the home sharing business, or are we in some kind of business that is even bigger and broader than that?" Ultimately, we came up with the idea that we were in the belong anywhere business.

**Chip Conley** (00:55:49):
Airbnb was not in home sharing, we were in belonging anywhere. Once you have that down, that was sort of the unrecognized need at the top of the pyramid. Then that becomes an organizing principle for how do you teach your hosts to create a sense of belonging? How does our marketing and advertising play up the belonging piece, especially and the everywhere piece, because hotels are not everywhere, but homes are? I would just say that this model, the idea of hierarchies is, I think, very helpful. Yeah, my book Peak has been around for 18 years, but I still am asked to give 20 or 30 speeches a year on it.

**Lenny Rachitsky** (00:56:30):
Oh, man. This pyramid of comp, recognition, meaning is really interesting, especially these days, because with all this AI researcher poaching, there's all this talk of just like, "Will people just go work wherever they get the most money, or is there a mission and meaning to the work they're doing that will keep them not taking a hundred million dollars offer?" Seems to be happening in a lot of cases, which shows you the power of meaning.

**Chip Conley** (00:56:55):
Yeah. If you know you're working for a toxic company, at some point, your conscience kicks in. Whether it's toxic in terms of the purpose of the company, toxic in terms of the leadership or the culture, life is too short.

**Lenny Rachitsky** (00:57:10):
Okay. You've had two major shifts in your career. You started the hotel chain, then you went to Airbnb. Most recently, the Airbnb experience, I imagine, led you to starting something called the Modern Elder Academy. Talk about what is the Modern Elder Academy?

**Chip Conley** (00:57:28):
Yeah, what is going on with that Modern Elder Academy? The Modern Elder Academy. There was a couple times where I was called the Modern Elder at Airbnb, and then I was told that a Modern Elder is someone who's as curious as they are wise. Jonathan Mildenhall, who is the chief marketing officer at Airbnb, used to call me the Modern Elder as well, and he said, "If you ever create a school, Modern Elder would be a good name."

**Chip Conley** (00:57:56):
We talked about it, and next thing I knew, I was saying, "Okay, this is called the Modern Elder Academy." We now call it MEA because elder is a fraught word on some level, it makes you sound elderly. What I really wanted to create was a place where people could come and do a workshop, they're five day workshops in Baja on the beach, or in Santa Fe on a big four square mile horse ranch, and reimagine and repurpose yourself, and navigate transitions.

**Chip Conley** (00:58:27):
We go through so many transitions in the middle of our life, let's say between, I define midlife as 35 to 75, guys. It's a very long life stage. We go through a lot of transitions. We are constantly evolving our purpose. We're building our wisdom. We have knowledge management tools out there, but we're the wisdom management tools. We're the tools that help us to get wiser over time, and then we need to reframe our relationship with getting older. Becca Levy has shown at Yale that when you shift your mindset on aging from a negative to a positive, you get seven and a half years of additional life, which is more life than any other biohack that's being done right now.

**Chip Conley** (00:59:08):
That's what we do, and we have 7,000 grads from 60 countries, and 56 regional chapters around the world. It's a bit of a movement, and I teach. I teach some of the workshops, and we have all kinds of famous people who come and teach. For me, creating the world's first midlife wisdom school just feels like the natural next thing for me to do. I love hospitality, so it's a very upscale kind of experience, but we have scholarships. I love retreat centers. I was on the board of the Esalen Institute in Big Sur for 10 years. I love wellness.

I've owned the Kabuki Springs and Spa for 28 years, which is the largest spa in San Francisco, and I love education. My book, Wisdom at Work: The Making of a Modern Elder, gave me a curriculum in which we've expanded quite a bit with Harvard, Yale, Stanford, and UC, Berkeley professors helping us create a curriculum around midlife. That's how MEA came about.

**Lenny Rachitsky** (01:00:14):
To your point, I forget who said, I think you said Jonathan has said this was a natural next step for you, I completely agree. It's like, looking back, this is the obvious thing you should be doing right now.

**Chip Conley** (01:00:22):
Yeah.

**Lenny Rachitsky** (01:00:23):
Also, I'm learning more things about you. I didn't know you were involved with the Kabuki Spa. I think Esalen and I knew, you just keep getting more interested.

**Chip Conley** (01:00:31):
Thank you.

**Lenny Rachitsky** (01:00:32):
There's a couple threads here I want to actually follow. This point you made about shifting your mindset to aging as a positive thing helps you live longer. That's such a powerful point. Can you just speak more to that, just what does that look like?

**Chip Conley** (01:00:43):
Yeah, there's lots of data points. I'll talk about two. One is this Becca Levy study, which has been going on for 15 to 20 years. If you sort of buy into the ageism of American society or Hallmark cards, when you get a card at age 40, 50, or beyond, there's a belief that life gets worse as you get older. If you can survive your midlife crisis, all you have to look forward to is disease, decrepitude, and death. The bottom line is there's a lot of things that get better with age. I wrote a book called Learning to Love Midlife.

The subtitle says it all: 12 Reasons Why Life Gets Better with Age. What I really wanted to do with that book, which is really, it summarizes the MEA curriculum, I wanted to write a book that sort of helped people to see the upside of aging, the unexpected pleasures of aging. They had a pro-aging, not just an anti-aging point of view. When you actually have a pro-aging point of view and you see the upside of aging, you take better care of yourself, both your mind and your body. You actually are willing to learn and try new things.

**Chip Conley** (01:01:51):
One of my favorite MEA questions is, 10 years from now, what will you regret if you don't learn it or do it now? It's a powerful question, really important question as we get older. When you're young, you've got all of your life left ahead of you. When I moved to Baja part-time in Mexico at age 56, I had a mindset which was, "I'm too old to learn Spanish. I'm too old to learn to surf," but when I said, "10 years from now, what will I regret if I don't learn it or it now?" I said, "Well, 10 years from now, I might still be living in Baja. I should learn Spanish, I should learn how to surf because we're right next to a surf break."

**Chip Conley** (01:02:29):
I did. What I believe is that anticipated regret is a form of wisdom, and it's a catalyst for taking action. That's one data point. The other data point is something called the U-curve of happiness, and it's been around for 20 years, and it shows the following. It has changed in the last couple of years because young adults are unhappy like never before. A 20 or a 22-year-old, really unhappy, 24-year-old, really unhappy.

**Chip Conley** (01:02:58):
Historically, the way it was is you were happy from 18 to 23 or 24, and then around 23 or 24, you start to see a long, slow decline in life satisfaction that actually bottoms out between 45 and 50. I'm sorry to tell you that, Lenny, since you're 44, but your mileage may vary.

**Lenny Rachitsky** (01:03:17):
You're saying I'm the least happy I'll ever be. That's only upside. That's great. Yeah.

**Chip Conley** (01:03:21):
Well, here's the part that's weird is that before this research was done, and it's global research across all demographics, what they found was starting around age 50 or 52, you get happier, so that you're happier in your fifties than your forties, sixties, fifties, seventies, happier than sixties, and the women in their eighties, happier than seventies. Wow. It's partly because we are in around 45 to 50, doing this thing called the midlife unraveling, what Brené Brown calls the midlife unraveling.

**Chip Conley** (01:03:55):
You're unraveling your expectations, what you define as success, your definition of what a beautiful body looks like, and you're liberated into freedom in your fifties and beyond. I can say that, yeah, I'm happier today at 64 than I was at 47 when I was going through my flatline experience, and not wanting to run my company anymore.

**Lenny Rachitsky** (01:04:19):
You used this term earlier, the midlife chrysalis, was that what it was?

**Chip Conley** (01:04:22):
Chrysalis. Chrysalis, yeah.

**Lenny Rachitsky** (01:04:23):
Chrysalis. What is that? Is that kind of along the same lines?

**Chip Conley** (01:04:27):
If you think about the caterpillar to butterfly journey, midlife is the chrysalis. It's that cocoon in which all of the change is happening. At the time, when you're going through it, it's like, "Oh, shit. My life is liquefying in front of myself." On the other side of it, there's a metamorphosis that happens.

**Chip Conley** (01:04:48):
I like to use the language, in fact, I have a podcast called The Midlife Chrysalis, because I want to help change the dialogue around midlife, so that the number one word attached to midlife is not crisis, but in fact, it's maybe chrysalis, and the idea that life is meant to be transformative during that era.

**Lenny Rachitsky** (01:05:09):
That is actually very empowering. I am sort of going through that, not necessarily in this intense way yet, but that might be coming. You said there's a bunch of upsides to getting older. It might be helpful just to share a couple of those things for folks that are like, "Oh, wow, I didn't realize that."

**Chip Conley** (01:05:23):
Emotional intelligence grows with age. Our wisdom can grow with age, although we know 70-year-olds who are not as wise as 30-year-olds, so it's a matter of what you do with your life experience. I define wisdom as metabolized experience, mindfully shared for the common good. What else gets better with age? You learn how to edit. You have no more Fs left to give, no more fucks left to give. That is absolutely true, especially for women as they age. You are more spiritually curious. The list is long, and so there are a lot of things that, actually, another one that I love is you're not compartmentalized.

**Chip Conley** (01:06:01):
When you're younger, you're compartmentalized. As you grow older, you are growing whole, and that means you're alchemizing curiosity and wisdom, introvert, extrovert, masculine, feminine, gravitas, depth, and levity, lightness. The people who I really admire who are 85 years old, they're so present and they're so whole. They are just who they are.

**Lenny Rachitsky** (01:06:25):
There's a quote I found from you along these lines, the societal narrative on aging is just don't do it.

**Chip Conley** (01:06:31):
Fantastic. Yeah. We sort of say we don't want to age, but we do want to live. Quite frankly, aging and living are the same thing, as are aging and growing.

**Lenny Rachitsky** (01:06:44):
Coming back to MEA, just for folks that are interested, curious about this, who's this for, would you say? Who should seriously look into this program?

**Chip Conley** (01:06:52):
MEA is really, the people who tend to come to MEA are in the midst of a transition. It could be selling their company, leaving a job, getting divorced, having kids, becoming an empty nester, taking care of parents till they're passing away, having a health diagnosis that's scary. Average age is 54, and it's people of all walks of life. It's not just the tech industry, but it's very popular in the tech industry. It's people who are looking to maybe do a reframe of their purpose, and maybe even a reinvention of their career.

**Chip Conley** (01:07:29):
Yeah, the two campuses are just gorgeous. It's been called the Four Seasons meets Blue Zones meets the Esalen Institute, which I like. We have online programs too, and so you don't have to come to either of our campuses in Mexico, or on the beach, or in New Mexico. You can actually do it online.

**Lenny Rachitsky** (01:07:54):
Those three, yeah, that's the tagline. That's your tagline right there. Esalen meets Blue Zones, meets what was the first one?

**Chip Conley** (01:08:00):
The Four Seasons.

**Lenny Rachitsky** (01:08:01):
The Four Seasons.

**Chip Conley** (01:08:02):
Yeah.

**Lenny Rachitsky** (01:08:03):
Nailed it. Okay. I'm going to zoom out and take us to a recurring segment on this podcast. I want to see if this goes anywhere, AI Corner, and with AI Corner, ask guests, what's a way that you've found AI useful in your work or in your life, any kind of trick you've learned, any workflow, anything you've found useful?

**Chip Conley** (01:08:23):
Yeah, I have a daily blog. It's called Wisdom Well, and it's on the MEA website. When I'm looking for inspiration, AI does it for me, and ultimately, it gives me a first draft. That's good enough for me then to say, "Okay." There's times when I'm missing the inspiration. I tend to write really well in the morning.

**Chip Conley** (01:08:47):
If it's any other time of the day, I do not like writing creatively. If I have a deadline for tomorrow and it's five o'clock in the afternoon, it's like, "Okay, ChatGPT. I'm on my way to you." I tend to use ChatGPT the most because I don't know, I like Claude as well, but yeah.

**Lenny Rachitsky** (01:09:03):
Okay, awesome. I was going to ask which tool you use. What's your workflow there? Is it you use voice mode? Do you just type out, "Here's what I'm thinking about, write me a little drop blog post?"

**Chip Conley** (01:09:12):
The good news is that at this point, it knows me well enough and my blogs, and I've actually, it knows my weird sense of humor, so it's able to ape me pretty well. I'll just say, "I need a 250 word post on," like today, today's post was a post that ChatGPT helped me with it. I said, "I believe that there's a refrain that needs to happen with the soul. We tend to say, 'I have a soul, or I don't have a soul,' but what if my soul has me? What if in fact, my job is just to be this vehicle for my soul to go to the next lifetime?"

**Chip Conley** (01:09:56):
My job is to be this steward of the soul. I said, "Write me something around that." It was just a weird idea. Of course, not all my blog posts are so new age, and I like that. I write a lot on leadership, but that was one that within 30 seconds, I had a 250 word blog post that I then adapted, and there you go.

**Lenny Rachitsky** (01:10:19):
Amazing. Chip, we've covered a lot of ground. We've gone through your entire life. Maybe actually just the tip of the iceberg. With that, we've reached our very exciting lightning round. I've got five questions for you. Are you ready?

**Chip Conley** (01:10:31):
Yes. Let's do it.

**Lenny Rachitsky** (01:10:32):
First question, what are two or three books that you find yourself recommending most to other people?

**Chip Conley** (01:10:38):
My favorite book of all time, Man's Search for Meaning, Viktor Frankl in a concentration camp in World War II. When someone's going through a hard time in their life, I say, "Read that book. You'll realize it's not so bad, what you're going through." It also really speaks to this idea of despair equals suffering, minus meaning. I wrote a book called Emotional Equations that was a New York Times bestseller that spoke to this idea that what if you could take all of your emotions and turn them into equations?

**Chip Conley** (01:11:04):
Very engineering-minded of me. That's one. I love any book by Liz Gilbert, sort of the opposite. Elizabeth Gilbert wrote Eat, Pray, Love. Her book, she's on faculty at MEA. She teaches here. Big Magic is just a beautiful book about sort of how do you get in the flow to allow the genie to come through you. Her Ted Talk in 2009 was about the fact that genius is not about being the genius yourself. It's about being the receptacle for the genie to come through you.

**Lenny Rachitsky** (01:11:44):
I want to come back to this equation you shared. I was going to get to it, but I didn't, so this is a good opportunity to. There's a couple that are really interesting to me. This is, you wrote about these in a book. You have a bunch of these equations about living a happier life. The one you shared is despair equals suffering, minus meaning.

**Chip Conley** (01:12:00):
Yes.

**Lenny Rachitsky** (01:12:01):
The implication there is if you want less despair, increase the meaning.

**Chip Conley** (01:12:05):
That's right.

**Lenny Rachitsky** (01:12:05):
Or reduce the suffering.

**Chip Conley** (01:12:06):
Suffering, Buddhist philosophy, the first noble truth of Buddhism is that suffering is ever-present. If suffering's a constant and you have two variables, using some algebra, I guess, you know that if you have more meaning, you have less suffering. That's that one.

**Lenny Rachitsky** (01:12:25):
The other one that I love is anxiety equals uncertainty times powerlessness. Maybe talk about that one briefly.

**Chip Conley** (01:12:32):
98% of anxiety comes from two sources. One is what you don't know, and number two is what you can't control or influence, and based upon social science. You can create an anxiety balance sheet and create four columns. First column is what is it you do know about the thing that's making you anxious? The second column is, what is it you don't know? The third column is what is it you can control or influence? The fourth column is what is it you can't control or influence?

**Chip Conley** (01:13:01):
When you take free-floating anxiety and put it into an equation, it actually makes it more tangible, and you often are less anxious as a result.

**Lenny Rachitsky** (01:13:11):
Boom. Okay, so if you're feeling anxious right now, this is an exercise you can do and you'll feel less anxious in like five minutes is what I'm hearing.

**Chip Conley** (01:13:19):
Yes.

**Lenny Rachitsky** (01:13:19):
Okay, excellent. Very, very good nugget of advice. Okay, let's keep going with the lightning round. Come back from our tangent. Do you have a favorite recent movie or TV show you've really enjoyed?

**Chip Conley** (01:13:29):
Ted Lasso is, I'm a sucker for that show. When it comes to movies, I'm a total movie buff. We have an annual MEA Film Festival at our Santa Fe campus. I would say that the film that I'm most excited about that is coming out that most people have never heard of, it's called I'll Push You. It's the story of two guys, one of whom is in a degenerative health condition and in a wheelchair, and his best friend pushes him the 500 miles of the Camino de Santiago, and it's the relationship they build along that way.

**Lenny Rachitsky** (01:14:08):
Amazing. Very deep cut. Do you have a favorite product that you recently discovered that you really love?

**Chip Conley** (01:14:17):
Yes. Hair growing material. No. Do you know Viori shorts? I sound like Scott Galloway because he advertises this, but Viori shorts are like, I just love them. They're just breathe and they're comfortable.

**Lenny Rachitsky** (01:14:33):
I'm wearing Viori joggers right now. The one downside of Viori, not to make anyone mad, is they're kind of plasticky if you look at the material. I'm trying to like, I don't know, but I do love, there's nothing better. That's the problem. Anything else like this that is all cotton.

**Chip Conley** (01:14:50):
Yes.

**Lenny Rachitsky** (01:14:51):
I'm a fan. I have many Viori, I don't know if they're called joggers, just, I don't know, weekenders or something. Anyway, love you, Viori. Do you have a favorite life motto that you often come back to and find really useful in work or in life? I imagine you have many, but is there one that comes to mind?

**Chip Conley** (01:15:07):
My favorite one right now is your painful life lessons are the raw material for your future wisdom. The premise of that is that wisdom often comes through the school of hard knocks. When you're in the midst of a really challenging time, you are developing your future wisdom that's going to be valuable to you.

**Lenny Rachitsky** (01:15:28):
Okay, final question. You were on the board of Burning Man, or you still are?

**Chip Conley** (01:15:28):
I was.

**Lenny Rachitsky** (01:15:28):
Was.

**Chip Conley** (01:15:34):
I helped found the board of Burning Man. Yeah.

**Lenny Rachitsky** (01:15:36):
Okay. No big deal. I don't know if you know this, I got married at Burning Man. We had an unofficial wedding there on bicycles, so it's really meaningful to us. I've been there four or five times. What's something about Burning Man that maybe people don't know, some inside story or a really unexpected piece of the journey? I may imagine there's a lot, but what comes to mind?

**Chip Conley** (01:15:55):
I would say the best not well-known thing about Burning Man is that Burning Man own owns a place called Fly Ranch. Fly Ranch is about 10 miles from Burning Man. Now, when you go to the Burn, the event around Labor Day, you cannot go over there. It's locked off. It's 3,400 acres. If you look at Fly Ranch, FlyRanch.org, I think it might even be, or it's on the Burning Man site, Fly Ranch is the opposite of Burning Man. Burning Man is this alkaline desert.

**Chip Conley** (01:16:29):
There's no living life there at all. It's very masculine. Fly Ranch is porous, and lots of desert grasses, and hot springs, and hot pools, and birds, and wild horses, and it's one of my favorite hot springs places in the world. Just check it out, and you can go there when it's not during the event. It's quite beautiful.

**Lenny Rachitsky** (01:16:57):
It feels like it might've inspired MEA in many ways.

**Chip Conley** (01:17:00):
It did, yes.

**Lenny Rachitsky** (01:17:03):
Chip, two final questions. Where can folks find you online, and how can listeners be useful to you?

**Chip Conley** (01:17:08):
Online, MEAWisdom.com is the website for MEA. My website is ChipConley.com, C-O-N-L-E-Y, and I'm on LinkedIn. That's really, from a social media perspective, the thing that I do the most. I actually take my daily blogs and put them on LinkedIn. Then what your community can do, just come say hi, come check me out. If wisdom's interesting to you, and I think wisdom should be interesting to everybody here, on the MEA website, at the very bottom footer, you'll see a bunch of free resources.

**Chip Conley** (01:17:43):
One of them is called Why Successful Leaders Value Wisdom. It is a free resource, and there's also a free resource down there called The Anatomy of a Transition. Those two free resources, understanding how to build your TQ, your transitional intelligence, and understanding how to develop wisdom are two, to my mind, two of the most important modern skills that we can have.

**Lenny Rachitsky** (01:18:05):
It's funny when you say you're on LinkedIn. It doesn't resonate with me. Chip Conley on LinkedIn, posting on LinkedIn, something about...

**Chip Conley** (01:18:13):
I don't know. Why? Because I'm a little too Burning Man?

**Lenny Rachitsky** (01:18:15):
You're just, yeah, exactly. It feels like that's not your vibe, but I love that you do it, because that's where the people are.

**Chip Conley** (01:18:21):
Oh, I put wild, weird stuff up on LinkedIn, and thank God somebody's doing that.

**Lenny Rachitsky** (01:18:28):
For some reason, I don't see it. I need to fix that. Chip, this was incredible. Everything I was hoping it'd be, thank you so much for being here and for sharing-

**Chip Conley** (01:18:34):
Thank you, Lenny.

**Lenny Rachitsky** (01:18:34):
... your wisdom.

**Chip Conley** (01:18:35):
I am so proud as I go back, like have your proud Papa who just loves to see you in your element, and I just want to make sure everybody knows the following. Lenny was so good to work with. Whenever you were assigned to a project as a PM, I appreciated it because I just knew that we were going to have great conversations. You're just an interesting dude.

**Lenny Rachitsky** (01:19:00):
Well, I appreciate that, Chip. That's going to be the beginning of this whole episode. We're just going to put that up front. Just kidding. That was awesome, Chip. Really appreciate it.

**Chip Conley** (01:19:01):
Thanks.

**Lenny Rachitsky** (01:19:08):
Thanks everyone for listening. Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at LennysPodcast.com. See you in the next episode.

---

## [7/18] Inside ChatGPT: The fastest-growing product in history | Nick Turley (Head of ChatGPT at OpenAI)

**Lenny Rachitsky** (00:00:00):
You were a product leader at Dropbox, then Instacart. Now, you're the PM of the most consequential product in history.

**Nick Turley** (00:00:05):
I didn't know what I would do here because it was a research lab. My first task was I fix the blinds, or something like that.

**Lenny Rachitsky** (00:00:11):
When someone offers you a rocket ship, don't ask which seat.

**Nick Turley** (00:00:13):
We set out to build a super assistant. It was supposed to be a hackathon code base.

**Lenny Rachitsky** (00:00:16):
What was it called before?

**Nick Turley** (00:00:17):
It was going to be Chat with GPT-3.5 because we really didn't think it was going to be a successful product.

**Lenny Rachitsky** (00:00:21):
And then Sam Altman is just like, "Hey, let me tweet about it."

**Nick Turley** (00:00:23):
This is a pattern with AI, you won't know what to polish until after you ship. My dream is that we ship daily.

**Lenny Rachitsky** (00:00:28):
By the time people hear this, they're going to have their hands on GPT-5.

**Nick Turley** (00:00:31):
About 10% of the world population uses every week. With scale comes responsibility. It just feels a little bit more alive, a bit more human. This model has taste.

**Lenny Rachitsky** (00:00:38):
Kevin Weil, your CPO, said to ask you about this principle of, "Is it maximally accelerated?"

**Nick Turley** (00:00:43):
I just really want to jump to the punchline, "Why can't we do this now?" I always felt like part of my role here is to just set the pace and the resting heartbeat.

**Lenny Rachitsky** (00:00:49):
Everyone is always wondering, "Is Chat the future of all of this stuff?"

**Nick Turley** (00:00:52):
Chat was the simplest way to ship at that time. I'm baffled by how much it took off, even more baffled by how many people have copied.

**Lenny Rachitsky** (00:00:58):
ChatGPT is now driving more traffic to my newsletter than Twitter.

**Nick Turley** (00:01:02):
That is the type of capability that has been incredibly retentive. I've been really excited about what we've been doing in search.

**Lenny Rachitsky** (00:01:06):
Can you give us a peek into where this goes long-term?

**Nick Turley** (00:01:09):
ChatGPT feels a little bit like MS-DOS. We haven't built Windows yet, and it will be obvious once we do.

**Lenny Rachitsky** (00:01:15):
Today, my guest is Nick Turley. Nick is Head of ChatGPT at OpenAI. He joined the company three years ago, when it was still primarily a research lab. He helped come up with the idea of ChatGPT and took it from 0 to over 700 million weekly active users, billions in revenue, and arguably the most successful and impactful consumer software product in human history. Nick is incredible. He's been very much under the radar. This is the first major podcast interview that he has ever done, and you are in for a treat. We talk about all the things, including the just launched GPT-5.

**Christina Cacioppo** (00:03:31):
Great to be here. Big fan of the podcast and the newsletter.

**Lenny Rachitsky** (00:03:34):
Vanta is a longtime sponsor of the show, but for some of our newer listeners, what does Vanta do and who is it for?

**Christina Cacioppo** (00:03:41):
Sure. So we started Vanta in 2018, focused on founders, helping them start to build out their security programs and get credit for all of that hard security work with compliance certifications, like SOC 2 or ISO 27001. Today, we currently help over 9,000 companies, including some startup household names, like Atlassian, Ramp, and LangChain, start and scale their security programs, and ultimately build trust by automating compliance, centralizing GRC, and accelerating security reviews.

**Lenny Rachitsky** (00:04:12):
That is awesome. I know from experience that these things take a lot of time and a lot of resources, and nobody wants to spend time doing this.

**Christina Cacioppo** (00:04:20):
That is very much our experience, but before the company, and some extent, during it, but the idea is, with automation, with AI, with software, we are helping customers build trust with prospects and customers in an efficient way. And our joke, we started this compliance company so you don't have to.

**Lenny Rachitsky** (00:04:36):
We appreciate you for doing that, and you have a special discount for listeners. They can get $1,000 off Vanta at vanta.com/lenny, that's vanta.com/lenny for $1,000 off Vanta. Thanks for that, Christina.

**Christina Cacioppo** (00:04:50):
Thank you!

**Lenny Rachitsky** (00:04:55):
Nick, thank you so much for joining me, and welcome to the podcast.

**Nick Turley** (00:04:59):
Thanks for having me, Lenny.

**Lenny Rachitsky** (00:05:00):
I already had a billion questions I wanted to ask you, and then you guys decided to launch GPT-5 the week that we're recording this. So, now, I have at least 2 billion questions for you. I hope you have a lot of time. First of all, just congrats on the launch. It's coming tomorrow, the day after recording this. Just congrats. How are you feeling? I imagine this is an ungodly amount of work and stress. How are you doing?

**Nick Turley** (00:05:22):
It's a busy week, but we've been working on this for a while, so it also feels really good to get it out.

**Lenny Rachitsky** (00:05:27):
So, by the time people hear this, they're going to have their hands on GPT-5, the newest ChatGPT. What's the simplest way to just understand what this is, what it unlocks, what people can do with it? Give us the pitch.

**Nick Turley** (00:05:39):
I'm so excited about GPT-5. I think for most people, it's going to feel like a real step change. If you're the average ChatGPT user, and we have 700 million of them this week, you've probably been on GPT-4o for a while. You probably don't even think about the model that powers the product. And GPT-5, it just feels categorically different. I'll talk about a lot of the specifics, but at the end of the day, the vibes are good, at least we feel that way. We hope that users feel the same. And increasingly, that is the thing that I think most people notice, right? They don't look at the academic benchmarks. They don't look at evaluations. They try the model and see what it feels like. And just on that dimension alone, I'm so excited. I've been using it for a while, but it is also the smartest, most useful, and fastest frontier model that we've ever launched.

**Nick Turley** (00:06:33):
On pure SMARTs, one way to look at that is academic benchmarks on many of the standard ones, whether or not it's math, or reasoning, or just raw intelligence. This model is state of the art. I'm especially excited about its performance on coding, whether or not that's SWE-bench, which is a common benchmark, or actually front-end coding is really, really good as well, and that's an area where I feel like there's the true step change improvement in GPT-5. But really, no matter how you measure the SMARTs, it's quite remarkable, and I think people are going to feel the upgrade, especially if they weren't using o3 already.

**Nick Turley** (00:07:13):
And the second thing beyond SMARTs is it's just really useful. Coding is one axis of utility, whether or not you have coding questions or you're vibe coding an app, but it's also a really good writer. I write for a living, internally, externally. I just wrote a big blog post that we published Monday, and this thing is such an incredible editor. And compared to some of the older models, it's got taste, which I think is really exciting. And to me, that's something that is truly useful in my day-to-day. And there's a bunch of other areas, like it's state of the art on health, which is useful when you need it, but again, the thing you can't really express in use cases or data is the vibe of the model. And it just feels a little bit more alive, a bit more human in a way that is hard to articulate until you try it. So, feel good about that.

**Nick Turley** (00:08:06):
And yeah, as mentioned, it's faster. It thinks, too, just like o3 did, but you don't have to manually tell it to do that. It'll just dynamically decide to think when it needs to. And when it doesn't need to think, it just responds instantly, and that ends up feeling quite a bit faster than using o3 did. And then maybe the thing that's most exciting is that we're making it available for free, and that's one of those things that I feel like we can uniquely do at OpenAI. Because many companies, I think, if they have a subscription model like us, they would gate it behind their paid plan. And for us, if we can scale it, we will, and that just feels awesome. We did that with 4o as well. So, everyone is going to be able to try GPT-5 tomorrow, hopefully.

**Lenny Rachitsky** (00:08:46):
How long does something like this take? I don't know if there's a simple answer to this, but just how long have you guys been working on GPT-5?

**Nick Turley** (00:08:51):
We've been working on it for a while. You can view GPT-5 as a culmination of a bunch of different efforts. We had a reasoning tech, we had a more classic post-screening methodologies, and therefore, it's really hard to put a beginning on it, but it really is the end point of a bunch of different techniques that we began for a while.

**Lenny Rachitsky** (00:09:14):
Can you give us a peek into the vision for where ChatGPT is going, GPT in general is going? If you look at on the surface, it's been the same idea with a much smarter brain for a long time. I'm curious where this goes long-term.

**Nick Turley** (00:09:28):
So, to maybe back up a bit, now, you think of ChatGPT as, "Is this going to be ubiquitous product?" Again, about 10% of the world population uses every week.

**Lenny Rachitsky** (00:09:37):
Holy shit.

**Nick Turley** (00:09:39):
I think we have 5 million business customers now. It's an established category in its own right. But really, when we started, we set out to build a super assistant, that's how we talked about it at the time. In fact, the code base that we use is called SA Server. It was supposed to be a hackathon code base, but things always turn out a little bit differently. So, yeah, in some ways, that is still the vision. The reason I don't talk about it more than I do is because I think assistant is a bit limiting in terms of the mental model we're trying to create. You think of this very personified human thing, maybe utilitarian, maybe a... And frankly, having an assistant is not particularly relatable to most people, unless they're in Silicon Valley and they're a manager, or something like that. So it's imperfect.

**Nick Turley** (00:10:24):
But really, what we envision is this entity that can help you with any task, whether or not that's at home, or at work, or at school, really any context, and it's an entity that knows what you're trying to achieve. So, unlike ChatGPT today, you don't have to describe your problem in menu to detail because it already stands your overarching goals and has context on your life, et cetera. So, that's one thing that we're really excited about. The inverse of giving it more inputs on your life is giving it more action space. So, we're really excited to allow it to do, over time, what a smart, empathetic human with a computer could do for you. And I think the limit of the types of problems that you can solve for people, once you give it access to tools like that, is very, very different than what you might be able to do in a chatbot today. So, that's more outputs.

**Nick Turley** (00:11:19):
And I often think, "Okay, I'm a general intelligence. What happened if I became Lenny's intern, or something?" And I wouldn't be particularly effective despite having both of those attributes that I just mentioned, and it's because I think this idea of building a relationship with this technology is also incredibly important. So, that's maybe the third piece that I'm excited about is building a product that can truly get to know you over time. And you saw us launch some of those things with improved memory earlier this year, and that's just the beginning of what we're hoping to do so that it really feels like this is your AI. So, I don't know if supersystem is still the right exact analogy, but I think people just think of it as their AI. And I think we can put one in everyone's pocket and help them solve real problems, whether or not that's becoming healthy, whether or not that's starting a business, whether or not that's just having a second opinion on anything. There's so many different problems that you can help with people in their daily life, and that's what motivates me.

**Lenny Rachitsky** (00:12:16):
So an interesting between the lines that I'm reading here is the vision is for it to be an assistant for people not to replace people. It feels like a really important piece of the puzzle. Maybe just talk about that.

**Nick Turley** (00:12:29):
AI is really scary to people, and I understand there's decades of movies on AI that have a certain mental model baked in. And even if you just look at the technology today, everyone, I think, has this moment where the AI does something that was really deeply personal to them and you're thought, "Hey, AI can never do that." For me, it was weird music theory things where I was like, "Wow, this thing actually understands music better than I do," and that's something I'm passionate about. And so it's naturally scary. And I think the thing that's been really important to us for a long time is to build something that feels like it's helpful to you, but you're in the driver's seat, and that's even more important as the stuff becomes agentic, the feeling of being in control, and that can be small things.

**Nick Turley** (00:13:15):
We built this way of watching what the AI is doing when it's in agent mode. And it's not that you actually are going to watch it the whole time, but it gives you a mental model and makes you feel in control in the same way that, when you're in a Waymo, you get that screen, for those of you who've tried Waymo. You can see the other cars. It's not like you're going to actually watch, but it gives you the sense that you know how this thing works and what's happening, or we always check with you to confirm things. It's a little bit annoying, but it puts you in the driver's seat, which is important. And for that reason, we always view technology and the technology that we build as something that amplifies what you're capable of, rather than replacing it, and that becomes important as the deck gets more powerful.

**Lenny Rachitsky** (00:13:53):
Okay. So you mentioned the beginnings of ChatGPT. I was reading in a different interview. So you joined OpenAI. ChatGPT was just this internal experimental project that was basically a way to test GPT-3.5, and then Sam Altman is just like, "Hey, let me tweet about it, maybe see if people find this interesting," yada yada, yada. It's the most successful consumer product in history, I think both in growth rate in users and revenue, and just absurd. Can you give us a glimpse into that early period before it became something everyone is obsessed with?

**Nick Turley** (00:14:24):
Yeah. So we had decided that we wanted to do something consumer-facing, I think, right around the time that GPT-4 finished training, and it was actually mainly for a couple of reasons. We already had a product out there, which was our developer product. That's actually what I came in to help with initially, and that has been amazing for the mission. In fact, it's grown up. And now, it's the OpenAI platform with, I don't know, 4 million developers, I think. But at that time, it was early stage, and we were running into some constraints with it because there was two problems. One, you couldn't iterate very quickly because, every time you would change the model, you'd break everyone's app. So, it was really hard to try things.

**Nick Turley** (00:15:03):
And then the other thing was that it was really hard to learn because the feedback we would get was the feedback from the end user to the developer to us. So it was very disintermediated, and we were excited to make fast progress towards AGI and it just felt like we needed a more direct relationship with consumers. So we were trying to figure out where to start. And in classic OpenAI fashion, especially back then, we put together a hackathon of enthusiasts of just hacking on GPT-4 to see what awesome stuff we could create and maybe ship to users, and everyone's idea was some flavor of a super assistant. They were more specific ideas, like we had a meeting bot that would call into meetings, and the vision was maybe it would help you run the meeting over time. We had a coding tool, which full circle now, probably ahead of its time. And the challenge was that we tested those things, but every time we tested these more bespoke ideas, people wanted to use it for all this other stuff because it's just a very, very generically powerful technology.

**Nick Turley** (00:16:04):
So, after a couple of months of prototyping, we took that same crew of volunteers, and it was truly a volunteer group, right? We had someone from the supercomputing team who had built an iOS app before. We had someone on the research team who had written some backend code in their life. They were all part of this initial ChatGPT team, and we decided to ship something open-ended because we just wanted a real use case distribution. And this is a pattern with AI, I think, where you really have to ship to understand what is even possible and what people want, rather than being able to reason about that a priori. So, ChatGPT came together at the end because we just wanted the learnings as soon as we could, and we shipped it right before the holiday thinking we would come back and get the data and then wind it down. And obviously, that part turned out super differently because people really liked the product as is.

**Nick Turley** (00:16:56):
So I remember going through the motions of like, "Oh, man, dashboard is broken. Oh, wait, people are liking it. I'm sure it's just going viral and stuff is going to die down," to like, "Oh, wow, people are retaining, but I don't understand why." And then eventually, we fell into product development mode, but it was a little bit by accident.

**Lenny Rachitsky** (00:17:14):
Wow. I did not know that ChatGPT emerged out of a hackathon project. Definitely the most successful hackathon project.

**Nick Turley** (00:17:21):
I like to tell this story when we do our hackathons because I really do want people to feel like they can ship their idea, and it's certainly been true in the past, and we'll continue to make it true.

**Lenny Rachitsky** (00:17:32):
If you don't want to share these things, but I wonder who that team was.

**Nick Turley** (00:17:34):
The team is largely still around. Some of the researchers working on GPT-5, actually, were always part of the ChatGPT team. Engineers are still around. Designers are still around. I'm still here, I guess. So, yeah, you've got the team still running things, but obviously, we've grown up tremendously, and we've had to because with scale comes responsibility. And we're going to hit a billion users soon and you have to begin acting in a way that is appropriate to that scale.

**Lenny Rachitsky** (00:18:06):
Okay. So let me spend a little time there. So, I don't know if this is 100% true, but I believe it is that ChatGPT is the fastest growing, most successful consumer product in history. Also, the most impactful on people's lives. It feels like it's just part of the ether of society now. It's just my wife talks to it. Every question I have, I go to it, voice mode. My wife is just like, "Let me check with ChatGPT." It's just such a part of our life now, and I think it's still early. So many people don't even know what the hell is going on. Just as someone leading this, do you ever just take a moment to reflect and think about just like, "Holy shit"?

**Nick Turley** (00:18:45):
I have to. It's quite humbling to get to run a product like that, and I have to pinch myself very frequently, and I also have to sometimes sit back and just think, which is really hard when things are moving so quickly. I love setting a fast pace at the company, but in order to do that with confidence, I need at least one day every week that I'm entirely unplugged and I'm just thinking about what to do and process the week, et cetera.

**Nick Turley** (00:19:14):
And the other thing is I've never ever worked on a product that is so empirical in its nature where, if you don't stop, and watch, and listen to what people are doing, you're going to miss so much, both on the utility and on the risks, actually. Because normally, by the time you ship a product, you know what it's going to do. You don't know if people are going to like it, that's always empirical, but you know what it can do. And with AI, because I think so much of it is emergent, you actually really need to stop and listen after you launch something and then iterate on the things people are trying to do and on the things that aren't quite working yet. So, for that reason alone, I think it's very important to take a break and just watch what's going on.

**Lenny Rachitsky** (00:20:03):
Okay. So you take a day off every week... not off. Okay, that's not the right way to put it. You take a day of thinking time, deep work.

**Nick Turley** (00:20:12):
I need it. Yeah, yeah, yeah. And I need to hard unplug on a Saturday, or something like that. Obviously-

**Lenny Rachitsky** (00:20:16):
On a Saturday [inaudible 00:20:16].

**Nick Turley** (00:20:16):
But it's just not possible otherwise. This has been a giant marathon for three years now. Yeah.

**Lenny Rachitsky** (00:20:25):
Like a sprint marathon.

**Nick Turley** (00:20:26):
Sprint marathon, that's right, or interval training, or something. I don't know how to exactly describe the OpenAI launch cadence, but you've got to set yourself up in a way that is sustainable. Even if this wasn't AI and it didn't have the interesting attributes that I just mentioned, I think you would need to do that. But especially with AI, it's important to go watch.

**Lenny Rachitsky** (00:20:45):
So, along those lines, I talked to a bunch of people that work with you, that work at OpenAI. Joanne specifically said that urgency and pace are a big part of how you operate, that that's just something you find really important, to create urgency within the team constantly, even when you are the fastest growing product in history, growing like crazy. Talk about just your philosophy on the importance of pace and urgency on teams.

**Nick Turley** (00:21:08):
Well, it's nice of her to say that. Two things, with ChatGPT, when we decided to do it, we had been prototyping for so long and I was just like, "In 10 days, we're going to ship this thing," and we did. So, that was maybe a moment in time thing where I just really wanted to make sure that we go learn something. Ever since then, I spent so much time thinking about why ChatGPT became successful in the first place, and I think there was some element of just doing things where there was many other companies that had technology in the LLM space that just never got shipped. And I just felt like, of all the things we could optimize for, learning as fast as possible is incredibly important. So I just started rallying people around that, and that took different forms.

**Nick Turley** (00:21:55):
For a while, when we were of that size, I just ran this daily release sync and had everyone who was required to make a decision in it, and we would just talk about what to do and to pivot from yesterday, et cetera. Obviously, at some point, that doesn't scale, but I always felt like part of my role here, obviously, was to think about the direction of the product, but also to just set the pace and the resting heartbeat for our teams. And again, this is important anywhere, but it's especially important when the only way to find out what people like and what's valuable is to bring it into the external world. So, for that reason, I think it's become a superpower of OpenAI, and I'm glad that Joanne thinks that I had some part in that, but it really has taken a village.

**Lenny Rachitsky** (00:22:38):
I love this phrase, "the resting heart rate of your team". That's such a perfect metaphor of just the pace of being equivalent to your resting heart rate.

**Nick Turley** (00:22:46):
I actually learned that at Instacart, when I showed up there, because we were in the pandemic and it was all hands on deck. For a while, there was this... I think there was a company-wide stand-up because we disbanded all teams. We were just trying to keep the site up. And for me, I had been used to taking my sweet time and just thinking really hard about things, and that's important, but I really learned to hustle over there, and I think that's come in handy at OpenAI.

**Lenny Rachitsky** (00:23:12):
Okay. So, along these same lines, I asked Kevin Weil, your CPO, what to ask you, and he said to ask you about this principle of, "Is it maximally accelerated?" Talk about that.

**Nick Turley** (00:23:22):
That's funny, we have a Slack emoji, apparently, for this now because I used to say that. Now, I try to paraphrase. Sometimes, I just really want to jump to the punchline of like, "Okay, why can't we do this now?" or, "Why can't we do it tomorrow?" And I think that it's a good way to cut through a huge number of blockers with the team and just instill... especially if you come from a larger company. At some point, we started hiring people from larger tech companies. I think they're used to, "Let's check in on this in a week," or, "Let's circle back next quarter to see if we can go on the plan." And I just, as a-

**Nick Turley** (00:24:00):
... on the plan and I just kind of as a thought exercise, always like people asking, "Okay, if this was the most important thing and you wanted to truly maximally accelerate it, what would you do?" That doesn't mean that you go do that, but it's really a good forcing function for understanding what's critical path versus what can happen later. And I've just always felt like execution is incredibly important. These ideas, they're everywhere. Everyone's talking about a personal AI, you might've seen news on that and I really think that execution is one of the most important things in the space and this is a tool. So, it's funny that that became a meme. It's like a little pink Slack emoji that people just put on whatever they're trying to force the question.

**Lenny Rachitsky** (00:24:45):
I was going to ask, what theme [inaudible 00:24:47]. So, it's a little pink, is there something in there like-

**Nick Turley** (00:24:48):
It's a Comic Sans emoji that says, is this maximally accelerated?

**Lenny Rachitsky** (00:24:53):
Okay. And so, the kind of the culture there is when someone is working on something, the push is, is this maximally accelerated? Is there a way we can do this faster? Is there anything we can unblock?

**Nick Turley** (00:25:02):
Yeah. And we use that sparingly, right? Because it needs to be appropriate to the context. There's some things where you don't want to accelerate as quickly as possible because you kind of want process. And we're very, very deliberate on that where your process is a tool. And one of the areas where we have an immense amount of process is safety. Because A, the stakes are already really high, especially with these models, GPT-5 which is a frontier in so many different ways. But B, if you believe in the exponential, which I do and most people who work on this stuff do, you have to play practice for a time where you really, really need the process for sure, sure, sure. And that's why I think it's been really important to separate out the product development velocity, which has to be super high from, for things like frontier models, there actually needs to be a rigorous process where you red team, you work on the system card, you get external input, and then you put things out with confidence that it's gone through the right safeguards.

**Nick Turley** (00:26:02):
So, again, it's a nuanced concept, but I found it very, very useful when we needed and for everything product development, you're a dead on arrival, so it's important to get stuff out.

**Lenny Rachitsky** (00:26:11):
We got to open source those memes so that other teams can build on this approach.

**Nick Turley** (00:26:16):
Absolutely.

**Lenny Rachitsky** (00:26:17):
So, interestingly with ChatGPT, and it's not a surprise, but not only is it the fastest-growing, most successful consumer product ever, retention is also incredibly high. People have shared these stats that one month retention is something like 90%, six month retention is something like 80%. First of all, are these numbers accurate? What can you share there?

**Nick Turley** (00:26:39):
I'm obviously limited on what exactly I can share, but it is true that our retention numbers are really exciting and that is actually the thing we look at. We don't care at all how much time you spend in the product. In fact, our incentive is just to solve your problem and if you really like the product, you'll subscribe, but there's no incentive to keep you in the product for long. But we are obviously really, really happy if over the long run, three month period, et cetera, you're still using this thing. And for me, this was always the elephant in the room early on. It's like, "Hey, this may be a really cool product, but is this really the type of thing that you come back to?" And it's been incredible to not just see strong retention numbers, but just see an improvement in retention over time even as our cohorts become less of an early adopter and more the average person, so.

**Lenny Rachitsky** (00:27:29):
Yeah. So, that note is something that I don't think people truly understand how rare this is when a product... The cohort of users comes, tries it out and then retention over time goes down and then it comes back up, people come back to it a few months later and use it more. It's called a smiling curve, a smile curve, and that's extremely rare.

**Nick Turley** (00:27:48):
Yeah, yeah. Yeah. There's some smiling going on that's just on the team and I feel like have technology, some of it is not the product. I think people are actually just getting used to this technology in a really interesting way, where I find, and this is why the product needs to evolve too, that this idea of delegating to an AI, it's not natural to most people. It's not like you're going through life and figuring out what can I delegate? Certain sphere of Silicon Valley does that because they're in a self-optimization mode and they're trying to delegate everything they can. But I think for most people in the world it's actually quite unnatural. And you really have to learn, "Okay, what are my goals actually and what could another intelligence help me with?"

**Nick Turley** (00:28:26):
And I think that just takes time and people do figure it out once they've had enough time with the product. But then of course there's been tons of things that we've done in the product too, whether or not it's making the core models better, whether or not it's new capabilities like search and personalization and all that kind of stuff, or just standard growth work too, which we're starting to do. That stuff matters too, of course.

**Lenny Rachitsky** (00:28:49):
So, you might be answering this question already, but let me just ask it directly. People may look at this and be like, "Okay, they're building this kind of layer on top of this God-like intelligence. Of course it will grow incredibly fast and retention will be incredible. What do you guys actually doing that sits on top of the model that makes it grow so fast and retain so much?" Is there something that has worked incredibly well that has moved metrics significantly that you can share?

**Nick Turley** (00:29:18):
One thing we've learned, I'll answer that question in a minute, but one thing we've learned with ChatGPT is that there really is no distinction between the model and the product. The model is the product and therefore you need to iterate on it like a product. And by that I mean obviously you typically start by shipping something very open-ended, at least if you're OpenAI [inaudible 00:29:38] that's kind of a playbook. But then you really have to look at what are people trying to do? Okay, they're trying to write, they're trying to code, they're trying to get advice, they're trying to get recommendations and you need to systematically improve on those use cases. And that is pretty similar to product development work. Obviously the methodology is a bit different, but discovery is the same. You got to talk to people, you got to do data science and you got to try stuff and get feedback.

**Nick Turley** (00:30:04):
So, that's one chunk of work that we've been very consciously doing is improving the model on the use cases people care about. And there's also such thing as vibes because I'm sure you know and that's one of the things that I'm excited about in GPT-5 is that the vibes are really good. So, that too is, we have a model behavior team and they really focus on what is the personality of this model and how does it speak and talk. So, there's that kind of work. I would say that's maybe a third of the retention improvements that we see or so just roughly. And then I think another third is what I would call product research capabilities. They're research driven for sure. They have a research component, but they're really new product features or capabilities. And search is one example of that where if you remember in the olden days, maybe 20 months ago or something, you would talk to ChatGPT and it'd be like, "As of my knowledge cut off..." Or, "I can't answer that because that happened to recently," or something like that.

**Nick Turley** (00:31:00):
And that is the type of capability that has been incredibly retentive and for good reason. It just allows you to do more with the product personalization, like this idea of advanced memory where it can really get to know you over time is another example of a capability like that. I think that's another good chunk. And then the third stuff is the stuff you would do in any product and those things exist too. Not having to log in was a huge hit because it removed a ton of the friction. I think we had this intuition from the beginning, but we never got to it because we didn't have enough GPU or other constraint to really go do that. So, there's the traditional product work too. So, I often think about it as roughly a third, a third, a third, but really we're still learning and we're planning to evolve the product a ton, which is why I'm sure there's going to be new levers.

**Lenny Rachitsky** (00:31:52):
You mentioned something that I want to come back to real quick. You said that it was something like 10 days from Hackathon to Sam tweeting about ChatGPT being live?

**Nick Turley** (00:32:01):
The Hackathon happened much earlier and we were prototyping for a long time, but at some point we basically ran out of patience on trying to build something more bespoke. And again, that was mostly because people always wanted to do all this other stuff whenever we tested it. So, it was 10 days from when we decided we were going to ship to when we shipped. And the research we'd been testing for a long time, it was kind of an evolution of what we'd called instruction following, which was the idea that instead of just completing the sentence, these models could actually follow you instructions. So, if you said summarize this, it would actually do so. And the research had evolved from that into a chat format where we could do it multi-turn. So, that research took way longer than 10 days and that kind of baking in the background, but the productization of this thing was very, very fast and lots of things didn't make it in.

**Nick Turley** (00:32:50):
I remember we didn't have history, which of course was the first user feedback we got. The model had a bunch of shortcomings and it was so cool to be able to iterate on the model. The thing I just talked about, treating the model as a product was not a thing before ChatGPT because we would ship in more hardware where there'd be a release GPT-3 and then we would start working on GPT-4 and these weird giant big spend R&D projects that would take a really long time and the spec was whatever the spec was and then you'd have to wait another year. And ChatGPT really broke that down because we were able to make iterative improvements to it just like software. And really, my dream is that it would be amazing if we could just ship daily or even hourly like in software land because you could just fix stuff, et cetera. But there's of course all kinds of challenges in how you do that while keeping the personality intact while not regressing other capabilities. So, it's an open field to get there.

**Lenny Rachitsky** (00:33:42):
That's such a good example of is it maximally accelerated? Okay, we're going to ship ChatGPT 10 days.

**Nick Turley** (00:33:48):
[inaudible 00:33:48]-

**Lenny Rachitsky** (00:33:48):
Holy moly. We've been talking about ChatGPT. Clearly it's kind of a chat interface. Everyone's always wondering is chat the future of all of this stuff? Interestingly, Kevin Weil made this really profound point that has always stuck with me when he was on the podcast that chat is actually a genius interface for building on a super intelligence because it's how we interact with humans of all variety of intelligence. It scales from someone at the lower end to a super smart person. And so, it's really valuable as a way to scale this spectrum. Maybe just talk about that and is chat the long-term interface for ChatGPT, I guess it's called ChatGPT.

**Nick Turley** (00:34:27):
I feel like we should either drop the chat or drop the GPT at some point because it is a mouthful. We're stuck with the name, but no matter what we do, the product will evolve. I think that I agree that there's something profound about natural language. It just really is the most natural form of communicating to humans and therefore it feels important that you should be communicating with your software in natural language. I think that's different from chat though. I think chat was the simplest way to ship at the time. I'm baffled by how much it took off as a concept. Even more baffled by how many people have copied the paradigm rather than trying out a different way of interacting with AI. I'm still hoping that will happen. So, I think natural language is here to stay, but this idea that it has to be a turn-by-turn chat interaction I think is really limiting.

**Nick Turley** (00:35:24):
And this is one of the reasons I don't love the super system analogy, even though we used to always use it is because if you think that way, then you kind of feel like you're talking to a person and GPT-5 it's amazing at making great front-end applications. So, I don't see a reason why you wouldn't have AIs that can render their own UI in some way. And you obviously want to make that predictable and feel good. But it feels limiting to me to think of the end-all-be-all interface as a chatbot. It actually kind of feels dystopian almost where I don't want to use all my software through the proxy of some interface. I love being in Figma, I love being in Google Docs. Those are all great products to me and they're not chatbots.

**Nick Turley** (00:36:07):
So, yes on natural language, but no on chat is where I would describe my point of view. And I'm just hoping in general that we see more consumer innovation on how people interact with AI because there's so many possibilities and you just got to try stuff. That's why chat stuck is we just did it and people liked it. So, I'm hoping that we see more there and we'll try to do our part.

**Lenny Rachitsky** (00:36:31):
So, you mentioned that you kind of got stuck with this name ChatGPT. Maybe this is part of the answer, but I'm curious just are there any accidental decisions you guys made early on that have stuck and have essentially become history changing?

**Nick Turley** (00:36:45):
There's so many and it is funny, because you have no time to think about them and then they end up being super consequential. The day was one, we went from chat with GPT-3.5 to ChatGPT the night before, slightly better but still really bad.

**Lenny Rachitsky** (00:36:58):
What was it called before?

**Nick Turley** (00:36:59):
It was going to be Chat with GPT-3.5 because we really didn't think it was going to be successful product. We were trying to actually be as nerdy as we could about it because that's really what it was. It was a research demo, not a product. So, we didn't think that was bad. But I think that in the original release, making it free was a big deal. I don't think we appreciate that because the GPT-3.5 model was in our API for at least six months prior to that. I think anyone could have built something like this. It might not have been quite as good on the modeling side, but I think it would've taken off. So, making it free and putting a nice UI on it, very consequential in the way that you take for granted now. And this is why I think that A, distribution and the interface are continuously important even in 2025.

**Nick Turley** (00:37:48):
The paid business, which now it's a giant business both in the consumer space and in the enterprise space. The birth of that was just to turn away demand originally. It was not like we brainstormed, "Oh, what is the best monetization model for AI?" It was really what monetization model or what mechanism would allow us to turn away people who are less serious than the people who are really trying to use it? And subscriptions just happened to have that property and it grew into a large business. I think shipping really funky capabilities before they were polished is another thing where that feels like a tactical decision, but it became a playbook because we would learn so much. Remember when we shipped Code Interpreter, we learned so much after we shipped it. Now it's known as I think data analysis in ChatGPT or something like that just because we actually got real world use cases back that we could then optimize. So, I think there's been a lot of decisions over time that proved pretty consequential, but we made them very, very quickly as we have to, so.

**Lenny Rachitsky** (00:38:53):
The $20 a month feels like an important part of this. Feels like everybody's just doing that now and-

**Nick Turley** (00:38:57):
On that one actually, I remember I had this kind of panic attack because we really needed to launch subscriptions because at the time we were taking the product down every time. It was, I don't know if you remember, we had this fail whale, there's a little [inaudible 00:39:09] generated poem on it. So, they were like, "We had to get this out." And I remember calling up someone I greatly respect who's incredible at pricing and I was like, "What should I do?" And we talked a bunch and I just ran out of time to incorporate most of that feedback. So, what I did do is ship a Google Form to Discord with, I think the four questions you're supposed to ask on how to price something-

**Lenny Rachitsky** (00:39:32):
[inaudible 00:39:32]?

**Nick Turley** (00:39:33):
Yeah, exactly. It literally had those four questions and I remember distinctly A, you [inaudible 00:39:38] a price back and that's kind of how we got to $20. But B, the next morning, there was a press article on you won't believe the four genius questions the ChatGPT team asked to price their... It was like if only you knew. So, there's something about building in this extreme public where people interpret so much more intentionality into what you're doing than might've actually existed at the time. But we got with the $20. We're debating something slightly higher at the time. I often wonder what would've happened because so many other companies ended up copying the $20 price point. So, I'm like, "Did we erase a bunch of market cap by pressing it this way?" But ultimately I don't care because the more accessible we can make this stuff, the better. And I think this is the price point that in Western countries has been reasonable to a lot of people in terms of the value that they get back.

And most importantly, we were able to push things down to the free tier semi-regularly and we always do that when we can [inaudible 00:40:35], but-

**Lenny Rachitsky** (00:40:35):
So, the survey, just to give the official name, the Van Westendorp survey is how you guys ended up pricing ChatGPT?

**Nick Turley** (00:40:42):
It was the top Google result. This was before ChatGPT has real-time information. Otherwise, it could have maybe price itself, but it was Discord plus Google Form plus a blog post on that methodology that got us there.

**Lenny Rachitsky** (00:40:54):
That is incredible. What a fun story. This is the survey that Rahul Vohra at Superhuman popularized in his first- round article-

**Nick Turley** (00:41:00):
Yeah. Yeah, yeah, that's right. That's right. Definitely don't bring me on here as a pricing expert, I think you have got better people for that.

**Lenny Rachitsky** (00:41:08):
Whether it was right or wrong, it is now the fastest-growing, insane revenue generating business in the world. So, I wouldn't feel too bad.

**Nick Turley** (00:41:16):
No, it worked out. Yeah.

**Lenny Rachitsky** (00:41:17):
It worked out. And by the way, I'm on the $200 a month tier, so there's clearly a room-

**Nick Turley** (00:41:22):
Thank you. Thank you.

**Lenny Rachitsky** (00:41:25):
... [inaudible 00:41:25]-

**Nick Turley** (00:41:25):
The story of that one is interesting too because originally the purpose of the Plus plan was to be able to ship first uptime and then be able to ship capabilities that we couldn't scale to everyone. And at some point it got so many people in the Plus tier that had just lost that property. So, the main reason we came up with the $200 tier is just we had so much incredible research that's actually really, really powerful. Like o3 Pro or tomorrow GPT-5 Pro and just having a vehicle of shipping that to people who really, really care is exciting even though it kind of violates the standard way a SaaS page should look, it's a little jarring to see the 10X jump. So, thank you for being a subscriber on that and thank you everyone else who's watching you subscribed to any tier, it's great.

**Lenny Rachitsky** (00:42:10):
I'm just going to throw a fishing line into this pond of are there any other stories like this? You shared this incredible story of Chat with GPT-3.5 being the original name, how you came up with pricing. Is there anything else?

**Nick Turley** (00:42:22):
Enterprise is interesting one too because we've seen so much incredible adoption in the Enterprise and it's sort of objectively crazy to try to take on building a developer business and a consumer business and an enterprise business and all at once. But the story there is in like month one or two, it was very clear that most of the usage was work usage, actually much more than today where you've got so many consumers on the product and it's kind of sort of transcended into pop culture. But at the time it was writing, coding, analysis, that kind of stuff. And we were pretty quickly in organically in 90% of Fortune 500 companies in a way that I had seen maybe at Dropbox back when that was my two jobs ago where we had a similar story. And since then there's been more PLG companies. But the real reason we did Enterprise, remember we were debating should we do enterprise or should we launch an iOS app because that's how small the team was.

The reason we did is we were starting to get banned in companies because they all felt rightfully or wrongfully that the privacy and deployment story, et cetera wasn't there. So, I was just like, "Man, we have to do something. We're going to miss out on a generational opportunity to build a work product." And we've literally defined AGI as outperforming most humans at economically valuable work or I'd probably [inaudible 00:43:45] that, but I think that's the way we put it. And so, I feel like we had to be present there and it was a fairly quick decision at the time, but it's grown into an immense business. We just hit 5 million business subscribers up from 3 million, I think a month or two ago. So, it is kind of the spinoff that it's taking a life of its own that I'm really, really excited about for [inaudible 00:44:11]-

**Lenny Rachitsky** (00:44:11):
That is a lot to be handling the platform essentially the API, the consumer product, the fastest-growing, most successful product in history and also the B2B side, which is clearly a massive business. Do you have any kind of heuristics for how to make these trade-offs do all this at once and stay sane and be successful?

**Nick Turley** (00:44:30):
That's a good question. And first off, I don't run the developer stuff anymore. We found someone way more competent to do that and he's amazing. So, I still look after the various forms of chat, but luckily you don't have to make that trade-off OpenAI does. And I can get into that too, but it keeps me a little bit more sane. I will say that you kind of have to practice in two different ways when you're building on this AI stuff. One is sort of working backwards from the model capabilities and that is much more art than science, where I think you really need to look at what tech do we have available and what is the most awesome way to productize it? And if you applied to some sort of PM framework to that, I think you would do something horrible wrong. Because if you have tech that's, for example, GPT-5 is really, really good at front-end coding now, I think that means you've got to reprioritize it.

**Nick Turley** (00:45:27):
You got to actually bring that capability to life. Maybe that's making ChatGPT better at vibe coding and rendering applications. Maybe that's more like leveraging the taste of the model to make the UI more expressive. There's a number of things we could do, but you kind of have to replan and reprioritize and that is more important than any particular audience segmentation. It's really just looking at what is the magic thing we have and how do you make it shine. Voice is a similar thing. It wasn't like our customers need voice, they're begging for it or something like that. It was like, "Wow, we figured out a way how to make these things anything in, anything out." What is a creative awesome way to productize that and then we can see what people do. So, I think that's one chunk of it. But then the other chunk of it really is more like classic product management where you need to listen to customers and then when your customers are really different, that can be confusing because ChatGPT is a very general purpose product.

**Nick Turley** (00:46:23):
We see when you look at end users, there's actually an immense amount of overlap in terms of what they want. Primitives like projects or history search or sharing and collaboration, all those kinds of things. They are actually very, very present. Whether or not you're talking to people at work or you're talking to people at home, at school, there's slightly different mechanics sometimes, but they're largely similar investments that I think we can get a lot of mileage out of. And then there's Enterprise-specific work that we just have to do. You've got to do HIPAA, you got to do SOC 2, you've got to do all those things if you want to be a serious player. And those are just non-negotiable. So, it's complex as you correctly identified, but it's kind of the curse of working on a very open-ended and powerful technology.

**Nick Turley** (00:47:11):
One analogy that someone at OpenAI who I really respect, he's like, "We're kind of like Disney, where Disney has this one kind of creative IP, which is their content, and they have cruises and they have theme parks and they have comics and they have all these different things." And I think we have amazing models, but there's all these different ways that you can productize them and we kind of just have to maximize the impact in all these different ways.

**Lenny Rachitsky** (00:47:38):
As you were talking, I was thinking about how usually horizontal platforms that are just so general and can do so much take a long time to take off because people don't know what to do with them. They're not amazing at anything. And this is an amazing counter example where it took off immediately and everyone figured it out and then over time they figured it out more and more.

**Nick Turley** (00:47:54):
But I think the reason why is because it just went live. Talk about another consequential decision actually. We were debating waitlist, no waitlist because we-

**Nick Turley** (00:48:00):
Actually we were debating waitlist/no waitlist because we really knew we couldn't scale the engineering systems. And the fact that there was no waitlist, which no open AI release had worked like that before, ended up being consequential because you were able to watch what everyone else was doing live. So I think when you launch these things all at once for everyone, there really is a special moment where you can see what other people are doing and learn from that.

**Nick Turley** (00:48:25):
And a lot of that is actually out of product. There's these crazy TikTok posts that go viral and they have like 2, 000 use cases in the comments. And I go through those in detail because it's not like I knew about those use cases either. They're very, very emergent and I just go through the comments and process because there's so much to learn. And for that reason, I think we get to skip the empty box problem a little bit because so much learning is happening out of product as people are watching each other either in IRL or online.

**Lenny Rachitsky** (00:48:55):
That is so interesting because you think about Airtable, you think about Notion, all these companies, they took years to just build and craft and think and go deep on what it could be.

**Nick Turley** (00:49:04):
It's like they compare Airtable, which they had to do templates, they had to do all these kind of things of taking the horizontal product and making it use case driven. They compare it to the Instant Pot, which there's recipes being shared everywhere online. There's this whole ecosystem around it. I think we were really lucky with ChatGPT that that happened where there's just users sharing use cases with other users everywhere. And therefore I think we got very lucky by jumping ahead on that journey.

**Lenny Rachitsky** (00:49:40):
And it feels like a quarter there is Sam had big following and everyone would pay attention to something you launch. So that's a really interesting new strategy for launching horizontal product. With a huge distribution channel, just launch it and see what comes up.

**Nick Turley** (00:49:51):
Yeah. And of course I'm actually really excited to take some of that into the product. I think we shouldn't rest on the fact that there's so much out product discovery happening. I actually think for the average consumer, it would be amazing if the product did a little bit more work on really exposing to you what is possible.

**Nick Turley** (00:50:07):
I still feel like ChatGPT feels a little bit like MS-DOS, like we haven't built Windows yet. And it'll be obvious once we do, but there's something that feels a little bit like... Imagine MS-DOS had gone viral and you were just trying to hack little conversation starters onto it. That might've missed sort of the big picture in terms of how to really communicate affordances and value to people. And so I think there's actually a ton more product work to do in addition to just seeing use cases spread.

**Lenny Rachitsky** (00:50:33):
Are you able to share just what you think that might look like? This Windows version of ChatGPT?

**Nick Turley** (00:50:37):
I'll let you know when we figure it out. We're hiring. I think there's so many interesting product problems here.

**Lenny Rachitsky** (00:50:42):
Okay, got it. By the way, I also love that TikTok was like your feedback channel.

**Nick Turley** (00:50:49):
Those common threads, they're just so wild. And also the love that people have for it, the excitement with which you're sharing their product, I feel like it's special that people are so excited to share what they're doing with your product. And I don't take that for granted either.

**Lenny Rachitsky** (00:51:06):
**Nick Turley** (00:52:22):
Before I built the product team, I actually built the data science team because I was getting frustrated. I was talking to as many users as I could. And my calendar the weeks after ChatGPT, it was just 15 minute user interview the whole week through. It was usually I stopped doing interviews when I can predict what the next person's going to say. That's how I know I've talked to enough users, but it just wasn't happening. I just kept getting new stuff.

**Nick Turley** (00:52:46):
So data is one way out where I think we have conversation classifiers that without us having to look at the conversations, allow us to figure out what are people talking about, what use cases are taking off, et cetera. And I think that's very, very helpful. The quality of the stuff is important for empathy. Even though you're never going to get a rap on all the use cases people have, I still spend a huge amount of my time doing that. And then yeah, things like those TikToks, collections of threads, I think they're really, really useful. It's just fun to watch people talk to each other about the various use cases that they have.

**Lenny Rachitsky** (00:53:22):
Is there kind of a new margin use case that you're excited about or is there a really unusual use of ChatGPT that you think about that'd be fun to share?

**Nick Turley** (00:53:30):
I mentioned this earlier, but I had always conceptualized ChatGPT as a worky product, whether or not you're at home or you at work. I feel like getting help with your taxes is very similar to the types of things you do at work where planning a trip is actually very similar to planning an event for work. So I always felt like, "Okay, this thing is going to kind of be a productivity tool."

**Nick Turley** (00:53:51):
And I think something has happened, I realized, a few months where that has begun to change and I really do think the fact that you have consumers turning to this thing for day-to-day advice, helping them have better relationships... People talk about how this thing saved their marriage is really exciting to me because they use it to process their own emotions, get feedback on their communication style. They just have a buddy to talk to about really difficult things. And that comes with a ton of responsibility and work that we have to do to make those things like life advice great, but it also is really, really important to me because you can't run away from those use cases. You have to run towards them and make them awesome. And that's part of what we're trying to do. So that emergent behavior is really, really cool.

**Nick Turley** (00:54:41):
And more broadly, I'm so excited about education. I'm so excited about health. I think it would really be a waste if we didn't take the opportunity of using ChatGPT to really, really help people. And I think we've just begun to scratch the surface on that. So there's many aspirational use cases that I want to make happen.

**Lenny Rachitsky** (00:55:05):
Along those lines, an interesting use case I've recently had, I feel like it's going to be really helpful for couples that are disagreeing about something when they need a third opinion. I just had this recently where my wife's like, "You can't heat a whole thing that you're going to only eat part of in the microwave and then put it back in the fridge." It's like, "What's the problem? I'll heat it up, I'll put it back in the fridge." And she's like, "No, that's really dangerous." I'm like, "Let's ask ChatGPT." And that fact that she so trusts ChatGPT now and relies on it throughout the day, it's such a valuable third independent party that we can go to.

**Nick Turley** (00:55:35):
Yeah, yeah, totally. And a lot of those micro-interactions talk about interesting product work, right? Those are micro-interactions that are important. Did it definitively weigh in or did it help you guys think through that disagreement and solve it on your own? I think those details actually matter a lot and it's where we're spending a bunch of time.

**Lenny Rachitsky** (00:55:54):
Along those lines, there was this whole launch of the very sycophantic version of ChatGPT where it was just, " You are the best person in the world. Everything you tell me is amazingly correct." Are you able to tell us just what happened there?

**Nick Turley** (00:56:08):
Yeah, we have all kinds of collateral online because we really felt like we should over-communicate on how we discovered it, what we did about it, et cetera. So I encourage people to check that out. We have a whole retro on that model release.

**Nick Turley** (00:56:24):
But basically what happened is that we pushed out an update that made the model more likely to tell you things that sound good in the moment, "You're totally right. You should break up with your boyfriend" or something like that. That's just really dangerous. We took it more seriously than you even might expect because again, at current technology levels, you can kind of laugh about it. Maybe it's like, "Ha-ha. This thing's always complimenting me. I thought it was just me. I saw all those comments online." But it actually is really important to make sure that these models are optimized for the right things.

**Nick Turley** (00:57:01):
And we have an immense, I think, luxury to have a mission that affords us to really help people, a business model that does not incentivize maximizing engagement or time spent in the product, right? So it's really important to us that you feel like this product is helping you with your goals, whether not that's your current goals or even your long-term goals.

**Nick Turley** (00:57:25):
And oftentimes being extremely complimentary with the user isn't actually in service of that. So we instilled new measurement techniques. Whenever we put these models in contact with reality and we learn about a problem, we actually go back and make sure we have good metrics for this stuff. So we measure sick efficiency now with every release to make sure we don't regress and actually improve on that metric. GPT-5 is an improvement, which is really exciting for me, but we have more work from there.

**Nick Turley** (00:57:54):
And more broadly, it caused us to articulate our point of view. I actually spent a bunch of time on a blog post that we just published on Monday on what we're optimizing ChatGPT for. And it really is to help you thrive and achieve your goals, not to keep you in the product. And so there was a bunch of good outcomes from that incident. It's a good example of how contact for reality is not just important for the use cases, but also for learning what to avoid because you would've never discovered this issue purely in a lab unless you actually heard from physicians.

**Lenny Rachitsky** (00:58:26):
I am excited to read that blog post then. I was going to ask you this. Just like how you-

**Nick Turley** (00:58:29):
Yeah, have your feedback on it.

**Lenny Rachitsky** (00:58:31):
Yeah. I guess is there anything more there of just how you... Because this tension is so difficult, helping people feel supported, but not just letting them believe everything they want to believe. Is there anything more you can share there? Just trying to find that middle ground.

**Nick Turley** (00:58:43):
Incentives are important. There is a famous saying, "Show me the incentive and I'll show you the outcome."

**Lenny Rachitsky** (00:58:48):
Charlie Munger maybe?

**Nick Turley** (00:58:49):
Yeah, I think that's where it came from, right?

**Lenny Rachitsky** (00:58:50):
Yeah.

**Nick Turley** (00:58:52):
Yeah, I think that's very, very important. So I would take a good look at our mission, our business model, the type of product we're trying to build. And I really think that ChatGPT is a very special product because I think in vast majority of cases, it makes you leave it feeling better or not worse and feeling like you're achieving something you're trying to do. So I think that those incentives really matter because it helps you reason about, "Okay, when there isn't behavior in the wild, that's not good. Was that a bug or was that by design? And with [inaudible 00:59:29] I can very much say that to us that's a bug.

**Nick Turley** (00:59:31):
And then on the forward-looking work, there's so many kind of challenging scenarios to get right. And you could easily run away from these use cases. Like you and your wife going to this thing for input on a relationship, a question or a dispute, you could very easily run away if you were totally risk avoidant and say, " Sorry, I can't help you with that." I think that's what most tech companies do when they hit a certain scale, they run away from these use cases. And I think it's a lost opportunity to help people.

**Nick Turley** (01:00:08):
So we want to run towards these use cases by making the model behavior really, really great. That can mean connecting you with external resources when you're struggling. That can mean not directly answering your question, but instead of giving you a helpful framework in the case of like, "Should I break up with my boyfriend?" ChatGPT should probably not answer that question for you, but it should help you think through that question in the way that a thoughtful companion would. So I think it's really important to do the work because I think the upside is immense.

**Lenny Rachitsky** (01:00:37):
That is a really profound point you're making there, that if most companies, if their users want to ask them something risky like getting medical advice or, "Should I break up with my partner?" or, "what should I do with this big problem I have?"

**Nick Turley** (01:00:51):
I feel like we would have immense regret if you had a model that was state-of-the-art on health bench, which is, GPT-5 is a state of the art on a bunch of these medical benchmarks, and you didn't use that to help people, you just disabled that use case because you wanted to avoid all possible downside. I think the duty is to make it awesome and to do the work, talk to experts, figure out how good it really is, where it breaks down, communicate that. And I think this technology is too important and has too much potential positive impact on people to run away from these high stakes excuses.

**Lenny Rachitsky** (01:01:27):
And fast-forward to today, it's saving lives regularly. It's probably saving relationships regularly. Such a consequential decision, which I imagine was made early on.

**Nick Turley** (01:01:36):
Yeah. We're just at the beginning of watching how this stuff can transform people. It's incredibly democratizing. If you compare, you roll out of this with the roll out of the personal computer, computers were so scarce when they first came out. And this stuff is ubiquitous in a way where you have access to a second opinion on medical stuff, you have access to a relationship buddy, you have access to a personal tutor on literally any topic that makes you curious. It's really, really special that we get to do that. Unique point in history.

**Lenny Rachitsky** (01:02:15):
Let me zoom out a bit and talk about OpenAI and just product in general. So you've worked at traditional, let's say traditional product companies, Dropbox, Instacart. Now you're at OpenAI. What's maybe the most counterintuitive lesson you've learned by building products from your time at OpenAI?

**Nick Turley** (01:02:33):
Each time I always tried to pick the maximally different job whenever I made a job change. So after Dropbox, I was craving a real world product because it was just so different than working on SaaS, et cetera. And after Instacart, I was craving on working on something that intellectually was interesting and had this kind of invoked the nerd in me. And so I've always looked for things that are really different.

**Nick Turley** (01:02:59):
And then once I showed up at these places, I tried to understand what makes that place successful, what is truly the thing that they cracked and how we can lean in that into that even more.

**Nick Turley** (01:03:11):
I think I spent a lot of time thinking about this with OpenAI, especially after ChatGPT. Before that it was kind of a moot point because we didn't really have much revenue or products or anything like that. There's a few things that come to mind that have driven many decisions. One is the empiricism. We talked about that a bit. The fact that you can only find out by shipping, which is why maximally lean into that. And that's a huge part of why we ship so much.

**Nick Turley** (01:03:46):
One of them is that amazing ideas come from anywhere. The thing about running a research lab is you really don't tell people what to research. That's not what you do. And we inherited that culture even as we become a research and product company. So just letting people do things who have amazing ideas rather than being the gatekeeper or prioritizer of everything or something like that has been proven immensely valuable to us. And that's where much of the innovation comes from, is empowered smart people on any function really. So that was a good inheritance from what I think made OpenAI successful and makes us successful.

The interdisciplinariness of really making sure that you put research and engineering and design and product together rather than treating them as silos. I think that's the thing that has made us successful and that you see come through in every product we ship. Like if we're shipping a feature and it doesn't get 2X better as the model gets 2X smarter, it's probably not a feature we should be shipping. Not always true. SOC 2 doesn't get better with [inaudible 01:04:48] models, but I think for many of the core capabilities, that's a good litmus test.

**Nick Turley** (01:04:52):
So I've always found you really have to lean into why is this place successful and then maximally accelerate that, so to speak, because it's what allows you to turn something that feels like an accident into something that is a repeatable label.

**Lenny Rachitsky** (01:05:07):
So you talked about this kind of collaboration between researchers and product people. And you've been at the beginning of ChatGPT from day one to today, from zero to 700 million weekly active users. Not just registered users, weekly active users. How have you approached building out that team over time?

**Nick Turley** (01:05:24):
One of the other inheritances of being in a research lab is that you take recruiting really seriously. That's something that AI labs know every person matters. But many tech companies that go through hyper growth and they kind of lose their identity, they lose their talent bars, they just have chaos. So we've always had this tendency to run relatively lean.

**Nick Turley** (01:05:51):
So it is a small team that is running ChatGPT. I take co inspiration from WhatsApp where it was a very small team running a very global-scale product. And then more importantly, you have to treat hiring a little bit more like executive recruiting and less like just pure pipeline recruiting where you really need to understand what is the gap you're trying to fill on each team, what is the specific skill set and how do you fill it.

**Nick Turley** (01:06:17):
To give you an example, I'm a product person at heart, but sometimes a team doesn't need a product person because there's already someone doing that role. In many cases, we have a really talented engineering leader who has amazing product sense, or we have a researcher who has product ideas. And in my mind they can play that role. And maybe we have something else missing instead. Maybe we need a little bit more front-end or something like that.

**Nick Turley** (01:06:41):
In other cases, maybe what you're missing is incredible data scientists. So I really like to go through every single team and figure out what is the skill sets that that team needs and how do you put it together from principles rather than just assuming, "Hey, we're going to do a bunch of pipeline recruiting for all these different roles" and then people will find a team later. So I think that's always felt really important to me. And it's the way that you keep your team really small, yet super high throughput.

It also allows you to hire people who I think Keith Rabois calls us like barrels, I think. [inaudible 01:07:15] barrel's an ammunition where he thinks... I think this comes from him, but the idea being that sort of the throughput of your org depends on how many barrels you have, which is people who can make stuff happen. And then you can add ammunition around them, which is people helping those people. I think that's been really true for our recruiting too where we try to maximize the number of empowered people who can ship because that's how you have a small team and still get the ton done.

**Nick Turley** (01:07:43):
So there's a couple of things, and I spent a lot of time on vibes too with each team because I think one of the things that is challenging when you try to do research and product together is that the cultures are different. People have different backgrounds. And I think to make that go super well, you need to spend time team building and making sure that people have a huge amount of trust for each other's skill sets, feel like they can think across their boundaries. I really believe that product is everyone's job, for example. And for that reason, the recruiting doesn't stop when the people are on the door. It actually starts because you have to start making the teams awesome.

**Lenny Rachitsky** (01:08:24):
Is there something you do with team building that would be fun to share? Just like something you do to create [inaudible 01:08:28]?

**Nick Turley** (01:08:28):
I just love whiteboarding with teams. I just love getting into a generative mindset. It breaks down everything. So that's the thing that I try. It's not particularly creative, but I found it to be a universal tool where the minute you can get people to stop thinking about what's my job versus the other person's job and more like we're all in a room trying to crack something together, that is incredible.

**Lenny Rachitsky** (01:08:50):
You mentioned this idea of first principles. This came up actually when I talk to a lot of people about you, is this something you're really big on. A lot of people talk about first principles, most people are like, " I don't really understand," or they think they're amazing at thinking from first principles. Is there something you can share of just what it actually looks like to think from first principles as maybe an example that comes to mind where you really went to first principles and came up with something unexpected?

**Nick Turley** (01:09:15):
Yeah, this is not something I'd ever say about myself. It's nice that someone else would say it, but it's a mysterious thing. Yeah, I think you just really got to get to ground truth on what you're really trying to solve. For example, as I mentioned with the recruiting thing, I'm not dogmatic that you have to have a product manager and an engineering manager and a designer or whatever. We're just trying to make an awesome team that can ship. So in that case, first principles means just really understanding what we actually need and what we're missing rather than applying a previously learned process or behavior. So I think that's a good example.

**Nick Turley** (01:09:54):
Another good example of I think being first principles in this environment is, does this feature need to be polished? We get a lot of crap for the model chooser, and I own it. I've tried to say that to everyone who will listen. For those who don't know model chooser, it's this giant drop down in the product that is literally the anti-pattern of any good product traditionally.

**Nick Turley** (01:10:16):
But if you are actually recent from scratch, is it better to wait until you got a polished product or to ship out something raw even if it makes less sense and start learning and getting into people's hands? I think a company with a lot of process or a lot of just learned behaviors will make one call, which is, we have a quality bar when we ship, and that's what we do. If your first principle is about it, I think you're like, "You know what? We should ship. It's embarrassing, but that's strictly less bad than not getting the feedback you wanted."

**Nick Turley** (01:10:51):
So I think just approaching each scenario from scratch is so important in this space because there is no analogy for what we're building. You can't copy an existing thing. There is no, "Are we an Instagram or are we a Google or a productivity tool or something like that?" I don't know. But you can learn from everywhere, but you have to do it from scratch. And I think that's why that trait tends to make someone effective at OpenAI, and it's something we test for in our interviews too.

**Lenny Rachitsky** (01:11:23):
So this theme keeps coming up, and I think it's just important to highlight something that you keep coming back to, which is this trade-off of speed and polish and how in this space, speed is more important, not just to stay ahead, but to learn what the hell people actually want to do with this thing. Is there anything more that you think people just may be missing about why they need to move so fast in the space of AI?

**Nick Turley** (01:11:46):
Yeah. I mean, the boring answer would be, oh, it's competitive and everyone's in AI and they're trying to compete each other. I think that's maybe true, but that's not the reason that I believe this. The reason really is that you're going to be polishing the wrong things in the space. You absolutely should polish-

**Nick Turley** (01:12:00):
You're going to be polishing the wrong things in this space. You absolutely should polish things like the model output, et cetera, but you won't know what to polish until after you ship. And I think that is uniquely true in an environment where the properties of your product are emergent and not knowable in advance. And I think that many people get that wrong because they think the best product people tend to be craftspeople and they have a traditional definition of craft. I also think it would be easy to use all what I just said as an excuse not to eventually build a great product. So I often tell my teams that shipping is just one point on the journey towards awesomeness, and you should pick that point intentionally where it doesn't have to be the end of your iteration at all. It can be the beginning, but you better follow through.

**Nick Turley** (01:12:50):
So we've been doing a bunch of work, especially over the last quarter of really cleaning up the UI of ChatGPT. I'm really excited to do the same for the sort of the response layouts and formats next. Simply because once you know what people are doing, there's no excuse to not polish your product. It's just really, in a world where you don't know yet, you might get very distracted.

**Nick Turley** (01:13:09):
So it's situational. Again, you kind of have to be first principles about it. But I do think using velocity, especially early on, as a tool... Actually this has been said about consumer social for example. It is not the first space where people have said, "Hey, you just got to try 10 things because you're probably going to be wrong." So I don't think this has never existed before as a dynamic either, but I do think with AI, it's important to internalize.

**Lenny Rachitsky** (01:13:32):
And there's also an element of the models are changing constantly and so you may not even realize what they're capable of, I imagine.

**Nick Turley** (01:13:38):
Totally. The models are changing and the best way to improve them, whether or not you're a lab or actually just someone who's doing context engineering or fine-tuning a model maybe, you need failure cases, real failure cases, to make these things better. The benchmarks are increasingly saturated. So really you need real-world scenarios where your product or model is not actually doing the thing it was supposed to do, and the only way you get that is by shipping, because you get back to use case distribution and you can make those things good. And therefore, it's actually the best way to then go articulate to your team, especially your ML teams, what [inaudible 01:14:17] climb on? It's like, "Oh, people are trying to do X and the model's failing in ways. Why? Now let's make those things really good."

**Lenny Rachitsky** (01:14:23):
This point about failure cases makes me think about something that both Kevin Weil and Mike Krieger shared, which is that evals are becoming a huge new skill that product people need to get good at because so much of product building is now writing evals. Is there something there you want to share?

**Nick Turley** (01:14:41):
My entire OpenAI journey has been this journey of rediscovering eternal product wisdom and principles in like slightly new contexts. So I remember I started writing evals before I knew what an eval was because I was just outlining very clearly specified ideal behavior for various use cases until someone told me, "Hey, you should make an eval." And I realized there was this entire world of research evaluation benchmarks that had nothing to do with the product that I was trying to make. And I was like, "Wow, this might be the lingua franca of how to communicate what the product should be doing to people who do AI research." And that really clicked for me.

**Nick Turley** (01:15:23):
And at the end of the day, it's not that different from the wisdom of, you ought to articulate success before you do anything else. It's just a new mechanism for doing that. But you can do it in a spreadsheet, you do it anywhere, and I really wanted to mystify it for people who hear that term. It's not some technical magic that you have to understand. It's really just about articulating success in a way that is maximally useful for training bots.

**Lenny Rachitsky** (01:15:50):
Awesome. I have a post coming out soon that gives you a very good how-to for PMs have how to write evals.

**Nick Turley** (01:15:56):
I would love to read it. And I hope you agree with what I just said because maybe there's [inaudible 01:16:02] to it.

**Lenny Rachitsky** (01:16:02):
Absolutely. Absolutely. And now there's all these tools that make this easier for you.

**Nick Turley** (01:16:04):
Totally.

**Lenny Rachitsky** (01:16:04):
Okay, so this basically backs up this point that this is just a very important skill that product teams and builders need to get good at.

**Nick Turley** (01:16:12):
Yeah. Yeah.

**Lenny Rachitsky** (01:16:13):
Okay. Just a few more questions. I know you have a lot going on today. One is that this trend of ChatGPT being a big driver of growth for traffic to sites, for products. For example, ChatGPT is now driving more traffic to my newsletter than Twitter, which completely shocked me. I just was looking at my stats, I'm like, "What the hell? This is not something I knew was coming." So just I guess thoughts on the future of this, how you think about just ChatGPT driving growth and traffic to products and sites?

**Nick Turley** (01:16:48):
I'm really excited about it because in the same way that I find it dystopian to talk to everything through a chat bot, I also find it dystopian to not have amazing new high quality content out there. And for that reason, I talked a little bit earlier about search and have that solved a really important user problem early on because you had this knowledge cutoff thing and you suddenly could talk about anything. Very obvious in retrospect. A, it wasn't just a user problem, it was an ecosystem problem where the original ChatGPT, it didn't have outlinks, it would just answer your question, it would keep you in the product. And even if you wanted to keep reading or go deeper, there was no way for us to drive traffic back to the content ecosystem. And I've been really excited about what we've been doing in search, not just because it gives people more accurate answers, but because it allows us to surface really high quality content, like this podcast, to people who want to see it.

**Nick Turley** (01:17:47):
And of course there's so many interesting questions about, well in the Google era, there was the search engine optimization and there was clearly understood mechanisms of how to show up and get more traffic. So I get a lot of questions from people, like, "What is the equivalent of that? The AI era, if I'm Lenny and I want to 10X the traffic to my podcast, what do I actually need to do?" And the truth is we don't have amazing answers there simply because the way to appeal to an AI model ideally is the same way that you would appeal to a real user, because the model's supposed to proxy the interest of the user and nothing else. At least that's how I want our product to work. And for that reason, my advice is super lame, which is make really high quality content, which is not as actionable as I think people making content would ideally like. And I think this is why we have more work to do because maybe there's a better mechanism or protocol that we could come up with.

**Nick Turley** (01:18:42):
But I'm excited this is driving meaningful traffic for you, and I hope that other people making great content start to feel this way because, again, it's a very new scenario.

**Lenny Rachitsky** (01:18:52):
There's two acronyms people have been using for this specific skill of AI driven SEO. I think one is AEO, which is answer engine optimization. The other is GEO. I forget the G one.

**Nick Turley** (01:19:04):
Generative... Yeah, I don't know.

**Lenny Rachitsky** (01:19:06):
Generative, yeah, AI optimization.

**Nick Turley** (01:19:08):
Yeah.

**Lenny Rachitsky** (01:19:08):
Do you have a favorite of those two? [inaudible 01:19:10]-

**Nick Turley** (01:19:10):
No, no. I try to shy away from these terms unless they become inevitable just because I'm not entirely sure yet if that should be a concept or not. Again, I think ideally, ChatGPT understands your goals and therefore understands what content would be interesting to you. And the content creator's job is to share enough information and metadata about that content such that the AI model can make a user-aligned decision. And therefore, I'm not sure if giving this thing a name and making a thing is what we should be doing or not. I'm very eager to learn from folks making content about what this could look like because. Again, we're still working through.

**Lenny Rachitsky** (01:19:59):
Along these lines, another question people think about is you have GPTs, which are kind of these custom GPT apps that you can build to answer very specific use cases. There's always this question of, you're going to build an app store where I can plug in my product into ChatGPT, monetize that. Is there stuff there that you could talk about that might be coming someday?

**Nick Turley** (01:20:19):
GPTs are cool. They're kind of ahead of their time in the sense that we built that kind of concept before you could really build very differentiated things. At least in the consumer space, your learning GPT is going to be pretty similar to what the model could already do out of the box. So it's mainly a way of articulating a use case to people, but it doesn't have enough tools yet to make something that feels like an app, so to speak.

**Nick Turley** (01:20:47):
Different in the enterprise by the way. We're seeing a ton of adoption of GPTs there because just every single company has very bespoke business processes and problems, etc. And it's a really, really useful tool there. They also have unique data that they can hook up to these things that it can retrieve over. So we've seen a lot of success there.

**Nick Turley** (01:21:05):
I think the idea is the right one, and I think we're going to figure out a good mechanism for it. Because when you have so much capability packed into AI, it feels really powerful to allow people to package that up in ways that have a clear affordance, a clear use case, and are differentiated from each other. I also would love it if you could start a business on ChatGPT. I think there really is a world where, as this thing hits a billion user scale, it can get you distribution, it can get you started on making something in the same way that people built on the internet and there was entirely new businesses to be built.

**Nick Turley** (01:21:41):
So I think we'll have more to share there in the future. GPT's was an early stab. And I'm just excited to evolve the thinking there as the models get good and our reach increases as well.

**Lenny Rachitsky** (01:21:51):
Amazing. That is really cool. I'm really excited to see what you guys do there. Okay. Completely different direction. Something that I know about you is you studied philosophy in college.

**Nick Turley** (01:22:02):
I did.

**Lenny Rachitsky** (01:22:02):
Computer science and philosophy, right? A combo.

**Nick Turley** (01:22:05):
Yeah. I started as a philosophy major and took one coding class because I really liked logic, and programming was most similar to that. And then I fell in love with coding and then eventually computer science, and I just kept doing more and more of it. But until then, I'd never really thought of myself as a technical person, so it was kind of a late discovery in my life that I'm very grateful for.

**Lenny Rachitsky** (01:22:27):
What an incredible combination for someone leading this product [inaudible 01:22:30].

**Nick Turley** (01:22:30):
It's true. It is really coming in full circle in a way that I couldn't have predicted. The amount of questions you have to grapple with are truly super interesting. And philosophy, it's not a traditionally practical skill, but it does really teach you to think things through from scratch and to articulate a point of view, and I think that has come in handy numerous times.

**Lenny Rachitsky** (01:22:51):
Is there a specific philosopher or school that has been most handy to you, or is there more just the general [inaudible 01:22:57]?

**Nick Turley** (01:22:56):
Oh, there's so many.

**Lenny Rachitsky** (01:22:56):
okay.

**Nick Turley** (01:22:57):
I wrote my senior thesis on whether and why rational people can disagree, which also comes in handy when a lot of people with very different values have opinions on your model behavior or on how things should work. So I really like 20th century analytical philosophers. It's kind of dirty stuff, but I don't know if I have a favorite. It's too many to count. But that's the kind of stuff I like. And some of it ends up being quite analytical. You have let P be this theory of love and let Q be this other theory of love, and then you do some sort of symbolic manipulation. So it is just as much a brain thought exercise as it is... Or it's much more that than practical, but it taught me how to think in a way that continues to be pretty valuable.

**Lenny Rachitsky** (01:23:48):
Incredible. What a cool combo of skills and background. Last question before we get to very exciting lightning round. So you were a product leader at Dropbox, then Instacart, now you're the PM of arguably the most consequential product in history. How did you land in this role? What was the story of joining OpenAI and taking on this work?

**Nick Turley** (01:24:10):
Every single career decisions I ever made, including my first one out of college, was just figuring out who are the smartest people I know that I want to hang out with and learn from, and can I work with them? And I don't know how to vet companies, I don't know how to really logically think through what space is going to take off or something like that, but I just do feel like I have a sense on people. And for Dropbox, I followed the head teaching assistant for a class that I was TA-ing. And for Instacart, I followed some of the smartest product people I knew. And for OpenAI, the person who recruited me, Joanne, I had messaged her about getting off the DALL·E waitlist and she said, "Only if you interview here." So she turned it into a reverse recruiting thing.

**Nick Turley** (01:25:02):
And initially, honestly, I didn't know what I would do here because it was a research lab and I was a product person and they said, "Don't worry, we'll figure it out." And they were sort of being cagey. And I thought they were being cagey because it's OpenAI and they can't share anything, but they were being cagey because we actually just didn't know yet at the time. So I showed up and I did everything under the sun and it definitely wasn't product. It was like, I think my first task was fix the blinds or something like that. And then I started sending out NDAs for people because they needed some operational help. And then I started asking, "Wait, why am I sending out NDAs? Oh, so we could talk to users." And I was like, "Talking to users, that sounds like the thing I know how to do." And I quickly stumbled into doing product work, and then eventually leading a bunch of product work. But it was organic by just showing up and doing what had to be done because, again, the company I joined was not a product company by any.

**Lenny Rachitsky** (01:26:00):
Wow. This is such a good example of, I don't know if you think of it this way, but when someone offers you a seat on a rocket ship, don't ask which seat. [inaudible 01:26:07].

**Nick Turley** (01:26:08):
Yeah, so I didn't know it was a rocket ship. I kind of got nerd sniped is what I would describe it as. Where as I prepared for the conversation to get off the DALL·E waitlist really, I just started reading about the space and that piqued the philosophy brain and then also actually the computer science brain. I was like, "Wait, this is cool." Then I started reading all the academic papers of that era. So it was intellectual itch and the people, but then I stayed for the product opportunity, obviously. Post ChatGPT, when that took off, realized that we'd built a rocket ship where we'd launched it while building it, maybe is the analogy. But I can't say that it felt like a hyped job or anything like that when I applied.

**Lenny Rachitsky** (01:27:00):
So a lesson there is, as you said, follow the smartest people you know. There's also just this thread of follow things that are interesting to you. Just you playing with DALL·E led to this opportunity.

**Nick Turley** (01:27:10):
Yeah, yeah. And actually that's something we still test for is curiosity is an attribute that we think matters so much more than your ML knowledge. I'm not making a comment on research hiring. I think you do need some ML knowledge, I'm afraid. But for product and engineering and design people, and those kinds of functions, I actually think that if you are just curious about the stuff works, it doesn't matter at all if you've never done it before. In fact, if you were to filter for people who've done it before, you would have a very narrow filter of very lucky people rather than necessarily the best people you can get. So I think we've scaled that. Certainly what got me here, but I think it's actually, just generically, been a good predictor of success at OpenAI.

**Lenny Rachitsky** (01:27:50):
Nick, I told you I had a billion... I said I had 2 billion questions to ask you. I feel like I've asked a lot. I feel like I still have a billion left. But I know, you told me right after this you, have a big GPT- 5 check-in that you got to get to. So-

**Nick Turley** (01:28:01):
We got to ship.

**Lenny Rachitsky** (01:28:03):
We got to ship. Better ship now that this is recorded and we're putting this out.

**Nick Turley** (01:28:08):
This is true. [inaudible 01:28:08].

**Lenny Rachitsky** (01:28:09):
This is the forcing function. Okay, so before we get to a very exciting lightning round, is there anything else that you want to share, leave listeners with, think is important to share?

**Nick Turley** (01:28:20):
I try to share a little bit about how I made decisions because I hope to... I'm not that far out of school. I relate a lot to people who are coming in the job market, who are trying to figure out what to do with their life right now. And I feel very confident that if you surround yourself with people that give you energy and if you follow the things you're actually curious about, that you're going to be successful in this era. So my parting advice to folks really is put yourself around good people and do the things you're actually passionate about. Because in a world where this thing can answer any question, asking the right question is very, very important. And the only way to learn how to do that is to nurture your own curiosity. So it worked for me and it's the one repeatable thing that I can share. Everything else is luck.

**Lenny Rachitsky** (01:29:15):
This is counter to what a lot of people are doing right now, which is follow the money. Where can I make the most? How do I grow this thing and make $100 million? All these people that are getting these crazy offers were not planning to make a lot of money doing this.

**Nick Turley** (01:29:27):
It's quite interesting to see that stuff play out because I think all these people entered school for genuine reasons. They were excited about the space, they were researching it, they were pursuing knowledge, and I'm happy that that's being rewarded. And I don't know what the rewards will look like in the future, especially in a post-AGI world. But I just a feeling that if you follow that advice, you'll end up okay.

**Lenny Rachitsky** (01:29:54):
With that, Nick, we've reached our very exciting lightning round. I've got five questions for you. Are you ready?

**Nick Turley** (01:29:59):
Sure, yeah.

**Lenny Rachitsky** (01:30:00):
What are two or three books that you find yourself recommending most to other people?

**Nick Turley** (01:30:04):
In the product space, probably things like High Output Management or The Design of Everyday Things, or those kind of classic type things because I think they're extremely applicable in AI.

**Lenny Rachitsky** (01:30:13):
We talked about philosophy. I don't know, is there a philosophy book you're like, "Here's the one to read if you're getting into this."

**Nick Turley** (01:30:17):
Oh man. Anything by Rawls and Nozick. I like the political stuff. I think it's really fun. That is a type of thing I recommend. I don't think there's a practical reason to read that stuff, but I will nerd out about it with you. So at your own peril.

**Lenny Rachitsky** (01:30:32):
Do you have a favorite recent movie or TV show you've really enjoyed? If you've had time to watch anything.

**Nick Turley** (01:30:36):
I think you've got to do a little bit of sci-fi to be in this space. You shouldn't copy any of it, but I think you learn from it. So regularly re-watch Her and Westworld. Severance was great. I think that's the stuff that, when I have time, I'll meddle with.

**Lenny Rachitsky** (01:30:56):
That is awesome. I love that those are the two. Of all the sci-fi movies, those are the ones you resonate most with and find most interesting and valuable.

**Nick Turley** (01:31:03):
Yes, but that's probably my own limitation, so I'm sure there's more to discover.

**Lenny Rachitsky** (01:31:08):
By the way, have you read Fire Upon the Deep, that sci-fi book?

**Nick Turley** (01:31:08):
No.

**Lenny Rachitsky** (01:31:13):
Okay. I don't know if you have time to read this book, but I think you would love it. It's such a good-

**Nick Turley** (01:31:16):
Oh, man. Okay.

**Lenny Rachitsky** (01:31:17):
... AI oriented sci-fi space opera sort of book.

**Nick Turley** (01:31:20):
Great.

**Lenny Rachitsky** (01:31:21):
Yeah.

**Nick Turley** (01:31:22):
I'll check it out, thank you.

**Lenny Rachitsky** (01:31:22):
Okay. Off tangent.

**Nick Turley** (01:31:22):
Yeah, yeah, yeah. For sure.

**Lenny Rachitsky** (01:31:26):
Okay. Do you have a favorite product you've recently discovered that you really love?

**Nick Turley** (01:31:29):
I actually don't. I am at extreme capacity. It's kind of interesting. API developers ask me like, "Hey, are you going to copy all of our products?" It's like, I actually just do not have time to follow up what's going on outside of OpenAI because the pace here is so, so intense. So don't have good recs for you, I'm afraid.

**Lenny Rachitsky** (01:31:54):
That's a comforting answer, I think, to a lot of product companies. Go figure. Nick has no time to even listen to our stuff. Oh man. Okay. Do you have a favorite life motto that you find yourself using when things are tough, sharing with friends or family that other people find useful?

**Nick Turley** (01:32:10):
Being the average of the five people you spend the most time with is a thing that I really internalize, both in my personal life, where there's people who give me energy and who lift me up and make me a better person. My fiance is one of those people, but there's many people in my life. But then there's also just, at work, there's the equivalent. And again, that's how I've made all the career decisions. It's like who do I want to learn from? So I apply that principle constantly.

**Lenny Rachitsky** (01:32:36):
Final question, everybody I talked to told me that you are a very good jazz pianist. You have won competitions. I think you were planning to do this as your main thing and then you somehow took the side quest.

**Nick Turley** (01:32:47):
Yeah, I chickened out that at the very last minute, but I was going to go to school for music. And that's still my, hopefully, chapter two.

**Lenny Rachitsky** (01:32:55):
Wow. I love that that might still happen.

**Nick Turley** (01:32:58):
Might still happen. Now I'm in some for fun bands and we will kick from time to time. It's like the one thing I can do when I'm otherwise super tired and can't think anymore because it balances me out in good ways. But yeah, hopefully I'll get to do more of it in the future.

**Lenny Rachitsky** (01:33:16):
Is there any analogs between music and your job? Anything that you find-

**Nick Turley** (01:33:20):
Yeah, actually. I feel like you could think of software development as, or being a product person, as you could be a conductor of an orchestra or you could be in a jazz band. And I think of it as a jazz band where I don't believe in the idea of everyone having this set part that they have to play and me kind of telling people when to play. I love how in jazz, or other forms of improvised music, you're kind of riffing off of each other and you listen to what one person played and then you play something back. And I think that great product development is like that, in the sense that ideas could come from anywhere. It shouldn't be a scripted process. You should be trying stuff out, having fun, having play in what you do. So I use that analogy a lot. For those who like music, it tends to resonate.

**Lenny Rachitsky** (01:34:13):
Nick, I am so thankful that you made time for this. I know today is insane. Tomorrow's going to be even more insane for the entire world. They have no idea what's coming. Thank you so much for doing this. Two final questions. Where can folks find you if you want them to find you online? Where can folks find GPT-5 potentially. And then just how can listeners be useful to you?

**Nick Turley** (01:34:31):
Just use the product. You don't even have to pay. Should be your default model starting tomorrow and just use it and don't think about models anymore. Unless you want to and you're a Pro user, in which case you get all the old models. So rest assured. And useful, honestly, I learned so much from people at large and ChatGPT users, et cetera, so just keep doing your thing. I am watching and learning, and I appreciate all the feedback. So I'm sure after we fix the model chooser, you guys will roast me for something else and I'll take it. So keep it coming.

**Lenny Rachitsky** (01:35:05):
Amazing. Nick, thank you so much for being here.

**Nick Turley** (01:35:08):
Thanks for having me, Lenny.

**Lenny Rachitsky** (01:35:09):
And good luck tomorrow.

**Nick Turley** (01:35:10):
Thanks.

**Lenny Rachitsky** (01:35:11):
Bye everyone.

**Lenny Rachitsky** (01:35:13):
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lennyspodcast.com. See you in the next episode.

---

## [8/18] The one question that saves product careers | Matt LeMay

**Matt LeMay** (00:00:00):
More product managers and teams are getting laid off. The problem is the message that Daniel Ek from Spotify sent out with their layoffs in 2024, we still have too many teams doing work around the work.

**Lenny Rachitsky** (00:00:11):
Even if you are told to build the thing that the execs are really excited about, you're still going to get fired eventually.

**Matt LeMay** (00:00:15):
If you were the CEO of this company, would you fully fund your own team? Frankly, most of the people I ask that question to don't know the answer right away.

**Lenny Rachitsky** (00:00:24):
Which is something you call the low impact PM Death Spiral.

**Matt LeMay** (00:00:26):
Is the dynamic in which every medium to large company I've ever worked with finds itself in, one way or another.

**Matt LeMay** (00:00:33):
It starts with adding little features here and there, making little cosmetic improvements until the next round of layoffs.

**Lenny Rachitsky** (00:00:39):
You have three steps to become more of an impact first product team.

**Matt LeMay** (00:00:42):
So the first is in setting team goals, no more than one step away from company goals. Don't let it get cascaded into oblivion.

**Lenny Rachitsky** (00:00:50):
You're an ICPM. It's up to you. No excuses.

**Matt LeMay** (00:00:52):
You can follow all the best practices, but if your company goes out of business, they're not going to keep writing your paycheck for two years because all of your OKRs were a 0. 6 or a 0.7.

**Lenny Rachitsky** (00:01:02):
Today my guest is Matt Lameé. Matt is a longtime product leader, author of one of the most popular and practical books in the field of product management called Product Management in Practice.

**Lenny Rachitsky** (00:01:12):
Over the course of his consulting practice, he's worked with hundreds of product teams, helping them improve how they operate and drive more impact, more consistently.

**Lenny Rachitsky** (00:01:20):
From that experience, he wrote and recently published a new book called Impact First Product Teams, that I could not agree more with.

**Lenny Rachitsky** (00:01:26):
In our conversation, Matt shares why it is so essential to align all of your work with business critical outcomes, especially if you fear layoffs at your company.

**Lenny Rachitsky** (00:01:35):
We talk about the low impact death spiral that many product teams fall into. What steps an individual product team can take to align their work to business critical outcomes, regardless of how their organization approaches product development. Tips for how to push back on stupid ideas that execs ask you to build, and so much more.

**Lenny Rachitsky** (00:01:52):
The message in this episode is one that I believe every product manager needs to hear, especially if you don't work at a high-flying Silicon Valley tech company.

**Lenny Rachitsky** (00:02:00):
A huge thank you to Martin Erickson, Adrian Joselow, and Dan Corbin for suggesting topics and questions for this conversation.

**Lenny Rachitsky** (00:02:07):
If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. Also, if you become an annual subscriber of my newsletter, you get a year free of a bunch of incredible products, including Replit, Lovable, Bolt, NNN, Linear Superhuman, Descript, Wispr Flow, Gamma Perplexity, Warp, Granola, Magic Patterns, Raycast, JetPRD, Mobbin, and more. Check it out at Lenny'snewsletter.com and click bundle.

**Matt LeMay** (00:04:29):
Thank you so much for having me. I'm really, really excited to be here.

**Lenny Rachitsky** (00:04:32):
I'm even more excited. You have a really interesting background. I usually don't spend time on background, but I thought this would be fun.

**Lenny Rachitsky** (00:04:38):
I was looking at your LinkedIn and your background. The beginning of your career you spent 13 years at Pitchfork reviewing music.

**Matt LeMay** (00:04:46):
Yes.

**Lenny Rachitsky** (00:04:46):
Reviewing artists. Pitchfork was a massive deal. I don't know where it is today, but it was the most influential music review site. Is it still a big deal or is it less so?

**Matt LeMay** (00:04:55):
It is still a big deal. It was much less of a big deal when I started working for them and I was a 16-year-old music nerd, writing record reviews in my bedroom at my parents' apartment.

**Lenny Rachitsky** (00:05:06):
Okay, so what I'm hearing is you helped make them a big deal. Excellent. That's impacts. We're going to talk all about impact.

**Matt LeMay** (00:05:11):
If I were better at telling my own story, that's exactly what I'd be saying, is that before I started, I was product manager number one at Pitchfork.

**Lenny Rachitsky** (00:05:17):
Funny enough, you look at your resume, you went into product management from Pitchfork. Very typical career path.

**Lenny Rachitsky** (00:05:23):
By the way, if you're on YouTube watching, there's a lot of musical paraphernalia behind you, an awesome guitar, so clearly there's a thread of music throughout your background.

**Matt LeMay** (00:05:31):
Yes.

**Lenny Rachitsky** (00:05:31):
Let me ask you this question: what is most similar about making music, and even critiquing music, and building great product?

**Matt LeMay** (00:05:39):
The magic lies in the way people work together. That's really what I think has been the consistent thread between all the work that I've done.

**Matt LeMay** (00:05:47):
I spent years as a touring musician. I still occasionally tour in my friend Will Chef's band. And it's that magic of people with different perspectives, different ideas, building into something that is somehow greater than the sum of its individual parts and perspectives.

**Matt LeMay** (00:06:03):
That's what makes this interesting. That's what makes music magical. That's what makes product development interesting, especially in the age of AI.

**Matt LeMay** (00:06:13):
When people have the opportunity to, I think, close themselves off from the messier parts of human interaction, from those moments where you realize that your perspective might be limited, that somebody else might be able to expose you to something, show you a way of working, show you a way of seeing that you haven't seen before, I think that those interactions, that ability to learn from and build with each other, is only going to become more valuable and more precious, frankly, as technology continues to do what technology does.

**Lenny Rachitsky** (00:06:44):
Wow, what a beautiful answer.

**Matt LeMay** (00:06:45):
Thank you.

**Lenny Rachitsky** (00:06:47):
I love that. So let's get to the task at hand.

**Matt LeMay** (00:06:50):
Yes.

**Lenny Rachitsky** (00:06:51):
So you're known for writing one of the most popular books in the field of product management called Product Management in Practice.

**Matt LeMay** (00:06:58):
I suppose I am.

**Lenny Rachitsky** (00:06:59):
Yes. Basically describes the job of a product manager very practically, very specifically, better than any other book out there.

**Lenny Rachitsky** (00:07:09):
You have a new book out called Impact First Product Teams. I asked you what the goal of this book was and you said the goal was what are the steps that individual product teams can take to align their work to business critical outcomes, regardless of how their organizations approach product development?

**Lenny Rachitsky** (00:07:25):
And I thought this was a really nice way of framing this conversation. There's two important parts to the sentence that you gave me.

**Lenny Rachitsky** (00:07:32):
One is aligning your work to business critical outcomes, and the other is steps an individual product team can take, regardless of other business functions.

So let me start with this first part. This is kind of you named your book after this concept of impact, first: Why is aligning work to impact and to business critical outcomes so important?

**Matt LeMay** (00:07:51):
It's important because at the end of the day, it is those business critical outcomes against which you and your team will be evaluated. That's the reality of working for a business, right?

**Matt LeMay** (00:08:02):
If you are contributing to the business in a way that the business at large can understand, that the CEO can understand, that the CFO can understand, if your team is a good investment for the business, then the business will continue making that investment.

**Matt LeMay** (00:08:16):
If your team is not a good investment for the business or you don't really know if your team is a good investment for the business, but you figure you're showing up, so probably it's fine, you guess, then that puts you in a really tenuous position, and a position that I think we've all seen not work out terribly well for everybody in recent history.

**Lenny Rachitsky** (00:08:36):
And so, what is this a reaction to? Because I imagine many people hearing this are, like, "Of course, PMs, product teams we're driving business impact, well, that's why we're here."

**Lenny Rachitsky** (00:08:44):
That's not actually the case in most cases. So first of all, just what is this reaction to? Is this the reaction to people being, like, "Oh, we're just going to build great products, we're going to listen to customers,"? Talk about the flip side of this.

**Matt LeMay** (00:08:55):
I mean, there's a couple of things. I think the main one is just more product managers and teams are getting laid off. That's a really scary reality right now.

**Matt LeMay** (00:09:05):
And if you look at the messages that are being conveyed by CEOs when they lay off these teams, some of them are pretty clear.

**Matt LeMay** (00:09:13):
The message that Daniel Ek from Spotify sent out with their layoffs in 2024 said, "We still have too many teams doing work around the work and supporting work, rather than focusing on opportunities with real impact."

**Matt LeMay** (00:09:25):
And that really struck a nerve because I've been on those supporting teams before. I've done the work around the work. And I've assumed that if I was given work around the work to do, that surely it must be critical to the business, otherwise they wouldn't have hired me to do it.

**Matt LeMay** (00:09:40):
I don't think that is a safe assumption to make. I think especially given that we are no longer in a zero interest rate environment, given that hiring and retaining employees is expensive, given that companies are looking at cost savings, I think we are in a moment where it is really incumbent upon each product team and each member of each product team to be able to understand, articulate, and work towards the top line business impact of their own work.

**Lenny Rachitsky** (00:10:12):
Awesome. So where this comes from is, one, if you are not driving something that is directly driving business growth, you're not as valuable to the company. You're the kind of person that they'd lay off.

**Matt LeMay** (00:10:12):
Yeah.

**Lenny Rachitsky** (00:10:23):
There's also a flip side of finding a job. It feels like if you have shown impact in your resume, "Here's all the impact I've driven," this also helps you get hired. True?

**Matt LeMay** (00:10:31):
Absolutely. Yeah. I mean, it's the most common resume advice I see, is show the number, show the impact, say what you did.

**Matt LeMay** (00:10:40):
I have a friend who does resume coaching, and I showed her my resume a couple of years ago. And she said, "I love your resume except for one thing. Why do you write like a little girl? Say what you did, say what your contributions were. Don't say 'I helped. I may have maybe helped people do this'. Say the impact. Put the number on it." And I think I needed to hear that from her because I think there is a tendency, especially for those of us who do really thrive in that collaborative environment, who our nature is to not want to take credit for work that could only exist via the contributions of many people with diverse experiences and perspectives.

**Matt LeMay** (00:11:22):
It's really hard to say, "Well, I did this and I contributed to that," but that's part of why I wrote this book at the team level, because I think for folks, like myself, who really enjoy and thrive in that team environment, if you can look at the product team as the foundational unit of impact and say, "We delivered this. We were able to work together to have this impact for the business," that feels really good and also speaks directly to the work that that team did together.

**Lenny Rachitsky** (00:11:49):
That's a really interesting insight. I think a lot of this comes from just PMs naturally are trained to deflect credit and to include, say 'we' in everything and just this person did that. So I get where that comes from. So most listeners will probably still be thinking, "Yes, I'm driving impact, I'm great. I don't have a problem here. I don't to change, need to change anything." What are some ways to stress test your thinking? Maybe some questions to ask yourself or maybe your team of just, like, "Okay, maybe I'm not?"

**Matt LeMay** (00:12:17):
So the first question I ask most of the teams I work with is if you were the CEO of this company, would you fully fund your own team?

**Matt LeMay** (00:12:29):
And that is a small question that can provoke some big reactions. Because, for me, when I was working as a product manager, I truly believed that my job was to find the next most defensible thing to build, build it, celebrate it, find the next most defensible thing to build, and so on and so forth. Rinse and repeat forever and ever.

**Matt LeMay** (00:12:53):
I kind of assumed that the existence of my team was this righteous fact of the universe, right? My team would always be there for all time, and it was just a matter of us finding the next thing to work on, which meant doing discovery and making concessions to what leadership wanted and doing all that stakeholder management.

**Matt LeMay** (00:13:13):
But the idea that my team came at a cost to the business, that at some point somebody might look at my team on a spreadsheet and say, "Who are these very expensive people? Why do we need them? Do we actually need them?" It just wasn't something that crossed my mind because I wasn't in the room for those conversations. That wasn't how I was thinking about my team's work.

**Matt LeMay** (00:13:34):
So I like asking this question because it's a perspective shift. It gets you thinking if you were personally accountable for the existential market-level success of this business, would you invest in your team?

**Matt LeMay** (00:13:52):
And frankly, most of the people I ask that question to don't know the answer right away. They're not sure. They say, "I think so," or, "Yeah, yeah. Yeah, because... " And then you can see the gears turning.

**Matt LeMay** (00:14:05):
And again, to me that is a risky situation to be in because if you can't answer that question as a member of the team, nobody else in the organization is going to be as well-equipped to answer that question as you are.

**Matt LeMay** (00:14:19):
So bringing that question to the team, asking it and answering it proactively and collaboratively and being ready to say, "Okay, if we can't answer this confidently, what are the changes we need to make? Do we need to change our purview? Do we need to change our goals? Who do we need to talk to? What do we need to do in order to make this happen, in order to answer that question confidently?" We are much better off having that conversation at the team level than waiting for someone else to have it for us, or at us.

**Lenny Rachitsky** (00:14:50):
So is kind of the signal here is if you can't confidently say yes, asking yourself this question or your team talking through this and having a clear answer, there's maybe a potential problem that you need to work on?

**Matt LeMay** (00:15:01):
Absolutely. I think that's exactly it, yeah.

**Lenny Rachitsky** (00:15:04):
And we'll talk about what to actually do and the steps you recommend to fix this problem, but I want to keep talking about impact because sometimes it takes a while for people to be convinced, "Okay, I need to really do this," versus, "Okay, here's the ways to do this."

**Matt LeMay** (00:15:15):
Yeah. It has taken me until now, until writing this book to be convinced, and I still feel like I need to be convinced a little bit sometimes.

**Lenny Rachitsky** (00:15:22):
I think the layoffs are a really good convincing function for people.

**Matt LeMay** (00:15:26):
Yes.

**Lenny Rachitsky** (00:15:26):
Where they're seeing all these PMs being laid off and just, like, "How do I avoid that?" And this is basically the answer to that, is here's how you avoid getting laid off.

**Lenny Rachitsky** (00:15:33):
There's an interesting implication in what you describe here where you encourage the PM and the product team to think like the CEO.

**Lenny Rachitsky** (00:15:39):
This touches on something I've been meaning to write about, but I haven't, which is a lot of people, there's this argument are PMs the mini CEO?

**Lenny Rachitsky** (00:15:46):
In my mind, the PM is 100% the mini CEO, in terms of they think like the CE or they should within the team. They're the same; their job is to think about the business like a CEO within the team.

**Lenny Rachitsky** (00:16:03):
Obviously they're not in charge of everyone. They can't let get anyone go. The reporting lines are not the same. But I feel like that's a really good lens for the job of a PM, is just you're like the mini CEO on a team, making sure you're building the right things, thinking about the business, solving real problems that matter.

**Matt LeMay** (00:16:19):
I think the way I've come to think about it is that the product manager is responsible for the whole team thinking like a CEO, if that makes sense.

**Lenny Rachitsky** (00:16:26):
I love that.

**Matt LeMay** (00:16:27):
So if the product manager takes that on as their sole responsibility and says, "I'm the mini CEO. I think about the business," that's a missed opportunity.

**Matt LeMay** (00:16:35):
In my consulting work, it's more often than not been an engineer who's actually able to get to that specific impact-level goal for the team, sometimes before the product manager does, because the engineer is thinking in systems, is thinking about, "Okay, I know this system, I know this other system. I've had this conversation with this person. I think I see how this can all fit together."

**Matt LeMay** (00:16:55):
Sometimes it's a designer or a UX researcher who understands what customers need and who has that direct firsthand knowledge.

**Matt LeMay** (00:17:02):
So I think the way that I certainly initially misinterpreted the mini CEO thing was you are solely in charge of this, as opposed to you are the person who brings that kind of CEO-level commercial thinking to the team. And I've been much more inclined of late to see the product manager's job as to bring in and facilitate that conversation, rather than to be the sole owner of that conversation.

**Lenny Rachitsky** (00:17:29):
That is a really good Nuance to what I said, and it makes all the sense in the world.

**Lenny Rachitsky** (00:17:34):
Following these lines of what is the job for product manager, you have a really nice way of just thinking about it. What does a business expect from a product manager?

**Matt LeMay** (00:17:42):
Yeah. My favorite definition of product management comes from Melissa Perry's book Escaping the Build Trap, where she describes product management as facilitating a value exchange.You're facilitating a value exchange between a business and its customers.

**Matt LeMay** (00:17:57):
The actual shape and nature of that value exchange depends on so many things. It depends on your funding model, it depends on your business model, it depends on whether you're B2B or B2C, it depends on whether you're publicly traded or privately held. There are so many different variables that go into what a business expects from product teams.

**Matt LeMay** (00:18:19):
And one thing I've come to is that I'm probably too hesitant to generalize in general, aside from that generalization I just made, but I think that the best product managers and teams I've worked with are really curious to understand what success means to their particular business.

**Matt LeMay** (00:18:38):
So if that's a startup, how much runway do we have? What do we need to raise that next round? Are our investors looking for growth? Are they looking for profitability? What are they looking for in order for us to reach that next existential milestone?

**Matt LeMay** (00:18:52):
If we're a publicly traded company, what are investors looking for? What have we told shareholders? What's in our next prospectus? What are we saying we're going to deliver to the market at large?

**Matt LeMay** (00:19:01):
This information usually exists somewhere, but it's really easy to get disconnected from it as we go about the day-to-day work of doing product management.

**Matt LeMay** (00:19:11):
I've been in a lot of rooms having workshops and sessions with teams I work with, where they're pulling up the last town hall meeting to look for slides from their CEO because they're so busy, they have so many day-to-day things to deal with, that for them that town hall day is, like, "Okay, finally I can just not have to deal with people emailing me every 10 seconds." It's a chance to take a deep breath and then get back to the stuff that really matters.

**Matt LeMay** (00:19:38):
But those top-level things, those promises that have been made to the market at large, to the board, to the shareholders, those are often a great place to start for understanding what exactly your business is looking for, so you can figure out how your team's work helps contribute to success in the terms that matter most to the business.

**Lenny Rachitsky** (00:20:01):
That's an amazing segue to exactly where I'd wanted to go, which is this quote that is exactly what you just said, the question you should be asking, it's kind of a different version of the one you shared earlier, which is what are we doing to contribute to the success of the business?

**Matt LeMay** (00:20:14):
Yeah.

**Lenny Rachitsky** (00:20:15):
So what is it that you think people are thinking instead most often?

**Matt LeMay** (00:20:21):
It's interesting. We live in a world where for the last 20 years of product development, we've been all about best practices. We love best practices.

**Matt LeMay** (00:20:30):
And I love best practices too. I think best practices can be really valuable, can be really useful. I love how much knowledge sharing there is out there. When I started 15 years ago, it was so much harder to get any sense of how anybody was doing anything.

**Matt LeMay** (00:20:44):
But I think the downside of this is that there are a lot of folks who are really, really fixated on doing things the right way. And a lot of the writing and thinking about doing things the right way exists in kind of this middle layer, between impact and day-to-day work.

**Matt LeMay** (00:21:03):
So if you have impact, the overall success of the business, revenue, growth, profits. Then from there you set objectives and you do initiatives, and then you do bets, and then you make a strategy, and then you do your day-to-day work.

**Matt LeMay** (00:21:15):
I think the tendency among a lot of product teams is to key to those middle pieces, to say, "Okay, what we're doing is on strategy. What we're doing is part of objectives. We did OKRs, right? We spent three weeks coming up with our OKRs for three months. It's OKR season, everybody. We have to pretend to care about our goals, and I read a book that says we have to have five to seven objectives with five to seven key results, and we want the score of each one to be a 0.6 or a 0.7."

**Matt LeMay** (00:21:41):
And at the end of the day, you've actually spent so much time and energy being really clever and cascading these things into these intermediate steps, that each step has incurred more and more risk that we might not achieve that impact we seek to drive.

**Matt LeMay** (00:21:59):
Each time we abstract things out, each time we cascade things from impact to something else until we wind up with a checklist of features to build, we're assuming that if we then build that checklist of features... We will have a successful business.

**Matt LeMay** (00:22:14):
But honestly, most of the companies and teams I work with, if I ask them how their lowest-level goals add up to their highest-level goals, they look at me like I just asked them the most impossible question in the world.

**Matt LeMay** (00:22:27):
Because by the time you get things down to that level, maybe it's going to work, maybe it's not going to work. There are assumptions baked into each of those levels.

**Matt LeMay** (00:22:37):
You might go through the motions of doing OKRs the right way or doing strategy the right way, but that doesn't guarantee that your strategy is going to work. It doesn't mean that your OKRs are going to add up to what your investors or your shareholders need to see.

**Matt LeMay** (00:22:52):
So I think there's a tendency among product teams to really sweat the middle, and to get really stressed out if the middle is not in perfect theoretical harmony with itself. If the objectives are a little out of line with the initiatives, are a little out of line with the bets, are a little out of line with the domains.

**Matt LeMay** (00:23:14):
But at the end of the day, I don't think I've ever worked for a company where these things add up to a perfect breadcrumb trail you can trace back from, "We prioritized these bits of work in this sprint because it adds up to this objective, which is part of this strategy, which adds up to this initiative, which is part of this bet, which will deliver this much impact." It's just not how things work in the real world.

**Lenny Rachitsky** (00:23:37):
I feel like a lot of listeners are, like, "That sounds very familiar," exactly what you just described. Again, we're going to talk about what to change and how to make work better to actually align your teams with impact. But I want to first talk about something that follows this thread you're on, which is something you call the Low Impact PM Death Spiral. Talk about what that looks like.

**Matt LeMay** (00:23:55):
The Death Spiral.

**Lenny Rachitsky** (00:23:56):
I love the hands.

**Matt LeMay** (00:23:59):
The Low Impact Death Spiral is the dynamic in which every-

**Matt LeMay** (00:24:00):
... impact death spiral is the dynamic in which every medium to large company I've ever worked with finds itself in one way or another. And honestly, a lot of small product companies do as well, and it goes something like this. It starts with teams taking on low impact work. Adding little features here and there, making little cosmetic improvements because it's easier, it invites less scrutiny and you're less likely to mess up something important, right? The analogy I use sometimes is if you're working on a car, if you put your hands in the engine, you might make the car run really well or you might make it so that the car doesn't run at all. If you have the option of doing that or decking out the car with a paint job and rhinestones and making it look really, really cool, what are you going to choose to do? I know for me, I would choose the rhinestones every time because then the car runs however it runs. Whoever else is the expert in engines can fix the engine, but I'm going to do this thing which everyone can see. All the executives can look at it and say, "Wow, look at how cool this car looks," and it's not going to do any harm, right? What's the worst that could happen? We can always take the paint off.

**Matt LeMay** (00:25:15):
The problem is if you have 10 teams adding rhinestones to a car, eventually the hood of the car is going to be so heavy that you can't lift it to get to the engine anymore, and that's exactly what happens in most product organizations. As you have more and more teams adding in these little features, making little enhancements, adding features, which might get some usage, but they're not going to take away anything meaningful, they're not going to mess with the commercial engine of the business. You have more folks building in and the hood of the car gets heavier and heavier, the product gets more and more complicated.

**Matt LeMay** (00:25:50):
You have all these little bits that have little dependencies on each other and the product becomes as, I think most modern products are a collection of loosely connected features rather than a single guided experience. What does that mean? That means internally, building anything becomes a lot harder, right? Because there's 10 product teams that have to manage dependencies whenever you want to build something, so companies start adding program management layers. They're like, "We have too many meetings. We need to reorg for program management because reorgs fix everything and we need to make sure that we have all these layers and people are connected and we're doing dependency management." But as you add more of those things, it becomes harder and harder to do high-impact work.

**Matt LeMay** (00:26:33):
The hood just gets heavier and heavier and heavier, which in turn sends teams deeper and deeper down the rabbit hole of low-impact work. Low-impact work begets low-impact work. The more low-impact work you do, the harder it is to do high-impact work, the more likely you are to do low-impact work and so on and so forth. It goes and goes and goes until the next round of layoffs, and this is a real problem.

**Matt LeMay** (00:27:01):
I've experienced this at again, pretty much every company I've worked with, and it's been really interesting to see how often the breaking of that cycle comes down to individual product teams being brave enough to look at their own work and look at what matters to the business and say, "You know what? We are not going to incur the risk of being a low-impact team anymore. We are going to proactively seek out high- impact work and we're going to deliver, even if it means we have to coordinate with a lot of other teams. Even if it means that suddenly the executive team takes a really big interest in the work that we're doing. We're going to see that as signs that the work we're doing matters not as things that are going to be an impediment to getting the work done."

**Lenny Rachitsky** (00:27:47):
As you just kind of alluded to, there's a big implication here of even if you are told to build a thing that the execs are really excited about, that is low impact in your opinion, that's no excuse. You're still going to get fired eventually, even though they're telling you to build it. It sucks, but that's how it goes.

**Matt LeMay** (00:28:04):
It's funny, I worked with MailChimp for three years leading up to their acquisition by Intuit, when they were on this really interesting journey of going from a single product company to a platform company, which is a journey that a lot of companies are on. Where they've found success in one part of the market and they say, "You know what? We recognize that there's a value add in working more horizontally and giving people a suite of tools they can use. We want to build these tools and really make sure that we are meeting the needs of our small and medium business customers."

**Matt LeMay** (00:28:34):
So I was brought on shortly after they did a reorg. They went into a domains model. They had different teams that were supposed to be experts on different parts of this marketing platform that were building in new features and their VP product, Natalia Williams, who then became their CPO, she said, "This is great that we're doing this, but I'm worried that we're going to lose sight of the commercial fundamentals. We have these teams building this really cool stuff and we've been so clear that we're excited about the teams that are working on initiative, that are building these new things, that are building out this platform, but let's make sure we stay focused on making sure that the entire platform grows in a healthy and sustainable way. And that as we are adding in these new features and functionalities, we're not making this so crowded and complicated that it's harder for people to get value out of it."

**Matt LeMay** (00:29:31):
So she set a very clear goal for the product team. She wanted a specific change in the rate of users who successfully send their first email. She was like, "As we build these new things, email is still where people are coming in. We want to make sure that they can do this successfully." And she set a really ambitious goal for the entire product organization. And she put it forth to the product teams and a lot of product teams were understandably hesitant to take this on, for a number of reasons. They said, "If we mess this up, that's the entire commercial heart of the business. That's really risky."

**Matt LeMay** (00:30:12):
Other ones said, "That's great, but we're building out these new parts of the platform that are really important. We don't want to risk slowing those things down." There was one product manager I worked with who really understood how important this was. She had worked on enough different parts of the app at that point. She had been on a growth team, so she really had that kind of user-centric, how do we hit the metrics we need to hit mindset? And she came to me and she was like, "I want to make sure we do this because I'm worried about what'll happen if we don't do this, right? I don't want to be on the product team that failed to deliver what matters most to the business."

**Matt LeMay** (00:30:53):
So she went and worked with some user researchers and found where people were getting stuck. She did this amazing study finding where people were getting stuck before they sent that first email and she gathered together a group of product managers. I'll never forget this. I was on site in Atlanta and she gathered together a group of product managers and she was like, "Look, these are the things we need to fix if we're going to hit this, but if we fix these things, I think we can do it. I think we can get there." And there was a moment of silence where again, for totally understandable reasons, people were like, "Okay, this is great. I see it, but I'm kind of afraid that if we do this," and this product manager, I will never forget that.

**Matt LeMay** (00:31:34):
She looks around the room and she says, "You know what? I'm going to haunt all of you in your dreams if you do not take this on. This is the most important thing for the business." And everyone laughed. It was a strong comment, but it was a strong comment made in really good faith that in its own way, I think acknowledged the fear that people had about taking this on. She was like, "Look, this is really important and if we work together, if we care less about the constraints and parameters and borders of our domains and care more about how we work together to deliver the thing that our leader has told us is the most important thing for the business, we can get this done."

**Matt LeMay** (00:32:15):
And they did get it done, and they got it done through subtracting. They streamlined the experience. They took out steps that people were getting stuck on. They made things easier. They did the kinds of things that are rarely celebrated in the way that traditional feature launches were celebrated, but because they had this clear, impactful, specific sense of what success looks like, they were able to take on that work themselves and it made the product better, made the business better, and it was just a great moment of working with a great team.

**Lenny Rachitsky** (00:32:48):
That was an awesome example. I love just how much of a hero's journey this example is. Along those lines, it's interesting how much courage it takes to do this. Because to your point, the easy thing is just, "Okay, everyone's just telling us to do this thing. We're going to build this feature. Great, easy. I don't have to put myself out there." And what you're pointing out there is it's up to you to recognize what you're doing is not high impact and to have courage to push back and suggest something else.

**Matt LeMay** (00:33:15):
The funny thing is, it's courage, but it's also a kind of radical acceptance of reality. There's a very kind of emotional philosophical dimension to this conversation because I think it is in human nature to not want to be accountable for things outside of our control. Impact is not something we can just check off a list. If we're really working towards business impact, that means that there are things that might happen in the market that are going to affect us, right? Our competitors might launch something, there might be a pandemic or a war or something we didn't expect. We're reliant upon customers to change their behavior, to convert or to upgrade or to do whatever it is we hope they will do.

**Matt LeMay** (00:33:58):
I use the word hope because again, we can do our best to influence that behavior, but we can't just check a box. And I think there is this very understandable human fear of being accountable for things that are beyond our control. I get that so hard, but what I think the last couple of years have shown us is that we are accountable for things outside of our control. The success of the businesses we work for is ultimately beyond our immediate control.

**Matt LeMay** (00:34:26):
And you can follow all the best practices and have the highest velocity and do everything quote-unquote "right", but if your company goes out of business, they're not going to say, "Yeah, we're going out of business, but we're going to keep writing your paycheck for two years because all of your OKRs were a 0.6 or a 0.7." That's not how it works. The reality is that things outside of our control affect us, and I think that if we can internalize that, there's a real freedom in it. The thing that surprised me most when I was researching this book is that the commercially minded PMs I interviewed were also the happiest, which I did not expect at all.

**Matt LeMay** (00:35:06):
As you said, I come from a music background. I was expecting the business people to be really stressed out, and like, "I have to show up at work 92 hours a day to crush it and make sure that we hit the metrics and crush the numbers." But they were just like, "Yeah, I work for a business and I do the best I can, and then I go home. When I'm at work, my job is to help the business succeed. And I do that by understanding our business model, by trying to do work that contributes to what the business wants. In order to do that, if my goals are really impact level goals, I have to learn about our customers. I have to do discovery, I have to do all these things that we're supposed to do, not because they're abstract best practices, but because that's how you build a successful product."

**Matt LeMay** (00:35:49):
And then at the end of the day, I'm not fighting my company to do product the right way, and if they can't do it the right way, I'm willing to die on this hill. It's like, " I work for a company. I don't get to make every decision. There are things outside of my control. I'll do the best I can, and I get to free up a little bit of energy to live the rest of my life."

**Lenny Rachitsky** (00:36:12):
To make this even less scary, hopefully, I'm curious what your answer is going to be here. Do you find that people who take this leap and push back and try to change the way their teams are thinking and what they take on, do you find that even if they fail, even if they impact wasn't there and the project fails, they're seen more favorably and their career ends up doing better and they end up doing better even if the project fails?

**Matt LeMay** (00:36:36):
I hope so. I'm always super, again, it's outside of my control. I can't say, if you do these things, then you will be looked upon favorably by the people who manage you because I don't know those people, and maybe they're not going to react well. Maybe they're not actually... People are not always rational, and these things don't always play out the way you hope they will. What I will say is there's a story in the book, in Impact-First Product Teams about somebody who worked for a dating app, and they were given a project to work on, which had a very specific ambitious revenue goal. And at a certain point, it became clear that they were probably not going to be able to hit this because of other things they needed to do.

**Matt LeMay** (00:37:15):
So they went to the finance team and said, "Hey, we have to adjust this down." And the finance team said, "Yeah, of course you do. That makes sense. Let's adjust it down together." Would that conversation have played out the same way if they had not taken that step of saying proactively, " Based on what is happening in the broader world in the market, our product, we no longer believe we are on track to hit this, but we want to adjust so that the company at large can make better predictions and resource things accordingly"?

**Matt LeMay** (00:37:42):
I don't know. But I think again, if you are embracing that mindset that there are things outside of your control and your job is to do what you can from where you sit to contribute to the success of the business, I think you're certainly doing so much more by telling the business honestly what you think is possible and what's going to happen, as opposed to stepping back from it and saying like, "Oh, well, if we talk about it, we're going to get in trouble and nobody's going to believe us." And just that simple withholding of information does its own kind of harm. So I can't guarantee any kind of outcomes. I can't say, "Yes, this will be received with generosity and grace," in all cases, but I think it's the right thing to do.

**Lenny Rachitsky** (00:38:29):
My sense is either you will be respected more highly because you are kind of solving problems for people, like higher ups that you're seeing, things that they should be seeing, and you're right, and they're like, "Oh, wow, Matt's amazing. He saw all this stuff and changed." Or you realize this is not the company for me. They don't value this. They just want me to ship this widget and that's all they want from me. I could do more somewhere else.

**Matt LeMay** (00:38:56):
Well, I think you touched on something really important there, which is that when you understand the business model, you understand the real world ethics and priorities of the company you work for, right? If you understand what the company's optimizing for, what they really care about, what trade-offs and sacrifices they're willing to make in order to achieve those goals, that tells you a lot more than the mission statement. Which usually sounds great, but if you follow the money, if you follow the decision-making, sometimes it paints a different picture.

**Matt LeMay** (00:39:28):
And I think the more you can understand how the business model works, what that value exchange is, what value is being delivered, what value is being extracted, that I think puts you in a better position to decide, if I were to help the company achieve success, do I feel good about that? Do I feel like this is a company that if it achieves its goals has made the world a better place or not? And the more you understand the business model, the better equipped you are to make those decisions with all the information available to you, I think.

**Lenny Rachitsky** (00:40:05):
That's such an important point.

**Matt LeMay** (00:41:51):
Yeah, I mean, the reality is that most companies that product managers work at aren't big tech companies. And when I worked with product teams at those companies, at insurance companies or at banks or at CPG companies, or I worked with a company a while ago that makes audio equipment, which was obviously really fun for me, the first thing they usually tell me is, "Well, we can't do product the right way because we're a regulated industry," or, "because we're B2B," or, "because we have these quarterly earnings targets we need to hit."

**Matt LeMay** (00:42:22):
And that always makes me a little sad because yes, there's a way of looking at it where you can say, "All right, if these five famous companies do product management the right way, then anything that puts us in a different situation from those companies must mean that we're doing product management the wrong way," which is demoralizing and distressing, and again, leads people to these kind of quixotic battles of like, "I have to turn a bank into a search engine," which is not going to happen.

**Matt LeMay** (00:42:49):
But the other way of looking at it is that those constraints are how you do product management. Those are what guides you to the commercial decisions you should be making. If you're regulated, your competitors are regulated too. If you can really understand those constraints, figure out how to work well within them, that's a huge commercial advantage for your business. If you're B2B, I honestly prefer working with B2B companies in a lot of situations because they have such immediate access to their customers and they can understand the whole system of their customers and their users and how these things connect in a way that's really immediate and actionable.

**Matt LeMay** (00:43:24):
Similarly, if your company has quarterly financial targets to hit, congratulations, you know what the business really cares about. You don't have to go looking much farther than that. You don't necessarily need to cascade that 10 different levels into 45 OKRs if you can do work that you know feeds directly into what the company cares about the most.

**Matt LeMay** (00:43:47):
So yes, these things are constraints, but they're the constraints that shape the work that we do. And if we embrace them as such, if we treat them as guides, not restrictions, if we say we're working with not against the commercial realities of our business, there's so much we can do. There's so much that product managers and product owners and project managers that everybody working on product at every organization can do if we again work with, not against the realities of the companies we work for.

**Lenny Rachitsky** (00:44:23):
Okay. So I think that's really empowering. You're not saying, "You need to change the way product is built at your company. It's up to you. You're an ICPM, it's up to you. No excuses." What's a simple way to describe what you're actually saying, which I think is going to make people feel a lot better?

**Matt LeMay** (00:44:38):
What I'm actually saying is that the things you think you're fighting against are usually the things that are giving your work shape, if you let them be.

**Lenny Rachitsky** (00:44:50):
Awesome. Let's talk about, you have three steps, three handy steps to become more of an impact-first product team. So let's talk through them. What is step one?

**Matt LeMay** (00:44:59):
Yeah, so the first step, and I think the most important and where a lot of my work plays out is in setting team goals, close to company goals. No more than one step away from company goals. This idea comes from Christina Wodtke's book Radical Focus. She had a great episode on your podcast that I just re-watched for inspiration before I came on here.

**Lenny Rachitsky** (00:45:22):
Nice move.

**Matt LeMay** (00:45:24):
But in that book, she has this visual where she says that most companies approach cascading OKR as kind of this multi-level, like at the top is company goals, then department goals, then big team goals, then small team goals, and smaller team goals. Which leads to the very situation I described where you can't add those small team goals back up to the company goals. She has a visual that she compares that to, which is the company goal as a center of gravity, and then every other team goal orbiting one level around it. And this blew my mind when I saw it for the first time, because whether you're doing OKRs or KPIs or north star metrics, whatever framework you're ostensibly using, if you use it well, it looks like that.

**Matt LeMay** (00:46:09):
I worked with a team a while ago that was a traditional feature team at a FinTech company. They were the team that was kind of like the team I worked with at MailChimp, kind of given new features to build to take this product and build it more into a multi-product platform. And the business would say, "Go build this." They'd build it, they'd celebrate, they'd say, "Go build this." They'd build it, they'd celebrate. And then the business said, "Okay, we're going to do things a little bit differently. Now we have this top line revenue target. We want to make this much more money next year, and we want you to tell us how you're going to contribute to this."

**Matt LeMay** (00:46:42):
And this was a really challenging conversation for this team because as much as people complain about being feature factories, again, it feels safer, right? There's more control over that. If you're told what to build, you have a lot more control over whether or not you build it than you do over whether or not it generates revenue. So we started having this conversation, what framework should we use? How should we do this? And I was not asking very good questions. I was kind of panicking too, to be honest. And then I asked them, I was like, "Okay, why does this team exist, right? Why are we doing this at all? Why is the business investing in this big team to build all these features? Why do we care if we're a single-product company or a multi-product platform company?"

**Matt LeMay** (00:47:26):
And one of the team leaders said, "Oh, well actually, the customer lifetime value of a multi-product user is much higher than that of a single-product user." I was like, "Oh, do we know how much higher?" And they said, "Yeah, we actually have a number on that." And I was like, "Wow. So if we had a goal of converting a certain number of single-product users to multi-product users, we know exactly how much revenue is on the line." They were like, "Yeah, I guess so." So we were able to set a goal, and from there we had this conversation about what's the number we would need to hit to make this team feel like a good investment? And we came to a specific number and-

**Matt LeMay** (00:48:00):
To make this team feel like a good investment. And we came to a specific number and that number informed every decision that was made by that team for the rest of the year. We said, "By the end of the year, we need to convert this many single product users to multi-product users because if we do that, there's this much additional revenue which contributes this much to the company's top line revenue goal." It's one step away.

**Matt LeMay** (00:48:22):
And when they went and shared that goal with company leadership, they got it right away. They didn't have to say, "Well, why is this valuable?" They didn't have to say, "What are your other OKRs? Show us the deck." They got it. And that gave them so much leeway to make bolder decisions, to work more closely with other teams to ask for the resources they needed. And that process of arriving at that goal, that goal that is impactful and specific and one step away from what the company cares about the most, is probably the most valuable way I have spent my time with teams of late. Because once you get to that, you see how everything else flows from there.

**Lenny Rachitsky** (00:49:01):
And the implication here is that if it's too far away from people understanding what the hell are you guys even doing to help the business? That's the problem you want to avoid.

**Matt LeMay** (00:49:10):
Exactly.

**Lenny Rachitsky** (00:49:11):
So one extreme is, I don't know, it probably doesn't make sense for your team to have exactly the same goal as the company. Say your company's trying to grow a hundred million revenue rate, that's not going to be your team's goal. So it's almost like that extreme is what's one step away from that? What's your contribution to that company goal?

**Lenny Rachitsky** (00:49:29):
The other end is just some 10 level deep, we're going to improve conversion of this random step, which is going to grow this thing which grows that, and then that grows revenue.

**Matt LeMay** (00:49:37):
Exactly.

**Lenny Rachitsky** (00:49:38):
What is your guidance on revenue-based goals as goals for teams?

**Matt LeMay** (00:49:43):
I think, again, it depends. I know people have very strong opinions about teams should always or should never have revenue-based goals. I don't think it's that simple. I think sometimes yes, if your company is working towards revenue, having a revenue-based goal makes sense, or at least being able to put some value in terms of revenue on the work that your team is doing. I've worked with some startups where they actively are not going after revenue because right now the name of the game is growth, right? They want to show that they have enough users to raise that next round of investment. I actually worked with one startup where their revenue was going up, and their number of users was going up, and we had to have a really interesting conversation around which one actually matters to our investors. And we kind of wound up developing these two scenarios, the high growth scenario and the low growth scenario.

**Matt LeMay** (00:50:33):
And talking to the product team, it was really clear that there was no way they were going to hit the high growth scenario. So they pivoted to focus on profitability so they could build more runway and raise that next round as a smaller business with a sustainable revenue base.

**Matt LeMay** (00:50:46):
So I get why people are hesitant to commit to revenue goals. I think that there are certainly situations where revenue goals would be too broad or not exactly what the business needs, but at the end of the day, money is how the impact of most teams is going to be measured. And if you don't have a point of view on why your team is a good investment, either in the money return it brings, or in another return that is equally or more important to the business, then again, that feels like a tenuous and risky position to be in.

**Lenny Rachitsky** (00:51:25):
To kind of get people's brains noodling in the right direction. What are some examples of good goals, like specific goals that your teams have set that are one step away from a company goal? What are just some examples without naming companies?

**Matt LeMay** (00:51:38):
I have a story to tell to that effect, which also sets up the second step here, which is to keep impact first at every step.

**Matt LeMay** (00:51:47):
So, I worked with a growth team at a tech company that is beloved by many the people. And I was brought on to help them set OKRs, right? The company was doing OKRs across the board, and this team was going to set its growth OKRs. So they had done a lot of reading, a lot of research into how to do OKRs the right way. They had read, unfortunately, they had not read Radical Focus or they hopefully would've done things differently, they had read a lot of other books. They had printed out a lot of articles and they said, "We're going to make sure we get this right. We're going to put together a slide deck that has five to seven objectives with five to seven key results each, and each key result is going to have an owner and we're going to make sure that people are accountable for it. And we're going to really, really do this the right way. We're going to get that middle piece perfect."

**Matt LeMay** (00:52:39):
So they spent a long time working on these OKRs, OKR season ends. They file away the OKR deck not to be seen until the next OKR season. We reconvene, and spend a whole day scoring all of the OKRs. And there are heated conversations about what's a 0.6 versus a 0.7? Is this a 0.9? Who really should be the owner of this key result? Who contributed the most to it? And we get through this whole conversation, everyone's tired. And I'm like, "There's just one question I need to ask. I'm so sorry to do this to y'all, but how does this all add up to the company level growth goals?"

**Matt LeMay** (00:53:23):
People look around and they're like, "Well, that wasn't what we did. We set our team level OKRs and we went through the process and we made sure, and we followed the OKRs, we thought about the OKRs," and I said, "Yeah, but the company has a goal of let's call it bringing on a million new users in this year. We're a growth team. How many users did we bring on?"

**Matt LeMay** (00:53:50):
And there's a moment of silence. And the leader of this team's a really, really good leader. She stands and she's like, "Okay, we're going to do things differently." She steps up to the whiteboard, puts 1 million on the right end of the whiteboard, draws a line, "This is the year," draws a little tick on the line. "This is where we are now. It's a smaller number than a million. This is where we need to get to. If our conversation doesn't start with this, I don't want to have the conversation. And if you are 51% sure that a different approach is going to get us 1% closer, I expect you to advocate for that approach. And this is now how we are going to do everything."

**Matt LeMay** (00:54:34):
And I think about that all the time, because it's subtracting steps. It's making things less complicated. It actually is doing exactly what you described. It's saying, "This is what the company cares about. We're the growth team. We need to figure out how everything we're doing contributes to this."

**Matt LeMay** (00:54:55):
That story really stuck with me because it seems like it should be obvious, right? If you're the growth team at a company with a growth goal, obviously that's going to be something you care about. But it's so easy in the day-to-day work of setting these intermediate goals, of doing strategy, of doing the things we do to make things just a little bit more complicated than they need to be. So for this particular team and this particular moment, a good goal was looking at the company level goal and saying, "If we're the team that's driving growth, we're accountable for this. We're going to step up and we're going to do everything we can to make sure, the company meets its ambition."

**Lenny Rachitsky** (00:55:38):
And it's so obvious if you come at it from the perspective you shared earlier. If you're the CEO, would you fund this team? If you see our goal is a million users, this team will drive us a hundred thousand users. Assuming that's a lot for you. Obviously we want that team to do great and we're going to give them the resources to do that. They're going to contribute meaningfully to this goal. It's very obvious as you see it. The trick is a lot of people think they're doing that and they're not. Maybe one heuristic I think about as you're talking is just like, how do you know if you're doing this? There should be a very simple spreadsheet formula of just here's the company goal, here's your goal. There should be one simple formula that translates up that adds to it.

**Matt LeMay** (00:56:20):
One Y statement and or one mathematical operator is usually the way I think about it.

**Lenny Rachitsky** (00:56:24):
There we go.

**Matt LeMay** (00:56:26):
In some cases, like the story I told about that FinTech company, it's something like upgraded users. That's one step, right? You multiply that by the increased customer lifetime value. In some cases, it's a smaller unit of the same thing. So it's a subset of the revenue or growth that the company's trying to deliver.

**Matt LeMay** (00:56:47):
I worked with a company recently that had a team rebuilding their ad tech platform. This is a big company that does a lot of different things, and at the heart of all of their different offerings is this ad tech platform. And I was brought on to do a day long workshop with this company because this was a really important team and they were struggling to figure out what success looked like.

**Matt LeMay** (00:57:06):
So I said, "Okay, what are your goals?" And they said, "Here's the deck with our goals." I said, "That's a lot of goals. I wouldn't be able to make decisions with those goals." They said, "Oh wait, wait, wait, no, we also have a Miro board. We went through this and we decided the deck was too much. So we also have a Miro board," but as Miro boards tend to do, it kind of grew and grew and grew in all directions. So now it's a big board. Turned out they had two Miro boards because they had tried to do this twice.

**Matt LeMay** (00:57:33):
So we sat down and we did this really interesting kind of subtractive exercise, where everybody got a post-it note to write down what they think the most important goal of the team is. And then they would pair up and they would have to make it smaller each time, make it more concise. If there were two things, get it down to one thing. If there were 10 words, bring it down to eight words, make it smaller and more concise each time.

**Matt LeMay** (00:57:56):
And we came up with two options. One was build a fantastic, wonderful, magical, beautiful platform to increase our team's profits by 20%. Fair enough. The other one was, build a stupendous, wonderful, fantastic platform to increase our team's profits in the next financial year from 20 million pounds to a hundred million pounds.

**Matt LeMay** (00:58:18):
And I'm looking at these and I'm like, "Okay, so we know that there is an expectation of revenue here. There's an expectation that we will drive profitability, but one of these is 20% and the other is 500%. Where do these numbers come from?" And the folks who had said 20%, they were like, "That feels like a good return." And what's interesting was when I was telling the story, when I was giving some talks recently, and people would kind of laugh like, "Oh, they named the wrong number." But they were so brave to do that, and them saying 20% is what actually unlocked the conversation at the team level, because somebody else in the room heard that and said, "Wait, I do remember something about profits. There was some target we were supposed to hit, but I think I remember it being a lot bigger than that." And they actually went back and found a town hall that their VP had given and buried in a slide in that town hall was this hundred million pound profit target by the end of the next financial year.

**Matt LeMay** (00:59:22):
So they have this number and they start to go like, "Oh gosh, that's a lot, but it's a whole financial year. It'll be fine." And we do, again, inspired by that leader I worked with, I just put this out on a whiteboard. I'm like, "All right, a hundred million in profits by the end of the next financial year right now we're here. What else should be on this timeline?" Somebody said, "Well, we're launching this in October." So I put October up there. I go, "Okay, so if we don't ship anything until October, and right now it's March, how much profit do we make until?" And they said, "About 10 million." I'm like, "Okay. So that means that between October and March, we need to generate 90 million pounds of profits."

**Matt LeMay** (01:00:06):
And this team, I was worried they were going to freak out or they were going to be like, "Oh, we can't do this. It's impossible." But they were like, "Okay, it's time to step up. It's time to do all these things that we need to do. We need to ship sooner. We need to do more validation. We need to do more testing. Can we commercialize things sooner? Can we do more discovery to make sure that this is really going to have the impact we need to have? Why are we doing this whole big relaunch? Why don't we rebuild individual things? What impact will we have if we brought more people in top of funnel now? Can we begin building our user base and then upgrade that user base? What are the trade-offs if we do that versus if we bring people in?"

**Matt LeMay** (01:00:41):
All those things that I think is really the heart and the intent of these best practices. They came to life because this team had clarity. They understood like, "Okay, we are accountable for something which is specific enough to drive some real urgency, and which is impactful enough that we have to get out there and do all these things that we're supposed to do."

**Matt LeMay** (01:01:05):
So that to me, is a really good example of how if you start with this question of like, well, what's expected of this team? What does success look like for us? It'll often lead you to the right neighborhood of what kind of unit of measure you want to look for, how the business is going to look to you, whether they're looking directly at revenue, whether they're looking at something else.

**Lenny Rachitsky** (01:01:29):
Okay. These stories are awesome. So basically, so far, the two steps that you've illustrated here of doing this correctly, step one is set team goals that are close to company goals. And then step two is, do this at every step of the product building process. It's not just a one-time thing at the beginning.

**Matt LeMay** (01:01:46):
Right. As you're doing OKRs, like that team I talked about, when you're thinking about your company strategy, when you're writing epics, whatever it is, keep that impactful goal top of mind. Don't let it slip away. Don't let it get cascaded into oblivion.

**Lenny Rachitsky** (01:02:02):
Because to your point, the examples you shared where you think you're just like, "Cool, we have our goal, let's go." And then you spend months doing this thing and then you forget, okay, this actually got disconnected from the company goal.

**Matt LeMay** (01:02:12):
Exactly.

**Lenny Rachitsky** (01:02:12):
Everything you've worked hard to do is just, no one cares.

**Matt LeMay** (01:02:15):
Yeah.

**Lenny Rachitsky** (01:02:16):
Okay. What is step three?

**Matt LeMay** (01:02:18):
Step three is to connect every bit of work back to impact. So this is when we get to the prioritization piece, right.

**Matt LeMay** (01:02:26):
So impact estimation is something that is written about quite well, quite extensively in product management stuff, but it's something that it's easy to not do. I work with a lot of teams that want to know, again, the right best practice for doing prioritization, whether that's should we use ICE or RICE or MoSCoW or whatever it is, what's the right way to do prioritization? And again, it depends, but if you don't have a clear and specific sense of what impact means, you can't really prioritize effectively. So this is where, again, everything comes back to that first step of the way you set goals.

**Matt LeMay** (01:03:13):
So that team I told you about that had the goal of converting single product users to multi-product users, one of the product managers on that team came back to me a few months later and said, "Hey, Matt, I could really use your help figuring out how to do prioritization. Should we do ICE? Should we do RICE? Should we do MoSCoW? I put together an ICE matrix where I scored impact and I scored certainty and I scored effort, and I multiplied each one by each one, and I have a list of the things we should do."

**Matt LeMay** (01:03:38):
And I said, "That's awesome." And I do really think it's awesome when people take the initiative to do the thing to say, "I put this together, let's walk through it." So when they showed me this, I said, "Let's start with impact." They're like, "Okay, well here's the score for each one." I was like, "What's the score?" They're like, "Oh, it's a lot of different things." And I said, "Okay, well, we have a team goal of a certain specific number of users we need to convert by the end of the year, so let's try to express impact as a function of that goal. So rather than an abstract score, what is the most number of users we could convert in each of these different scenarios?"

**Matt LeMay** (01:04:18):
And understandably, they were kind of like, "Oh, you know what? I don't know exactly." I'm like, "Let's, just work through it. We don't have to be precise. It's called impact estimation, not impact getting perfect number." So for one of them, they were like, "Well, we can do this landing page." I'm like, "Okay, well how many people hit the landing page?" And they were like, "Not that many, maybe a few hundred." I'm like, "Okay, cool. So that's maybe a few hundred."

**Matt LeMay** (01:04:43):
And there was one thing which they were really reluctant. It had a high effort score and a medium certainty score. And what they basically said was like, "Look, if we really wanted to do this, we could redesign the entire onboarding experience. We can make it so that rather than us trying to retroactively shuttle people into multi-product usage, we can understand them when they join and get them into the right category of user right away."

**Matt LeMay** (01:05:08):
I'm like, "That sounds amazing." What's the most impact that could have? They were like, "It would get us all the way there if we did it right, but we'd have to work with these other teams and we'd have to work with marketing. And there are so many dependencies and so many things outside of our control that we don't know if we could do it." And I said, "That's okay. These are the trade-offs we make as product managers, you will have some things to work on that are going to have less impact, but might be easier and more certain to deliver. But if this thing could get us all the way there, if it's really the only thing we're looking at, which has really any chance of getting us to our goal, which remember we came up with to answer the question, what does this team need to deliver to be a good investment for the business, then I think it's worth going for it.

**Matt LeMay** (01:05:55):
"It's worth having those conversations with other product teams. It's worth talking to folks in marketing. It's worth stretching ourselves a little bit and saying, 'This is going to be challenging to do, but if we can do it, we can really deliver on something which we're going to feel great about.'"

**Matt LeMay** (01:06:11):
So the way that third step manifests itself the most frequently, tactically is are we estimating and measuring impact in the same unit of measure as our goals? Because if we're doing that, then we're keeping ourselves honest and we're saying, "Does this actually have a chance of contributing?" Whereas again, if we get really clever and we come up with a proprietary scoring system and all these little intermediate steps, we can feel like we're doing really awesome work and being very, very terribly smart. But we again, run the risk that we are going to lose our connection with why we're setting about to do this work in the first place.

**Lenny Rachitsky** (01:06:51):
Such an important and often easily overlooked point. This idea that when you're doing RICE or whatever version of ICE that you do, that impact should be based on impact to your goal, which is one level removed from the company goal. It feels so obvious as you talk about it, and I hope most people know this, but I feel like that's something people forget. Just like, what is impact? Oh yeah, customers will be happier.

**Matt LeMay** (01:07:16):
Again, this is not easy work, right? Working in product means that we have so many different perspectives, so many different folks who have so many different priorities. The stakeholder management piece of this is relentless and exhausting, and it's really, really easy to lose sight of this stuff. And it takes work and discipline and as you said, bravery to keep bringing it back. But I think again, it is not only the right thing to do, but it helps you at the end of the day, have a little bit more of a boundary between the work you're doing at your job and the stuff you're good at in your life.

**Lenny Rachitsky** (01:07:58):
Speaking of boundaries, a lot of the advice you share will involve people saying no, and pushing back, and convincing people they're wrong, which is very hard. That's not a natural happy place for most people.

**Matt LeMay** (01:08:12):
Nope.

**Lenny Rachitsky** (01:08:15):
What advice do you have? What have you learned works well for pushing back, helping people see how they're not... What they want you to build is wrong.

**Matt LeMay** (01:08:24):
So this is where I have the benefit, I suppose, of being a conflict-averse, people pleaser. So I've been through this, and I think the thing I am proudest of saying in product management and practice is that if you're doing product management really well, you never have to say yes and you never have to say no. You're giving people options and you're helping them understand the trade-offs.

**Matt LeMay** (01:08:52):
So in keeping with this idea that there are things outside of your control, as a product manager, you're not always going to be the one who makes the decision. There are going to be other people who come to you and say, "Build this," or who say, "I don't see the value in that. Do this, do that. I want this, I want that." If those people are in a position of formal authority in the organization, I've certainly never had luck telling them no.

**Matt LeMay** (01:09:16):
Furthermore, I've found myself in situations where they just know things that I don't know. Where they're pursuing an acquisition, so there are certain things that we have to do or can't do. Where there's some serious financial challenge that they're working their way through right now, which is shifting around priorities, but they are not at liberty to discuss that with the broader organization. It is quite possible that the folks who are asking you to do things that don't make sense, have access to information that you don't. And that in trying to convince them otherwise, you are not only going to fail, but you are going to lose some of that trust and degrade that relationship that could in time be really important for you. So I think the challenge is to say, "All right, you want me to build this thing? Maybe. Let's look at the different things."

**Matt LeMay** (01:10:15):
If you're going through these steps, if you say, "I understand, here's what our team's goals are, here's the stuff we're prioritizing, here's the impact we think it could have." If you're coming to me with something which is going to have more impact, heck yeah, I would love to build that. That sounds amazing.

**Matt LeMay** (01:10:29):
But if it has an unclear impact or if it doesn't tie into this, then we might need to adjust the impact we're seeking to have. If this is suddenly really important, as with that story that was told in the book, right? Sometimes it's like, all right, we got to do this. Sure. That means that our impact projection is going to have to be adjusted this much.

**Matt LeMay** (01:10:48):
You're not saying no, you're not saying yes. You're helping someone who is in a position to make a decision, understand the trade-offs that go into that decision. And again, if you can do that, you're going to go home at the end of the day feeling better. You're going to feel less like you lost a battle or that you were overridden by an unjust tyrant, and it's going to be more like, "Okay, I made sure they had all the information I would want to have if I were making that decision. That decision is ultimately theirs to make, and I'm going to do the best I can."

**Lenny Rachitsky** (01:11:19):
Tell me if you disagree. In my experience, you also want to recommend a path. Because a lot of people don't want PMs just, "Here's all the options. I have no opinion, no bias of any kind. You decide." I find that people want you to have a perspective.

**Matt LeMay** (01:11:33):
Absolutely. Options and a recommendation is kind of the magic formula. I did when I was working with MailChimp, we actually did a workshop with all the product managers at the company where we had people roleplay trying to sell in a single option versus multiple options with a recommendation. And it was amazing how when you present a single option, people's instinct is just to, well, I don't know, you start poking holes in it. Not because you're a jerk, but because-

**Matt LeMay** (01:12:00):
You start poking holes in it, not because you're a jerk, but because you're looking at one thing and you're like, "Oh, well, what about this? What about that?"

**Matt LeMay** (01:12:07):
Again, all of your own anxieties and fears and the things that you don't feel represented in there, and where you're worried, "What if this happens? What if that happens?"

**Matt LeMay** (01:12:14):
All of that kind of detracts from the momentum, whereas if you come in and say, "Here are three options, here are the trade-offs, here's the one we recommend," then every little bit of back and forth makes those options clearer and stronger.

**Lenny Rachitsky** (01:12:30):
I feel like people might miss that nuance, in the way we talked about it, so I think that's a really important point to make there.

**Lenny Rachitsky** (01:12:35):
Let me come back to the three steps, just for folks that are, "Cool, I'm excited, I'm motivated. I'm going to change the way we operate." I'll summarize real quick.

**Lenny Rachitsky** (01:12:44):
So step one is, set team goals close to company goals. Step two is keep impact first at every step. Don't forget, as you're building throughout the year. And then, step three is connect every bit of work back to impact.

**Matt LeMay** (01:12:57):
Yeah.

**Lenny Rachitsky** (01:12:59):
I think, again, it's important to remind people, this may be hard and annoying, and maybe things are going fine. Everyone's telling you what to build, you're building and shipping, and things are, it seems to be going all right.

**Lenny Rachitsky** (01:13:11):
I think an important part of this is just, if you believe what you're building is not helping the business grow and succeed, either, they'll let you go someday, because you're not doing something that matters. Or the business will not work out, and you'll be in trouble.

**Lenny Rachitsky** (01:13:27):
So it's like, even if you don't want to do this something, you need to really seriously think about this, even though it's really hard.

**Matt LeMay** (01:13:36):
Yeah.

**Lenny Rachitsky** (01:13:37):
But maybe let me just think of that a little bit. Say the business is doing great, everything's great. The CEO's telling you all the things to build, that sales guys and girls are just like, "Here, ship these features, everything's great."

**Lenny Rachitsky** (01:13:46):
And everything's working, business is growing. What thoughts there? Is it just, "Nah, leave it alone?" Or is your sense, "Okay, most often this will not last," and you should really start to work on these skills?

**Matt LeMay** (01:13:56):
It's funny, the teams I've worked with, when things are going well, are sometimes just as anxious as the teams I've worked with, when things are going poorly, because they have a hard time attributing success to what they're doing, right?

**Matt LeMay** (01:14:08):
They'll look at it and they'll say, "Yeah, we're doing great. But I think that honestly has more to do with marketing than it does with us."

**Matt LeMay** (01:14:14):
Or, "Yeah, we're doing great, but we're just in the right place at the right time. Our business is doing really well, our sector is doing really well. We're just growing along with everyone else who's growing."

**Matt LeMay** (01:14:24):
And I think, as with most things, if you approach it with curiosity, there's a lot to learn from it. So if you look at it, and you say, "Yeah, we can't necessarily take sole credit for this, but Marketing's doing awesome. Let's talk to them. What are they doing? Why is this going so well? What have they learned from this, that we can work with them more closely?"

**Matt LeMay** (01:14:46):
If your sector is just doing really well, be like, okay, well what about this is working right now? What might change in the future? How can we understand this?"

**Matt LeMay** (01:14:54):
So I think the challenge, when things are going well, is to keep that curiosity going, and to keep celebrating the things, not just that your team is doing, but that everyone is doing, and to take that time, where you have a little bit more leeway, where there's less pressure on you in the moment, to build those connections with other parts of the organization, to see where that really good work is happening.

**Matt LeMay** (01:15:21):
One thing that makes me sad is when teams look upon each other with envy and disgust, right, when it's, "This team's doing, they're doing great work, but they're not held to the same standards that we are, and it's not fair, and they're the team that everyone likes."

**Matt LeMay** (01:15:37):
It's like, "Go talk to them, because there's probably something you can learn from them. And if they're the team that's getting positive attention, that's doing meaningful work, there's probably a reason for that. And if you can understand that reason, and you can align your work, that team's work, that's probably going to be really good for you."

**Matt LeMay** (01:15:58):
Again, I think, in a lot of cases, it comes back down to that control thing, right? If we can't feel a sense of control over, "Things are going well because we did this," we tend to disconnect ourselves from things going well and say, "Well, I can't. It almost feels dangerous to engage with it because, what if we didn't do anything, but good things are still happening."

**Matt LeMay** (01:16:21):
And I think, just keeping that openness, keeping that curiosity going, puts you in a much better position for when things inevitably begin to either plateau or decline, which is the cyclical nature of all things.

**Lenny Rachitsky** (01:16:34):
Wow. Profound there. To start to wrap up our conversation, what's maybe one question from the book, or even from this conversation, that you think product leaders, product teams should ask themselves to help them solve problems, that maybe they're not seeing, and to help their team operate better?

**Matt LeMay** (01:16:53):
I mean, I love the, "Would you fund this team, if you were the CEO question?" I know that that can feel a little confrontational sometimes. So, a gentler way of bringing that up is to say, "What's one sentence you'd want to be able to say at the end of this year, that would leave you feeling awesome about this team's work? What's something?"

**Matt LeMay** (01:17:11):
At the end of the year, everything went really well for this team. You run into the CEO in the hallway, and you say, "I feel awesome, because we did this." '.

**Matt LeMay** (01:17:20):
And you'll sometimes get very different answers from different people on the team, building on those answers, saying, "Okay, this is what you care about. This is what I care about. This is what I'm worried about. This is what I'm excited about."

**Matt LeMay** (01:17:33):
That's how you get that magical, people in a room building together, or musicians in a room writing their next song, right?

**Lenny Rachitsky** (01:17:40):
What a call, back to the beginning.

**Matt LeMay** (01:17:40):
And then-

**Lenny Rachitsky** (01:17:40):
I love this.

**Matt LeMay** (01:17:43):
It's getting that conversation out in the open, and getting people to talk about their perspectives and their expectations, with the goal of bringing it together, to create something better.

**Lenny Rachitsky** (01:17:55):
Bam. Matt, is there anything else that you wanted to cover or leave listeners with before we get to a very exciting lightning round?

**Matt LeMay** (01:18:03):
I think we covered it. I just leave people with, again, this hope that you can start to see the commercial realities and constraints of your business as opportunities, as guideposts, as things that shape and structure your work, not as things that you single-handedly need to change, or break down, or transform, because you probably can't. And that's okay.

**Lenny Rachitsky** (01:18:31):
That's empowering stuff. With that, we've reached our very exciting lightning round. Are you ready?

**Matt LeMay** (01:18:37):
As ready as I'm going to be.

**Lenny Rachitsky** (01:18:39):
Question one. What are two or three books that you find yourself recommending most to other people?

**Matt LeMay** (01:18:43):
Radical Focus, by Christina Wodtke, is obviously, I cite it a lot in my book, I think it's the best book about OKRs. I love her style of writing. I love the way she tells stories.

**Matt LeMay** (01:18:56):
I love the focus on focus, this idea that again, the way you do it is less important than the result of the way of doing it. I just find that so inspiring, always.

**Matt LeMay** (01:19:08):
The other book, this is a bit of a curveball, is by Alan Watts, the Buddhist philosopher. It's called The Wisdom of Insecurity. And a lot of the way I talk about control, and giving up control, comes from that book.

**Matt LeMay** (01:19:23):
It also has a concept. It opens with a concept called the law of reverse effort, which is that sometimes, the harder you try, the worse you make things. When it comes to setting goals, doing OKRs, doing strategy, I think about this a lot, that there is a tendency sometimes to put effort towards the overcomplication of things, which in turn makes those things less effective, what they're ultimately intended to achieve. So, highly recommend that book. It helped me a lot through some dark times, mentally.

**Matt LeMay** (01:19:59):
There's an idea in the book. He talks about how trying to keep yourself perpetually safe in an imagined future is effectively killing yourself to the present. And ooh, that hit hard. That hit me really, really, really hard. So, highly recommend that book.

**Matt LeMay** (01:20:18):
If the things we've talked about today make you even a fraction of, as anxious and fearful as they make me when I navigate them in my work, then I highly recommend The Wisdom of Insecurity, by Alan Watts.

**Lenny Rachitsky** (01:20:32):
I was going to ask what the title is, because that sounds really powerful. Amazing, amazing pick.

**Lenny Rachitsky** (01:20:37):
Second question, do you have a favorite recent movie or TV show you've really enjoyed?

**Matt LeMay** (01:20:41):
God. So especially, when I'm in work-y write-y mode, my brain shuts off to all things that are not trash. So I have been watching a lot of trash.

**Matt LeMay** (01:20:53):
I watched Temptation Island, the new season of Temptation Island on Netflix. I loved it so much. It's Mark L. Wahlberg, who is the host of Antiques Roadshow, just weaponizing therapyspeak to torture people, in this bizarre scenario.

**Matt LeMay** (01:21:15):
And I should not love it, but I am only a human, and a messy bitch who lives for the drama, sometimes, so ...

**Lenny Rachitsky** (01:21:25):
So good.

**Matt LeMay** (01:21:26):
I know. I enjoyed that very much.

**Lenny Rachitsky** (01:21:27):
I think it's the first Temptation Island call-out, and I love it. Okay, favorite product or ... Yeah, favorite product you've recently discovered, that you really love?

**Matt LeMay** (01:21:37):
Yeah, so I'm really glad you asked this question, because as a musician, I live in this world of niche products, which is so much fun, because the economies of scale for music creators are just totally different, right?

**Matt LeMay** (01:21:48):
You don't need to build a product that's going to have a billion users, and create a huge return. You can do something that a few hundred or a few thousand people get a lot of use out of and love. You're in this fun realm of hybrid, physical and digital, and different types of things. And when you asked me this question, the first thing that came to mind was this, which is made by an amplifier builder who goes by Milkman Sound, in the San Francisco Bay area, who took, basically, the circuit of a Fender amplifier, and made a version of it that fits into this handy, compact, lightweight form factor.

**Matt LeMay** (01:22:35):
So for me, as somebody who is no longer 22 years old, and still occasionally tours with my friend's band, and has back problems, the fact that I can carry this really powerful amplifier around, in this really well-designed, aesthetically pleasing form factor, just feels like such a joy. It's one of those products that really understands what quality means to its particular audience.

**Matt LeMay** (01:23:09):
And I can go on tour, and have things sound really nice, without having to lug a giant, heavy thing. And I love that. It's something for which I am truly grateful.

**Lenny Rachitsky** (01:23:24):
This makes me want to play music randomly, but just so I understand how this works, do you still plug that into a speaker, or is that-

**Matt LeMay** (01:23:30):
Yeah, so I have a little speaker.

**Lenny Rachitsky** (01:23:32):
Okay. Oh, okay.

**Matt LeMay** (01:23:32):
You can also plug direct into the soundboard at a club, or for recording.

**Lenny Rachitsky** (01:23:36):
Okay, got it.

**Matt LeMay** (01:23:37):
And it has a little headphone jack, too. So if I want to practice in the hotel room, before you play, I can also use this.

**Matt LeMay** (01:23:45):
It does everything you would need this device to do, and nothing more, and it does it really well. And that is so delicious to me, to have a product-

**Lenny Rachitsky** (01:23:59):
Another great pick.

**Matt LeMay** (01:24:01):
... that I'm like, "This does exactly what I need. It does it so well, and it's a delightful product."

**Lenny Rachitsky** (01:24:11):
Two more questions. Do you have a favorite life motto that you often come back to, find useful in work, or in life?

**Matt LeMay** (01:24:17):
I come back to something my mom told me. I'm sure you've experienced this, too. When you're a person creating things in the world, there's a tendency to look at the numbers, and look at, "How am I doing? And did this resonate? Is this good? Did this work?"

**Matt LeMay** (01:24:31):
And I've definitely worked on things, especially in my musical life, where the amount of effort and love, and time and care that went into something, didn't feel like it showed up in the result of that thing, or the way it was received.

**Matt LeMay** (01:24:48):
I was talking to my mom about this once, and she said, "I really believe that no good work is wasted. Even if you don't see it right away, when you put good work out into the world, even if it only finds a few people, it's never wasted. If you do work you really believe in, it's going to have some positive effect on the world."

**Matt LeMay** (01:25:10):
And that really stuck with me. I've carried that with me, even when I think about how to write, and what to say, and when I start to worry about outsmarting myself, and should I be more strategic in the way I message this, and should I say something I don't really totally believe?

**Matt LeMay** (01:25:34):
There've been a few times, not recently, but a few times, 10 years ago, where I dabbled in writing click bait-y headlines, because I was like, "Ooh, is this going to get more attention?" And there was one in particular, that got really thoughtful critique from some people I really respect.

**Matt LeMay** (01:25:49):
I was like, "I don't want to do this. I don't want to put work I don't believe in out into the world," and even if it doesn't do what I hoped it would do, I'm going to go forth with the belief that no good work has wasted.

**Lenny Rachitsky** (01:26:03):
I totally believe that. Final question. You wrote a ton of music reviews over the years. I'm curious if there's one that was just very, people super disagreed with it, or just one that you're super proud of?

**Matt LeMay** (01:26:19):
I mean, I am a villain, in certain corners of the internet-

**Lenny Rachitsky** (01:26:22):
Say more.

**Matt LeMay** (01:26:23):
... for the review I wrote of Liz Phair's self-titled album. I wrote this review when I was 19, when my worldview was toxic, and very much in progress. It was a really condescending, arrogant review, written from a place of unearned condescension and arrogance.

**Matt LeMay** (01:26:46):
Not that condescension and arrogance should necessarily be earned, but it felt righteous at the time, in a way that is dangerous, right? That toxic righteousness where you're like, "I'm fighting something bad." But then you realize that what you're fighting is not actually bad.

**Matt LeMay** (01:27:09):
You're actually punching sideways and down, but you don't have the wherewithal and the maturity to realize that you are not where you think you are, in relation to people and culture, and yourself. So I apologized for that review in 2019, and Liz Phair and I actually had a really nice little Twitter back and forth.

**Matt LeMay** (01:27:34):
This was back when Twitter was still a place where this kind of thing could happen, and it was nice to wrap that up in that way. I still don't feel good about it, because it was playing into some really toxic and hurtful dimensions of discourse.

**Matt LeMay** (01:28:06):
But what can you do, if not look back on it and say, "Yeah, I was wrong. I was just flat-out wrong, and beyond being wrong, I was being a jerk." Somebody posted this, a meme creator on Instagram, posted that review a couple months ago, with a caption, " Matt LeMay, you will be dealt with."

**Matt LeMay** (01:28:31):
And I was like, "Yeah, yeah. If I read that now, I would probably say something pretty similar." So I think, as I said, I'm a conflict averse people pleaser.

**Matt LeMay** (01:28:48):
I went through my kind of edgelord phase. And I don't think it did any good for me, or for anyone else.

**Lenny Rachitsky** (01:28:57):
Wow. That was a much more profound answer than I expected. I can't believe people are still referencing this review from a long time ago. How many years? How ...

**Matt LeMay** (01:29:08):
22? It was 2003, so it was 22 years ago. It was more than half my life ago. But this is the wild thing, is that on the internet, it still looks new.

**Matt LeMay** (01:29:19):
The first talk I ever gave, in London, actually, was at a Red Monk event, and I brought my teenage diary with me, to make the point that if something has 10 years of age on it as paper, it's contextualized by its own material existence, right?

**Matt LeMay** (01:29:36):
But somebody will go, "Oh, I wonder what Pitchfork thought of this record," they'll see my jerk-ass review, and be, "Pitchfork published this jerk-ass review. Who was the jerk-ass reviewer who wrote it?"

**Matt LeMay** (01:29:48):
And they'll find me, and be like, "Hey, you, you're a jerk-ass, because you wrote this jerk-ass review." And I'll be like, "Yes, I did write that jerk-ass review, from the place of being a jerk-ass." But I like to think I am a person who has lived 22 years of life since then.

**Lenny Rachitsky** (01:30:11):
Your frontal cortex has developed more since then. I think it doesn't stop-

**Matt LeMay** (01:30:13):
God, I hope so.

**Lenny Rachitsky** (01:30:14):
Until 25, 26. Matt, this was awesome. We covered so much ground. We've touched on everything I was hoping to, and more.

**Lenny Rachitsky** (01:30:22):
Two follow-up questions. Where can folks find you online, talk about the sort of work you do with companies, and how can listeners be useful to you?

**Matt LeMay** (01:30:29):
Yeah, so you can find me online at matlemay.com, pretty straightforward, and in terms of how folks can be useful to me, hopefully I can be useful to them, whether that's through the books, or through coming into work with your company, and help your teams focus.

**Matt LeMay** (01:30:47):
I find that facilitating these conversations is a really big value add. It's hard to have these conversations to do this without some outside help sometimes. So again, whether it's in book form or in me form, I hope that I can be useful to you, and having these conversations in a way that's less scary, more accessible, and leads you down that road to doing impactful work.

**Lenny Rachitsky** (01:31:11):
Awesome, and mattlemay.com. And then, what's the name of the book? Where do they find it

**Matt LeMay** (01:31:15):
Impact-First Product Teams. You can find it at all the places where you find books, on Amazon. There's an author read audiobook, which I recorded right here in this room, with this microphone. I did all the musical interludes myself ...

**Lenny Rachitsky** (01:31:28):
Why?

**Matt LeMay** (01:31:30):
Because it was a great way to procrastinate, when I did not actually want to read the book. But I hope you will find it fun.

**Lenny Rachitsky** (01:31:39):
Matt, thank you so much for being here.

**Matt LeMay** (01:31:41):
Thanks so much, Lenny. This was great.

**Lenny Rachitsky** (01:31:43):
Bye, everyone.

**Speaker 1** (01:31:44):
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.

**Speaker 1** (01:31:52):
Also, please consider giving us a rating, or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes, or learn more about the show at lennyspodcast.com. See you in the next episode.

---

## [9/18] How Intercom rose from the ashes by betting everything on AI | Eoghan McCabe (founder and CEO)

**Eoghan McCabe** (00:00:00):
You don't have a choice. AI is going to disrupt in the most aggressive violent ways. If you're not in it, you're about to get kicked out of all of it.

**Lenny Rachitsky** (00:00:08):
You have very successfully shifted late stage SaaS business to an AI-first agent-based business.

**Eoghan McCabe** (00:00:15):
Fin is our AI agent who will pass 100 million ARR with Fin in less than three quarters.

**Lenny Rachitsky** (00:00:20):
Let's talk about how you made this actually happen.

**Eoghan McCabe** (00:00:22):
We were about to hit $0 net new ARR, which means we would've been in negative growth territory.

**Lenny Rachitsky** (00:00:28):
So, ChatGPT launches. Was it just like this is it, we got to go all in on this thing?

**Eoghan McCabe** (00:00:33):
I said, we need to become a wartime company. If we don't fight for this, we are dead. I jumped hard on AI, but I also restarted the culture. I rewrote the values designed to be a sharp knife to cut out the parts of the company that I just knew wouldn't be effective.

**Lenny Rachitsky** (00:00:48):
If you're trying to make the shift and it's just not moving, you may need to go hardcore founder mode.

**Eoghan McCabe** (00:00:52):
The way that greatness is created is that you find a CEO who's willing to make brave hard decisions and own the results.

**Lenny Rachitsky** (00:00:59):
What percentage of the employees kind of turned over during this period?

**Eoghan McCabe** (00:01:02):
Ultimately like 40%.

**Lenny Rachitsky** (00:01:04):
You said there was a soft coup. Is there more you could share about that?

**Lenny Rachitsky** (00:01:09):
Today my guest is Eoghan McCabe. This is the first in a series of conversations that I'm having with founders who have successfully transformed their established SaaS or marketplace businesses into an AI first company that is growing like crazy and overtaking their decade plus old business. So, many companies and product teams and founders are trying to navigate this very tricky time where every industry is being disrupted by AI, and my goal here is to help you essentially disrupt yourself before somebody else does. The story of Intercom's transformation into Fin is incredible. Their traditional business was valued at billions of dollars, was making hundreds of millions of dollars in ARR, but growth started to plateau and was even about to go negative. Six weeks after GPT-3.5 Came out, they had a working prototype of what is now Fin and Eoghan and the team decided to go all in on AI.

**Eoghan McCabe** (00:05:09):
Thank you. Great to be here.

**Lenny Rachitsky** (00:05:10):
You have done something quite extraordinary with Intercom, something that a lot of founders and product teams are trying to do, which is to navigate this very scary disruption that's happening as a result of AI to most businesses. You have very successfully shifted, as you described, a late-stage SaaS business to an AI-first agent-based very successful business. I want to use the time to extract as much as I can out of your journey so that people that are trying to navigate this and having a hard time can have less pain, less suffering and will hopefully get to something that works. To give people a sense of just how well things have gone. Can you share some stats about the current state of the business, how it's going?

**Eoghan McCabe** (00:05:53):
Currently across the business we benchmark ourselves against all public software companies. There's like 120 something B2B software companies. We're like in the 15th percentile for ARR growth, so we're up there. Fin, which is our AI agent, which is the future of the business, the thing that will disrupt the old business. It's growing north of 300%. It took off really fast like all these other AI companies you hear of the first year, it grew from one to 12 million ARR. We're now in solid mid eight digit ARR growth there, we'll pass 100 million ARR with Fin in less than three quarters. And yeah, Fin, we're in the customer experience category with Fin, so it's one of these agents that helps do all your customer work and they all started with service and in that category we are the biggest by customer count, biggest by revenue, best by performance benchmarks. We win all our head-to-heads and our direct competitor bake-offs. We're rated number one on G2, so I think we're doing pretty well. We're doing far better than we imagined at this point.

**Lenny Rachitsky** (00:07:00):
Okay. This sounds like the dream for a lot of founders, especially ones that are stuck with their existing business that isn't going very far. So, let's get to that. Let's talk about the beginning of this journey. You had a business that was working, people used it, loved it, over 100 million ARR I believe. Talk about just the state of the business at the point roughly when you decided I really need to make a big change and go AI first.

**Eoghan McCabe** (00:07:23):
It was already in the hundreds of millions, Intercom is 14 years and change now. Part of the story is that in 2020 I had been sick for a couple years. The background is I had mold toxins and later I found out that I got a tick bite and that messed me up. And so I left the CEO role in 2020 and a lot of the mistakes I had been making when I was sick got worse. We became what a lot of late stage software companies are today, which is a bit bloated. We lost some energy. Our strategy was diluted and unfocused. We're trying to do all the things for all the people.

**Eoghan McCabe** (00:08:06):
We didn't know what problems we were already solving and for who, and the result was very slow revenue growth in the low single digit percent and I was away for two years unsatisfied where the business was going. We had this post COVID sugar rush, which a lot of big companies at that stage did in 2021. Everyone's valuation and revenue was through the roof and that hit a lot of problems in a lot of these companies and we had five quarters of success of sequential decline in our net new ARR and we were about to hit $0 net new ARR which means we would've in negative growth territory. We never got there. I managed to stop it before we got there, but we were falling each quarter and I found that I, despite my wishes to go and have new adventures, still had a lot of pride for this damn thing and didn't want to see it.

**Eoghan McCabe** (00:09:15):
And in a way that was so different from the way it started, it started with so much hope and optimism like so many companies do, and it was about to fade away. So, that was when I felt like I need to go back and I need to make a change. I went back and one month later ChatGPT was announced, so it would be really neat and tidy to be able to say that the AI transformation came, I knew I couldn't be on the sidelines, I had to save this thing from the coming disruption. Actually, I got whacked across the head by this AI thing, but it also ended up being a gift.

**Lenny Rachitsky** (00:09:55):
So, ChatGPT launches, was it just like this is it, we got to go all in on this thing? Was it like, hmm, let's watch this thing? How quickly was it clear that this is the future, this isn't working what we're doing?

**Eoghan McCabe** (00:10:06):
We and I were very lucky and that we had an AI group already. We were in the customer communication business, chiefly doing customer service. We were building bots, but they were rudimentary AI. We had a bunch of our own machine learning that did Q&A for customer service, but it required a phenomenal amount of setup and was kind of crappy. But we had a number of AI engineers in the company already, and so when GPT 3.5 came out, they said, this is different and it didn't take long for people to start to imagine that this is going to be pretty disruptive to service. And it started where we imagined that this was going to just wreck everyone selling seats, everyone in the conventional SaaS game.

**Eoghan McCabe** (00:10:52):
And we believed that was quite possible for some couple years after that moment. But we were only six weeks into the launch of GPT 3.5 when we actually had a beta version of Fin. I got a text from Des, my co-founder, a week or so after the launch of GPT 3.5 and he said, "The AI team have something interesting and they actually think we could make a product out of this." And this was long before there's now no doubt 100 service agents. We had something very early working and part of what we had to our advantage also was that we had this giant base 300, sorry, 30,000 paying customers, hundreds of thousands of active users, millions of their users, billions of data points.

**Eoghan McCabe** (00:11:42):
So, we had a lot to play with and so we jumped on it. Now obviously it's fun to tell that once again to the idea of this brave Maverick move, and I won't discount the fact that we were brave, but we were coming from a point of having nothing to lose. So, we certainly are unique. I don't know a single company of our size and age that has pivoted this hard to AI and being as successful as we have been, but we also previously were screwed. We were in a really tough spot, so had no choice. So, I'll take the kudos and credit, but also have a lot of empathy for companies that weren't as in as much trouble as we were, and so try to thread the needle and sustain the old business while adding to it with the new AI stuff.

**Lenny Rachitsky** (00:12:34):
Something I heard from someone that worked at Intercom, correct me if this is not correct, you've always been very anti-bot in the customer support business because you didn't like how impersonal it was. It just didn't feel like the way you wanted to build a business, and then now that's what you do. Talk about just that transition.

**Eoghan McCabe** (00:12:50):
Yeah, I know it's a fun and ironic twist. Our mission from the early days was make internet business personal. And when I came back and we started to lean into AI, I started to wonder, does that mission make any sense anymore? Now, part of our lean into AI is that we had no choice not only for the business, we needed something new, but also we saw that this is the future and you can't fight the future. You must be part of it. And so okay, fuck, we're going to be part of it. And ultimately, and it's very easy to tell yourself these little stories. So, I'm open to anyone telling me this is bullshit, but when I interrogate myself, my soul and my mind, I don't think it is, when I interrogate my heart and my mind, I don't think it is.

**Eoghan McCabe** (00:13:43):
But I'm now of the belief that providing a customer with a highly engaged, instantly available expert, consistent, fast, charismatic, funny, friendly, personal agent available for literally every single customer every minute of the day around the clock is so much more personal than making them wait 2, 3, 4 days for a crappy canned response. And so that's the irony and the magic and the wonder of AI, even if it does make us ask some hard questions of ourselves and think carefully about its impact on humanity, it actually is superior at the things we describe as personal and human, relative to humans themselves. And so that's where I'm at today. Yeah, maybe it's a bunch of fancy post rationalization, but honestly that's really where I stand.

**Lenny Rachitsky** (00:14:59):
I think data has shown people often prefer not to talk to humans just to solve problems that can just be solved. It's a lot of stress to try to figure out how to talk to some support agent that doesn't know anything about what's going on.

**Eoghan McCabe** (00:15:10):
Yes, and the AI is just better. Look at Waymo. So, Waymo doesn't crash. It has 3.5 times less crashes than humans. It doesn't bother you or bug you. I like to chat with an Uber driver as much as the next guy, but not always. It doesn't have hygiene problems. It doesn't take wrong turns. I mean it just doesn't do all these things that really bug people. And it's really interesting to see Uber now offer women the option to call only female drivers, and I guarantee the reason they're doing that is because women love Waymo because they feel safer. AI is so often superior and humans are going to be far better at other things. I'm pro-human, I love humans. I really want humans in the mix for all things in the rest of my life, but when it comes to practical, productive, efficient, and effective value, the glue in between the human parts of our lives, I actually want AI and robotics.

**Lenny Rachitsky** (00:16:21):
Before we start talking about how you actually made this transformation a success, one other piece of history is just your pricing strategy historically has been not liked by people. For example, I once had a Twitter poll or a survey on my newsletter just like what products do you pay the most for of all your SaaS products? And Intercom's by far the most. I know people constantly complain about just how unclear it was and how high it was, and now you guys are at the forefront of how to price AI products. So, we're going to get to that, but just talk about the lessons and what happened there with pricing back in the day.

**Eoghan McCabe** (00:16:53):
Yeah, so I want to just validate your survey data. Yes, people abhorred our pricing. It was a meme. There were actual funny popular viral memes on Twitter that were making fun of our pricing. Part of the problem, if not all of the problem, well, there's two problems. One was our strategy, super unfocused. As you said, we're trying to do all the things for all the people, and when you're trying to do all the things for all the people, your efforts to capture all that different types of value are going to necessitate pretty complex pricing. If you're like customer service and you're selling seats and you're doing outbound messaging and you need to charge for messages and you're doing like SD or Messenger on a website and you need to charge for leads, already that's just metrics in every direction.

**Eoghan McCabe** (00:17:49):
And then if you're trying to sell to many different sizes of customers, you need tiers and gates and it just became a behemoth. So, part of the problem was the unfocused strategy, and then the other part of the problem was an unwillingness to frankly make bold decisions, say no, pick a lane and actually take pain in the short term for the long term. We rolled out this new pricing, and this is even before the Fin pricing you're talking about. When I came back and I said, "Yes, we're going to lose a lot of revenue here." I can't remember how much we wrote down, but we actually have already given away something like $50 million in ARR.

**Eoghan McCabe** (00:18:32):
We've reduced the prices for a lot of customers just to give them way simpler pricing because surprise, surprise, when people feel like they have far simpler, more predictable, fairer pricing, they'll stick around longer and it creates so much more ease in the company and promotes a healthier relationship with the customer too. When our people saw that we were screwing customers effectively in every direction, it starts to erode the idea that we care about our customers and then they make other customer-unfriendly decisions. And so one of the values I promoted when I came back was that we would be customer-obsessed, and so we had to kill our old pricing and give away a lot of revenue. So, that was the spirit behind the changes. But we can talk about the Fin pricing if you want to also.

**Lenny Rachitsky** (00:19:22):
Let's save that because that's a really important topic that I think people need to hear. Let's talk about the shift and how you made this actually happen. You make it sound like, oh, not fully, but it's oh, we have to do, it wasn't working anyway. There's no risk to go all in on this AI thing. You're making $150 million a year ARR, you're worth at least a billion dollars at that point as a business. Yeah.

**Eoghan McCabe** (00:19:45):
I mean multiple billions.

**Lenny Rachitsky** (00:19:46):
Multiple billions.

**Eoghan McCabe** (00:19:47):
We were making more money than that, so we were like multiple hundreds of millions.

**Lenny Rachitsky** (00:19:50):
Yeah, okay. Very difficult to actually do even if things don't feel like they're growing anymore. So, first of all, just what was the moment, if there was one, of just like, okay, the six-week experiment of someone building a Fin internally, was that being like, this is it, or was there another moment of like, let's go all in on this?

**Eoghan McCabe** (00:20:07):
It was the combination of the company being older, us all, me and the founders being impatient like, are we going to make something out of this? We went through a time when the company was worth a lot. We're private so we don't have a daily mark to market, but all the other public software companies dropped 80%, 85, 90%. We saw our revenue growth crater. We were used to nice double digits. We were in low single digits. And so part of it was, let's do something here. Another part of it was my own anger and dissatisfaction with how the company was being run and the mistakes that I made myself. I made a lot of compromises as a lot of founders and founding CEOs do to placate employees or do it out fear to bring investors along, following advice in the industry and best practices.

**Eoghan McCabe** (00:21:12):
You betray your intuition in little bits and pieces over the years when the bright spark of your original idea turns into this big, unstoppable, scary corporate beast and a little bit of you dies every single time you go and betray yourself in that way. If you could pick in your mind three or four tech darlings from 10 years ago when you meet the CEO and talk to them privately, very few of them feel outstanding about the state of their culture and the decisions that they make and the way in which they have to work. All of them have betrayed themselves in little ways, and I had left the business, I was super sick, I was burned out frankly from the revenue even having started to slow down before I left.

**Eoghan McCabe** (00:22:05):
I had been attacked unfairly in the press, just all of me was just fed up and I decided to take a very authoritarian, top-down, aggressive founder-first approach to all the things, and I found that deeply cathartic and that was the thing that led to me in part. The other was just good old-fashioned logic and the other was desperation saying, we're doing the AI thing, the AI thing, exciting and sexy. We need some new energy thing here. The new AI thing makes sense. And also just my intuition says, go for it. And so when people tell these stories, they rewrite history in their minds for the stories to be elegant and also so that they support their own self-aggrandized narratives about their brilliance. Actually, it's a big messy cocktail of things. And anyway, that's my attempt at explaining the cocktail.

**Lenny Rachitsky** (00:23:10):
I saw a stat that when you first launched, when you first had this kind of prototype, you were losing money on every transaction that you're charging like a dollar, it cost you $120, something like that.

**Eoghan McCabe** (00:23:21):
That's right. 120 cents. Yeah, yeah.

**Lenny Rachitsky** (00:23:24):
120 cents. Okay. So, there's a lot of vision here if this is going to get to a place where this actually will be great and affordable.

**Eoghan McCabe** (00:23:33):
It's really funny. We charge 99 cents to resolve tickets, customer problems, and we have a higher resolution rate than anyone else, and we are proud of that and we obsess over that. It is the metric by which these agents are assessed, and we wanted our revenue to be 100% aligned with the value that they attained because we had all this scar tissue from pricing prior that felt unfair to customers. So, we said, what's the most fair that we can possibly find? Now, when we did all our research, we found that many SaaS businesses were spending between 20 and $30 per ticket resolved. We were spending 22. Now, consumer businesses, maybe they go down to $5. We were thinking, can we charge $10? That seems fair. It's half price. Can we charge $5? Can we even charge two and a half dollars? But early on we started to sense that people just wouldn't value the digital work as much as the human work, even though the digital work is better, more consistent, always available, makes the customer far happier.

**Eoghan McCabe** (00:24:49):
And so we actually started to lean into a price that we thought would be was the nexus between us earning the most and it being the most palatable. We basically said that if someone is not prepared to pay 99 cent for us to rapidly and elegantly perfectly and excellently solve their customer's problem, we need to wrap this up. We don't have a business here. So, that was where the 99 cent came from. I always believe that that pricing should come from value and not from costs. The cost is our problem. We just had this sense and intuition early on that this thing will get cheaper and it got a lot cheaper. The margin moves around, but we make a margin that makes this more than worth our while, and we know our customers get an excellent deal and are able to deliver to their customers a level of service that they never could before.

**Lenny Rachitsky** (00:25:50):
It's a very clear pitch. We just had Madhavan the podcast and the pricing expert, and he has this phrase, beautifully simple pricing is where you want to get to. Also, he's a huge fan of outcome-based pricing, which is what you're describing here, where you pay for an outcome. So, you guys are in the magic quadrant of his pricing advice.

**Eoghan McCabe** (00:26:07):
Yes, thank God our pricing wars are over.

**Lenny Rachitsky** (00:26:10):
Finally.

**Eoghan McCabe** (00:26:11):
Yeah.

**Lenny Rachitsky** (00:26:12):
Okay, so going back to how you actually did this thing. So, basically you described what many people think of now as founder mode, just top down, as you said, the third area, and just here's what we're doing. We're not going to sit around waiting for you to give me ideas. What did you do? What did that look like internally?

**Eoghan McCabe** (00:26:31):
There was a couple of things. One was we were burning a lot of money, so I cut a lot of costs aggressively. Canned a bunch of different projects. We had this big glorious office we were about to fit out and I'm like, we're about to hit negative growth territory, stop it. And a lot of companies were really stuck in the prior world where they just were used to being super successful, rich and wealthy and spent like drunken sailors. So, I stopped all of that, got really frugal in ways I never thought I would. I still haven't touched the interior design of this office. I'm in here, even though I call it the hotel Marriott, I'm sick of it. Anyway, that was one. Another was picked a lane. Strategically we were all over the place and I said, "We're doing service." Zendesk had been acquired a couple years prior.

**Eoghan McCabe** (00:27:20):
They were strategically, energetically, culturally dead. They were upsetting customers in the market. There's an opportunity there. We're doing service, forget all the other stuff, even though there was a lot of people in the company saying, well, shit, we still have $80 million of ARR that we're getting from the other thing and we're really good at that, and there's a big opportunity. There's other companies in this space worth billions. It was the type of decision that where I had to practice the professional CEO approach, which is, "Hey folks, what do you all think? Let's take everyone's input. Let's put it all down on a spreadsheet." Everyone had color beside all of the different options that we may take. Let's make a group decision. I said, "Sorry, this is what we're doing." So, I was very dictatorial in that respect. We had no one making decisions, so somebody needed to, even if I had some qualms about the decisions myself, I couldn't predict the future, but someone had to make a call.

**Eoghan McCabe** (00:28:16):
Obviously as soon as AI came around, I jumped hard on AI and announced that we were going to spend nearly $100 million of our own cash on that. We allocated a lot of capital, but I also restarted the culture. We had just a very comfortable culture as a lot of companies did. There was a lot of focus on social issues and a lot of complaining and dissatisfaction, and I rewrote the values designed to be a sharp knife to cut out the parts of the company that I just knew wouldn't be effective. So, I said that people must be resilient, that we had very high standards, that we'd work incredibly hard, that shareholder value was the most important thing that we'd optimize for. A lot of things that were controversial for this prior crowd. And then I designed these quarterly performance processes where not only would you get a mark or a grade for your performance against your goals that quarter, but you'd also get a score for your behavior against the values.

**Eoghan McCabe** (00:29:10):
And I hard coded a formula myself, and so I took it out of the manager's hands to say, if people got below a certain mark, respectfully and lovingly, we would say, thank you for your service. We're going to go forward without you. And so you do that just a small number of quarters and you can start to shape an organization that's design and the image of the values you want to create. And obviously there was a lot of pain, a lot of satisfaction. There was attempts at soft coup, there was letters sent to the board, people really unhappy, but on the other side of it, the people left were the most incredible entrepreneurial, brave, inspiring, happy individuals you could possibly imagine. And then you hire in their image. We ran an anonymous employee survey, I think 15 or 16 months after I started aggressively working through the org and rebuilding the org and rebuilding the culture, and we had a 98 to 99% approval of management, leadership and new strategy.

**Eoghan McCabe** (00:30:19):
And this is coming from me having the lowest Glassdoor rating for a CEO I had ever seen when I came back. So, I just want to explain that being that deliberate about your culture and upsetting a lot of people is the path through which you can create a culture where people are super happy, super engaged, super aligned, and now we have just this highly performant organization. Yes, we're messy in many ways. So, that was a big part of it too. So, it was kind of strategically picking a lane. Remaking how we go to market, the pricing was a really, really big piece that had a big effect. Betting on AI and then culture. And I kind of buried the AI thing because frankly none of this would matter if we didn't bet on AI. So, the story could all be summed up by saying, when you ask what did I do, it was that we built Fin and that changed everything.

**Lenny Rachitsky** (00:31:19):
You said that this was very unpopular. I imagine many people were not happy with all the change and how top-down this was, you said there was a soft coup. Is there more you could share about that? I never heard that story.

**Eoghan McCabe** (00:31:30):
When you make that degree of change and you tell people that they're in control like we did in the previous generation of late-stage businesses, there's going to be some friction when you change the rules. And it's my strong belief that great employees and great companies want and are constructed out of a very clear and strong hierarchy where it is the responsibility of the CEO to make brave and hard decisions unilaterally, yes, using their experts as inputs and be responsible for the outcome. If I make decisions that propel the company in the way that thankfully my decisions have, I get rewards and kudos and I get to go back to the board and say, I want a bigger grant. If I don't, I get fired and I should get fired. If my big, brave, unilateral decisions put us in the toilet, then I have to take responsibility for that also.

**Eoghan McCabe** (00:32:30):
So, that's how in my humble opinion, it should work. And I, for one, don't know of a great company that doesn't work that way. You'll see from time to time, I did this a couple of years ago, people will construct these indexes of the performance of companies that are founder led, and of course this is a self-serving statement, but it's also true. And surprise, surprise, the founder led companies perform substantially better because they have the moral authority and the willingness to take the risks that the professional CEOs don't have the remit for. The professional CEOs are typically told, don't mess things up, and the founders are bored if they're not taking the risk of messing things up from time to time. And so that's in my opinion, what creates greatness and great innovation. But like I said, there will be friction changing a company that's configured for democracy and committee decisions and soft and gentle interactions and communication to be properly founder led and top down.

**Lenny Rachitsky** (00:33:32):
So, a big lesson here is if you're trying to make the shift and it's just not moving, there's a lot of resistance. You may need to go hardcore founder mode and make some significant change. What percentage of the employees kind of turned over during this period?

**Eoghan McCabe** (00:33:46):
Could be something ultimately like 40%. So, it was a big, big turnover over some couple number of years. Often the culture is set by a very small number of people, so it only took a quarter to really start to change the tenor of the conversations that were happening, but to bring in the people that were that new level of ambition and wanted to work as hard as the rest of us and work in a mature and engaged in excited way, that took a little longer time. There's such a thing as product market fit. There's a thing as founder market fit, there's a thing as founder, product market fit. That's how you're doing it right, but there's also such a thing as employee, founder, product market fit. You have to have the right employees for the type of business you're creating, and there are companies that want the need to be more stable and they're going to want the need to hire more stable individuals.

**Eoghan McCabe** (00:34:48):
There's going to be companies that want to do the highly collaborative, more democratic thing. I wouldn't invest in them, but there's companies that want to do it. If you're an employee that enjoys that, there are a lot of positions out there. There are big companies like Google that do that. There are startups that hire the crazy, young, wild, messy, early startup people, and that's great for them and the company too. So, it's really all about having the right individuals and when you create that, not only do you create great success, but you just create a lot more happiness and balance and harmony. Ultimately, the employees who wanted a more gentle democratic environment, they're not going to be happy in a company like Intercom or Coinbase or any of these strong organizations. They'll be more happy somewhere else. So, even if it requires a little bit of a loving push out the door, I know that you're actually doing them a favor in the medium to long run.

**Lenny Rachitsky** (00:35:49):
I was going to say that a lot of these people will be happier working in a different company.

**Eoghan McCabe** (00:35:52):
Absolutely. Who wants to go to war every day with your organization and in Slack? That's just not fun. That's not good for the nervous system or the soul.

**Lenny Rachitsky** (00:36:02):
Yeah, so this whole period sounds very stressful for you. Did you ever regret coming back and just like, what the hell did I get myself into? What am I doing to myself?

**Eoghan McCabe** (00:36:12):
I never regretted coming back, but I have many moments where I don't enjoy the job. I didn't regret coming back because it was deeply cathartic for me. When a founder runs away from their business, it is the ultimate betrayal of their heart and the dream that they have. Now, it's okay to wrap things up and quit, but when you kind of run away, like I kind of had to because I was sick and burned out and kind of disenchanted, I don't know, it didn't feel good. So, especially when I had done that, having betrayed in a million or a thousand small ways, my intuition, there was something I needed to exercise. So, it has been deeply meaningful in that respect. And then of course, I'm fortunate that it worked out. I get to be on the second most popular podcast in tech. I get to pat myself on the back in front of all these people. Who wouldn't want that?

**Eoghan McCabe** (00:37:13):
That said, the reality is that for particularly people like me who like the adventure and the high agency being unilateral, day-to-day movement where you're trying to make big, wild, bold decisions, the reality is that if you're successful, most of your days will not be that. It'll be reviewing the bonus policy for next year and reviewing the comp proposal for your execs for the next year. It will be showing up for accountability meetings and stepping through the status of different work streams. It'll be rushing from meeting to meeting, having 8, 9, 10 meetings a day. I don't happen to believe that that's a great way to live your life. It'll be trying to get to all the emails you need to get to such that all those people aren't offended and hurt and trying to communicate in the ways with your staff and your team that is empathetic and thoughtful and keeps in mind that they may be having as shit a day as you are.

**Eoghan McCabe** (00:38:33):
You're giving me an opportunity to paint story of this maverick led adventure that you might imagine in a comic. I'm for some reason picturing TinTin sail the seas, this swashbuckling adventure. It's not. It's corporate life kind of sucks particularly for people like me. So, I have many of those days, and so the only reason I'm still around is that I have a broader mission that makes it worthwhile for now, but that's why you see so many of our best founders get to a point where they're like, okay, I've had enough corporate fun. So, that's the most authentic answer I could give you. No regrets coming back, but plenty of pain on a day-to-day basis.

**Lenny Rachitsky** (00:39:22):
**Eoghan McCabe** (00:40:35):
The first thing I'll say is that CX is deceptively large given it's hidden behind just two words, two letters. Customer experience really is service success, sales and marketing, in my opinion. It's all engagement with all customers. It's the biggest part by headcount of any business. Any consumer business and any B2B business the biggest organizations are sales, service, success. So, I'll talk about things other than CX in a moment, but I want to emphasize that CX is the majority of business operations. Of course, it'll go beyond CX. Any function that requires a lot repetitive operational mechanical work will be automated, whether it's chasing or collecting or issuing invoices, it could be onboarding or offboarding employees.

**Eoghan McCabe** (00:41:47):
There are so many repetitive jobs in an organization that it'll start to replace. One of the interesting questions is how much will be generic operations bots, how much will be expert agents? There are expert agents for law and contract review. There will probably be expert agents for accounting, but you'll need the glue in between all of these agents too. But future organizations will be agents everywhere. I've spent quite a bit of time thinking about what does it all look like in the future, and I imagine it as a medley of humans and agents, and I don't think it's obviously going to be humans on the top and the agents all in the IC roles. I think that'll be more of a complex mix where you're going to have people that are like managers and leaders, but they'll be in IC roles, working with agents to configure them for success and monitor and manage their progress, kind of add that oversight and cover for edge cases. And so I think we're going to be surprised in which the way that these organizations go, they'll definitely be smaller, they'll be flatter because of that.

**Eoghan McCabe** (00:43:17):
I won't be surprised if there are agents at the highest level too. I mean, I've been thinking about how, and we do have a great human chief of staff here, but imagine a future human chief of staff that understands your priorities and actually talks to you and does a check in each day and reaches out to different people and ask for updates and helps organize your priorities and helps you remember who you need to keep accountable. Clearly there's an opportunity for that. And so you can imagine agents in specific roles like customer service in operational roles being glue and in being kind of like co-pilot or assistant roles like that, which I mentioned. But what I think that all brings is just epic levels of efficiency. It's going to be super deflationary. There'll be a lot more competition. AI itself is insanely competitive right now.

**Eoghan McCabe** (00:44:18):
It's so intense in a way that was never before that's going to come to all industries when so much of their inner workings becomes automated. And ultimately I think it's going to be great for the consumer. They'll have more options, cheaper options, and I can't but see that be great for the economy, a lot of economic lubricant as it were, and a lot of new movement and activity. And if we were to really go off the reservation, but I'll stop here, that means that we need more humans too. We need population growth to show up for this big growth economically. And yeah, I just see the future as just a beautiful collaboration between humans and agents in every direction.

**Lenny Rachitsky** (00:45:02):
I love the optimism. Someone described this once as a society ... What is it? Agentic society where it's us and agents living together.

**Eoghan McCabe** (00:45:11):
Right.

**Lenny Rachitsky** (00:45:12):
This begs the question around just jobs. We had Marc Benioff on the podcast. He's all agent force, agent force, agent force, and asked him just like, what jobs do you think are going away? He's just like, "CX, going away, gone. Sales not going away. We need sales people." Just what's your sense? I know it's like touchy subject. No one ever wants to say jobs are going away, but just what's your sense of where jobs might be disappearing more, most?

**Eoghan McCabe** (00:45:34):
Yeah. Well, I don't find it to be particularly touchy because jobs have always gone away and technology has done a really good job at stealing jobs that we're repetitive, demeaning, dangerous. We have less people losing limbs and dangerous factories or dying and suffocating down mines because of the technology that we now have available to us. People breaking their backs on farms or just doing things that's highly demeaning to the great, beautiful creative potential of each human individual life. So, I won't apologize for competing with or competing for shit work because all the while technology has done that in the past, population has increased. GDP has increased, longevity, crime rates have diminished in the western world, the world that has enjoyed the most technology. So, we have no good reason to not believe that that won't continue. Even while there is difficulty and there has been in the past, no doubt, people who were gainfully employed in dangerous work in mines had to find new work.

**Eoghan McCabe** (00:46:47):
And so I don't take that for granted, but I think that this is part of the long arc of humanity flourishing and getting healthier and happier. What are the types of work that will go away? It's all the demeaning, crappy stuff, and that exists in digital businesses. You ask a human to sit at a keyboard answering the same question day in, day out, and you get to a point where you don't even ask them to answer the question manually. You ask them to click the button that brings up the macro. Like what a horrible use of a human life. I've met thousands of people that have worked at Intercom, a broad range of talents. People who they might not describe themselves as particularly high IQ. Maybe they were suited at that point in their life for this highly repetitive work. You talk to them for two or three minutes, you'll see the bright spark of a beautiful human that if they got to do the right thing, they would light up and bring so much happiness and joy to the world.

**Eoghan McCabe** (00:48:00):
And so that's the mission we're all on. I'm not pollyannaish here, like I said, and I'm suggesting that there won't be friction, but for the most part, we're doing good. And to get specific, they will be CX roles and a lot of basic repetitive roles. There is a lot of repetitive stuff in sales, and so you'll do more sales with less people. There are SD or roles qualifying basic questions. You're not going to need as many people in sales organizations. So, I'm a little misaligned with Marc in that respect. But what he's getting at is that what sales people bring to the table is human connection and trust, and that is not about to go away anytime soon. And thank God for that.

**Lenny Rachitsky** (00:48:46):
I had Ben Mann, the co-founder of Anthropic on the podcast recently, and he said that he's like, "Even my job is probably going to go at some point." He was like, "Lenny, your job is going to be replaced by AI at some point." That was pretty compelling. Did not expect him to say that.

**Eoghan McCabe** (00:49:02):
Yeah, I don't know. It will in many ways. We're going to have agents in AI to aggregate content and create content, but humans, as much as when it comes to productivity, value, efficiency, efficiency is not the number one thing that we value. If efficiency was the number one thing we value, I'd always buy the cheapest clothes, furniture, computers, even paper for my printer. But I think humans value things like beauty and human stories and human heart and connection. And not only will they still want those and they'll still want a Lenny that has his own story and his own take and opinions and is a little imperfect, but they'll pay more for it. The abundance of AI is going to make automated things worth zero. Just like the value of cheap content on YouTube. Why do people subscribe to some channels and pay more? Why do people pay to rent movies? Because some things have more quality, more beauty, more craft, more art, more humanity. So, I think there'll always be a place for that.

**Lenny Rachitsky** (00:50:33):
Phew. All right. I've got a couple more years at least.

**Eoghan McCabe** (00:50:36):
Yeah.

**Lenny Rachitsky** (00:50:38):
Before I move on to a different topic. Just kind of reflecting back on this shift to Fin and the success that you've had, are there any other just lessons that we haven't touched on that you think might be helpful for folks that are trying to go through this journey?

**Eoghan McCabe** (00:50:52):
I think it's ultimately that you don't have a choice. My co-founder Des is writing a book at the moment, and that's core to the idea here. You don't have a choice. The story of the technology industry or digital technology is really short and it's punctuated by a small number of things, microprocessors, personal computers, the internet, maybe mobile. Now there's AI. I think AI is bigger than all these things. And all of these things disrupted essentially all categories. So, not only is this likely to disrupt all the categories, it's going to disrupt it in the most aggressive violent ways. And if you're not in it, you're about to get kicked out of all of it. And so my strongest advice is roll your sleeves up, figure out what's going to disrupt you, have fun with it. You need to bring in actual talent. We and I will be nothing if we didn't have actual AI scientists and leaders.

**Eoghan McCabe** (00:51:59):
It's the only way we can be successful here. We have an incredible person who by the time this is out will have received a promotion to chief AI officer. I keep announcing all these things and that's great confidence to you. Fergal Reid, and he's just one of the very best in AI applications, and we happened to be working with him for many years. So, part of it is finding the talent and part of it is bringing in the young talent too. AI is kind of a young man's game, and I'm young, but I'm not as young as a lot of the kids building AI. And so learning to empower and enable them and learn from them too is a really big deal. And unfortunately, part of what you learn from them is the only way you're going to win right now is if you work your ass off, because all these little AI companies run by kids in their twenties are literally working 12 hours a day, literally 365 days a year.

**Eoghan McCabe** (00:53:01):
No joke, all of them. And that's not a fun idea for many of us, especially those who've grown up. Some people in our generation have kids or a lot of them do. There's comfort and stability in your life. You don't want to work like that, but if you want in, that's part of the price and that's how so many of these young new AI are going to win because very few of the previous generation companies are willing to make all of those changes and go all the way in. And so my actual advice, which is not that helpful, is that if founders of previous generation companies are themselves not willing to roll up their sleeves and get into it and work as hard as the kids, hire a kid. You can be a chairperson like I was, have a lot of fun. You can mentor the kid, hire a kid because you're in the wrong job, buddy.

**Lenny Rachitsky** (00:53:58):
I love how pragmatic this advice is and what's interesting as you talk about 12 hours a day every day, it's like we're trying to get close to what agents are doing, which is half, that's basically 50% of agents.

**Eoghan McCabe** (00:54:08):
But that's not just a poetic cute thing to say, that comes from something very real, which is these younger companies know how to use AI in ways that the older companies don't. The younger companies are vibe coding and using AI for their creative work and for their job descriptions. I guarantee you go to companies of our generation and even we have had to push people, you go to companies of our generation, most people in most organizations, particularly non-technical organizations, they're not using any AI. Maybe they're starting to use ChatGPT to write a job description, but they're not doing it by default. And so that's more than a joke. You're competing with young companies that are in part AI.

**Lenny Rachitsky** (00:55:02):
This reminds me, I did an interview with the Perplexity founders. It was, I just checked, April 2024, so just over a year ago. And they were saying that the way they operated, and this sounded it's so crazy at the time, is anytime they had a question for anyone else on the team, they first asked ChatGPT about it, and then they go ask the person as like, that is insane. And now this is just obvious. That's what we all do now. Just like, hey, I'm just going to talk voice.

**Eoghan McCabe** (00:55:28):
It's a prime example. They're doing many such things. When I say 365 days a year, they're the company I think of because they're doing exactly that. All these young companies are doing wild, weird and ridiculous things that people you are in my age kind of chuckle at, but it's business as usual for them. So, there's just a big mind shift, cultural shift, and there's a culture clash of the previous generation versus the new generation. And the sooner you kind of wrap your head around that, the sooner you can start to unstake yourself, I think.

**Lenny Rachitsky** (00:56:04):
And just to build on that, the sounds crazy to work this hard, it sounds very stressful, not fun. Why would I do this? This sucks. But at the same time, this is, as you said, such an unusual rare opportunity. There's so much opportunity. There's so much wealth being created. There's so many businesses being created. This is the time, if you were to ever work really hard, this is a good time to do it.

**Eoghan McCabe** (00:56:26):
I think so. I don't actually generally promote working that hard. I try to not fetishize it. I actually think a life well-lived includes taking slow walks in nature where you're not thinking about ARR growth or hiring your chief revenue officer, not going to eight meetings a day. Maybe you should go to no meetings a day, certainly not working 12 hours a day. I don't actually promote that in general as a thing one should do with their life. I'm simply saying that if you want to compete and enjoy success in this age, which means you need to be doing AI, that is the price.

**Eoghan McCabe** (00:57:16):
So, you either decide to pay the price or get out. Don't half-ass it. You see all these companies saying, we do AI and they've just sprinkle a little bit of crappy AI and they've got the same cultures. It won't work. The one thing I will say, the one little asterisk to my first point is that all great people and great things have been achieved through hard work. And so I'm speaking out of both sides of my mouth here, to younger people to let them know that every way of living is valid, but people who have achieved things have always worked hard and they find a way to enjoy it too. And particularly in 2025 in AI.

**Lenny Rachitsky** (00:58:01):
I want to follow this thread. I was going to ask you this earlier, but I didn't, and I want to see if this takes us somewhere interesting, just watching you speak and talk. You're very self-reflective, very centered. You have these really good breaths you take when you think about something. I met you a long time ago, randomly at a party when you were just starting Intercom. I don't think you were like that. During this kind of two-year period was there kind a transformation that you went through to kind of become this?

**Eoghan McCabe** (00:58:28):
Absolutely. Yeah. There's a couple things. First, I mean, there's three things that come to mind. Working in a startup for 14 years has a certain way of kicking you in the head many times a day that either kills you or makes you far stronger. So, that's one piece. There's no elegance to that point, but I think we can all intuit that that level of experience teaches you something, you grow up very fast. Point two is I did a lot of therapy. I found this amazing guy 12 years ago. He started a couple of his own tech companies and talked in public. He only coached and was a therapist to CEOs. He's now kind of in a later stage of his career. But this amazing guy, his name is Yosi Amram, amazing guy. I just landed on my feet. I just didn't know who I was dealing with.

**Eoghan McCabe** (00:59:24):
But one of the greatest minds and teachers of the last, I don't know, many decades, people don't even know him, but he's taught and worked with many CEOs and he just helped me get to know me and take time for myself. And people like to hate on therapy right now. I think a lot of therapy sucks and a lot of therapists are not good. And they fear that actually therapy will lobotomize them and turn them into thumb sucking, navel-gazing, soft, irrelevant losers that won't have that edge anymore. And the interesting thing about 12 years of weekly therapy and spiritual work is that it takes your edges off, but they're all edges that are super counterproductive. All the edges that made you an asshole, got you triggered, miscommunicated or fought back when you were insecure, they take all the edges away then help you see yourself and love yourself so much more for who you are.

**Eoghan McCabe** (01:00:42):
Be completely unafraid to acknowledge the things you're not good at, but own the things you are. And in understanding yourself, you understand others better and can communicate in a substantially more connected and authentic way. Great, great therapy and it has to be great, is a recipe for brilliant leadership in my opinion. And then the third part is two years away where I ran away, where I was sick, revenue growth wasn't doing so hot. I unsuccessfully tried to defend myself from a bunch of fake bullshit in the newspapers. I mean I was beat up. And in a moment like that, your ego, any sense you have of your greatness is eviscerated. And that's painful. It can be so painful that many people don't come back from it, and I credit the 10 years at that point, or nine years of therapy I did at that point, plus the support of this therapist, the coach that I had, to surviving it.

**Eoghan McCabe** (01:01:48):
But if you can survive it, what you end up with on the other side is all of those insecure, a lot of the insecurities and all that ego bullshit that made you super ineffective, jealous, or triggered for all sorts of different reasons, it's gone. And your image that you are this perfect, brilliant leader that all successful founders form when they are successful had to die. And the reason that's so good is that that's so limiting. When you have this ego identity of yourself about how fucking amazing you are, then any moment that challenges that is super scary. Anyone who questions it is offensive. And so I credit wherever I am today and I have decades of learning still to go to those three components. And I feel super fortunate to have had all of them, even though the last one sucked, I can finally say, wow, it really helped.

**Lenny Rachitsky** (01:02:52):
Thank you for sharing all that. I'm glad I went there. I want to show you something that I randomly have in my office, my wife just got me that I think you're going to love. It's a piece of art that I think will resonate with [inaudible 01:03:02].

**Eoghan McCabe** (01:03:02):
Yeah. What am I looking at here? So, it's a hand?

**Lenny Rachitsky** (01:03:04):
It's a hand with a snap and then let me see if you can see what it says.

**Eoghan McCabe** (01:03:08):
I can't see what it says.

**Lenny Rachitsky** (01:03:10):
It says ego death now.

**Eoghan McCabe** (01:03:12):
Right. Look at this. Good. Exactly.

**Lenny Rachitsky** (01:03:15):
There it is.

**Eoghan McCabe** (01:03:16):
May all our egos peacefully become smaller and leave this mortal coil. The reality is none of our egos ever die. And even great ... Ram Dass is this great spiritual teacher who died a few years ago and someone asked him on his deathbed something like, "How did you get over your bullshit or your ego?" And he said, "I never did. Just the edges got smoothed away." And this is a guy who had 70 years of the deepest, wildest spiritual work, he acknowledged, no, still my same self. So, the ego is still there and we actually need to acknowledge it and love it. And when you acknowledge it, then it's not a surprise when you're like a little jealous and you're like, huh, I'm jealous. That's funny. Okay. And it's all good.

**Lenny Rachitsky** (01:04:09):
Reminds me Daniel Kahneman who wrote all these books about biases that we have and here's all the ways we're flawed. If people ask him, "Have you learned to live more rationally?" He's like, "Not at all." Knowing all these things about how we're flawed in the way we think all these biases doesn't actually, I can't use it in life.

**Eoghan McCabe** (01:04:24):
We're human. We should let ourselves be human. I think it's beautiful. We're logic systems, but we're also heart systems and body systems and soul systems. So, all of it is good.

**Lenny Rachitsky** (01:04:35):
Okay, I want to go in a completely different direction. The last thing I want to talk about, I needed to mention this. I don't know if you've seen this, but I've been doing research on which companies produce the best product leaders. And I've been doing this by looking at which alumni of companies go on to become CPOs at the highest rate, get promoted the most at their next job, become the first product manager at a future startup, start their own companies. Intercom is coming number one across this research next to Palantir and Stripe, Revolut. So, the question this begs is, what are you guys doing that produces such great product leaders? There's the hiring piece and then there's what they do at Intercom piece. So, what do you think is creating these sort of really big successes from your alumni group?

**Eoghan McCabe** (01:05:20):
Yeah, I don't have a really succinct answer unfortunately. I can say in the abstract, our culture is a very producty culture. So, myself and Des, there was four founders and me and Des Traynor drove a lot, like all the strategy. We're product guys. I was a software designer. I studied computer science, so I'm technical, but never did it professionally. So, the first part is that just product innovation, design just was just core to our culture and people always picked up on that. So, I think good people wanted to work here and we were good at finding good people. The other part was that because we had this sprawling strategy, we had all these products that we needed a complex structure for it and that included lots of PMs and PM groups that we gave a lot of autonomy to. And so the product of our big messy strategy was that we had PMs that got to act like mini CEOs.

**Eoghan McCabe** (01:06:27):
And so I think that they got to learn the broader skill sets beyond designing wireframes and interviewing some customers. They really own it like a mini CEO to some degree. I think there's one other thing which is part to our approach was this deeply first principles thinking methodology almost to a fault, although I don't think it's a fault. I and we would create frameworks for everything. It's like, okay, we want to do these events. Who are the events for? What is the ultimate goal of the event? What's the mechanism by which events work? What are other mechanisms that can achieve that same goal? How do we define success for an event like that? How does the user or the attendee define value? What other things do those people find valuable? We create these complex systems to try and approach everything, but the net effect was we'd have really joined up considered strategy and it's everywhere.

**Eoghan McCabe** (01:07:36):
Like Paul Adams, our chief product officer, I didn't even plan to show this. He made this book recently, The AI Age and the Transformation of Customer Service and it's a bunch of frameworks for how to think about AI, et cetera. So, it's part of what we do. And so we would hire people who are good at that, but we teach that. That's teachable and not everyone does that. And so the conversations that Des and I would have, we still love being on whiteboards. Our very first office, our own office in Dublin, it was a tiny office. One wall was four, five computers, the other wall was just all whiteboards. We loved that we had a whiteboard wall. In our next office we had a room, square room and all walls were whiteboards.

**Eoghan McCabe** (01:08:24):
So, we just love to draw diagrams so you can teach all that stuff. So, yeah, it's just all that good energy product, product energy, first principles, the people we chose. And on the founders side, I was talking to Des about this morning, why have so many Intercom people gone on to be founders? I think it's because we hired founder types and my pitch to people was always come to Intercom, figure out how great companies are built and build it with us and then go on to start your own. I would say that often at all hands. But the irony is that the people we hired back then, the founder types were probably not great employees. They were better founders. I'm not a good employee. And so it'll be interesting to see if this current cohort, we'll get many founders out of this current cohort, but will they convert as well as they did before?

**Eoghan McCabe** (01:09:16):
Because we're now hiring people who want to be part of something bigger. They're more mature and grown up, more stable and consistent. They're part of, they have a certain expertise and a certain lane they want to work in. And maybe they're not the crazy types that went on to start companies, but it's wild. I did see some of that research by you, particularly the one where you show the companies ranked by the number of founders that they have. And I'm like, what is happening? I was as surprised that we were that high as you were because there are many other great companies on that list. So, surprised and proud.

**Lenny Rachitsky** (01:09:53):
I love when people say, I don't really have a clear answer. And then you have exactly a clear answer and it resonates a lot with other companies on this list that I've had on of what the themes are, and I'll just reflect back a few of them. One is complexity. That comes up a lot. And interestingly, most of the other companies in the list, I'll read them real quick. Intercom, Palantir, Revolut, N26, Dropbox, Chime, Stripe, and then Coinbase and Notion is down there. So many are FinTech. Almost all are FinTech. And the complexity there is really high. So, there's a really interesting trend there. Just complexity. Ownership is another one that comes up a lot. Many CEOs, GMs kind of roles, first principles thinking and just going to the bare metal comes up a lot in these conversations.

**Eoghan McCabe** (01:10:38):
Yes.

**Lenny Rachitsky** (01:10:39):
And then hiring senior people, hiring founder types.

**Eoghan McCabe** (01:10:42):
Yes. Like Stripe did a lot of that. I think Stripe did a lot of first principle stuff and founder types.

**Lenny Rachitsky** (01:10:49):
The other thing, we didn't even talk about this, but you guys invented RICE. You guys popularize jobs to be done. Like speaking of frameworks, you guys are a wealth of frameworks that we all use.

**Eoghan McCabe** (01:10:58):
Drowning and frameworks, yeah.

**Lenny Rachitsky** (01:11:00):
Drowning slash changing the way everyone builds product in a really positive way. Okay, is there anything else that you wanted to touch on or leave listeners with before we get to a very exciting lightning round?

**Eoghan McCabe** (01:11:13):
When someone like me comes on a podcast like this, they always have an ulterior motive and that's healthy and good. It's part of the transaction. Some of it is to enjoy feeling like an expert. But my ulterior motive today is to make sure that people understand that Intercom is a fundamentally different type of late stage company. We are a large old startup. Every single way in which we work is as a startup and are competing with and crushing the actual startup competition in our agent categories. And the reason that that's important for people to know is just like I said earlier, that the handicap that good but late stage companies have is that their late stage and people don't mentally put them in the same box.

**Eoghan McCabe** (01:12:10):
They just don't imagine these older companies. If I told you that IBM had made the most wildly innovative coding assistant, you'd find it hard to believe, most people would. It's maybe so interesting such that it would stick in your mind, you need to go look at it. But by default people aren't going to look at IBM. And so I want people to take a new look at Intercom because it's a brand new company and our mission is to help every single type of business deliver impeccable, incredible, beautiful personal service to every single one of their users and people, many thousands of people are using Fin for that today. So, go check out Fin please, fin.ai.

**Lenny Rachitsky** (01:12:58):
And I don't know if you mentioned this at the beginning, but let's mention that you predict that you'll be the fastest growing company across if you were to look at all public software companies next year.

**Eoghan McCabe** (01:13:10):
So, two years ago we were in the low single digits growth rate. We doubled our growth rate and last year we were in the low double digits. This year we're in the 15th percentile of all public software companies. So, you take the 120 something public software companies, we're in the 15th percentile. So, we're getting up there fast and if we sustain this trajectory, and it's obviously dangerous to put these types of things out publicly, but I'll tell you, I look at the charts and it's hard not to imagine where this goes. I think we're going to find ourselves being the fastest growing out of all, relative to all public software companies. So, let's see. But that's the level of shock, surprise and transformation that has actually happened here all because of Fin. So, check in with me in a year and maybe I'll be embarrassed or maybe I'll be feeling like a genius.

**Lenny Rachitsky** (01:14:03):
Or underselling it. This just reflects back on exactly how I started our conversation. You've done something extraordinary at Intercom. I'm really happy that we're here and we're sharing this story.

**Eoghan McCabe** (01:14:11):
Thank you.

**Lenny Rachitsky** (01:14:12):
With that, we've reached our very exciting lightning round. I've got five questions for you. Are you ready?

**Eoghan McCabe** (01:14:16):
Please. Ready.

**Lenny Rachitsky** (01:14:18):
What are two or three books that you find yourself recommending most to other people?

**Eoghan McCabe** (01:14:21):
So, I found I lost the habit of reading as I started to get more and more stressed with my startup. And so I would listen to audiobooks here and there, but the most recent book I read is a book called Nuclear War: A Scenario, and it's a very much a nonfiction and scared the shit out of me. So, if you like nightmares, it'll be beautiful bedtime reading.

**Lenny Rachitsky** (01:14:46):
Excellent.

**Eoghan McCabe** (01:14:47):
Yeah.

**Lenny Rachitsky** (01:14:48):
What's a recent movie or TV show you've really enjoyed?

**Eoghan McCabe** (01:14:50):
I love movies. I want TV to be better, but I very rarely find TV to be great. The first and last TV show I loved was True Detective one that was just incredible. But the last movie I watched was 28 Years Later, and that's by Danny Boyle. I was born in the eighties, grew up as a kid in the nineties, and so grew up with Trainspotting. It was 28 Days Later. Then he made a movie called Sunshine. So, 28 Years Later is a type of movie that's just not made anymore. It's the most nineties movie made since the nineties. It's like very rock and roll and also deeply touching. So, I was really surprised by that. I bet I would love to know that younger generations that are watching this, what they may think, they may hate it, but I love 28 Years Later.

**Lenny Rachitsky** (01:15:49):
So, this is the same person that made 28 Days Later, then 29 years later [inaudible 01:15:51].

**Eoghan McCabe** (01:15:51):
Yeah, Danny Boyle.

**Lenny Rachitsky** (01:15:53):
Wow. Okay. Very cool. I didn't know about that. Do you have a favorite product you've recently discovered that you really love? Could be a gadget, could be an app, could be clothes.

**Eoghan McCabe** (01:16:01):
I very rarely like products, I'm such a perfectionist that it has to be really simple with very little moving parts, like a bowl, to actually be like, good. I've started to get more into coffee. I've been buying products by Fellow. They're remarkably good for consumer products, different, it's on a different level. So, there's some sort of level of taste and craft happening there that I don't see in basically any other consumer hardware type products. And of all things, I'd bought a Porsche 911 recently and that is a beautiful product. The interiors are exquisite and there's still a bunch of shit that is going to annoy you. And so it's far from perfect. So, yeah, perfectionism is sometimes a gift if you're in the business of creating products, but also quite the curse. You're never happy, including with the Porsche 911.

**Lenny Rachitsky** (01:17:02):
I think that's the third time someone recommended a car. Someone recommended, I think Boz at Facebook recommended a fancy Mercedes, and then someone once suggested a Rivian, so now we got Porsche on the list. I always thought maybe one day I'll give someone all the prizes, all the products people have ever mentioned in this. And those are getting, Porsche might be a little high. Okay, two more questions. Do you have a favorite life motto that you find yourself repeating, coming back to in work or life sharing with friends?

**Eoghan McCabe** (01:17:30):
It's trite. It's not sophisticated and it's more of a concept than a phrase, but it's something around the idea that life is short. I'm just so aware that time ticks by and we all live on autopilot. So much of what we do is inspired by either our insecurities or things that other people we look up to or envy do. Very rarely making contact with what we really want and following our hearts and our heads. And we just kind of get stuck in these lanes and just live out our days. And certainly when you get 41 now, you get to 41 and thankfully still very young, anyone in their forties, congrats, should feel good about that.

**Eoghan McCabe** (01:18:18):
But I know if you're in your twenties or thirties, 40 feels old. But when you're in your forties, my experience is that the weeks and the months and then the years go by. It's not a big deal. I'm back at Intercom two and a half years now to any of these kids in AI in their twenties. If they don't get something done or achieved by next month, they'll be so disappointed themselves and so impatient. And in some ways, at least when it comes to productivity, they're better at getting more out of the time. But I'm now trying to get more life out of the time too. So, just, if there is a motto, it's like life is short or memento mori, we're all on the way out. So, make the use of what you've got.

**Lenny Rachitsky** (01:19:05):
Fun fact, I built an app once called Savorable that helped you savor the moment, it was called Savorable and it sent you a text every few hours, I don't know, maybe once a day with a little reminder of way to savor the moment. And one of the texts was just like, remember, you will die.

**Eoghan McCabe** (01:19:21):
Yeah. And the problem is that even that idea, we forget it instantly. And if you start getting text every day, you'll ignore the text. Try to fault. We just don't want to acknowledge that reality on a day-to-day basis. Maybe that's important.

**Lenny Rachitsky** (01:19:39):
Yeah, maybe for the best.

**Eoghan McCabe** (01:19:40):
Yeah.

**Lenny Rachitsky** (01:19:42):
Okay, Final question. Speaking of apps, I was doing research on you in preparation for this and I didn't realize you built Quitter back in the day. I love Quitter. I found it so fun. It basically told you anytime someone unfollowed you on Twitter. So, the question just what happened to that app?

**Eoghan McCabe** (01:19:58):
I think we eventually sold it for 14K.

**Lenny Rachitsky** (01:20:03):
Wow, that's cool.

**Eoghan McCabe** (01:20:06):
On one of these, I think there's a website called Flipper where you could sell websites. It really blew up. It was like a little experiment, kind of a social experiment. It was the first time that I had this feeling that there's no reason someone wouldn't want to use this. Obviously people are going to want to use this. And it was really instructive for me because it taught me that that feeling is possible. You meet so many founders, young founders particularly, and they don't have a sense within themselves about the value of the stuff they're building. Will this be good? Let's get customer feedback. And it is possible to build things that you deeply know makes sense. And that's why my formula for building things was to always build things for myself. And that was what Quitter was like. Followers go up, followers go down. At that point in time, people had 100 followers or 200 followers, and you'd want to know who's not my friend anymore.

**Lenny Rachitsky** (01:21:04):
Oh man, I love that that was your bar that led you to the success later if it's as good as Quitter in terms of product market fit.

**Eoghan McCabe** (01:21:12):
I mean, it had the best fit ever. About everyone on Twitter tried to sign up for it and it broke.

**Lenny Rachitsky** (01:21:21):
Well, I loved it. Eoghan, thank you so much for doing this. I love just how real and open you are about everything and just how much insight you have to share. I also just love the vibe. I feel like I just am more centered just watching you-

**Eoghan McCabe** (01:21:34):
Oh, thank you.

**Lenny Rachitsky** (01:21:35):
... speak. Two Final questions. Where can folks check out Fin, follow you if they want to follow up on anything? And then how can listeners be useful to you?

**Eoghan McCabe** (01:21:42):
Check out Fin, fin.ai. If they want to follow me. I'm E-O-G-H-A-N on Twitter, so it's Irish spelling of Owen. But if they want to be helpful to me, I'd love them to try Fin. I'd love them to have their friends that run any kind of customer operations, try it too. This AI thing is noisy. There's so much hype, but it's also really real. And the weird thing about Fin, even relative to the coding apps, the coding apps are blowing up, and yet there's a lot of people experimenting and kicking tires. You can't kick tires with Fin. We only deliver value when you expose it to your customers and it closes tickets and makes them happy. And so AI is really, really happening. And so if you know anyone out there that has customers, they should be using Fin. It's the smartest, cheapest, easiest way to dramatically enhance their business. So, if they do that, they'll be helping me sincerely.

**Lenny Rachitsky** (01:22:46):
I'm sold. Eoghan, thank you so much for being here.

**Eoghan McCabe** (01:22:50):
Thank you, sir. Pretty fun.

**Lenny Rachitsky** (01:22:51):
This was amazing.

**Eoghan McCabe** (01:22:51):
Yeah, thank you.

**Lenny Rachitsky** (01:22:53):
Bye everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.

---

## [10/18] Inside the expert network training every frontier AI model | Garrett Lord (Handshake CEO)

**Garrett Lord** (00:00:00):
There will never be a time like this. I've never seen anything like it. I doubt I'll ever feel anything like this in business again where there's unlimited demand. How do you make sure that three months from now, six months, you have no regrets? Get on the plane to go talk to a customer, make the late night push, check the data six times over again.

**Lenny Rachitsky** (00:00:15):
Your company creates new data to continue advancing the intelligence of models. This is a business that you built on top of a business you've already had. 

**Garrett Lord** (00:00:24):
We're the largest expert network in the world. We have this massive strategic advantage, which is like no customer acquisition costs. The only moat in human data is access to an audience. 

**Lenny Rachitsky** (00:00:33):
You guys come in after the model's trained to tweak the weights based on additional data that you you've created. 

**Garrett Lord** (00:00:38):
The models have gotten so good that the generalists are no longer needed. What they really need is experts. 

**Lenny Rachitsky** (00:00:44):
There's this tension between all these students training models to become smarter, and then there's that they will have harder time potentially finding jobs. 

**Garrett Lord** (00:00:52):
That's not what we're hearing from our employers, this is just enabling human beings to be even more productive. You used to put a Google Search on a skill on your resume because you grew up with Google. Being AI native, young people are at a huge advantage.

**Lenny Rachitsky** (00:01:05):
Today my guest is Garrett Lord. Garrett is the co-founder and CEO of Handshake, which is one of the most interesting and incredible AI success stories that you probably haven't heard of. Handshake has been around for over 10 years, they're essentially LinkedIn for college students, it's a place for students to connect with companies to find a job. They are the platform of choice for every single Fortune 500 company. Over 1,500 colleges, over 20 million students and alumni, and over 1 million companies use them to hire graduates. At the start of this year, Garrett and his team realized that their huge proprietary network of students, including tens of thousands of PhDs and master's students, is extremely valuable to AI labs to help them create and label high quality training data. So, they launched a new business from zero to one in January. Four months later, they hit 50 million ARR. They're now on pace to blow past 100 million ARR within just 12 months. They'll exceed the revenue that they're making with their decade old business in under two years. 

**Lenny Rachitsky** (00:02:04):
This is a truly incredible and rare story, and one that I think a lot of teams can learn from because AI is creating a lot of opportunity but also a lot of potential disruption, and this is an amazing story where the company basically disrupted themselves. This episode is packed with insights, including a primer on what the heck are people actually doing when they're labeling and creating data to train models? A huge thank you to Garrett for making time for this, his wife just had a baby this week. He's also in the middle of scaling this insane new business. So thank you, Garrett. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. 

**Garrett Lord** (00:05:07):
Yeah, thanks for having me. A long-time subscriber.

**Lenny Rachitsky** (00:05:09):
I appreciate that. Okay, so before we get into the insane trajectory that your data labeling business is on, which is just an amazing story that I think a lot of founders and product teams that are trying to navigate this AI disruption that's happening will have a lot to learn from. I want to first help people understand what the hell data labeling actually is. Just like, what are people actually doing? Why is this so valuable? Some of the most, I don't know, fastest-growing companies in the world today, including you guys are just, this is what you do. Clearly there's something really important here. I sort of understand it, probably not really. I think a lot of listeners feel the same way. So let me just ask you this, what is data labeling actually? What are people actually doing? And then, just why is this so valuable to frontier AI labs?

**Garrett Lord** (00:05:55):
Yeah. So, I think it's helpful to take a step back of what does training a model look like? So, there's really two primary functions. There's a pre-training and a post-training process in training a model, and for a long time these AI providers, or LLMs, or Frontier Labs we're focused on basically sucking up more and more information on the pre-training side of the house. And that's basically the entire corpus of written human knowledge. So, that's not just written, but every YouTube video, every book, basically the pursuit of sucking up everything that was on the internet, and that was the pre-training side. And there was a lot of gains from pre-training, like models continue to get better. And about 18 months ago, 24 months ago, we started to really see an asymptoting of gains coming from, because they had essentially sucked up all of the knowledge on the internet. And so, labs really shifted towards most of the gains now coming from the post-training side of the house. 

**Garrett Lord** (00:06:50):
And what post-training is, is it's augmenting and improving the data they have across every discipline or capability area that they care about. So take coding, or mathematics, or law or finance, they are focused on collecting high quality data that really improves the state of our capabilities, their models, and you can see a lot of these popular benchmarks on what are called model parts. When Llama IV is released, you'll see the benchmarks across various domains, and each one of the research teams inside of the labs have different use cases. Basically they're running experiments, almost think like the scientific process. They have a hypothesis around how to improve the model. They're trying to collect small pieces of data to see if that hypothesis works out. If that hypothesis is proving true, then they expand the overall collection of the data in that advert. And it could look like reinforcement learning environments, it could look like trajectories, it could be audio and multimodal, it can be text-based like prompt-response pairs. 

**Garrett Lord** (00:07:58):
It can also be reinforcement learning with human feedback, which is like preference ranking data. And so, that's the state of the art of models. And most of the gains that are happening from models right now are coming from the post-training side of the house. And there's just an incredible amount of demand to stay at the absolute frontier of where models are going.

**Lenny Rachitsky** (00:08:20):
So training, pre-training is feeding it, say the entire internet. Here's like all the data that the humans have ever created, figure out knowledge and facts, and how to reason and all these things. Post-training, is it correct to say there's essentially two buckets of things to do? There's reinforcement learning, human feedback RLHF, and then there's kind of this bucket of fine-tuning?

**Garrett Lord** (00:08:41):
I mean, yes and no because take for example trajectories, or you want to be able to do, people use flight search or an accounting end-to-end process, or you want to be able to conduct biological experiments, you need actual trajectory data. There's still very much, a lot of the labs are still, they have points of view on what data collect. It's evolving very quickly. But I think reinforcement learning is really preference ranking, like which question do you like more, question A or question B? SFT data is a prompt and a response, and obviously the labs are very focused on these thinking or reasoning models. So, in order to improve a reasoning model you need to actually have the step-by-step instructions, of which when you interact with a lot of these frontier models they struggle in very advanced domains. And so, I think there's a variety of data that they're working with to improve capabilities in their models.

**Lenny Rachitsky** (00:09:39):
What I'm hearing is there's other ways to post-train. Which of these are you guys focused on? Where do you help models most of these three-ish buckets?

**Garrett Lord** (00:09:48):
Our real unique proposition as a business is the fact that we have an engaged audience. We have 18 million professionals across, we have 500,000 PhDs, we have 3 million master students, we're a global platform. And so, depending on what you're looking for across any area, academic knowledge, what is the definition of a PhD? How do you get your PhD? You defend your thesis. Defending your thesis means, generally speaking, you have proven that you have extended the world's knowledge in a particular domain. And so, the ability to hyper-target this audience into chemistry, math, physics, biology, coding and really touch parts of human knowledge that have never before made it to the internet is really where we excel. And I would say that when you talk about the labeling market, something to make it more abstract is like it used to be generalists' work. 

**Garrett Lord** (00:10:52):
A lot of the market before the model started to get better was leveraging talented international lower cost labor to do basic generalist tasks. But really what's happened is the models have gotten so good that the generalists are no longer needed. What they really need is experts, experts across every area that the models are focused on. And really, you could think about these model builders as they're focused on the most economically valuable capability areas in the economy. And so that, generally speaking, right now is focused on advanced STEM domains, advanced science and math domains, and then the derivative functions of accounting, law, medicine, finance, where they want to make the models more capable. And then the work that we're doing, I think to come full circle to your question, we're doing work across so many domains. I mean, we have millions of bachelor students that are being used for work in audio, work in customizing a model depending on the voice and tone, where you are geographically in the country, what do women versus men prefer? All the way to the most advanced PhD STEM domains out there.

**Lenny Rachitsky** (00:12:08):
Okay. So, is it fair to say essentially all the data that is available has been trained on, and your company creates new data, new knowledge to continue advancing the intelligence of models?

**Garrett Lord** (00:12:22):
Yep. And I also say we help point out where the models are weak. So, in order to break a model, it's pretty tough for the average person to break a model and get an incorrect response. But if you are a PhD in physics, you can go in multiple subdomains of physics and prove where the model's actually breaking, either breaking in its reasoning steps or it's where it's broken in its ground truth right answer, or we start throwing tools in there or needing to follow some step-by-step process. And I wouldn't say it's easy for them, but the average person cannot break the models and that's where we really come in.

**Lenny Rachitsky** (00:13:03):
So, essentially it's just catching mistakes that the model has made. Okay. So, what are these people actually doing? I know there's all kinds of different types, you described all the ways that data's generated, what kind of data is useful? So, maybe just the most common examples. Let's say a PhD person is sitting there doing stuff, what are they actually doing?

**Garrett Lord** (00:13:22):
A great example is a public paper called GPQA. So, for the engineers out there that want to read about it, essentially the crux of the paper is you break the model, you provide a ground truth, the right answer to the question, you provide the step-by-step a reasoning steps. So, you might imagine because models are non-deterministic, the model can get the answer right once, but it might not get the answer right three out of five times. So, you actually prove where the model's failing. You actually break down into where is it failing? Maybe it can get, it knows the question, it can get the right answer, but the actual steps to get there are wrong and they really focus on the steps to get there. Say there's 10 steps in a math problem, step 6 through 10 is wrong. So, how do you fix the actual steps? 

**Garrett Lord** (00:14:11):
And what are they doing? So they're going in, we're really focused on calling this branding the experience and treating people like experts. PhD students expect to be treated different than a lower cost international labor with a different work expectation. And so, these PhDs come into a community, we have a instructional design team and an assessments team that's going through and basically iteratively helping them understand how to use the tools that we built, and how to interact with the latest models. Then they go in and start actually creating data. And that process is, on our side the model builders, they want to know that the data we're producing is high quality. Som we have our own research team, our own post-training team. 

**Garrett Lord** (00:14:53):
I hired a gentleman from Meta that went along on the post-training over there, and I-

**Lenny Rachitsky** (00:14:57):
Hope you paid him well. 

**Garrett Lord** (00:14:59):
Yeah. So, war for AI talent is very expensive, but super, super privileged and proud to be working with him. And so, each unit of data, we have to build it an environment for them to actually create the data. Then we have to understand in a unit level we're trying to approximate the actual gain from that piece of data and whether it can improve in a particular capability area. And then, we're also focused on evolving the use cases to also follow what the model builders want, which is they want more more real world tool use and trajectory based data as well.

**Lenny Rachitsky** (00:15:33):
Okay. There's so much here, and we could go infinitely down here but I think that this is really interesting because just like people hear so much about all of this and they barely understand what the hell it actually is. So, this is for me really interesting. I think it's going to help a lot of people. So essentially a PhD, say a biologist, biology PhD is just their job is find flaws in what, say ChatGPT is producing, and then come up with here's the correct answer. And that is used to fine tune the mode, here's something you are doing incorrectly, here's the correct answer and that improves the model.

**Garrett Lord** (00:15:33):
Yep. 

**Lenny Rachitsky** (00:16:03):
Is that a simple way to think about it? And please correct anything I'm saying that is incorrect because I don't want people to misunderstand it.

**Garrett Lord** (00:16:08):
Yeah. I mean, a great example, let's take a non-verifiable domain like education. So there's a PhD student, Rachel on the network, she got her PhD from the University of Miami, spent two decades as a teacher teaching students in the eighth grade. And she was an adjunct professor at a local community college in the field of education. And so, she is interacting with the state-of-the-art models in educational design. So, actually trying to understand what is the best way to teach people, and how do you spot incorrect issues in a model in the way that they're training people, and help the models understand the forefront of educational design with the hands-on experience of being an eighth grade teacher for 10 plus years and having a PhD in education? So, that's an example of you can have that all the way down to a verifiable engineering problem that you're seeing the latest models fail on.

**Garrett Lord** (00:17:12):
Yeah, I think that gives you the gamut. You also have, we talk about professional domains like these reinforcement learning environments, there's a bunch of papers out there that basically speak to people narrating over their step-by-step tool use. So, as they go to solve a problem from start to finish, interact with multiple different service areas, interact with multiple different tools, they're like, there's papers that talk about this, talking over what they're doing, actually following and screen recording where their mouse is going, how they're problem solving. When they run into a roadblock, what do they do? So, they really want to understand how humans think.

**Lenny Rachitsky** (00:17:50):
You mentioned this term trajectory. Can you just explain what that actually means? Because it feels like you've mentioned that a few times and that feels important to all this.

**Garrett Lord** (00:17:56):
Yeah. A trajectory is basically just like the entire environment that is collecting what you're doing. So it's your screen, it's your mouse-

**Lenny Rachitsky** (00:18:04):
Oh, I see. Oh, wow. 

**Garrett Lord** (00:18:05):
Yeah. 

**Lenny Rachitsky** (00:18:06):
Including this voiceover, okay. And then, this might be too technical, but what is the output of all this work? This, say teacher, is it just like a JSON file, an XML file, like a text file?

**Garrett Lord** (00:18:15):
Yeah, it can be managed JSON data.

**Lenny Rachitsky** (00:18:17):
JSON data? Okay. 

**Garrett Lord** (00:18:18):
And then, we also have multimodal work like audio, like classifying music and understanding ... We're engaging thousands, or not thousands, probably hundreds of top music students at the music schools in the country who are improving models of understanding of music. And you also have the thing called, which we haven't talked about here, a rubrics, and rubric models are, you can put a model in as a judge. What is a good educational design, or what's a good MRI result? In some of these domains, you actually don't have a guaranteed correct right answer. And so, models can sit in the middle as a judge and actually understand what is ... Think back on your school days. How do you get A on your 5,000 word paper? Well, there's a great introductory statement and there's scientific proof. So, you can build a rubric that allows a model to sit in the middle and actually auto-evaluate responses. We're seeing a lot of rubrics work as well. 

**Lenny Rachitsky** (00:19:28):
And you would think, why would you trust this one teacher's opinion that this is the right way to do it? But what's cool is the market speaks for itself. If these models are being used more and more, and people love them and value them, I imagine steps in between to verify this is good and other people think this is a good idea. It feels like the market dynamics will tell you if the data you're providing is correct at what people want. Is there something more there?

**Garrett Lord** (00:19:52):
I didn't get a PhD in AI, or math, or physics, and I haven't trained myself, we have frontier models, but there is a lot to each unit of data whether it's improving. There's a ton of science and research out right now around how do you make sure that the data that you're producing is improving the model? And it's very hard for model builders to understand. They really care about, to zoom out, they care about three things. They care about quality first and foremost. You have to have high quality data. And if you imagine you're training a model, like teaching a student and you're giving it the wrong data, it's extremely challenging to overcome that. So, quality is first and foremost. And then, the other huge problems you have is volume. How do you generate thousands of pieces of data in the most advanced domains of chemistry, and mathematics, and physics, and how do you ensure that it's high quality?

**Garrett Lord** (00:20:48):
Well for us, say in physics, we just reach out to students at Stanford, and Berkeley, and MIT, and they're at the top GPA at the best physics schools in the country. And so, our ability to get to scale or volumes of data, to produce very high quality data, is something they care deeply about. And then, the other thing I would say model builders care about is speed, because they have all these hypotheses and they're constantly testing different pipelines. And so, you might have three or four bets going at once, and then as soon as one is actually showing a gain, imagine you're a researcher or you're assigned to the processes, once you're running a gain then you're trying to grow that pipeline and grow that piece of data that's actually improving it, and you're maybe ditching two or three other projects you had that weren't showing improvement. 

**Garrett Lord** (00:21:31):
So, your ability to quickly turn around for them in a period of days, and then get to high volumes of data that are high quality is the number one thing they care about. And so, there's quite a bit of technology we built on our side to assess each unit of data. We have our own post-training teams, we're renting our own GPUs, and we're trying to make sure that we can sit directly with these researchers and help share what we're seeing with the data that we're creating and how it could improve their model, how they could best train with it. So, hopefully that helps.

**Lenny Rachitsky** (00:22:03):
Going back to the types of post-training, just because I think this might be helpful, at least for me the mental model of there's pre-training, there's post-training, within post-training there's reinforcement learning, human feedback, there's this concept of fine-tuning. There's also evals and stuff like-

**Garrett Lord** (00:22:03):
There's SFT, yeah. 

**Lenny Rachitsky** (00:22:19):
SFT, which is supervised fine-tuning? Is that-

**Garrett Lord** (00:22:22):
Yeah.

**Lenny Rachitsky** (00:22:22):
Yeah. So, the stuff you've been describing, would you mostly describe that as supervised fine-tuning?

**Garrett Lord** (00:22:29):
Yes, and we're doing all of the above. We don't do the auto eval, we produce rubrics which are used auto evals. But yeah. 

**Lenny Rachitsky** (00:22:39):
Okay, awesome. So essentially there's a model, it's trained on all this amazing data. You guys come in after the model's trained to tweak the weights based on additional data that you create. What's interesting is that this is a scalable system. I want to talk about just the supply of amazing people that you have producing this, but it's amazing that humans can do this. You would think it needs to be this infinitely scalable thing, but humans sitting there creating data is working and improving the intelligence of models significantly.

**Garrett Lord** (00:23:13):
Oh, yeah. I mean, I think maybe a funny joke is all the MBAs think this is all just going to go away. And I think for as long as models are improving, humans will be needed in this process. And when you talk to the lead scientists and researchers at these labs, it's like the data types will evolve and what they're trying to capture and collect, but there will be humans needed in the space for the next decade until we reach full ASI. So yeah, I mean, you think about in a lot of them will struggle to do basic trajectories right now. So, right now people are very focused on academic domains, and I think they'll continue to be focused on academic domains, but there will also be far, far more demand for professional domains as well across basically every trajectory or step-by-step problem that a knowledge worker solves in the workplace, it's the pursuit of these labs to make sure that they're trying to collect the data to help add as much value in that process for humans as possible. 

**Lenny Rachitsky** (00:24:17):
So, let me ask you about this. There's this tension, I imagine, people might feel between all these students training models to become smarter, and smarter, and smarter, and then there's that they will have harder time potentially finding jobs if models are so smart that people at entry level aren't being hired as much. How do you think about just that tension? Do you think this is a real problem or not, or do you think this goes-

**Garrett Lord** (00:24:42):
I'm probably in the camp of like GDP growth over universal basic income. I very much believe that this is going to improve and accelerate every human's ability to create an impact in the economy in the world, and that we're hearing from, there's like a million companies that use Handshake. 100% of the Fortune 500 uses Handshake, so we basically power the vast majority of how young people find jobs, and a lot of people are hyperbolic at saying that all young people won't have jobs, and that's not what we're hearing from our employers. What we're hearing is pick social media marketing, before you needed somebody that could do Photoshop, and take pictures, and create the videos. Then you needed somebody that understood marketing analytics platforms to track your posting on different social media forms. It's like now one person, one young, talented, AI native, Iron Man suit enabled young person can get on and they can build their own videos, produce their own creative assets, post across multiple social media platforms, run all of their own analytics. They don't need a data science degree to be able to do that.

**Garrett Lord** (00:25:47):
Or take an intern in our company, he had his first PR up I think the afternoon he started. You were a PM, you realize how challenging that would've been historically to get your dev environment set up and figure out where to add value. You just took a bug and squashed it. And so, I'm really a believer this is just enabling human beings to be even more productive and create more impact. And yeah, of course, hundreds of millions of jobs, the jobs will evolve. People will become displaced, they'll have to upscale and rescale, and I think Handshake has a huge role to play in helping knowledge workers evolve.

**Lenny Rachitsky** (00:26:24):
This has come up a couple of times this point that I think is really good, that younger people coming out of school are actually going to be much more likely to be successful because they're growing up with these tools, and are much more native to all these advanced tools and so they just come in as beasts just doing so much more.

**Garrett Lord** (00:26:42):
Well, do you remember when, this a little bit predates me, but you used to put Google search on as a skill on your resume. You were person, you were good at Googling, because you grew up with Google. It's like I think being AI native and having your Iron Man suit on, and understanding how to leverage these tools is like young people are at a huge advantage.

**Lenny Rachitsky** (00:27:03):
Yeah. And especially if they're involved in training these models, I imagine there's some other cool advantage there. 

**Garrett Lord** (00:27:08):
Yeah. Well I mean, just to hit on that, what we're hearing from our thousands of fellows is they're in the classroom, they're actually producing research. We're talking about PhDs at the top institutions in the country. They can make 100, 150, $200 an hour in their area, in their field of expertise. It's pretty sweet. You can make 25 bucks an hour being a teacher's assistant, or you can actually make $150 an hour breaking the latest models, and what we're hearing from our fellows is they're bringing a lot of those insights into the classroom to help them be more effective at teaching. More importantly, they're starting to learn how to leverage these tools to actually advance their area of research. So, they believe that these tools can help them advance their area of research by helping them be more effective with their time. And so, it is quite cool to get paid to learn a skill.

**Lenny Rachitsky** (00:27:58):
Before I get to the story of how this all emerged, because that is an incredible story, is there anything else about this whole field of labeling, of reinforcement learning that you think people just don't fully understand or you think that is really important? There's just so much happening. Like I said, some of the fastest growing companies in the world are in the space, Scale was just acquired for 30, sort of acquired for $30 billion. Just what else is there, if there's anything, that you think people need to understand?

**Garrett Lord** (00:28:26):
Generally speaking, anytime that you're interacting with a model and you're asking it to do really advanced things, and it's not performing your expectations, like somewhere there's probably an expert that is the top mind in that domain working directly for the best researchers in the world at the Frontier Labs trying to understand and go through the scientific iteration process of how to make that better. And that the assumption there is that they already have the entirety of human knowledge that's written and recorded. And so, for as long as there are problems in solving any problem with AI, any human problem, there will need to be humans in the loop helping advance that. And models don't generalize. I mean, obviously the field will advance a lot and the type of data they'll collect will evolve a lot, but it's pretty exciting at the frontier. 

**Lenny Rachitsky** (00:29:20):
Kevin Wheel was on the podcast, the CBO at OpenAI, and he made this point that really stuck with me that the model of today is the worst model you will ever use. 

**Garrett Lord** (00:29:29):
I love that line. 

**Lenny Rachitsky** (00:29:30):
Will only get better, just boggles the mind, and now we know why things are getting better because all the work you guys are doing. Just one quick question on this whole scale thing, I guess they were, I don't know, the main company doing this, now they're swallowed up and Alex is running superintelligence in Meta. Are they still a big player in this labeling space or are they out of it and that's a big opportunity?

**Garrett Lord** (00:29:50):
Yeah. I mean, kudos to the whole Scale team, a lot of respect for what they built, just many great companies operating the space. I think to the core of your question, I think if you viewed your research team and your model building team, and the experiments they're running to be really the cornerstone of how you're improving, you probably wouldn't want the latest research of what you're trying to work on being invested in by a peer. I mean, that's just generally what we hear in this space. And so, we have seen an incredible search and demand, and are I think extraordinarily well positioned. We like to say the only moat in human data is access to an audience. Basically, there are many, many small players in this space, some midsize players in the space, and they're basically running TikTok ads, running Instagram ads, paying money for Google Search display ads, YouTube ads, and they will be like, "Can you get me 200 physics PhDs?"

**Garrett Lord** (00:30:58):
What do they do? They only can do one thing. They have 100 recruiters on staff, they all get on LinkedIn, they all send messages, they spend a couple million bucks on performance advertising campaigns. Somebody's scrolling their Instagram feed that's a physics PhD of which you can't target them that well and they like see, "Come train a model." It's like, "I've never heard of this brand before." The huge advantage that we've had and why we've resonated so fast in the marketplace is we built a decade of trust with 18 million people, and they trust us, and we built a ton of brand affinity, and they use Handshake, and they have an active profile, and we have a ton of information around their academic performance and what they've done in school. And so, we're able to really target people really effectively, and get to scale and volume of high quality data faster than anyone else. And I think that competitive advantage of access to an audience is really resonating in the marketplace. 

**Lenny Rachitsky** (00:31:49):
**Garrett Lord** (00:33:40):
Yeah, I think it's been a pretty natural extension from helping people jumpstart, restart or start their career. Monetizing your skills and this new employment ecosystem is going to look very different in the future, and to zoom into how we discovered it, it's like because we have such a large access to this audience, and as the world shifted from generalists to experts, we're the largest expert network in the world. We have more PhDs, 500,000 of them use Handshake than any other platform. We have three million master students who are in school or alumni. And so, we started to see all what I would call middleman companies reaching out to us saying, "Can we recruit your PhDs and master's students?" And like any great marketplace we started sending them to these different platforms, and started to really realize that from hearing from our users that the experience was really frustrating. 

**Garrett Lord** (00:34:38):
Training was very transactional, it was very amorphous how you could get paid. There was immense amount of drop-off in the process to actual project like completion on these other platforms. So, we started to think the company was making tens of millions of dollars from helping these other platforms, and we started to realize what really kicked it off was hearing also from the Frontier Labs, they started to reach out to us and started to go direct and trying to almost cut out the middleman. And we started to realize, well, we could really serve our fellows, our PhDs, our experts, we could treat them. We just believe there will need to be a platform, an experts first platform in the pursuit of ASI and advancing AI, and there will need to be a place that everyone in the world could go to, to monetize their skills and their knowledge as these labs are focused on improving in all these multidisciplinary. And yeah, we entered the business in, really I started doing it over Christmas and New Year's. That's when I started flying around.

**Garrett Lord** (00:35:48):
My family thought it was a little wild that I was on planes trying to chase different leaders, but we built an incredible team of people that came from the human data world, and really started building out our platform in January, and then started really monetizing the relationships about five months ago. Fast-forward to today, we're working with seven of the Frontier Labs, basically every lab that's doing work and building the best large language models, and the team has exploded and revenue's exploded, and it's been really a incredible ride running back new company inside of a company for the second time over again. 

**Lenny Rachitsky** (00:36:28):
And just to share some numbers, tell me if this is correct or if you're sharing these, but I heard that you hit 50 million in revenue just four months into this? Today we're at eight months in and you're on track to hit $100 million in revenue in the first year.

**Garrett Lord** (00:36:43):
I think we'll blow through that number, but yeah. 

**Lenny Rachitsky** (00:36:44):
Okay. Incredible. And I didn't even know there were seven Frontier Labs, that's-

**Garrett Lord** (00:36:49):
Zero to 50 is pretty good in four months, I think. 

**Lenny Rachitsky** (00:36:51):
Think zero to 50 million in four months, that's something. It's like the bar has been shifting constantly. A year ago that'd be legendary. Now it's like, all right, well another one of these. 50 million in four months, no big deal. It's truly insane. Just to zoom out one second, for people that don't know a ton about Handshake, the original business, what was that? What was actually this network that you had, that you sat on top of?

**Garrett Lord** (00:37:18):
Yeah, that network does about 200 million. This will do about [inaudible 00:37:21]

**Lenny Rachitsky** (00:37:18):
200 million.

**Garrett Lord** (00:37:18):
Yeah. 

**Lenny Rachitsky** (00:37:18):
Okay. 

**Garrett Lord** (00:37:20):
So, we have 600-ish super passionate teammates that work on the core business, which I would separate those. These aren't two businesses, I think it's one business, but what is that business? If you're a young person in America that's graduated in the last five, six, seven, eight years, you probably have Handshake on your phone. You definitely know what Handshake is. It's a verb with young people in America, it's a verb with people that are in college in their PhD or master's program, and it is, I call it an unconnected graph, meaning you don't need to ... LinkedIn is very focused on who you know and what your experience is. The first question on LinkedIn is what's your job? And a lot of young people start off, they've never had a job before. They don't have 500 connections to add to their graph. 

**Garrett Lord** (00:38:13):
Whereas on Handshake, you start off trying to discover, and explore, and figure out how to navigate through school and figure out, "Oh, I'm an engineer. Maybe I want to be a PM, maybe I want to work at a startup, maybe I want to go to a larger company." What are the pros and cons you want to learn from your peers and young alumni? And so, Handshake's this I call a very social platform with groups, and messaging, and profiles, and short-form video and feed, all focused on your interests and helping really build your confidence in your early career to find your first job, your second job, and to manage 18 to 30, I would say.

**Lenny Rachitsky** (00:38:49):
And how long has that business been around? 

**Garrett Lord** (00:38:51):
It's been around 10 years.

**Lenny Rachitsky** (00:38:51):
10 years, okay. So it's just again, it just feels like such a holy shit, you guys are in the right place in the right time with the right network that is extremely valuable now. What an interesting story. I feel like it's just another interesting example of you've been doing something for a long time and then all of a sudden AI is just, opens up a whole new way of leveraging something that you have been doing for a long time. It makes me think a little better about Bolt, and StackBlitz, which was building for seven years this browser based OS where you could run an OS in the browser. And they're like, "I don't know, no one needs this. What are we doing?" And then, all of a sudden AI and they're like, "Oh, what if we build AI apps in the browser and just generate products for you with AI?" And now it's, I don't know, one of the fastest growing companies in the world. 

**Garrett Lord** (00:39:36):
Yeah. 

**Lenny Rachitsky** (00:39:37):
So interesting. And so, I think this is just an interesting time for our people to think about what have we done that may give us a new opportunity to build something huge based on this unfair advantage that we have?

**Garrett Lord** (00:39:49):
I think also as your company grows in size and headcount, and maturity, it's also hard to incubate something new inside of a business. It's hard in so many ways. The way that you build zero to one, and find product market fit, and scale a team very quickly and is very different than the way that you run a more mature business that has been around for 10 years with hundreds, and hundreds, and hundreds of people. So, I've really had a ton of fun and found a ton of passion in running it back again for the second time inside the business. And then yeah, we have this massive strategic advantage, which is no cost or acquisition costs, and we have much higher conversion rates and retention than any of the other platforms by a large margin because we have such consumer affinity. 

**Lenny Rachitsky** (00:40:43):
There's actually two threads here I want to follow, I'm going to follow the second one first, this idea of where this data labeling work can come from. This isn't a really clear, simple, understandable one, which is just experts sitting there creating data. Another one that I know a lot of other companies in this space use Scale, I know especially with just like low-cost labor internationally. Are there other methods for doing this that isn't one of those two? How are other companies doing this?

**Garrett Lord** (00:41:10):
I think if you care about building a really high quality business, and having good gross margin and high quality growth, the ecosystem here is, one of the leading players, they have 200 recruiters. It's unsustainable. There are like 200 people on LinkedIn sending individual messages to acquire these people, because there's no brand, there's no trust. They're spending tens of millions of dollars a month on performance advertising, Google Ads-

**Lenny Rachitsky** (00:41:39):
To find experts and to find folks. 

**Garrett Lord** (00:41:41):
Find experts. 

**Lenny Rachitsky** (00:41:42):
And it's experts mostly at this point. 

**Garrett Lord** (00:41:43):
Yeah. And then they put them onto an experience that is treating them like they're drawing boundary boxes around stop signs in the Philippines. The frontier tax accountants don't want to be treated like low cost international labor, and I don't think anyone enjoys that process. And so, the ability to build a experience that's rooted in community, that's rooted in high quality training. If you're getting your PhD at MIT, chances are you're just not being taught well enough on how to use the tools.

**Garrett Lord** (00:42:15):
It's not you can't break the models, it's just like the other platforms, they're spending thousands of hours to acquire an individual user and they're put right into a project with no training. So, we just started from day one at building this expert ... We believe there'd be a deep network effect here that's very connected to our core business of starting, jumpstarting or restarting your career. And you come in, you build a profile, you see the community, there's groups and a feed of here's how people are learning. You come into actual individual cohort with peers that look like you and have your similar background. You're being taught on how to interact, and there's a trial and error, and we have an instructional design piece so you can't do it. Then you're put on the projects where building ... There's certain swim lanes where we're actually pre-building data and selling that data to all the labs. 

**Garrett Lord** (00:43:03):
So, we can do this thing where we produce one unit of data ourselves. We pay for it, almost like a movie production. We pay for a unit of data, and then we make sure it's very high quality. We run our own post-training on it, and then we produce a bunch of specifications of the data, and we actually sell that individual package of data to many different labs. And so, you get put on a project like that. Once you're doing a really, really good job on our projects, oftentimes then we'll put you on customer projects where they only want the best of the best people in machine learning. And then they go from our projects to their projects. And so, there's a huge customer acquisition. You love going deep on your podcast, so just to talk about it, it's like you really have a couple of things that matter.

**Garrett Lord** (00:43:46):
You have a cost to customer acquisition, your CAC, and then you have your LTV, like the lifetime value of a user. And an LTV is calculated pretty simply in this business. It is based on the retention of a person, and how many projects they can participate in. So, if you treat people really well, you train them really well, well, A, we have no customer acquisition costs because we partner with 1,600 universities, power 92% of the top 500 schools in the country. We power almost every institution and community college in the country. We have no customer acquisition cost to acquire the people. We have a ton of brand and trust with them built up, so they convert at really, really high rates. And then, if you treat them really well, because what they expect from us, they know Handshake, their school buys Handshake, we care about treating these people well but the universities would not tolerate our partnership with these fellows unless we treated them well. 

**Garrett Lord** (00:44:41):
So, you put them into this process where our LTVs and repeat engagement rate and retention rate on different projects is really high. And so, these structural advantages are quite significant when you contrast a leading provider that has 200 individual contributing recruiters, and are spending tens of millions of dollars a month on performance marketing. So, that's I think why we've seen so much success.

**Lenny Rachitsky** (00:45:07):
That's extremely interesting. And it feels like, as you said, there used to be a big focus on generalists, which is people anywhere in the world for low-cost can do the work, like draw bounding boxes around things. And essentially the market has shifted from low-cost generalists to experts. And a lot of these companies like Scale, we're optimizing for general work model training data, and you guys are set up to be extremely good at expert based data. And so, you're in the right place at the right time, at the right supply. What a business.

**Garrett Lord** (00:45:43):
Yeah.

**Lenny Rachitsky** (00:45:43):
Nice work.

**Garrett Lord** (00:45:44):
I would say it's not been easy building business two inside of business one, but-

**Lenny Rachitsky** (00:45:48):
Yeah. So, let me follow that thread. That's where I wanted to go. What was just that like? So, you started noticing that model companies were coming to your people, that people were having hard times with some of these other companies in this space and you're like, "Oh, maybe we should be doing this sort of thing"? How did that just initial inception start, and how did you start to explore that idea and to see if it was a real thing?

**Garrett Lord** (00:46:09):
Tactically we were working with many of the middleman companies doing work. We started to see the demand, as I talked about earlier. We started to see direct outreach from the Frontier Labs reaching out to us, trying to cut out the middleman in their pursuit of getting higher-quality data. When we started to put together the dots on we could build a way better experience for our fellows, we could serve them directly to the labs and build a direct customer relationship with the labs, and basically cut out the middleman. And provide a better experience to the labs, provide a better experience to our fellows and provide a better experience to our million companies in the network.

**Garrett Lord** (00:46:48):
And you might think about just upskilling and reskilling, what's going to happen there. So, we walked into this space. We started in really December, exploring and learning more about it, like on expert calls and hammering down. I hired three expert firms, AlphaSights and GLG, and started doing a bunch of calls with the latest researchers, because we had resources. One of the cool things about being a larger company is our core business is $200 million ARR, so it's like we had resources to be able to accelerate the learning curve here. And then, we started working with arguably the number one lab about five months ago. 

**Lenny Rachitsky** (00:47:34):
I wonder who that is. 

**Garrett Lord** (00:47:35):
Yeah. 

**Lenny Rachitsky** (00:47:35):
Yeah, wonder who it is. 

**Garrett Lord** (00:47:39):
[inaudible 00:47:39] different answers working with the number one lab, and have just now we're working with Devin on the Frontier Labs and the number one thing we're trying to do is just focus on scaling up. And we've gone from four or five people working on this to 75 plus people working on it. I think we had 12 people start last Monday. It's like we are so bottlenecked on just meeting this opportunity, because in this market there's essentially unlimited demand. If you can produce high quality volumes of data, you most likely will be able to sell whatever you produce. And so on our side, it's like we're really focused on making sure that we pick the right longer term strategy, making sure that we don't grow too fast as to erode the trust that we've built up with these Frontier labs. Yeah, but it's been fun.

**Lenny Rachitsky** (00:48:38):
You said it's also been really hard to start those business within an existing business. What's been hard? What's been hardest? You touched on a couple of these elements already, but what else? 

**Garrett Lord** (00:48:50):
I think I just followed a lot more of my intuition around this, doing this. The story of Handshake was we had to sign up 1,600 universities, so I had to learn how to be the best ... We are the fastest growing higher education company in history. So, we signed up six 1,600 schools. Then we had to build an employer business, where we had to figure out how to sell the 100% ... All these Fortune 500 companies use it and 70% of it pay for it, so I had to learn about upmarket sales to Goldman Sachs, and General Motors, and Google and the biggest companies in the world, which is totally different than selling universities. And then we had to learn how to build an incredible student social network. What does the best feed look like? What does group messaging look like? So, I felt a little bit of familiarity in those zero to ones. 

**Garrett Lord** (00:49:39):
Oftentimes marketplaces are like many zero to ones. Sometimes I dream that we just, I actually don't dream, but I make a joke that I just wish we were a cybersecurity company and we had one buyer and just one product, and it was just like we had to ... In a marketplace, you have to serve three different sides, you know from your time at Airbnb. And so, one of my learnings in spinning up these three different businesses in starting Handshake was I was pretty hands-on. So, everyone reported directly to me. I really said in a lot of meetings, "I'm not trying to be the boss, I'm just trying to get another smart guy in the room." We've hired an incredible team of people that have spent a lot of time in the space and have been big leaders at a lot of the human data companies in the space.

**Garrett Lord** (00:50:27):
And so, everyone saw very clearly the structural advantages that we had, and a lot of the focus was on making sure that we could deliver high-quality data to one customer before we expand to anyone else. You had to say no to a lot of things. And then, you also had a lot of people in the core part of the business that, rightfully so, there's just checks and balances that there's a lot of people that try to get involved. Everyone wants to say, not everyone, this is a stretch, but it's easy to say no. It's easy to be like, "I can't prioritize that this week or this month. I have an existing set of priorities." So essentially, with the exception of a few things, everyone just came straight into this new org that I built, everyone did not have any responsibilities in the existing part of the business. It was extremely clear who was the directly responsible individual across each area of the new co. And now we've got deeper coupling and integration points across the rest of the business, but we sat in a separate part of the office. 

**Garrett Lord** (00:51:35):
Everyone's in the office five days a week, a lot of weekends. There's a totally different expectation in hiring talent too, where it's like, "Hey, this is a 24/7 job. This is an early-stage company." The compensation was also different too, and based on hurdles in this new business so people felt owners creating the new co. And yeah, it's still extremely nimble, very, very flat. Just because you run one function doesn't mean you're the directly responsible individual on a project. We pick the best person who's most capable of driving an initiative forward, regardless of the function to be the DRI. We're a lot more metrics-oriented. When I built Handshake, we resisted this operating cadence for a long time, this weekly, monthly, quarterly operating cadence. With Handshake AI, we've been way more focused on operating with data, and metrics, and rigor from an early stage. There's a gentleman named Sahil on our team who's been doing an incredible job with that. Shout out Sahil, shout out young, shout out Paco. Yeah.

**Lenny Rachitsky** (00:52:42):
Okay, this is incredible. So, a few elements of what allowed this to succeed within a decade-old company. And by the way, so you're at 200 million a year in revenue with the traditional business. You're going to, as you said, blow past 100 million in the first year of this new business. So, it's wild that in the first couple years, if things continue to go this way, you'll exceed the run rate of a business that took you 10 years to build. Incredible. To make this successful, a few of the things I noted as you were talking, one is clearly you were just in founder mode. You're the lead of this new business. You weren't delegating it to someone, "Hey, go start this thing." You dedicated people, "Here, we're going to pick people. You have nothing else going on, this is your new job. You're going to work on this stuff." 

**Lenny Rachitsky** (00:53:28):
You worked in different part of the office. There's a metrics-based cadence. It's just like, let's stay really diligent about here's how it's going, here's where we're going, here's our track, here's our KPIs, things like that. Anything else there that you felt really important to making this work? Because a lot of companies are going to try to do this, I imagine, and so I'm curious what else you found important to make this work.

**Garrett Lord** (00:53:50):
Yeah. I mean, I just really believe in separate and everything. Separate engineering team, separate design team, separate accounts and operations team, separate finance team. Early on, everything was separate. People only had one job and one job only, and that was making Handshake AI successful. We had a couple integration points more, and I had an incredible executive team and a core part of business, and now there's becoming more and more involvement. But our executives that have built Handshake for a long time ran the core business, and I focused 80 plus percent of my time and attention on just this. And we hired an incredible engineering leader like Avery, who ... We have a lot of entrepreneurs, people that have started companies inside the company. Or pardon me, people that started companies before. That was huge. A lot of familiarity with hiring talent that have only worked at early stage companies so [inaudible 00:54:44] that feels super comfortable with ambiguity. 

**Garrett Lord** (00:54:46):
We were also way more upfront around this is going to be chaotic. Just owning that narrative in front of all hands at the core company, owning it directly with the team. We have a separate all hands, we have separate onboarding, we have a separate recruiting team. I had some connection points, but mostly separate. And I think that was absolutely critical. We took some of the top people, I mean, we have great people in the core business, we took some great people from the core businesses and basically said, "Sorry, I know you love your old team. I know you love what you're doing. Will you join us in Handshake AI?" And they completely forego their historical responsibilities and came over. That became really critical with engineering when things started to scale and topple, and we're growing so quickly we took some of our top senior engineers, who were very entrepreneurial, and principal engineers, staff of engineers, parachute them in. It's been awesome to ask some of the most talented people in the core business like, "Hey, do you want to come over here and do this?"

And sometimes they say no. They're like, "I don't want to work most of the weekends." The number of 2:00 AM, 3:00 AM nights we done in this business, I mean, it's quite regular. People sometimes don't want to commit to that, but we've been up front, like here are the expectations for this team. It's an insane pace. If you want to be a part of one of the fastest growing businesses in Silicon Valley, you can join it. The ownership too has also been huge, like owning this outcome, and we have this motto to leave nothing to chance. For a while there we drew the number of days in the year on the whiteboard and it was like, there will never be a time like this. I've never seen anything like it, I doubt I'll ever feel anything like this in business again where there's unlimited demand and it's just our ability to execute against it.

**Garrett Lord** (00:56:39):
And so, we had this motto like leave nothing to chance. How do you make sure that three months are not six months? You have no regrets. Get on the plane to go talk to a customer, make the late night push, check the data six times over again, ship the extra feature that helps. And really, a huge celebratory culture too. It's very flat so there really isn't this principle of ... There's so many people putting up points, directly calling out the people that are putting up points and creating a really fun environment around impact I think has been awesome.

**Lenny Rachitsky** (00:57:14):
The leave nothing to chance piece I imagine speaks partly to the value of trust in what you're doing. You win if they can trust that your data's awesome, and great, and consistent, and I could see why that ends up being such an important part of what you're building. And just listening to you describe this, I understand it's obviously a massive opportunity, obviously a massive advantage you guys have, and just the stress that comes with that burden also imagine is very high of just like we can't screw this up.

**Garrett Lord** (00:57:44):
No. Yes, Handshake should be a ... Business does billions dollars revenue as a public company, we should be able to continue to ... I mean, and it also helps our core business. The longer term opportunity that we see is it's connecting, it's building the best job mashing marketplace on the internet. It's probably one of the largest problems in the world like labor supply mashing. It's where people spend most of their time and energy, just hours of their life they spend it at work. The process of searching for a job, applying to a job is going to be completely reinvented with AI. We've been leading the charge there. An AI interviewer that's collecting skills and actually asking about your experiences, doing work simulation experiences that help employers find the best candidates. I mean, I don't know the last time you've done this, but the hiring manager process, reviewing 200 resumes, are you kidding me? 

**Garrett Lord** (00:58:49):
I'm going to sit there and review 200 resumes? Not a chance five years from now. Students manually making cover ... Not a chance. So, there will need to be a marketplace that wins in connecting supply and demand, and talent with opportunity, and we think and get psyched about the opportunity for impact here. That's my story, I went to community college, I paid my way through school. I went to a no name school in Upper Peninsula of Michigan. I worked at Palantir as an intern, it totally changed my life, and I started Handshake because I wanted to make it easier for anyone regardless of who you knew, what your parents did, what school you went to, to find a great opportunity. And I think AI, totally step function improvement in matching. And I think that our human data business is really serving as the foundation for improving matching. 

**Garrett Lord** (00:59:44):
A lot of things that we're doing in the human data business are being integrated to our core business. I think that's going to improve outcomes for employers, save them in the aggregate like billions of dollars over time. And I think it makes the experience way better for students. So, it's just like we have to meet the moment. We still have the stamina, and the excitement, and the passion internally in our core and in the new business to go charge after this. And that's a lot of the message we've been sharing internally is it's time to amp it up. This is a once in a lifetime opportunity to be positioned as well, and we are going to make the moment as a team.

**Lenny Rachitsky** (01:00:19):
It really is. This very much feels like a once in a lifetime opportunity. Let me ask a few other questions along these lines that are something I've been thinking about, something that a lot of people think about, just while I have you, there's always this question of will we run out of data? Will model stop advancing? Are we going to hit some plateau and there's not actually going to be some AGI moment, SGI moment? So first of all, do you think we'll run out of data? There's a point at which we just can't produce more knowledge and data to feed these models? And along those lines, what do you think is the biggest bottleneck to advancing models faster and further?

**Garrett Lord** (01:00:50):
Yeah, I mean, it's just the type of data we're going to need is going to evolve. It's going to be CAD files, it's going to be scientific tool use data as they try to automate scientific discoveries and drug discovery. It's going to be esoteric operating systems that exist on scientific tools. So, I love this trajectory and stitching together step-by-step instruction following. The type of data we're going to need is going to evolve a lot. And we haven't even talked about multimodal, and video, and text and audio. There's a huge demand for audio data right now. So, the type of data's going to evolve.

**Lenny Rachitsky** (01:01:38):
Yeah, I use voice mode all the time. That's on my default ChatGPT experience, just talking to-

**Garrett Lord** (01:01:42):
It's amazing. It's amazing. I just had a baby on, or my wife had a baby on Sunday, and voice mode's been incredible. I mean, every night, every two hours it's like I have more questions. Voice mode's been huge. So, shot out voice mode. Yeah, so the type of data is going to collect a lot, or change a lot. I think synthetic data has a role to play and in verifiable domains, but what we consistently hear from companies it's like synthetic data is not going to dominate. There's billions, and billions, and billions of dollars of value to extract as a company over the next decade and following the frontier of AI development.

**Lenny Rachitsky** (01:02:24):
Let me first say just huge kudos to you for just having a kid, your wife just having a kid a few days ago, and building this business that is growing bananas and doing this podcast conversation. I really appreciate you. 

**Garrett Lord** (01:02:35):
Thank you.

**Lenny Rachitsky** (01:02:36):
Of course. Is there anything else that we haven't covered that you think might be helpful for folks to hear, or a part of your story that you think might be helpful for folks to learn from, or something you may want to just double down on that we've talked about before we get to a very exciting lightning round? 

**Garrett Lord** (01:02:52):
I mean, the thing I always love talking, I'm really passionate about people starting companies and helping them do so. I just think in this moment right now with AI, for young entrepreneurs that listen, that read this podcast, because I've been a reader since 2020. We looked. 

**Lenny Rachitsky** (01:03:07):
Yeah, we did check. That's incredible. 

**Garrett Lord** (01:03:08):
Yeah, been a long-term reader. I'm just so curious and love sucking up-

**Lenny Rachitsky** (01:03:08):
Appreciate it. 

**Garrett Lord** (01:03:11):
... your interviews. But it's like you just focus on doing something of meaning that really helps people. And I think with AI, there's going to be so many opportunities to improve the way people learn. I'm just really passionate about trying to make Handshake a platform that is not only an incredible business, but is also something that really helps solve a societal problem that matters. And yeah, that's be my one shout out here. If anyone wants advice on how to do that or wants to reach out, I'm happy to chat.

**Lenny Rachitsky** (01:03:43):
Okay, so this is an offer to share advice on starting companies within AI. Is that the offer here? Just so folks-

**Garrett Lord** (01:03:50):
Yeah, that'd be great. 

**Lenny Rachitsky** (01:03:50):
Okay. I don't know how much time you'll have for the hundreds of thousands of people coming your way, but I appreciate the offer. That's very cool. Anything else before we get to a very exciting lightning round?

**Garrett Lord** (01:04:00):
No. 

**Lenny Rachitsky** (01:04:02):
Well, with that Garrett, we reached our very exciting lightning round. We've got five questions for you. Are you ready?

**Garrett Lord** (01:04:06):
Ready.

**Lenny Rachitsky** (01:04:07):
What are two or three books that you find yourself recommending most to other people?

**Garrett Lord** (01:04:11):
I'm a sucker for Peter Thiel's Zero to One. I read it when I started the company, and watched Peter Thiel's startup school class at Stanford he taught back in the days where there wasn't everything written on the internet about how to start companies, and just think he was the coolest. Love Shoe Dog. I think it's the epitome of starting a company. Hard Things About Hard Things obviously, but these are all quite common books.

**Lenny Rachitsky** (01:04:38):
But also classics. Ben Horowitz is coming on the podcast, talk about Hard Things About Hard Things. 

**Garrett Lord** (01:04:42):
Super cool. 

**Lenny Rachitsky** (01:04:43):
The Hard Thing About Hard Things. Yeah. Okay, have you seen a recent movie or TV show you really enjoy it? I imagine you don't have much time for this, but-

**Garrett Lord** (01:04:49):
I'm going to get blasted for this, but I did start Game of Thrones with my wife, and I cannot-

**Lenny Rachitsky** (01:04:55):
For the first time. 

**Garrett Lord** (01:04:56):
Yeah. 

**Lenny Rachitsky** (01:04:56):
Okay, cool. 

**Garrett Lord** (01:04:57):
So, I got a lot of catching up to do.

**Lenny Rachitsky** (01:04:59):
Why would you get ... No, this is great. It's like people that have watched it-

**Garrett Lord** (01:04:59):
I've loved it so far. 

**Lenny Rachitsky** (01:05:02):
You've loved it so far? Okay. It's quite gruesome, that's the only downside of that show. Don't watch it before you go to bed, I don't know how many gruesome scenes you've seen already. Do you have a favorite product you've recently discovered that you really love? 

**Garrett Lord** (01:05:14):
The SNOO. The baby automated SNOO has really helped us a lot. So love the, shout-out SNOO team. 

**Lenny Rachitsky** (01:05:24):
Amazing. I had a SNOO as well. We never actually turned it on, we just ended up using it as a basinet the whole time. 

**Garrett Lord** (01:05:28):
Yeah, most of the time it's not turned on, but a couple of cries it's been turned on, it's been very helpful. 

**Lenny Rachitsky** (01:05:33):
Do you have favorite life motto that you find yourself coming back to, sharing with other people?

**Garrett Lord** (01:05:37):
I love that leave nothing to chance, leave it all out on the field. Grew up in a really hardworking family, and dad worked really hard to provide, make it happen for us and it's like just give it your all. Leave nothing a chance.

**Lenny Rachitsky** (01:05:51):
Okay. So the last question, I was researching you in prep for this podcast and there's a story that I love about your hustle early on is when you were going from campus to campus pitching schools to join Handshake, and there's a story where you had to shower in the Princeton's pool to save money because you just didn't have a place to stay. Is there something there? Is there a story there you could share?

**Garrett Lord** (01:06:13):
Yeah, so it was a tough one. I mean, I almost got arrested at Princeton, because I mean, I guess for entrepreneurs that are traveling around all the time, we were sleeping out of our car. We had this Ford focus, we would put 20, 30,000 miles on it, sleep in the back of like McDonald's parking lots because they're well lit and had good wifi back in the day. And instead of staying in a hotel, a way to freshen up ahead of your meeting is every university has a pool and the pool's almost always, it is always open. We never had a situation where it's always open for people to swim in the morning, like fitness. Faculty, students. And every pool, what do they have? They have a shower. 

**Garrett Lord** (01:06:49):
So, you could go to any pool at any university in the country, and you can get a free shower and freshen up. So, the Princeton campus security did not appreciate me showering as a non-student, but I think it meaningfully helped us because the Princeton campus security called the career service center director we were selling to, being like, "Who's Garrett Lord? Is he really here to pitch you software for your career center?" And it made the start of the meeting with the career center really stimulating and exciting, because they were like, "You showered in our pool, you drove here?" "Yeah, we drove here from Michigan." And so, I think that showed a level of commitment that was exciting for them.

**Lenny Rachitsky** (01:07:29):
Fast-forward to all these founders now starting to use this growth lever of getting in trouble with the campus police to get better meetings with the school leaders. Incredible. Garrett, this is such an insane, amazing, inspiring story, just like what you're building and the opportunity here, and just how it's fast, it's going, and all the advantages you have. If I was an investor in Handshake, I'd be like, "All right, 10 years, it's going great." And now it's like, "Whoa, holy shit. Where did this come from?" Incredible. And it's just also really meaningful. So, I'm really happy that you made time for this in spite of the madness you are in right now. Two final questions, where can folks find you if they want to maybe reach out or maybe if you're hiring, let us know. And then, how can listeners be useful to you?

**Garrett Lord** (01:08:12):
I mean, sign up for Handshake. If you want to message me on there, it's the easiest way to reach me. Just find me at garrettlord@Handshake, and you find me on Twitter. Love X, huge X guy. You can email me at garrett@joinhandshake.com and double R, double T. And how can you be helpful? We are trying to hire so many people. We have offices in New York and in San Francisco, and London and Berlin. If you have friends that are maybe passionate about this, you let them know, or you're interested in the learning more, please reach out. We'd love to talk to you. Hiring is like the number one problem we have right now to meet the demand. So, if you're talented and interested in learning more about Handshake, if you want to work on our consumer product, if you want to work on our employer products, cool PLG issues or the state-of-the-art consumer social experience, like reach out, or you want to work on the AI business we'd love to talk to you.

**Lenny Rachitsky** (01:09:06):
To make it even more clear for folks, what roles are you most hiring for? Is it every role? Is it engineering? 

**Garrett Lord** (01:09:11):
Engineers.

**Lenny Rachitsky** (01:09:12):
Engineering, all right. If you're an engineer and want to join one of the fastest growing AI companies in the world right now, here we go. We'll link to your careers page in the show notes. 

**Garrett Lord** (01:09:18):
Thank you. 

**Lenny Rachitsky** (01:09:19):
Yeah, of course. Garrett, thank you so much for being here. This was incredible.

**Garrett Lord** (01:09:22):
Of course. 

**Lenny Rachitsky** (01:09:24):
Bye everyone. 

**Lenny Rachitsky** (01:09:26):
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lennyspodcast.com. See you in the next episode.

---

## [11/18] How 80,000 companies build with AI: products as organisms, the death of org charts, and why agents will outnumber employees by 2026 | Asha Sharma (CVP of AI Platform at Microsoft)

**Lenny Rachitsky** (00:00:00):
He said that we're just starting to scratch the surface of what an agentic society actually looks like.

**Asha Sharma** (00:00:04):
We're approaching this world in which the marginal cost of the good output is approaching zero. We're going to see exponential demand for productivity and outputs. The way that you scale to that is with agents. When all of that happens, the org chart starts to become the work chart. You just don't need as many layers.

**Lenny Rachitsky** (00:00:23):
We were chatting about this concept you have that we're moving from product as artifact to product as organism.

**Asha Sharma** (00:00:29):
Because these models are so effective at this point, you want to start to tune them to certain types of outcomes. All of a sudden, these are these living organisms that just get better with the more interactions that happen. I think this is the new IP of every single company products that think and live and learn.

**Lenny Rachitsky** (00:00:45):
Planning right now is just crazy. How does anyone plan a roadmap when there's just like, "Okay, GPT-5 is out."

**Asha Sharma** (00:00:50):
We think about it as what season are we in? Season one might've been prototyping of AI and then it was all around models and reasoning models, and now it's the advent of agents.

**Lenny Rachitsky** (00:01:03):
Today, my guest is Asha Sharma. Asha is Chief Vice President of Product for Microsoft's AI platform where she oversees their AI infrastructure, foundation models and agent tool chains, while also leading applied engineering, responsible AI and growth for the core AI division. She was previously COO at Instacart and VPR product at Meta where she ran Messenger, Instagram Direct, Messenger Kids and Remote Presence. She also sits on the boards of the Home Depot and Coupang, and she's a second degree black belt in Taekwondo.

**Lenny Rachitsky** (00:01:32):
Asha has a really unique and rare role that allows her to see more than most anyone else in the world, where things are heading with AI and what works and doesn't work for companies that are building large-scale AI products. In our conversation, Asha shares a bunch of trends and predictions that she's seeing that I haven't heard anyone else talk about, why we're moving from a product as artifact to product as organism world, why GUIs are being replaced by code native interfaces, why post-training is the new pre-training, the coming age agentic society, what it takes to be a successful builder today and going forward, and also her single biggest leadership lesson that she learned from Satya who she works closely with.

**Lenny Rachitsky** (00:02:09):
If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. Also, if you become an annual subscriber of my newsletter, you get a year free of 15 incredible products including Lovable, Replit, Bolt, n8n, Linear, Superhuman, Descript, Wispr Flow, Gamma, Perplexity, Warp, Granola, Magic Patterns, Raycast, ChatPRD and Mobbin. Check it out at lennysnewsletter.com and click product pass. With that, I bring you Asha Sharma.

**Lenny Rachitsky** (00:02:35):
This episode is brought to you by Enterpret. Enterpret is a customer intelligence platform used by a leading CXN product orgs like Canva, Notion, Perplexity, Strava, Hinge and Linear. To leverage the voice of the customer and build best-in-class products, Enterpret unifies all customer conversations in real time, from Gong recordings to Zendesk tickets to Twitter threads, and makes it available for your team for analysis and for action.

**Lenny Rachitsky** (00:03:01):
What makes Enterpret unique is its ability to build and update a customer-specific knowledge graph that provides the most granular and accurate categorization of all customer feedback and connects that customer feedback to critical metrics like revenue and CSAT. If modernizing your voice-of-customer program to a generational upgrade is a 2025 priority, like customer-centric industry leaders like Canva, Notion, Perplexity and Linear, reach out to the team at enterpret.com/Lenny. That's E-N-T-E-R-P-R-E-T.com/lenny.

**Lenny Rachitsky** (00:03:35):
Today's episode is brought to you by DX, the developer intelligence platform designed by leading researchers. To thrive in the AI era, organizations need to adapt quickly, but many organization leaders struggle to answer pressing questions like which tools are working, how are they being used? What's actually driving value? DX provides the data and insights that leaders need to navigate this shift.

**Lenny Rachitsky** (00:03:58):
With DX, companies like Dropbox, Booking.com, Adyen, and Intercom get a deep understanding of how AI is providing value to their developers and what impact AI is having on engineering productivity. To learn more, visit DX's website at getdx.com/lenny, that's getdx.com/lenny. Asha, thank you so much for being here, and welcome to the podcast.

**Asha Sharma** (00:04:25):
Thanks for having me.

**Lenny Rachitsky** (00:04:26):
I want to start with something that we were chatting about before this that I've never heard about as a concept that I think is going to be really helpful for people to think about, which is this concept you have that we're moving from product as artifact to product as organism. Talk about what that means and what people need to understand here.

**Asha Sharma** (00:04:45):
It's been a pretty interesting shift, especially over the last year or so because when I got to Microsoft, it was right after OpenAI and the large foundation models happened, and then immediately after there was this explosion of models, proprietary open frontier models that were pushing the frontier curve and so they were both more efficient and then we're starting to see domain level expertise in a bunch of them and then even more recently, models now can tool call and they can function call and they can take action, and I think that's just giving way to a new type of products that are starting to see some success.

**Asha Sharma** (00:05:26):
And so all of a sudden products aren't just like these static artifacts that we start to ship that's not just like, "Hey, come up with an idea or an insight. Go solve a problem, ship it into the world, maybe make it a little bit better and then have a dashboard." All of a sudden, the whole KPI is what is the metabolism of a product team to be able to ingest data and then digest the rewards model and then create some sort of outcome? Because these models are so effective at this point, you want to start tune them to certain types of outcomes, whether it's price or performance or quality. And so it's pretty exciting because all of a sudden these are these living organisms that just get better with the more interactions that happen and in many ways I think this is the new IP of every single company and it's a completely different way to build product and to even think about products that think and live and learn, which is kind of exciting.

**Lenny Rachitsky** (00:06:21):
So when I hear this, what I'm thinking about is when I had Michael Truell in the podcast, the Cursor CEO, he talked a lot about how their big moat is the data that they capture from people using Cursor, accepting certain suggestions, not accepting other suggestions. Is that what you're talking about here? Just the proprietary data that companies gather from people using their product or is there something beyond that even?

**Asha Sharma** (00:06:40):
I think why we're seeing the rise of post-training happen is just that the models themselves are so powerful. As of this year, Nathan Lambert did this study that I thought was pretty interesting of all the top leader boards and it showed that once a model hits 30 billion parameters, the CapEx to actually train a model and put billions of tokens into a pre-run doesn't economically make sense and you can start to optimize on the loop. And so yeah, in many ways, I think using your own data is the best way to do that, but you can synthetically generate data.

**Asha Sharma** (00:07:16):
You have to come up with the rewards design, you have to actually roll it out, you have to A/B test it rigorously. You have to find the job to be done or the use case that it makes the most sense for. And then yes, that generates data that you can learn from. I haven't ever seen it be one loop for any product. I think it's multiple tracks running in parallel that are like assembly lines, if you will, and producing that.

**Lenny Rachitsky** (00:07:43):
And so is this thesis that we're moving towards product as organism, is this basically for model companies or is this also true for, I don't know, SaaS businesses and tools and user tools?

**Asha Sharma** (00:07:54):
Look, I think that software as a primitive is changing and the artifact inside of it is a model alongside the software components itself. And so in many ways I think that software products will all be model forward products, if you will.

**Lenny Rachitsky** (00:08:12):
This reminds me what I just had Nick Turley on the podcast who we were talking about before we started recording head of ChatGPT and I was asking just like how much does ChatGPT change with GPT-5 coming out, and he's just like, "It's the same thing, they're the same product. It's just the model tells us what to do in the product of ChatGPT."

**Lenny Rachitsky** (00:08:32):
And it makes me think about something else of just you would think why can't just GPT-5 build its own user interface just like as you use it, it just evolve. It's sort of what it's doing with Canvas and all these things, but that's another way I think about when you talk about this idea of product as organism is the product, the UX can shift based on how you're using it and evolve automatically without having product teams have to do anything.

**Asha Sharma** (00:08:53):
I 100% believe that's where the world is going, and then my experience should look and feel different than yours. That's been I've been in personalization, but now you can do it on the fly in the future. So I think that'll be a pretty fun world. I also think it will look different for agents and it will look different for power users and new users and all of those things too.

**Lenny Rachitsky** (00:09:12):
Let me zoom out a little bit and ask you this question. You work with a bunch of companies that are building AI products on your platform, other platforms. I imagine some just do an awesome job and are killing it, some are struggling. What do you find are common patterns across the companies that do really well and have a lot of success building really successful AI products and ones that don't?

**Asha Sharma** (00:09:35):
Yeah, so I think there's things that are more broadly applying to the organization themselves and then there's things that are applying to the people who are building the AI products too. So more broadly, I think there's a pattern that's starting to emerge for successful companies. One is they are embracing AI and everybody becomes AI fluent.

**Asha Sharma** (00:09:58):
So I think everybody is using some sort of co-pilot or sort of AI in their day-to-day workflows like job one, so everyone's not afraid of it, understands how we can raise the ceiling and lower the floor for all sorts of skills and tasks. Number two, from there, they start to say, "Okay, how can I take a process that already exists and apply AI to making it better?" That might be something like customer support or taking fraud down from 15 days to cure to 10 days.

**Asha Sharma** (00:10:26):
In going through that entire loop of mapping out the process, applying AI to it, seeing some sort of impact, and then feeling the P&L or the intrinsic benefits that looks like. The third thing then is like, "Okay, great. Now that you've seen impact, everybody is using it, how do you actually use it to inflect growth?" And that can be something like improving the customer experience, so your LTV or retention improves. It could be co-creating a new set of concepts or categories.

**Asha Sharma** (00:10:56):
It could be going from agents that are embedded to agents that are embodied and then being able to take on exponential number of tasks. I think that where companies fail is that they're doing AI for AI's sake. They have a ton of projects that they're kicking off at the same time without a blueprint to understand how it actually worked and what their Stack looks like and they aren't treating it like a real investment, and so they don't have the measurement and the observability and the evals all set up.

**Asha Sharma** (00:11:24):
It's going to do that end to end. I think the tricky thing is for enterprises is the technology is changing. There's something like 70,000 enterprise tools in the AI space launched last year. It's really hard to know which one you should use for what outcome. And so you really need to bet on a platform or some app server type layer that allows you to swap things in and out and not really be beholden to anything, any one technology or any one tool because the reality is the whole thing is going to change.

**Asha Sharma** (00:11:54):
I feel like you have to actually build for the slope instead of the snapshot of where you are. So that's kind of what I see at the enterprise level. I think the builders themselves are actually changing pretty fundamentally too. Every single advent change a technology has invented a changing set of roles like mainframes to PCs like the whole garage engineers, and then when we went from server to cloud and mobile, there was like SEO specialists and CDNs and growth VMs and UXR and front end, back end, and yada yada.

**Asha Sharma** (00:12:30):
And now I think we're seeing this advent of the polymath and where I think that full stack builders are kind of having their renaissance where if you take an average organization, it takes probably 10 steps to launch a product. It could be security review, it could be spec, it could be user research, and there's what? Five plus functions, maybe six or seven. I'm being generous for a normal organization, and then you have six or seven layers. So all of a sudden, you have 500 different touch points that have to happen to get a product out and when there are 500 models available a week or 500 new technologies, that is insufficient.

**Asha Sharma** (00:13:15):
And so I really believe in the concept of the full stack builder. You're seeing it with a bunch of the AI native companies that are coming up. I'm even seeing it in enterprises that have been around for 50 years starting to operate in that way. And I think that gives you velocity and throughput and then gives you the whole loop to start to actually metabolize and go through that much faster.

**Lenny Rachitsky** (00:13:35):
That's definitely a recurring theme in these conversations is just the Venn diagrams of PM engineering design or starting to converge and more and more of other disciplines within your role. So PM needs to level up on design or engineering.

**Asha Sharma** (00:13:50):
Yeah, I completely agree. I think it's all about the loop, not the lane here. And so I think that whatever function you are, you have to be obsessed with trying to understand the efficiency or the cost of the product, the actual rewards or system design that you're going after, the actual UI, UX, how that actually manifests for agents or people. You have to start to get really good at that really quickly.

**Lenny Rachitsky** (00:14:21):
I like this phrase that you just use, the loop and not the lane. Can you say more about that?

**Asha Sharma** (00:14:21):
Oh, it's just going back to our previous discussion on the signals loop and products evolving and becoming these living organisms and not these artifacts. And if you think about getting really good at that loop, I think that is the product, that is the IP, that is the future of every organization and I think feedback becomes continuous and observability becomes the culture, and I think that functions start to blur in future workforces.

**Lenny Rachitsky** (00:14:49):
To make this even more real, is there an example of a product or a company that is a really good example of doing this well, living this kind of loop life?

**Asha Sharma** (00:14:57):
I think most companies that we're seeing in the space from an AI perspective are doing this. I can tell you about a couple that we are working on. Obviously in the coding space, you mentioned Cursor. GitHub has very similar features that we're using as an ensemble of models that have been fine-tuned across 30 different countries. All of the languages to actually then go iterate in a loop for next set of suggestions or code completions and things like that.

**Asha Sharma** (00:15:25):
We've got in AI product called Dragon that's for physicians and we saw a massive difference from when we used synthetic fine-tuning to when we annotated 600,000 patient-physician interactions by experts and actually fed that into the model and continuously optimized it to then produce. I think we were sitting between 30 and 60 character acceptance rate depending on the run to something like 83%. And so that required a small group of individuals, not a large organization that were able to actually iterate in this loop across functions and all of those lines dissolving.

**Lenny Rachitsky** (00:16:07):
That's super interesting. So what I'm hearing here is if you can gather data on how things are going and then spend a lot of time creating high-quality labeling to feed back into it, to fine-tune it is basically the big advantage is how you win in a lot of this stuff. Okay. Along these lines, something else that you told me that you've been noticing that I want to hear more about is the shift from GUIs and you reference this from GUIs to code-native interfaces. Talk about what that means, what that looks like and what this means for folks building product.

**Asha Sharma** (00:16:38):
I think it goes back to what does it mean to be a product maker in the future. I think that everybody's instinct is a GUI, but if you think back in history, databases went from the desktop down into SQL, I think cloud was all about consoles and now it's about Terraform. And so I think we're literally just seeing the same pattern that's played out in history, start to play out in AI and everything else in AI, it's like Moore's law and it's getting faster. And so I think that's just accelerating and if you think about a stream of text just connects better with LLMs.

**Asha Sharma** (00:17:14):
And so I think that there's a bunch of trends that are working in the favor for the future of products being about composability and not the canvas. And I think that product makers really need to rewire their mindset around this because I think we spend an inordinate amount of time thinking about the UI of something rather than how something composes, how an agent's going to be able to read something. How do you actually get infinite scale? How does that collaboration start to work? And so I think it's just a new way of thinking even though it's long been a trend that's happened in these changes.

**Lenny Rachitsky** (00:17:49):
So is the prediction here that it's terminals like Claude code sort of experiences or is it that it's agents that are taking or is it both? Is that what you're just sharing?

**Asha Sharma** (00:18:02):
Yeah, look, if any of us knew, that would be amazing. I just think that the reason why terminals are great and it feels really great when you code is because of the way it can interact with an LLM with the text stream. And I think that both can be true that humans will continue to commit code and will find new ways to actually do that, whether it's in the IDE, whether it's in GitHub, Copilot, whether it's in some new development environment, and I think that we'll do that with agents and agents will do that with each other and we'll continue to evolve from there.

**Lenny Rachitsky** (00:18:36):
We had Bret Taylor in the podcast, founder of Sierra, and he had a similar prediction that all software companies are going to become agent companies and it's essentially what you're saying here is that your software will just be this thing that's running in the background and there's much less of a GUI. Do you think it still becomes this chat interface the way we're getting used to? Is that the primary interface with agents or is anything something else happening there?

**Asha Sharma** (00:18:58):
I think the conversation is a really powerful interface. I worked on messaging. I think it's great for lots of forms of communication, but it's not the only form of communication. We use email today to collaborate with each other. We use docs. Everybody uses Word and PowerPoint. There's a billion people living in places of artifacts that I think can become really important composable pieces of the picture and I think they should be. So I'm excited about that. I think that chat will be important, but certainly not sufficient.

**Lenny Rachitsky** (00:19:35):
What's interesting is ChatGPT, the number one fastest growing product of all time, maybe the most important consequential product of all time is chat.

**Asha Sharma** (00:19:44):
Yeah, it's great.

**Lenny Rachitsky** (00:19:45):
It works.

**Asha Sharma** (00:19:46):
I think the question we have to ask ourselves is will it only always be chat?

**Lenny Rachitsky** (00:19:49):
Yeah, yeah. The way Nick described it is we're in the MS-DOS era of ChatGPT, which is interesting. It's like the reverse of what you're saying, so it's like maybe if you start as that and then you have to move to GUI and then maybe it'll go back, but he said there's going to be a Windows version where it's much easier to understand what the hell is going on.

**Asha Sharma** (00:20:07):
Yeah. Look, I think that it's smart. Every company should be bringing AI to where their users are and ChatGPT has all of their users using chat and it's a phenomenal product and we've got lots of people around the world that do work in many different ways and we should be thinking about how we use AI to enable that.

**Lenny Rachitsky** (00:20:28):
So let's talk about agents. You spent a lot of time working with agents, building agents, helping companies build agents. Yeah. There's a really great quote that I love. You said that we're just starting to scratch the surface of what an agentic society actually looks like. I just love this idea of an agentic society. What does that actually look like in the future?

**Asha Sharma** (00:20:47):
Oh gosh. It's funny you were telling me about your two-year-old and I have my son Ron just turned one and I can't even imagine life at two. I'm just like that is so far away and what will have been developed. Look, I think that in the future, work will look really different. I think that we're approaching this world in which the marginal cost of a good output is approaching zero. And I think when that happens, we're going to see exponential demand for productivity and outputs.

**Asha Sharma** (00:21:20):
And I think that the way that you scale to that is with agents and it's agents that are embedded and their tools and their pieces of software. And I think there's going to be a ton of those far more than the software that we use today. And then I think there could be a set of embodied agents that are developed and we start to see that now, right? You can assign a pull request to Copilot. You can create a software development rep that's agentic that can do some of the lead generation and mining for you.

**Asha Sharma** (00:21:50):
And so I think that when all of that happens, the work chart starts to become the work chart. I think that tasks and throughput become more important than they have been before. I also think that you just don't need as many layers. I think the whole organizational construct might start to look different in a few years, and so I'm pretty excited about it. I think meetings will still be meetings and there'll be weird, but I think that will be a bit better and I think there'll be lots of changes.

**Asha Sharma** (00:22:24):
I think that for the average employee, my hope and my optimistic view is that they will be able to expand their skill set because now they have their own agents stack that they can bring with them to work just like you can bring your own device and you can start to have access to a set of skills that you never had before. And so if you think about the 20 million people that maybe sit in that space across America and they get 20% more skilled, it's pretty exponential for GDP, and so it's pretty fun.

**Lenny Rachitsky** (00:22:59):
This comment you made about the work chart becomes the org chart is such a profound concept because I don't know if this is what you meant, but what I'm imagining is you build these teams and here's your mission and goal and KPIs and it's humans and like, "Oh cool, go do this first." And what I'm recognizing as you're talking is like, "Okay, but if you have agents doing that, that is their prompt, go drive conversion." And then you have all these agents and that's the org. This is the conversion onboarding team and that's like a bunch of agents off doing their work. Is that what you mean?

**Asha Sharma** (00:23:33):
Yeah, I think today we think in terms of, "Hey, who reports to who in the org chart and who's responsible for these areas?" And I think at the end of the day, when you have a set of capable agents and people are capable of more things, you're not going to start to think in hierarchy and communicating up or during start to figure out outward task base type of opportunities. I think that humans will always decide in organizations how AI is used and what we want to apply it to.

**Asha Sharma** (00:23:58):
But yeah, it's exciting when a new issue comes up or new tasks comes up, how do you actually automatically decide where to route it? Who's working on that task? How do you actually go work on it? How do you observe if they, it's doing the right thing, how do you fine-tune it if they're not, all of those things. So I think that I'm just speculating that there's a world in which that could be pretty exciting and I think that's great because we can just accomplish more.

**Lenny Rachitsky** (00:24:25):
You touch on this point that reviewing the work is going to be increasingly important. If you have a thousand agents off doing work, it's just like holy moly, that's a lot to look at and make sure they're doing the right thing. How do you think that evolves? Just being able to scale your ability to review the work that's being done?

**Asha Sharma** (00:24:40):
Yeah, I think that the same kind of loop that we talked about becomes increasingly important, like fine-tuning and self-healing observability, really good evals, all of that. The good news is that there are systems that manage this for billions of people today that already exists, and so I think that we don't have to reinvent the wheel. There's certainly going to be a bunch of new things to learn if that world ever plays out, but I think managing devices and policies and group access, all those things are solved problems, which is good.

**Lenny Rachitsky** (00:25:17):
This episode is brought to you by Fin, the number one AI agent for customer service. If your customer support tickets are piling up, then you need Fin. Fin is the highest performing AI agent on the market with a 59% average resolution rate. Fin resolves even the most complex customer queries. No other AI agent performs better. In head head bake offs with competitors, Fin wins every time.

**Lenny Rachitsky** (00:25:39):
Yes, switching to a new tool can be scary, but Fin works on any help desk with no migration needed, which means you don't have to overhaul your current system or deal with delays in service for your customers. And Fin is trusted by over 5,000 customer service leaders and top AI companies like Anthropic and Synthesia. And because Fin is powered by the Fin AI engine, which is a continuously improving system that allows you to analyze, train, test, and deploy with ease, Fin can continuously improve your results too.

**Lenny Rachitsky** (00:26:07):
So if you're ready to transform your customer service and scale your support, give Fin a try for only 99 cents per resolution. Plus, Fin comes with a 90-day money back guarantee. Find out how Fin can work for your team at fin.ai/lenny, that's fin.ai/lenny. So a lot of this, it feels like it's in the future. I know a lot of this already happening, people are using agents in all these different ways. Is there any way you and your team have found a value in working with agents of some kind other than coding I imagine is a big part of it, but just anything there that's like, "Wow, that's a big deal."

**Asha Sharma** (00:26:40):
At this point, we have AI and agents and many of our workflows, one of my favorite ones, so right now are my engineering partners out. So I jump on the live site bridges when something goes down and as something as simple as you can automatically get a summary of everything that just happened because usually, there's 15 people talking, you don't actually know where the incident started, where it's going to end and everything and then all of a sudden I have that and I can figure out and ask questions and get updates. Awesome. I think that the entire DevOps areas is changing.

**Asha Sharma** (00:27:17):
We use use Spark to create prototypes so everybody on the team is expected to code, but sometimes just chatting in and talking in real words actually gets you to a prototype that's more interesting and more expressive and reflective of your creativity. So we use that. I think everybody's using AI to write. Everybody is using AI to find ways to have efficiencies and coming up with documentation and things like that, and so I think it's everywhere, which is cool. I think that we're just scratching the surface though for what's possible in terms of working with agents.

**Lenny Rachitsky** (00:27:58):
That's how I always feel when people ask me how I use AI. It's just like everywhere. It's just in every little sprinkled in everything I do now. I don't even know how to describe it.

**Asha Sharma** (00:28:06):
Yeah, it's hard to remember a world where it didn't really exist.

**Lenny Rachitsky** (00:28:09):
Yeah, there's a product manager that I collab with, Peter Yang who talks about how he just, "I don't even know how to do a strategy doc anymore without AI. How did people do this without having someone-"

**Asha Sharma** (00:28:20):
Do you think there will be strategy docs in the future? That's going to be interesting.

**Lenny Rachitsky** (00:28:25):
I wrote this post once of which skills of a PM job will be most replaced by AI, and strategy is the one that people are the most have the biggest debate on. You could argue, I don't know, let's get into it briefly. You would think if some AI had all of the information you had about where the market is going, your metrics, your product today, it would be so good at developing a strategy for you. Many people think that's the one thing AI will be really not good at for a long time because that's where we need all this human judgment stuff. I don't know, do you have any thoughts?

**Asha Sharma** (00:28:59):
I think that some of the most consequential products in the world required a bunch of deterministic, logical sets of inputs and sparks of creativity and imagination and judgment and vision that could not be achieved without humans. Microsoft is the vision of a software factory and creating what Microsoft did wasn't inevitable. Instacart, there was web bands and web bands didn't work, but Instacart did work because of a different way of thinking about it.

**Asha Sharma** (00:29:40):
That came through and iteration and a bunch of things that you couldn't have learned unless you actually went through the process, the iPod, you go forward. So I think it's there. I think docs themselves for every idea, for every need will just start to fade into applications and different artifacts in the productivity suite, which is just a different way of working.

**Lenny Rachitsky** (00:30:06):
Yeah. Your original question, which I didn't quite answer, but I think is important. You're asking do we even need strategy docs? And I guess it's just somehow everyone needs to be aligned on the strategy, maybe it's not a doc.

**Asha Sharma** (00:30:17):
Correct.

**Lenny Rachitsky** (00:30:17):
Yeah, it could be some other artifact.

**Asha Sharma** (00:30:20):
If you architect an organization the right way to keep up with AI, you need a different alignment mechanisms than traditional ways of actually work.

**Lenny Rachitsky** (00:30:34):
So let me ask you actually about that. So planning right now is just crazy. How does anyone plan a roadmap when there's just like, "Okay, GPT-5 is out." Okay, great. What works for you for setting actual a roadmap and a strategy for your team? How far out do you plan? How often do you have to rethink everything?

**Asha Sharma** (00:30:49):
I'll caveat this by saying everyone's just figuring it out and it's a lot harder to figure it out when you're a larger organization than when you're much smaller and you get to run something yourself and there's pros and cons to both. So here's what we do. The company historically, at least in our product teams had semesters that they planned against.

**Asha Sharma** (00:31:10):
So think of that as every six months there's a strategy to look back, look forward, all of those things. I think that's very valuable. I think the idea of six months though and really understanding what's changing out in front is truly challenging to have a overbaked situation. And so we think about it as what season are we in? And so a season which is very uncomfortable can be denoted by a set of secular changes that are happening in the industry or that are happening from customers.

**Asha Sharma** (00:31:40):
And so you can think about season one might've been the prototyping of AI and the early GPT work and then it was all around models and reasoning models and now it's the advent of agents and so that can last a year, that can last six months, that can last three months. But grounding everybody on the ethos of what are the secular changes? What are the customer problems we need to solve? What does winning look like?

**Asha Sharma** (00:32:06):
So everybody has that shared sense. What is the north star metric is something that we do. The second thing that we do is that we have kind of loose quarterly OKR. So like, "Okay, if we believe that, what do we need to do next quarter to actually put ourselves on a path to that?" And then from there, teams are operating in squads and they're kind of setting out four to six week goals that they're trying to go after for problem areas to go ladder up to that and especially as the platform for the company and the platform for our Azure customers with AI, I'll say we go through lots of changes to that all the time and I think we have to just have an openness that that is the business that we're in.

**Asha Sharma** (00:32:47):
I think the other thing is just we try to leave Slack in the system, not just for the unplanned, but for the slope. I think that we have to continuously be thinking about how we're going to disrupt the platform in our thinking and what we need to be investing in to make that possible. And so we try to do a little bit of both.

**Lenny Rachitsky** (00:33:06):
This is awesome. So what I'm hearing here is there's this concept of seasons and everyone's aligned, "Okay, this is time for agents, this is what's happening right now. We're going to center around our strategy around agents." And then there's these loose quarterly OKRs. You plan for three months roughly and then you leave some Slack in the system for things to change.

**Asha Sharma** (00:33:24):
Yes.

**Lenny Rachitsky** (00:33:24):
Is the current season agents, how would you describe what season we're in right now?

**Asha Sharma** (00:33:27):
Yeah, it's agents. The rise of agents.

**Lenny Rachitsky** (00:33:31):
The rise of agents. It sounds like a Terminator movie. Do you have a sense of what the next season might be? Is there any like, "Oh, this might be coming next."

**Asha Sharma** (00:33:39):
Gosh, I don't, but I think that, look, we have more than 15,000 agents that are deployed on our service today, at least at the Azure service. There's a bunch of other platforms in the company and I would just say that I think that we should really focus on making sure that we have all of the alignment, accountability, observability, evals to making those agents great.

**Asha Sharma** (00:34:11):
I think that Manus breakthrough in the space was that they could do these tool calling loops and have agents do longer running tasks that really no other platform was able to do. I think stuff like that is critical. Memory is critical. There's still a bunch of building blocks that I think are leaving agents incomplete in the wild that I think we have to really sweat the details on before we move on.

**Lenny Rachitsky** (00:34:37):
So it's just like agents until the end of time until super intelligence and then we're just on beaches chilling.

**Asha Sharma** (00:34:44):
Yes, agents until dank memes look. Yeah, I think the cool thing is something new could come in three months. Something new could come in 13 months. I think we have this conviction on a set of building blocks that we want to provide to enable these agents to endure and have high endurance and so that's what we're focused on.

**Lenny Rachitsky** (00:35:07):
When you said there's 15,000 agents, what does that mean? Is that 15,000 types of agents you can use or is it like that's how many processes are?

**Asha Sharma** (00:35:13):
No, that's customers. 15,000 I think I should re-reference the numbers. 15,000 customers who have produced agents. I think the number of agents is actually millions.

**Lenny Rachitsky** (00:35:24):
15,000 customers that are building a specific kind of agent on your platform and they're running and the number of agents is in the millions just running in the cloud.

**Asha Sharma** (00:35:32):
Yes. Exactly.

**Lenny Rachitsky** (00:35:33):
Okay. It's wild. Some crazy numbers here. Okay, so let me just go in a slightly different direction. You're in the center of the storm of a lot of AI, just seeing everything else going on. Is there something you wish you'd known before stepping into this role that you're just like, "Okay, I see. I didn't expect this."

**Asha Sharma** (00:35:52):
When I first took the role, it was described as the belly of the beast and I had spent most of my career building products at the center of machine learning and applications or businesses and I think that to my surprise, a lot of the learnings have translated in terms of what makes a great platform is what makes a great product. And the thing for me is it's often in the invisible work or not the pixels that actually drives that.

**Asha Sharma** (00:36:26):
So for example, one of the first companies that I worked at was a company called Porch Group. I was employee seven and we knew we wanted to help people take care of their home and I think we invented so many features like the home report or a way to manage your home or house style inspiration where you could see all of the houses and it's map every single room. And the single most important thing that we could have done and did during my time there was create a matching platform that matched the 6 million professionals with the 1,300 service types with the 37,000 zip codes and all of the homeowners in North America to actually take care of their home, and that was just the game of inches and optimizing that engine in order to create higher quality leads essentially.

**Asha Sharma** (00:37:15):
That's what got us to the first $500 million valuation. That's eventually what we built on to actually have other vertical services and software platforms that IP of the company. Same with messaging. The number one learning that I had was look like WhatsApp didn't win because it had stickers or stories or dark mode. In fact, I don't even think it had all of those things when it won. It won on a few premises because one was the phone book, you knew that when you use WhatsApp, you could reach every single person because you had their phone number and those are the people that you care about when you're using messaging.

**Asha Sharma** (00:37:54):
It was the reliability and how fast it was. I could text my grandmother in India and know that she would get my text message all the time, and then it was the privacy. When you are sending 200 messages a day to the four people you care about most, you want to make sure no one else can read the messages and so the end-to-end encryption really mattered. And so it wasn't the hundreds of features, it was all in the infrastructure and the platform.

**Asha Sharma** (00:38:21):
Same Instacart, there are so many loved features of Instacart, but at the end of the day, it's a billion items that updates 3,000 times every single minute to get homeowners their groceries from the store that they love. And so I think I wish I had known that because I think it would've curtailed my learning curve to say that it's not all the features for the platform that matters, it's the data residency.

**Asha Sharma** (00:38:46):
So the hospital in Germany that's fine-tuning the model can do so in confidence and the data isn't going to leave the region, it's the availability, it's the reliability. It's making sure you have the right selection of the tools that enterprises need and the right way to retrieve the knowledge and that's the platform that we've built but just didn't fully have that picture that those learnings would translate.

**Lenny Rachitsky** (00:39:07):
Mm-hmm. That's really interesting. So what I'm hearing is people undervalue who just the simple bottom of the Maslow hierarchy of things that help you win in platforms, especially in messaging platforms including so it's like reliability, privacy, I don't know, availability.

**Asha Sharma** (00:39:26):
Yeah, performance, reliability, privacy, safety, all of those things.

**Lenny Rachitsky** (00:39:31):
Mm-hmm. Let me ask you a totally different question. When we were going to record this previously and you're like, "Oh, I have a big meeting with Satya I got to do instead." And so we moved at a different time. Very few people get to work with Satya, he's quite a successful leader. What's something you've learned from him about? I don't know, leadership or product building?

**Asha Sharma** (00:39:52):
I've learned that optimism is a renewable resource. This company for 50 years has had every reason not to succeed and it has even as it's had early success in the AI era and challenges and other successes and the space is developing so quickly, I think that his ability to generate energy and to use his optimism to renew everybody's dedication to the mission is unbelievable and I think it's such an important part of the culture. Everybody talks about the growth mindset, that's real, huge part of the culture, but I think the ability to generate energy and clarity on what we need to go do and use optimism to renew the commitment every single day for every single person in an entirely competitive talent space is pretty amazing.

**Lenny Rachitsky** (00:40:54):
Is that something you think that is just innate to him or it's something that he's worked on to just generate this optimism on behalf of everyone?

**Asha Sharma** (00:40:59):
I have no idea. We should ask him, but I'm deeply impressed by it.

**Lenny Rachitsky** (00:41:04):
It's interesting that a lot of this comes down to just vibes. There's just this vibe of imagine it's not him just the words he uses, it's just this energy that he exudes optimism and energy.

**Asha Sharma** (00:41:16):
Think about it. We all choose to someone who just said this to me and I thought it was great, "We all choose to close the door on our kids every single day to go work on something." And so you have to work on something that is deeply moving to you and you have a deep belief that is going to make the world a better place and I think that's why it's vibes. I think you have to follow and have a sense of duty towards a mission that is bigger than yourself.

**Lenny Rachitsky** (00:41:45):
It makes me think of a line that I've referenced a couple of times on this podcast that hits people really hard that the only people that'll remember you working late are your kids.

**Asha Sharma** (00:41:54):
Okay. I don't know where we're going with that, but that was like, now you're like, yeah.

**Lenny Rachitsky** (00:42:02):
It's too much. We've gone too far. Oh man. Okay. Well let ask you this. What's driving you?

**Asha Sharma** (00:42:04):
We could have said our customers, we could have gone a different route on that one.

**Lenny Rachitsky** (00:42:09):
This is the real stuff. What's driving you? What's driving you? What's keeping you excited about the work that you're doing?

**Asha Sharma** (00:42:17):
What AI will help us do from a workforce perspective, what it will help us do from a healthcare perspective. My mom has cancer and I think a lot about how we might find a way to solve the form of cancer she has in my lifetime and I never thought that was possible three years ago. All of that's deeply profound and the thing that I personally think a lot about now that we know that we're living in this time working with such powerful technology is the effects of it and how I can best build a platform where people can make use of it.

**Asha Sharma** (00:42:52):
So the reason why I work at Microsoft is because the whole ethos of the company is how do I help people and businesses achieve more and more for me in the thing I think about at night outside of GPUs is I think about will my son have classmates in the future? And that's not because agents are going to replace them, it's because the fertility rates are declining. The average birth rate in the '90s when we were growing up was like three and now it's 2.3 and in 2050, it's estimated to be below replacement and I think that AI can have such a big effect on it and already is.

**Asha Sharma** (00:43:42):
It was just reading about a hospital in London that's able to improve pregnancy rates by using AI to match eggs and sperms and their cutting costs at the same time. You saw with the ChatGPT-5 launch yesterday. Such an amazing story about how ChatGPT is helping in healthcare. Stanford is one of our big customers with the platform that I build and they're working on using AI for tumor reviews and it's just like, it is these sets of things that will move humanity forward and expand our lifetime and give us the privilege to solve 100-year problems. And so that's why I'm excited and that's why I do what I do.

**Lenny Rachitsky** (00:44:22):
Yeah, especially in your role where you're building the platform that enables all of this, I could see how impactful that could be. Asha, is there anything else that you wanted to touch on or share or double down on of anything we've talked about before we get to our very exciting lightning round?

**Asha Sharma** (00:44:39):
We touched on it a little bit, but I think that with the advent of agents and products that think and can act and reason, there's going to be this new wave around RL and I have a deep belief that that will become one of the most important product techniques of the next season or at least the next few seasons.

**Lenny Rachitsky** (00:45:00):
And RL is reinforcement learning?

**Asha Sharma** (00:45:02):
Yes. Yes, exactly. I believe we will see just as much money spent on post-training as we will on pre-training and in the future, more on post-training. We talked a little bit about Nathan Lambert's study where his review was that when a model hits 30 billion parameters, it makes more sense to fine-tune and optimize that 50% of developers according to surveys are now fine-tuning and we know fine-tuning is good, but if you actually go through the full loop, you can get better results.

**Asha Sharma** (00:45:30):
So I think there's a bunch there and I think there's a whole new set of infrastructure and platforms and companies that will be created that are all around this part of the stack. And so I think it's an exciting time to be in the platform space, but it's also an exciting time to be starting companies and be thinking about those problems.

**Lenny Rachitsky** (00:45:48):
I want to make sure people truly understand what you're saying here because not everyone truly understands post-training, pre-training. What's the simplest way to understand the difference there and just why it's such a big deal that investment is moving to post-training?

**Asha Sharma** (00:46:02):
The way that I think about it is to create a foundation model, it requires a tremendous amount of compute, a tremendous amount of science. Expertise as we're seeing which the cost for scientists or the average value is raising dramatically and I think an expertise that we've seen isn't everywhere in the world right now. And so it's just a big CapEx investment to do that.

**Asha Sharma** (00:46:33):
With this explosion of models that we talked about in the beginning, there's a lot of good models to choose from for different domains. And so I think that you just get more leverage economically, you get more leverage from a taste perspective of how you actually want to steer a model if you're actually doing reinforcement learning or some sort of fine-tuning to actually start to optimize what's off the shelf for some outcome like price, performance, quality.

**Asha Sharma** (00:46:58):
If you think about that, that's not crazy, right? Ranking is an age-old optimization problem where you don't want to just take what's off the shelf because there's amazing frameworks and UI and components that the world is react components that are out there. You still want to tailor the experience to a set of use cases or a set of people. I think it's just the same industrial logic.

**Lenny Rachitsky** (00:47:22):
So in practice, what this means is there's a GPT-5 model. You're saying there's a lot of opportunity and a much more efficient way to spend money, which is take something like that and then train that on additional custom data that you have, whether it's data or just reinforcement learning, maybe even with humans to align it with what you wanted to achieve?

**Asha Sharma** (00:47:41):
Yep, and it could be your own data, it could be data that you buy, it could be synthetic data, it could be something else, but I think that we're going to start to see more and more companies and organizations start to think about how do I adapt a model rather than how do I take something off the shelf as is or invest a bunch of money and building my own models.

**Lenny Rachitsky** (00:48:06):
Yeah, I forget. I know Cursor, when he was on the podcast, he shared that they have a bunch of models that support your experience with Cursor and over time, they're just going to have their own thing. I forget who it was, Windsor for one of those guys just uses their own model now, they don't just plug into Claude.

**Asha Sharma** (00:48:22):
I'm much more in the model system camp. I believe in model diversity. I think that in experience like Claude, like Sonnet 4 is awesome for a set of use cases versus GPT-5 is different for different use cases. I think that there's some tasks where you care about the latency of the model. You're cool with the thinking time or you want a quick retrieval and things like that. I think the beauty is there's a lot of models that can help you achieve that, and so I'm much more in the model system rather than one model to rule them all.

**Lenny Rachitsky** (00:48:58):
Is that the right term? I've also heard ensemble model, ensemble of models.

**Asha Sharma** (00:49:01):
I think about an ensemble of models as a set of multiple models that then you can fine-tune and deploy independently, but at this point, we're all making up different terminology to define things that we have deep beliefs on that have limited sets of data points because everything is moving so fast.

**Lenny Rachitsky** (00:49:18):
Yeah. With that, we've reached our very exciting lightning round.

**Asha Sharma** (00:49:23):
I'm very excited for our lightning round and I'm turning down the lights

**Lenny Rachitsky** (00:49:27):
And then it'll come back on I imagine in one second. Okay. First question, what are two or three books you find yourself recommending most to other people?

**Asha Sharma** (00:49:34):
At work? It's probably Thinking Machines, so it's all about treating the cause, not the symptoms. The prototypical example is if you want to solve traffic, you don't actually put up speed bumps or speed limits, you actually have to solve walkability and mobility and why people actually use cars. Outside of that, personally, the CMO of Instacart recommended to me Tomorrow, and tomorrow, and tomorrow and I read it last month and last year and the year before because I love it so much. It's like this beautiful story over 10 years.

**Lenny Rachitsky** (00:50:12):
Mm-hmm. What are some favorite recent movie or TV shows you really enjoyed?

**Asha Sharma** (00:50:18):
Formula One, saw it twice for all mankind. For all Mankind, I like season four. I don't know, I like playing alternative theories to how the space race might have looked.

**Lenny Rachitsky** (00:50:32):
Do you have a favorite product? That you recently discovered that you really love? Could be tech, could be gadgets, could be clothing.

**Asha Sharma** (00:50:36):
So I just joined the board of the Home Depot and we're doing a little renovation project and so there's this new, well, new to me DEWALT power pack and they use pouch cells and so it's like 50% lighter, but with all the power and it's awesome for drills and things that I need to lift up with one hand that feel heavy. So I love that.

**Asha Sharma** (00:50:59):
We also are testing out this new brilliant, smart home kind of system. So it's four inches of high-res middleware that allows you to connect to everything and I've reached peak dissat with the explosion of all the technology required to actually use your home. So it just might be the middleware that sticks, but we'll see.

**Lenny Rachitsky** (00:51:22):
Did you say dissat? Is that short for dissatisfaction?

**Asha Sharma** (00:51:26):
Yes. Sorry. I'm speaking in acronyms.

**Lenny Rachitsky** (00:51:28):
Whoa, I've never heard that dissat. I love that. By the way, I love that you're on the board of the Home Depot. What a different part of the spectrum of work.

**Asha Sharma** (00:51:40):
Yeah, it's been awesome. The very first board meeting, the head of philanthropy has been at the company for decades and she said, "Welcome to the greatest company on the planet." It's pretty special.

**Lenny Rachitsky** (00:51:52):
They're like, "Microsoft." Is there something you've learned from working with them that you've brought to Microsoft?

**Asha Sharma** (00:52:00):
Look, it's new, it's this year, but I've long worked on products that had that impact. So when I was at Porch, it was pros. At Instacart, we had 600,000 shoppers and obviously, the Home Depot has associates. One of my favorite things about the company culturally is they have this inverted pyramid where instead of having executives at the top, the associates are at the top and the stores themselves are headquarters and then the traditional HQ is support.

**Asha Sharma** (00:52:35):
And so it's so customer-centric and when I think about amazing execution and creating these durable long-term institutions and how culture and ideology and leadership is formed, I think about that and I think about at the end of the day, AI is going to have an impact on every single person and every single job. And it's amazing to just spend time with people outside of our bubble and really try and learn what their real pain and problems and how they think about AI and how they think about technology and what we need to do.

**Lenny Rachitsky** (00:53:09):
Okay, two more questions. Do you have a favorite life motto that you find yourself coming back to? Sharing with friends or family?

**Asha Sharma** (00:53:18):
I used to use the minimize regret framework and it's great, and I've used that for a long time. I think that probably once I got into my adult years and started to have a family and things like that, my just worldview changed a little bit and it was all about maximizing option value and it just gave the things that I naturally cared about like family and health and trust and relationships.

**Asha Sharma** (00:53:51):
It was just a new level of value associated with those because all of a sudden, learning rest on the weekend can compound in the future or having good health can compound in the future. You don't have to trade that off of working extra hours or the importance of family and all of those things. And so I think that my worldview is when I'm 70, it's not about what do I look back on in my life and count the number of regrets, it's really about looking forward in the number of adventures I will still have because I have accumulated this wealth of skills and trust and people and family and impact and things like that.

**Lenny Rachitsky** (00:54:31):
Speaking of skills, the internet tells me that you're a second degree black belt in Taekwondo. Why? Oh gosh. Is this true? And then I have a question about it.

**Asha Sharma** (00:54:44):
This is true.

**Lenny Rachitsky** (00:54:45):
Okay. That's incredible. Why is this embarrassing? That's an incredible thing.

**Asha Sharma** (00:54:52):
I am generally embarrassed anytime anything is discussed about me.

**Lenny Rachitsky** (00:54:56):
Okay, great. No problem. What's something that you learned from Taekwondo that has helped you with life or work?

**Asha Sharma** (00:55:03):
Taekwondo is more mental than it is physical. And so I think that's the same with all of our jobs and making product. I think it's like mental clarity, it's courage. It's the ambition to see things through and be unwavering. And so I think that's literally what it taught me outside of meditating, which probably took me the entire time to actually learn to meditate and clear my head. But yeah, I think it's awesome. I think everybody imagines flying psychics or running up a wall and you can do those things too, but the real value is the mental pursuit of it all.

**Lenny Rachitsky** (00:55:47):
And you can do those things too. Wow. Okay. I'm good. I got to get into this. Asha, this was awesome. Is there, oh, actually two final questions. Where can folks find you online if they want to maybe follow up on anything, if you want people to reach out and how can listeners be useful to you?

**Asha Sharma** (00:56:02):
You can hit me up on LinkedIn or email or text. I think all of those are traceable. Look, how can you be helpful to me? I think we're all early in this journey and great platforms that are built on great use cases and built on great customers, and so if you have feedback, you have ideas, you have things want AI to be able to do to help you achieve more, I'd love to hear it. I think the thing about all of these changes is that all of these new products and use cases will be developed everywhere, and so I'm always just thinking about how can we be the platform to support that.

**Lenny Rachitsky** (00:56:40):
Amazing. Asha, thank you so much for being here.

**Asha Sharma** (00:56:42):
Thanks for having me.

**Lenny Rachitsky** (00:56:44):
Bye everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.

---

## [12/18] How we restructured Airtable’s entire org for AI | Howie Liu (co-founder and CEO)

**Howie Liu** (00:00:00):
If you were literally founding a new company from scratch with the same mission, how would you execute on that mission using a fully AI native approach? If you can't, then you should find a buyer and then if you really care about this mission, go and start the next carnation of it.

**Lenny Rachitsky** (00:00:12):
Or people that work for you, how have you adjusted what you expect of them to help them be successful?

**Howie Liu** (00:00:18):
If you want to cancel all your meetings for like a day or for an entire week and just go play around with every AI product you think could be relevant to Airtable, go do it.

**Lenny Rachitsky** (00:00:27):
Of the different functions on our product team PM, engineering design, who has had the most success being more productive with these tools?

**Howie Liu** (00:00:33):
It really does become more about individual attitude. There's a strong advantage to any of those three roles who can kind of cross over into the other two. As a PM, you need to start looking more like a hybrid PM prototyper, who has some good design sensibilities?

**Lenny Rachitsky** (00:00:49):
Do you see one of these roles being more in trouble than others? Today, my guest is Howie Liu. Howie is the co-founder and CEO of Airtable. I'm having a bunch of conversations on this podcast with founders who are reinventing their decade plus old business in this AI era, to help you navigate this existential transition that every company and product is going through right now. Howie and Airtable's journey is an incredible example of this, and there's so much to learn from what Howie shares in this conversation.

**Lenny Rachitsky** (00:01:20):
We talk about a very interesting trend that I've noticed that Howie is very much an example of, of CEOs almost becoming individual contributors again, getting into the code, building things, leading initiatives themselves. That's something that we call the IC CEO. We also talk about the very specific skills that he believes product managers and product leaders, also engineers and designers need to build to do well in this new world that we're in. Also, how he restructured his company into two groups, a fast thinking group, and a slow thinking group, which allowed their AI investments to significantly accelerate.

**Howie Liu** (00:04:14):
I'm so excited. Thank you, Lenny. I've been a listener from afar for a while now.

**Lenny Rachitsky** (00:04:19):
I'm really flattered to hear that. I'm also very excited. You've been on quite a journey over the last, is it 13 years, is it longer?

**Howie Liu** (00:04:27):
Yeah, right about 13.

**Lenny Rachitsky** (00:04:28):
13 years. I imagine there've been a lot of ups and a lot of downs. I want to talk about all those things. I want to talk about a lot of the lessons that you've learned along the way. I want to start with what I imagine was a very surprising down moment in the history of Airtable. This is something that, unfortunately, something I think about when I think of Airtable. I feel other people may feel this way, is there's this tweet that went super viral, maybe a couple of years ago at this point where someone just shared all this data and they're like, Airtable is dead.

**Lenny Rachitsky** (00:04:57):
They've raised way more money than they're worth. They're not making enough to get from underwater. Yeah, Airtable RIP. What happened there? How much of that was true? How did that go?

**Howie Liu** (00:05:06):
Yeah, so basically none of it was true. I mean, the surprising thing to me was how viral this tweet went when ... Frankly, I actually look back at this person's other tweets. I think they worked at CB Insights, and the irony is the whole point of that business is to have good data, good data quality around private company data. And they just literally had incorrect numbers by a strong multiple on what our revenue scale was, what our growth rate was. And if it gave me some consolation, I look back and this person had also tweeted about other companies, like Flexport was the last take-down tweet.

**Howie Liu** (00:05:45):
They have like, "Oh, Flexport's dead" and their evaluation is too high, and blah, blah, blah. And so, I think that the more surprising thing was just like this person has been tweeting a bunch of spicy takes that are not substantiated by real data or correct data, and yet this particular tweet went super viral and that was the perplexing part to me. And then, actually, I think what really gave it legs was on the All In podcast, which is obviously super popular. And I listened to it. They covered it. They were like, "Oh, latest on this week's news, this tweet about Airtable. What do we think about this?"

**Howie Liu** (00:06:22):
And it almost, I think became a way to talk about a broader theme of what happens to this last generation of highly valued companies, maybe decacorn companies in this new ... And at that point, it was the recent moment for both public and private markets. They did also issue a correction though. All In, did a follow-up episode, a few, I think weeks later saying, "Hey, we got the numbers wrong. We are revising our case and a view on Airtable."

**Lenny Rachitsky** (00:06:53):
What's that line about how a lie gets around the world some number of times before truth has even this time to get out of bed?

**Howie Liu** (00:07:02):
Yeah. Well, I think I learned about memes and morality very quickly in that experience. Not a very good social media person, but I think I learned a little more.

**Lenny Rachitsky** (00:07:10):
Yeah, it's tough. Twitter is such ... The incentives are so misaligned. It's just I tweet something people want to share, not truth.

**Howie Liu** (00:07:17):
Well, especially ... I mean, there's a lot to like ... I would say, I like the post Elon Twitter more than the pre Elon Twitter because it is just bolder, and I guess I really admire bold product execution where you're not just stuck to the current laurels and they've made so many changes, but I do feel like I get injected into my feed very sensational content all the time, and I mean, it works on me. I can't help but to click on it and engage with it, but it does ... I think it does result in this kind of content, really spreading.

**Lenny Rachitsky** (00:07:52):
Yeah. Now, Nikita running the show, I don't know if you saw this, there's a new ... We don't need to keep talking about Twitter, but there's a new feature where you take a screenshot of a tweet and it has a huge X.com logo watermark on the top, right? Yeah, just to ... People are sharing these tweets all the time. Yeah.

**Howie Liu** (00:08:06):
Yeah.

**Lenny Rachitsky** (00:08:06):
Man. Never a dull moment over there.

**Howie Liu** (00:08:07):
For sure.

**Lenny Rachitsky** (00:08:08):
Okay. I want to go in a completely different direction, something that I'm really excited to talk to you about, which is this very emerging trend that I've noticed that I feel like you're at the forefront of CEOs becoming ICs again. It's kind of this move of, IC CEOs. CEOs getting their hands dirty again, building again, getting in the weeds, coating again. I feel like you're again at the forefront of this. Talk about just why you've done this, why you think this is important, and just what that looks like day to day to you versus what your life was like a few years ago.

**Howie Liu** (00:08:39):
The underlying reason for this shift, at least for me, is that, as we started the company, I was very much in this mode. I was literally writing code both on the backend, thinking about the real time data architecture of our platform, also the front end, the UX. And I would argue that in that founding moment, the initial product market fit finding, and especially for a product that is pure software, we weren't building an operationally heavy business like a dog walking marketplace where the tech is only an afterthought.

**Howie Liu** (00:09:11):
The tech was the product, right? And in a very Meta sense, Airtable is the platform for other people to build their own apps. So it's all about the attack, like the very intimate design decisions, again, both architecturally and on the front end and the product UX choices. That is the product's value prop. You can't separate those two. You can't say, "Okay, I researched the jobs to be done. Here's the workflow, here's the process, and then, okay, some engineer can just build it as an afterthought."

**Howie Liu** (00:09:40):
It's those little decisions and really be able to be at the bleeding edge of what's possible both in the browser and with the real-time data architecture. That made the product what it was. I think the same is true for Figma, which actually had a very parallel timeline to us. We both were founded around the same time, both spent two and a half years building the product, hands-on that early team before launching. And when I think now to both the era in between that founding moment and then now as well as now the new gen AI moment, I think there was a maturing era of both SaaS overall and Airtable specifically.

**Howie Liu** (00:10:19):
Where, as you scale up and you learn how to build teams and organizations and you have to scale up stuff that's not actually those intimate details, but process and people and so on, you kind of get by default further and further away from those details, right? And maybe for some businesses that's fine because no longer is it about finding the details that make for a magical new product market fit. And it is really just about scaling up an existing thing that works and using what I would call more blunt instruments to scale it up, like a more blunt roadmap, a more blunt go-to market execution strategy.

**Howie Liu** (00:10:56):
Regardless, I think that now, we're entering this moment where ... Certainly every software product in my opinion, has to be refounded because AI is such a paradigm shift, it's not even just like the shift from desktop to mobile or on-prem to cloud where that was more like a very one time and somewhat predictable change in form factor. I think AI is so rapidly evolving that with every evolution, every new model release and every new type of capability that's released, it actually implies novel form factors and novel UX patterns to be invented to fully capitalize on those capabilities.

**Howie Liu** (00:11:36):
And so to be continuously relevant and to refine product market fit in this era, I think you have to be of the details. There is no looking at it from 10,000 foot view and saying, "Oh, we're just going to throw a bunch of people at this problem." It's actually understanding what is the right product experience and the right business model that backs it up and the right ... everything else to support that engine to take advantage of the capabilities in our product domain.

**Lenny Rachitsky** (00:12:07):
You have this phrase somewhere where you talk about being the chief taste maker.

**Howie Liu** (00:12:11):
Yeah.

**Lenny Rachitsky** (00:12:11):
And to do that, you have to do exactly what you're describing.

**Howie Liu** (00:12:14):
That's right. I mean, I think that, and I would also say it's actually now also hard to taste the soup without participating in at least some part of creating the soup. Meaning With AI, you can kind of look at the final product and say, "Okay, this feels right or not, or it feels like we're being bold enough and we're properly productizing these new capabilities." But I think to really understand the solution space of what's possible, you have to be in the details.

**Howie Liu** (00:12:46):
I mean, literally, you can't just look at screenshots or a pre-recorded video of a new product feature. AI is something you have to play with, and ideally you're playing with both the packaged up app or solution that you've built with it, but you're also playing around directly with the underlying primitives who are using the models either via API or via a chat interface. You're really pushing them to the boundaries. Because that's the only way that you really understand what these new ingredients. It's like as a chef, you just gained access to amazing new ingredients, but you have to actually get comfortable with them to put them into a new dish.

**Lenny Rachitsky** (00:13:23):
And we had Dan Shipper on the podcast, he runs this newsletter and podcast to product a company called Every. And they work with companies to help them become more AI successful and adopt AI and all that stuff. And I asked him, what's the signal that a company will have success adopting AI and seeing huge productivity gains? And he said, does the CEO use ChatGPT or Claude daily?

**Howie Liu** (00:13:48):
Yeah.

**Lenny Rachitsky** (00:13:49):
And I feel like you're describing exactly, hourly,

**Howie Liu** (00:13:51):
Literally hourly, or you could even have a measure of inference costs, right? Like the equivalent underlying inference compute cycles, right?

**Lenny Rachitsky** (00:14:02):
How many tokens you use?

**Howie Liu** (00:14:03):
Yeah, I mean, I'm proud to say I am pretty sure I'm still the ... I just checked this recently, but I take pride in being the number one most expensive in inference cost user of Airtable AI, not just within our own company, but I think for a long time I was globally across all our customers vault. I mean, I'm extremely intentionally wasteful. Wasteful in the sense of I'll do something that costs maybe hundreds of dollars of actual inference costs. For instance, doing a lot of LLM calls against long transcripts of let's say, sales calls to extract different types of insights like here's the product apps, identify or here's summaries, et cetera.

**Howie Liu** (00:14:49):
And we also have now a capability that's basically like an LLM map reduce. So effectively, even if you can't fit the entire corpus of content into one LLM call, because the context window limitations, we'll map through all of this content and break it up into chunks and then perform an LLM call on each one and then perform an aggregation LLM call on those chunks. Very expensive, because you're basically running a highly expensive model against a lot of data and then running it again on the aggregates of that. But for me, hundreds of dollars spent on this exercise is trivial compared to the potential strategic value of having better insights.

**Howie Liu** (00:15:29):
It's as if a really, really smart chief of staff has gone through and read every single sales call transcript that we've had in the past year and giving me very astute product insights, marketing insights, kind of positioning insights and segmentation insights. That's invaluable. You could pay a consulting firm literally millions of dollars to get that quality of work. So to me, I still think the value versus the actual cost of AI when applied greedily but smartly, it's a crazy ratio. And more people should be aggressively throwing compute cycles at these very high value problems.

**Lenny Rachitsky** (00:16:11):
Until somebody tweets how you're costing the company so much on AI compute and you guys are going to be underwater.

**Howie Liu** (00:16:19):
I'm just kidding. It's like how we have personally taken down the cashflow profile of the business.

**Lenny Rachitsky** (00:16:27):
So CEO's, founders hearing this, they're probably like, okay, I should probably start doing this. What does this actually look like? I imagine you still have a lot of other stuff you got going on once, you got all these ... How do you change your day to day to do this?

**Howie Liu** (00:16:41):
Yeah, so I actually cut my one-on-one roster by default, and the idea is not that I don't want to spend time one-on-one with people, but rather that I found that the ... Just having more standing one-on-ones actually precludes me from engaging in more timely topics. I like to think of the best types of meetings as very urgency driven. And there's some timely topic, you've discovered some insight. Maybe I talked to some new startup and I learned something from their product or their approach.

**Howie Liu** (00:17:20):
And I want to bring that into how we're thinking about a new feature at Airtable or even just plant the seed with some different EPD people within Airtable, I want to make most meetings very timely and very informed by real alpha. There's got to be some kind of value and insight to seed that with. Now, in addition to that, I'll supplement with, when I'm in person with someone, I want to carve out time for a proper catch up and less structured, less timely, and just more of building a relationship with a human.

**Howie Liu** (00:17:53):
But I actually find that having that common .. It's almost a barbell approach where it's like if you're going to spend time with somebody in a freeform way, actually do it in a high quality, not forced weekly ritual way. Go for a longer lunch or coffee walk or whatever in person when you can. Maybe that's a once every month or two kind of thing. And then the in-betweens are either topical, so we do have standing meetings for ... Now, we have a weekly basically sprint check-in on all of our AI execution stuff, which now is half the company or half the EPD org is working on AI capabilities.

**Howie Liu** (00:18:29):
We're trying to ship very quickly, like I basically want to always ask the question, how would an AI native company, like a cursor or windsurf, et cetera, how would they execute? And are we executing as fast as them and taking advantage of all the new stuff as well as them? So bringing that level of intensity and urgency to how I spend my time within, that's been the biggest shift for me.

**Lenny Rachitsky** (00:18:55):
What's a change you've made to help the company move faster and match that sort of pace?

**Howie Liu** (00:19:01):
Yeah. I mean, we did do a reorg of the EPD org. So before we had ... we've gone through a few different reorgs over the past, call it, four years. The original state as we just proliferated, I think by default or incrementally, was that we had a bunch of groups that were each responsible for a feature or a surface area. So there was a group responsible for search within our table, and there was a group responsible for mobile experience and so on and so forth. And that has its benefits. Obviously, that team can go and get really ramped up on that part of the code base, that part of the product.

**Howie Liu** (00:19:36):
But it has the disadvantage of yeah, you tend to think incrementally when everyone's remit is actually a feature that they incrementally improve by definition as opposed to thinking about a mission or a outcome goal that might need to coordinate dramatic changes across a wider set of surface areas instead of just each one incrementally improving. And so, we reorged initially to basically different business units effectively. So I know Airbnb has done the functional to GM back, et cetera. This was more like saying, "Look, we have an enterprise business" and them MO there is more about scalability.

**Howie Liu** (00:20:19):
Can we support the larger scale data sets and use cases? Do you have the core capabilities needed to be able to push out an app to maybe 10,000 seats or 20,000 seats for product operations? A lot of architecture, a lot of scale, that kind of work. We would have, what we call the teams filler, which is more about self-serve, kind of the product UX, how easy it is to adopt the product on board, share, do all the kind of basic functionality. An AI pillar, solutions pillar, and basically infra. And what we found though with that approach is that there was still ... there was more kind of holistic bets being made.

**Howie Liu** (00:20:58):
So the team's pillar could think not just about one feature, but the overall onboarding experience where really about Nuxt in a way that touched multiple parts of the product, but it still felt like it wasn't ... Especially as we started to execute more on AI stuff, it wasn't allowing us to aggressively and quickly move as a AI native company would. I mean, when you look at the cursors of the world, they're shipping major new stuff every week. And it's not like, "Oh, well we have this separate roadmap for enterprise, we have this roadmap for this group."

**Howie Liu** (00:21:33):
And it just feels like one cohesive product that's shipping at a breakneck pace. So we did this recent reorg where now we have what I call the fast thinking group, which officially is called AI platform, but it really means we want to just ship a bunch of new capabilities on a near weekly basis. And each of them should be truly awesome value. You should drop your jaw, how awesome it is to use this new capability in Airtable. And then separately, we have the slow thinking group, and that's not meant to be better or worse. It's literally like you need fast and slow thinking in the common sense to operate as a human.

**Lenny Rachitsky** (00:22:12):
I have that book behind me.

**Howie Liu** (00:22:14):
Yeah, I love that book. But slow thinking it's like, it's just a different mode of planning and executing, right? It's like more deliberate that require more premeditation. We can't just ship a new piece of infrastructure that has a lot of data complexity like our data store HyperDB that now can handle multi-hundred million record data sets. That's not something you ship in a week in a hacky prototype. So we now have these two separate parts of the company, and I actually think what's really cool is they actually compliment each other very well, right?

**Howie Liu** (00:22:46):
Because the fast execution, the AI stuff, that creates the top of funnel excitement that also inspires new use cases and new users to come to Airtable, including in large enterprise, right? Enterprises can use this stuff too. It's not just like a SMB thing, but the slow thinking basically allows those initial seeds of adoption to Sprout and grow into much larger deployments. Whereas I think a lot of the challenge for many of the AI native companies I've seen is that they could have a very wide top of funnel, like get all of this AI, tourist traffic.

**Howie Liu** (00:23:19):
A lot of interest, a lot of early usage, but then sometimes the challenge is how do you turn that into more durable growth and get each of those adoption seeds to retain and expand over time.

**Lenny Rachitsky** (00:23:33):
That is super cool. I've never heard of this way of structuring teams, the fast thinking, thinking fast, thinking slow, the Kahneman. It's so interesting for the fast thinking team, do you find there's specific archetypes of people that are successful there? Is it a lot of bringing in new people that are not just used to the way of working at our table? What do you find?

**Howie Liu** (00:23:52):
We have a mix. So we brought in ... I mean, we're always hiring, right? There was never a point in the company's life where we stopped hiring. And candidly, even when we had to do two rifts, that's significantly reduced our head count. We had just way too quickly grown and overscaled the business at a certain point. But even when we did our rifts, we were still actively recruiting and hiring in ... I mean every major department, but especially in EPD, because it's always been my belief that it would be arrogant to say that we have all the people we ever need already in the roster today, right?

**Howie Liu** (00:24:29):
We're always going to need to find new, fresh perspectives, new skillsets, et cetera. And so, we've continued to hire ... I think we've learned as we've gone along of what is the ideal type of hire, and we've done some actual hires and learned from that as well. But I think the fast thinking part, it really just requires a lot of ... Somebody who's able to operate with a lot of autonomy, who's entrepreneurial in nature. Now, it doesn't mean they have to literally be a former founder. I know some companies are, like Rippling for instance, does a lot of actual acquisitions and gets actual founders into the company.

**Howie Liu** (00:25:04):
We found that that's great and we've done some of that as well. But also there are some really, really capable people who we didn't literally have to acquire in, and yet, they're just able to think full stack about the problem and the user experience. Problem, not just meaning the technical layers of the problem, but also, what is the wow factor we're trying to create. So tangibly we're doing this new thing that's about to ship, where not only can you describe the app you want to build and then iterate on it with our conversational agent Omni.

**Howie Liu** (00:25:41):
And it builds it with the existing air table platform capabilities, but we're also giving it the ability to actually do code gen, to extend those apps with really final mile very bespoke functionality or visuals. So you could say, "Hey, generate me a very, very specific type of map view with this kind of heat mapping and this kind of icons and ..."

**Howie Liu** (00:26:00):
It's kind of like heat mapping and this kind of icons. And when you click it, do this. And that's a capability that there's so much ambiguity in some of the design decisions around it. And you have to blend that design thinking with some of the technical constraints of what can the AI models actually one shot effectively?

**Howie Liu** (00:26:21):
And if not, how do you add in the right human workflow for approval and review, and the reprompting and so on? So just so many different design decisions, and you need somebody who can really think full-stack about that kind of product and is not overwhelmed by that kind of open-endedness, but relishes in it.

**Lenny Rachitsky** (00:26:38):
I was actually playing with it before we started chatting. I made a really cute startup CRM.

**Howie Liu** (00:26:43):
Oh, that's awesome.

**Lenny Rachitsky** (00:26:43):
Yeah, started talking Omni over here. It's like the colors are beautiful-

**Howie Liu** (00:26:47):
[inaudible 00:26:47].

**Lenny Rachitsky** (00:26:47):
... so that's what's standing out to me right now.

**Howie Liu** (00:26:49):
[inaudible 00:26:49] there is...

**Lenny Rachitsky** (00:26:50):
Yeah.

**Howie Liu** (00:26:50):
I will say just as a note, I consider myself at my core a product UX person. That's my passion. And everything else I've had to learn to run this company is almost like what was a necessary part of the journey. But my real passion is thinking about product UX. And I think of UX in a deeper sense than just the cosmetic design. What you could put into a framer kind of prototype. I think of it as literally what should this product do and how should it represent that and behave for the user? That is the product, in my opinion, right. And of course, then you have to figure out technically what's possible and how to implement it.

**Howie Liu** (00:27:36):
But I think to me what's under executed today in the world of AI products is there's so many awesome capabilities of AI, and most of them are really under merchandise, and there's very poor, actually, visual or otherwise metaphors or affordances given to users to help represent or understand what those underlying capabilities are. I mean, ChatGPT obviously extremely successful product, so not knocking it at all, but you come in and you just get this completely blank chat box by default, and now they have suggestions underneath and so on.

**Howie Liu** (00:28:13):
But the product UX part of me is just craving more visual metaphors or colors or some kind of use the canvas of a web interface and all the richness interaction you create there to better represent or show all the different things that you can do with the underlying model, right. And so that's something we've tried to do with Airtable, is show all of the different states and use colors even to play those up.

**Lenny Rachitsky** (00:28:44):
It's interesting how much of this connects with I just had Nick Turley on the podcast. He's head of ChatGPT at OpenAI, and he had these two really interesting insights that resonate directly with what you're describing. One is he has this concept of whenever something is being worked on, he's always asking, " Is this maximally accelerated? How do we move faster? If this is important, what would allow us to move faster?"

**Howie Liu** (00:29:06):
Yeah.

**Lenny Rachitsky** (00:29:06):
And I love that that's one of the themes that's coming up as you talk, is just this creating this very clear sense of speed. And you even call it the fast-thinking team, like, "You are going to move fast." And then the other one is just this insight that with AI, you often don't know what it can do and what people want to do with it until it's out. So there's this need to get it out, and that'll tell you what it should be.

**Howie Liu** (00:29:29):
I couldn't agree more with both of those, and particularly on the second point, I think it's interesting. Clearly, there have been companies that have both been successful in PLG and more sales-led distribution for AI products. The most notable ones I can think of are Palantir with their AIP deployments. That's obviously very sales-led. You're not PLG into a Palantir deployment. But even companies like Harvey and so on, they're doing very well. And it's primarily, from what I understand, sales-led.

**Howie Liu** (00:29:59):
You're not self- serving into a Harvey instance at a law firm. And yet, to me, the best way to get AI value out there is experientially, right. And so you can kind of get that in a sales motion. You can show a demo. Maybe you can do a POC, but it's so much more powerful when you just open up the doors and say, "Anyone who wants to come and sign up and trial this product can." And I think to me, it's a real proof point that ChatGPT is arguably the most successful kind of PLG product of all time, just in terms of sheer scale of users. Like they announced 700 million... Is it MAUs or week... I think it's actually-

**Lenny Rachitsky** (00:30:41):
Weekly active users.

**Howie Liu** (00:30:41):
Weekly.

**Lenny Rachitsky** (00:30:42):
10% of humans on earth use it-

**Howie Liu** (00:30:43):
That's insane.

**Lenny Rachitsky** (00:30:44):
... weekly.

**Howie Liu** (00:30:45):
That's insane. In how many years? A few years.

**Lenny Rachitsky** (00:30:48):
Three years. Under three years.

**Howie Liu** (00:30:49):
Yeah. So I mean, literally, that is just the most insane ramp curve. And I don't think they would've gotten there if you couldn't just come in and literally try the product out. And as a little bit of a rebuttal of the point I made earlier where I think ChatGPT doesn't do a ton right now, and even earlier they did even less to expose all the different ways you could use it, but they just made it so frictionless to just try it for yourself that you as a user could come in and just literally ask it anything and see how it did. And of course, people in the early days tried to stump it and showed, "Oh look, see, it's not that smart. It doesn't answer this hard question really well."

**Howie Liu** (00:31:26):
But clearly, the magical nature of it still appealed to you enough. Everybody used it. And so I think I do have a view. We've gone through that whole arc of we started PLG. I'd like to think Airtable was one of the PLG darlings of our era. And anyway, I started moving up market and doing more sales execution, although that was still always on top of usually PLG within an enterprise, but we started doing more and more sales execution. We still have that. That's still really important for our business. But I also think, me personally, one of my goals is to shift my attention back into that kind of builder-led adoption and literally showing in the product experientially, not telling in a deck, the value that you can get from AI and Airtable.

I think that's so key, and it's [inaudible 00:32:23], but it's also more than that. It's not just literally how do you onboard somebody into the product. It's literally thinking about the entire product experience itself, right. And in our case, we just made the entire product experience AI-centric. It used to be that we had kind of this secondary thing that you could ask questions to the assistant sidebar. We now made our agent the default way of doing everything in Airtable, and it's now the Airtable app, as you know, it is almost like an artifact that's manipulated by and can be tool used by the agent.

**Lenny Rachitsky** (00:32:58):
Let me follow that thread. So if you go to Airtable.com today, it looks like basically all the other AI app building sites. Now it's just tell me what you want to build. Thoughts on that, as just a thing everyone's starting to do is there... what do you think comes next? Is this... Is it working well?

**Howie Liu** (00:33:15):
There's clearly an incredible magic to vibe coding and app building with AI. And this is actually a prime illustration in my view of that concept we talked about a second ago, which is as capabilities of these underlying models evolve, the form factor in the product UX also needs to evolve with it. And so the earliest models, like the kind of original ChatGPT, like GPT-3.5 kind of era models were not nearly as smart as the current models. And so you couldn't really ask it to one shot a more complicated chunk of code, or certainly not like a full stack app, and expect it to work.

**Howie Liu** (00:33:56):
And so the right form factor for leveraging those models in a software creation context was GitHub Copilot, right. It's like auto-complete a few lines of code at a time. But you couldn't chat to it and tell it, "Build me this entire app from scratch." And I think that as the models got better and better, you saw that the new form factors emerge. I think Cursor did a great job of being an early pioneer of this more age agentic way of leveraging the models to do more complex things and generate more larger chunks of code.

**Howie Liu** (00:34:27):
And now with Composer, you can literally just go into Cursor and build an app from scratch, build me a 3D shooter game from scratch, and just watch it go and create all the files and fill out each file, and then the thing actually runs some of the time. And so to me, this is where the world is going. The models are clearly getting smarter. And if you think about the original vision of Airtable, it was always about democratizing software creation. We just strongly believed that the number of people who use apps far outweighs the number of people who can actually build their own or manipulate apps and harness custom software to their advantage.

**Lenny Rachitsky** (00:35:08):
That sounds very familiar, very familiar these days.

**Howie Liu** (00:35:10):
Yeah, exactly. And so I think this is, it's a different means to the same end. And so it's almost like we have to lean into this because if we started Airtable today, this is what we would be all in on. Now I think that the advantage that we have, and I do think you have to be realistic to yourself, especially as a company that predates GenAI and now has to find your new footing in the AI landscape. You can't fool yourself and just say like, "Okay, I'm going to throw in some AI stuff on the landing... on the marketing site, put in a couple AI features, and call it a day."

**Howie Liu** (00:35:43):
I think you actually have to take a clean slate approach to saying, "How would our mission best be expressed? If you were literally founding a new company from scratch with the same mission, how would you execute on that mission using a fully AI native approach?" And then, by the way, do you have useful building blocks that you can leverage from your existing product and your existing business, or are you literally worse off having this legacy asset versus starting something from scratch? And I don't think the answer is always yes or no. I think it just depends on the product.

**Howie Liu** (00:36:19):
And if you can't really introspect and say, "Look, I think I'm better off doing this with the pieces that I have for my existing business and product," then I think you should sell. You should find a buyer for that company and then go. And if you really care about this mission, go and start the next carnation of it. In my case, I really thought about this and really feel strongly that the building blocks that we have, these no code components, actually do allow us to execute better on this vision than if I had to start from scratch.

**Howie Liu** (00:36:50):
Meaning the problem with vibe coding, especially if we're building business apps... So I should clarify that we want to democratize software creation, but specifically, we are focused on business apps. We're not trying to be the platform where you create a cool viral consumer game. This is for like your CRM, right. Or if you want to build an inventory management system as a small restaurant or a lawyer trying to build a case management system, that's what we've always been focused on. And I think in this AI-native world, clearly, you should be able to generate those apps agentically.

**Howie Liu** (00:37:24):
And yet if you have an agent that has to generate every single bit of that app from scratch, from code, it's going to be very unreliable. There's going to be bugs. There's going to be data and security issues. And then you're also going to have a context collapse, as it just cannot manage all of the code that it's written, basically, as the app gets more and more complex. And what we actually have are basically these primitives that the agent can manipulate and use without having to literally write the code from scratch to represent, "Here's a beautiful crud interface on top of the data layer.

**Howie Liu** (00:37:57):
Ours is real-time and collaborative, and really rich, and has collaboration on it. And by the way, here's all these other view types and a layout engine for a custom interface, a layout, or automations and business logic." And so it's almost like in programming terms, the Airtable pieces in our Lego kit today can be used by this agent as almost like a more expressive DSL, like a domain-specific language to build business apps instead of literally having to write everything down to the SQL and HTML and JavaScript to build every part of that app from scratch.

**Howie Liu** (00:38:31):
And so if we can combine the best of both worlds, we have these very reliable, high-quality Lego pieces. Now, an agent can go and assemble them for you instead of you just using the GUI to do that. And by the way, if you do want to fall back to the GUI, there's a really great kind of way for the non-technical user to still understand and participate in what's going on. Whereas if you're not technical, you can't inspect the code underneath a v0 or Lovable or Revolut app, right.

**Howie Liu** (00:38:58):
It's just kind of opaque to you. And if you can't re- prop it to get what you want, you're kind of stuck. This is much more akin to a developer using Cursor can generate lots of code, but then can still drop back to the IDE to edit and manipulate it to the final production-ready state. So that's kind of the play that we're making. And if I didn't fully and truly believe we have a better shot at doing it with our existing product, I wouldn't be running this company in its form today.

**Lenny Rachitsky** (00:39:25):
I'm talking to a lot of founders that are going through the journey are going on, which is, "We've had a business for a decade, AI emerged, and wow, we got to figure out something that works... that could work even better." And so I'm trying to pull out the threads that are consistently working across these journeys because I think a lot of companies are trying to figure this out. So one that you just touched on is just if you were to start today, what will you do?

**Lenny Rachitsky** (00:39:48):
What would that business be? Plus, how can... do we have an unfair advantage with the thing we've done in the past? That feels like an important ingredient. And then the other... circling back to stuff you've shared already, there's just creating a sense of urgency and pace and getting people to understand this is how things move in AI, and we need to create this fast-thinking team. I love that metaphor in framing.

**Lenny Rachitsky** (00:40:11):
And then there's the point you made about just talking to AI regularly as the founder feels like an important element, just like to truly be this ICCO talking to AI, working with AI regularly. Just on that note a little bit more, just to give people a sense of what this looks like day to day. So you're talking to Omni all day trying to and undertook... flex the power of what you can do and iterate on it. Is there anything else you're doing day to day that helps you figure out what to do for the business?

**Howie Liu** (00:40:38):
One, I try to use as many different AI products, including not Airtable, as I can, and both literally for the novelty factor and just some new cool demo comes out. Like Runway release their immersive world engine, and so I'm going to go try it out. When Sesame AI put out their cool interactive voice chat demo, I tried that out because even though we don't have a direct and near-term need for really realistic and interruptible voice mode where it's not as core to our capabilities, I just want to understand and get a feel for everything that's out there.

**Howie Liu** (00:41:23):
And I try to invent little, almost like side projects of my own, to have a real reason to use these products. Like, "Oh, cool. What if I were to take... What if I were to try to create a funny little short... a funny video short using a combination of HeyGen avatars with a script, like a comical script generated by AI? And maybe it'll be on an interesting topic. So I'll do deep research on the topic with ChatGPT and pull together the results, have it compose kind of a little dialogue.

**Lenny Rachitsky** (00:41:58):
Did you actually do this? Is there something you made?

**Howie Liu** (00:42:00):
Yeah. That's literally an example of something, just a fun weekend project. And to be honest, these things only take you an hour if you become kind of pretty proficient with using the products. They're all so easy to use. You can literally do the deep research thing, kick off query, make a coffee, come back in 20 minutes. Okay, let me prompt it to generate me some dialogue. It's a little bit like what NotebookLM does for you out of the box, but sometimes I like to just do it myself. And then, okay, let me take the script and cut it up and turn it into a HeyGen avatar and then download the video and play it.

**Howie Liu** (00:42:32):
And just for fun. I'm not trying to make that into an actual YouTube video business. But I think coming up with these different fun weekend projects is a really useful construct to force myself to actually try these products in a more than just a Twitch click way. And what it gives me is, A, it's not just understanding the models, which is also very, very important, right. GPT-5 came out yesterday, and playing around with it a bunch just on a variety of different personal use cases, but there's a difference between just understanding the model but then also understanding the product form factors in which they can be placed, right.

**Howie Liu** (00:43:15):
Meaning when you apply the model in a more structured way, when you apply the model with different tool calling than maybe what ChatGPT has in its out-of-the-box form, when you apply it with a more agentic workflow, again, that might be different from what ChatGPT gives you out-of-the-box, that's when you kind of learn you really get to inspire yourself on what are the product's form factors that these new models can take. And plus, by the way, I find it to be really fun. There is to me a delight and entertainment value to just using AI, period, because A, it's not perfectly predictable.

**Howie Liu** (00:43:57):
So I think the element of you're not quite sure what you're going to get. It's like a box of chocolates. And B, it always blows my mind just to think about, "Wow, five years ago we didn't have any of this stuff." AI was like, okay, it's like we can do predictive analytics. There's some basically very advanced kind of regressions that we could run with AI, but it looked nothing like this in its current form, and it's just actually super fun, in my opinion, to get to play around with all the different types of products that come out.

**Howie Liu** (00:44:33):
I think that is a big part of it because on the point about the pace of the world moving so much faster in AI than any other landscape in SaaS, in the mature SaaS era, it was important to study your competition. If you were building a SaaS company, you'd be crazy not to follow Salesforce every year and see what the major releases they're putting out are, or ServiceNow, or so on.

**Howie Liu** (00:45:03):
This is the equivalent of that, but there's major new releases and products and so on every week, not every year. And so I just think you have to say abreast of all... of it all and combining this with our point earlier of a lot of this has to be experienced, not just read. You can't just read the write-up on TechCrunch or even a tweet about a new capability. You kind of have to try it to really get a sense of what it is.

**Lenny Rachitsky** (00:45:33):
**Howie Liu** (00:47:02):
One is really, really, really stressing this idea of go play with this stuff. And I mean, when I say play, I really mean play in the psychological sense of there's a difference when you go in and you're kind of just trying to check the box and get a job done. There's a difference when you come in with a curiosity and you're kind of exploring, right. And it's both more fun and energizing, but also, I think you learn more through that. And so I've really tried to stress the value of play with these AI products.

**Howie Liu** (00:47:36):
And I kind of try to lead by example, by literally going and sharing out links or screenshots of the things that I'm doing in these various products. So, as an example, I will go into one of the prototyping tools and show, "Hey, I built a marketing landing page for this new capability we're launching." I created a landing page for it in Replit, let's say, and now I'm sharing that link. Instead of what typically we would've done in the past is like, okay, we're going to write a doc about it and then share the doc, I'm just going to show you an actual landing page with visuals and everything in there.

**Howie Liu** (00:48:20):
Or I'll share the actual link to my deep research reports. Or instead of me writing a perfect memo on a topic, I'll actually just prompt my way into getting a chat thread or a chat output that basically covers all the content that I care about and maybe even ask it to, "Okay, summarize this all into a final memo output," and then intentionally share that rather than expose the fact that I'm using AI in this way and here's literally how I'm prompting it so you can follow along as well.

**Howie Liu** (00:48:49):
But really trying to encourage everyone to go and just play with these products. And I've even said, "Look, if anyone wants to just literally block out a day or frankly even a week and have the ultimate excuse, you could use... you could say that I told you to do it, right. If you want to cancel all your meetings for a day or for an entire week and just go play around with every product, AI product that you can find that you think could be relevant to Airtable, go do it. Period. So I think that's the most important thing is this play, this experimentation.

**Howie Liu** (00:49:25):
I think there's also a lot of other kind of shifts in how we execute prototypes over decks. I want to see actual interactive demos because, again, it's hard to... In a deck or in a PRD, you could say, "Okay. Well, we're going to make Omni really good at handling this kind of app building." Okay, those are just words. The real proof is in the pudding of like, "Okay, let me try it out on a few realistic prompts that I can imagine."

**Howie Liu** (00:49:49):
And in a demo, in a real prototype, you can instantly try it out on unrealistic rather than golden pathy scenarios and see how it feels too. Does it feel too slow? Do we need to expose more of the reasoning or steps that are happening behind the scenes? Create a progress bar or something like that. But it's really hard to get that feel of the product with anything but a functional prototype that really does, in an open-end way, use the AI to do whatever you put in.

**Howie Liu** (00:50:24):
So I think it's more like a experimentation playground it feels like how we need to execute, versus I think, in the past, it sometimes felt like a more deterministic resourcing and kind of timelines view of execution. We're going to put this many people on this problem, and this is the eight-week timeline to this milestone, and then we're going to ship in a quarter from now. And I think now the whole thing is just a lot more experimentation and iteration-driven.

**Lenny Rachitsky** (00:50:56):
Of the different functions on a product team, PM, engineering design, who has had the most success being more productive with these tools, and how do you think this will impact each of these three functions over time?

**Howie Liu** (00:51:07):
What I found is that it really does become more about individual attitude and maybe some polymathism. There's a strong advantage to any of those three roles who can kind of cross over into the other two, like the hybrid unicorn types.

So if you're a designer who can be just technical enough to kind of be dangerous and understand a little bit of how these models work and how does tool calling work and all of this stuff, then you can actually design a concept or even prototype a concept, including in these prototyping tools that's much more interesting and maybe realistic than if you're just stuck in the flat like let me put something in a static design concept because I think designs have to be more interactive. The whole... The value of the product and the [inaudible 00:52:04]-

**Howie Liu** (00:52:01):
... the value of the product and the product functionality is in the interaction of it, right? Think about the design of Chachi Petite. Again, it's the most basic design you could possibly imagine. The real design actually is happening underneath the hood in how it responds to different queries and what happens after you fire off a prompt, right? So I think I found that there are people within each of these functions, there are engineers who are very good at thinking about product and experience and can go and prototype out the whole thing. They're designers who can do the same. Even if they can't literally code, they can prototype something out literally using a prototyping tool.

**Howie Liu** (00:52:42):
I think that's where AI tooling is also giving more advantage to people who can think in this way by equipping them with an alternative to actually having to go through the long hoops of learning CS, right? Then PMs as well. I think there are some PMs who are really getting into the technical details and studying up on how does this stuff work and actually getting hands-on, rather than seeing the role as writing documents, writing PRDs.

**Lenny Rachitsky** (00:53:08):
Do you see one of these roles, I don't know, being more in trouble than others, just like you need fewer of these people in the future potentially?

**Howie Liu** (00:53:16):
I think overall you can get more done with fewer people, and that's not to say we want to go and make the team smaller, but rather ... the really cool thing for us and I think a lot of other companies is it's not like you have a finite set of things you need to do and execute on from a product standpoint, and okay, now I can do that with a 10th of people. I mean, you could do that in a lot of cases, but for us, maybe it's also because we're a very meta product, right? We are the app platform with which you can build now any AI app with AI, right? The apps themselves leverage AI capabilities at runtime, whether it's to generate imagery for a creative production workflow or leveraging deep research, or AI-based crawling of the web to search for companies that match a certain criteria for your Dealflow app or something like that.

**Howie Liu** (00:54:10):
We can effectively leverage all of these other AI capabilities in this app platform, because by definition we're enabling our customers to build apps that have this wide range of AI capabilities. But because of that, it's like we have a almost infinite set of possible AI capabilities that we could execute on, right? I'm always telling the team like, "Look, the great news is it's like we have all these fruit trees and there's so many crazy low-hanging fruit, and you've got literally massive watermelons literally sitting on the ground and all you have to do is walk over 20 feet and pick it up instead of having to climb the really tall coconut tree to grab a hard coconut from 50 feet up. So there's so many watermelons on the ground, just go out and start finding the biggest ones and attacking those, right?"

**Howie Liu** (00:55:03):
What that means is that if we can build this culture, and I do think it's a learnable way of operating, I really like to believe in the growth potential of any human and any individual. I think if you really have a growth mindset, and that's why one of our most important core values is growth mindset, right? If you really have that growth mindset, I think especially if you're willing to put in the nights and weekends hours, or in my case I'm literally telling people like, take a full day off, take a full week off and learn this stuff, you can become more fluent in this way. I think then what we get is a team that can just go and work on more things in a much more leveraged and fast way, right?

**Howie Liu** (00:55:48):
So, I like to think people who are willing to jump on the train are just going to become more and more effective. It's not like, oh, as a PM my role is becoming entirely irrelevant, right? No, it means that as a PM you need to start looking more like a hybrid PM prototyper who has some good design sensibilities. By the way, I think some of the best eng PM and design cultures respectively over the past even few decades have always been multidisciplinary in nature, right? The original PM spec at Google required the PMs to actually be somewhat technical so they could understand the engineering limitations of the product designs they wanted to make, and they had to be kind of designy, right?

**Howie Liu** (00:56:33):
I remember my co-founder, Andrew, when he was in the APM program was always reading books about design, even down to visual design and color theory and that kind of thing, right? So I think it's just a reminder that designers as well, some of the best designers through designer to Apple, including hardware designer, you have to understand some of the technical capabilities of how this stuff works, right? If you're an engineer, I think some of the best engineers and maybe Stripe always had a very good engineering culture of engineers who could think about the product and business requirements. In fact, on any given product group, at Stripe my understanding is that the DRI isn't always the PM as is traditionally the case in that triangle. Sometimes it's actually the engineer who's taking the product lead and saying, this is what we need to build.

**Lenny Rachitsky** (00:57:24):
So, what I'm hearing is essentially the trend across product engineering design is each of those functions needs to get good at one of the other functions at least.

**Howie Liu** (00:57:35):
Yeah.

**Lenny Rachitsky** (00:57:35):
Ideally you can do them all, but if you can just do one additional, so a PM becomes better at design, an engineer becomes better at product management.

**Howie Liu** (00:57:43):
Well, I would actually go further and say I think you need to get decently good at all three. There's just a minimum baseline of if you're any one of those roles, you need to be minimally good at the other two, and then you can go deeper into your own specialty, right? You could be a designer who's really good at thinking about UX and interaction design, and then just good enough to be dangerous on thinking about what's technically possible and what is the product story around this feature.

**Lenny Rachitsky** (00:58:17):
I love that. To do that, one piece of advice that comes up again and again in what you've been describing is use the tools constantly to see what's possible, and that will teach you a lot of these things.

**Howie Liu** (00:58:28):
I think, well, use the tools gives you exposure to what's possible, right? It's kind of like if you wanted to be a great industrial designer, and let's say, I mean, the chair is the ultimate hello world of industrial design, it's the canonical design object, you wouldn't just sit there in a vacuum with no familiarity with the materials that you can use, plywood, steel, whatever, or existing form factors of chairs trying to invent the world's best chair in a vacuum, right? You should go and first do a study of all of the best chairs out there today. Go look at an Eames chair, sit in it and try to examine it to reverse engineer how it was made, and just look at the prior art for that type of product. That's how I see the go out and play with these products, and also, I think actually going and designing or implementing or executing is the best practice.

**Howie Liu** (00:59:21):
So you can't just only go and look at other people's chairs, eventually you have to go and actually try building your own and then try building another one and another one and another one. So, I think that's where ... when I think about how I hone my own product UX sensibilities, I never ... and at that time that I was in school and then learning about this stuff, there wasn't really any good curriculum for UX, right? It's not like there were great college classes to learn product UX. I mean, even CS was very academic in nature at that time, it wasn't applied software engineering, like build an app or whatever. Maybe now at some of the schools like Stanford, MIT, they have actually UXy type courses, but it's still a rarity for most people to have access to that.

**Howie Liu** (01:00:03):
So, the way I learned all of my product sensibilities was just trial and error and also using and studying other products, and then going and trying to build my own weekend project ideas, right? Oh, I want to build a Yelp style app with a map view and then also a list view, and I want it so that when you pan around in the map for it to automatically update the list view. Maybe there's some UX improvements I can make on top of that, but I can also test my technical skills to figure out which parts of this are hard to implement and how do you make it work, and what are some of the design changes or affordances that you can use to map to the technical possibilities.

**Lenny Rachitsky** (01:00:43):
To do that, I loved your piece of advice, which I forgot to double down on, which I also find really powerful. The best tip there is find something to actually build that is useful to you and fun. Pick a project that's like, okay, this would be fun to do. Have a problem you're solving that forces you to actually do this thing.

**Howie Liu** (01:01:00):
For sure. Look, I think that can be night and weekend projects, it can also be the daytime job projects, right? I mean, I am basically telling our teams on the AI platform group especially, "Look, in that low hanging fruit metaphor, it's like I'm not being prescriptive with you on which watermelons you should pick, but you should go ..." We do have different pods within that group, but one of them for instance is what we call the field agents team and they're responsible for the agents that work within your app. So this is not the agent that builds your app, but these agents that run on a customer's behalf to do web research on your customers, or they can go and analyze a document and in the future maybe do things like actually generate a prototype of a feature from a PRD or from a feature idea.

**Howie Liu** (01:01:52):
I'm telling them, "Look, there's a almost infinite number of superpowers you can give these field agents. I'm not going to tell you which specifically to do. Now you can ask me to weigh in for sure, but you should go and just experiment and prototype a few different directions we could go." What if you prototype what it would look like to have a deep research implementation in field agents, so that for any given row of data, let's say in your case it's podcast guests, you can just click a button or click a button en masse across every speaker you have lined up to do deep research powered by ChatGPT's own deep research on each of the speakers and have them all laid out side by side in this table, right? Go prototype that and see how it feels and looks like. So I think some of this stuff can also be in your daytime job, especially if that daytime job is literally to go and build AI functionality.

**Lenny Rachitsky** (01:02:46):
I actually tried to do exactly that. The problem I ran into, I wonder if it's changed, is there's no API for ChatGPT deep research yet as far as I know.

**Howie Liu** (01:02:55):
There is now, there is now.

**Lenny Rachitsky** (01:02:56):
There is, there we go.

**Howie Liu** (01:02:58):
Sometimes it ends up being ... and I think they only recently exposed it. It ends up being something on the order of a dollar plus per research call, which-

**Lenny Rachitsky** (01:03:05):
What a deal.

**Howie Liu** (01:03:05):
... I mean, again, exactly. I mean, some people would say, oh my god, that's so expensive, and you rack up 50 of those, you've cost $50 a month. I think it's like, well, it just saved you hours of research by a human.

**Lenny Rachitsky** (01:03:16):
Not only that, I actually have a researcher that I pay to give me background on guests that was four or 500 bucks and the dollar sounds great. I've been doing this-

**Howie Liu** (01:03:28):
[inaudible 01:03:28]

**Lenny Rachitsky** (01:03:28):
... I've been doing this manually.

**Howie Liu** (01:03:29):
If he was being smart he would be using deep research and they just collected [inaudible 01:03:33]

**Lenny Rachitsky** (01:03:33):
They might be. They might just be. Oh, man. Okay, there's one more skill I wanted to talk about real quick. This comes up a lot in these conversations is evals.

**Howie Liu** (01:03:43):
Okay.

**Lenny Rachitsky** (01:03:43):
The power of getting good at evals, I know that's something you value highly. Talk about just why you think this is something people need to get good at.

**Howie Liu** (01:03:50):
Yeah, and I listened to your episodes with [inaudible 01:03:54] and Mike who talked about this. I think it's interesting that both heads of OpenAI and Anthropic have converged on this point. I mean, look, I think I would add a slightly different or additive take though, which is I think for a completely novel product experience or form factor, you should actually not start with evals and you should start with vibes, right? Meaning you need to go and just test in a much more open-ended way, like, does this even work in kind of a broad sense?

**Howie Liu** (01:04:28):
So as an example, for our custom code generation capability, instead of defining evals that get repeatably tested as you vary the prompt or the model or the agentic workflow used to generate these outputs, and you have to define what does good look like by definition for the eval, I would first start with a much more open-ended and ad hoc style of just throw stuff against the wall, try different prompts and see how well it does.

**Howie Liu** (01:05:01):
To me, evals are more useful, A, once you've converged on the basic scaffold of the form factor and you kind of know what are the use cases you want it to work well for and what you want to test against it. Whereas in the early days, especially if your product market fit finding either for an entirely new company or for a pretty dramatically new or bold new capability that doesn't really have ... it's not an incremental improvement on something that exists in Airtable today, I think you have to just be a little bit more creative initially and throwing stuff at it, seeing what works to understand, okay, let's use an example, we're implementing this new capability that can use basically a long-running AI crawler agent that goes and researches the web for a specific type of object or entity, right?

**Howie Liu** (01:05:55):
So it's similar to deep research, but what it actually does is instead of outputting a report, it's actually going and compiling a list of things. The things could be companies or people or anything else, right? Find me every Marvel movie ever made, find me every DC Comics spin-off series, literally anything. You have to go in and first just try out a bunch of random ... use your own brain to think of what's the range of use cases I can test this against, right? Then you get back some results and you're like, okay, well, it's clear that where it does really well are these types of searches, people and companies with this kind of parameter.

**Howie Liu** (01:06:42):
I think to me, evals are useful once you have a sense of what is that cluster of useful use cases, you can start then more programmatically measuring the changes that you're making to improve the output for that, right? But by that point, you've probably already scoped the product and maybe the way we would merchandise it in Airtable is not a completely open-ended capability, but hey, here's a specific capability that can research one of these X number of entity types including people and companies, and here's even the filter conditions or criteria that are more explicit that you can define to give it the prompting to search for that thing, right?

**Howie Liu** (01:07:25):
But I kind of think it's more useful as a way to iterate your way to improvement, and you can start really testing stuff empirically, right? You can A/B test, especially if you have the scale of a really large product like Anthropic or OpenAI, you can just test everything and see like, oh, this model actually performs better than this one, this prompt performs better than this one, but I think early on you don't have that luxury and you're in a much more open-ended discovery process.

**Lenny Rachitsky** (01:07:51):
That is very wise, evals could constrain you too early. I think about just the Double Diamond, I don't know, IDO framework of be divergent first, and then converge and then maybe-

**Howie Liu** (01:08:01):
Yeah. Yeah, exactly. I hadn't heard that before, but that completely resonates.

**Lenny Rachitsky** (01:08:06):
Okay, let me try to reflect back some of the advice I've been hearing about how to shift a company to be successful in this new world, and let me see if I'm missing anything that you think is really important. So, one is there's this sense of just reset the expectations on pace and urgency and help people understand in AI things move incredibly fast, this is how we need to operate. Then there's also a piece of get stuff out so that you can learn how people use it and what it's capable of versus polishing it endlessly. Forcing people almost ... I don't know if forcing's the right word, but encouraging people to play with the latest stuff and giving them a chance to take days off or block out calendars, cancel meetings, just stay on top of this stuff to play as you talked about it. Then sharing things they've learned, get the vibes of what's possible.

**Howie Liu** (01:08:54):
Yeah.

**Lenny Rachitsky** (01:08:55):
There's also this idea of just rethink, okay, if we were just start today in this world, what would we do to achieve the same mission we are trying to achieve? Ideally it leverages this unfair advantage we have with things we've been working on for a long time. Then there's just talk to AI constantly every hour as you described.

**Howie Liu** (01:09:16):
For sure. Yeah, multiple times an hour, if possible.

**Lenny Rachitsky** (01:09:16):
Multiple times an hour, it keeps going up. Is there anything else that I missed there that you're like, you need to do this too to have a chance?

**Howie Liu** (01:09:25):
I think just to really, really try to break down role silos, and I think that's true certainly for EPND in the typical EPD triangle, but I also think it's probably true even for non-product roles, right? I think it's true in marketing, right? Something I'm really pushing for in marketing and I think our marketing team is really leaning into actually is if you can just do all of the thing yourself ... traditionally how a marketing team might operate is like, okay, you have one person who's responsible for executing the performance marketing part of a campaign. They literally go into the Google AdWords interface and they're tweaking the parameters of targeting and budget and conversion tracking, et cetera, and then somebody else is actually responsible for coming up with the specific ad copy, and somebody else yet was responsible for coming up with the seed content or positioning guide written by a PMM that feeds into the ad creative, and so on and so forth, right? Maybe they're promoting some new demo asset that somebody else yet created.

**Howie Liu** (01:10:35):
I just think that in the same way that you can collapse the roles in EPD, and the ideal person, maybe they're very specialized and deep in one dimension like engineering, but they're well-rounded enough to be dangerous on the other two, I think that's kind of true in almost every other function, right? Sales as well, I think you should start to be able to play more of an SE role. Traditionally salespeople didn't necessarily know the product that well and relied on the SE to come in and be the product experts. I think it's really hard to sell any kind of AI product now without actually being fluent in the product and be able to demo the product, so AEs need to be SE fluent as well.

**Howie Liu** (01:11:21):
So I just think that that concept of collapsing roles, everybody needs to become more full stack to do the ... being more outcome-oriented, right? Your outcome as an AE is to convince customers of the value of your product and close deals, right? Okay, well, in order to do that, you used to have dependencies on having assets created by marketing and an SE to help you demo. Can you collapse more of those dependencies so that if you had to, you could do it all yourself, right? I just think it's a new operating mentality overall for every AI native company or company that wants to compete in this new arena.

**Lenny Rachitsky** (01:12:06):
That is a great addition. It almost feels like you go back to startup times when everyone's doing a bunch of stuff. There's no here's the head of product, here's the head of engineering, we're just doing stuff-

**Howie Liu** (01:12:06):
Totally.

**Lenny Rachitsky** (01:12:16):
... that needs to be done.

**Howie Liu** (01:12:17):
Totally.

**Lenny Rachitsky** (01:12:18):
Yeah, I'm kind of seeing it as this upside down T where there's the thing you're really strong at and then as you described, the minimum of being good at engineering design or ... and SE, by the way, sales engineering imagine is what that stands for. Adjacent roles, you need to start having a baseline. The baseline is increasing of how much you need to understand that, everyone's Venn diagrams are kind of converging.

**Howie Liu** (01:12:40):
Exactly.

**Lenny Rachitsky** (01:12:42):
Amazing. Okay, let me take a step back and zoom out and think about the broader journey you've been on over the past decade plus. Let me just ask you this, what's the most counterintuitive lesson you've learned about Airtable building and company building teams that maybe goes against common startup wisdom?

**Howie Liu** (01:13:02):
I heard your interview with Brian Chesky and then later you talked about founder mode in that YC retreat, and the points there really, really resonated with me. I feel like maybe less eloquently I deduced some of the same principles just in my own experience, which is I think when you're scaling up, and this relates also to what we talked about before around the early days of building a company, you're in the details, you're finding product market fit, you kind of have to be pretty versatile, right? All these decisions from a technical standpoint to design, to even commercial, and what's the freemium model going to be like? And how are we going to market this product? What does the website look like? They're all very intertwined, right? You can't compartmentalize and then almost factory produce each of these things separately. They're all intertwined and you have a very small tight-knit team that's thinking full stack about all of this combined.

**Howie Liu** (01:14:04):
Obviously that's the only way, in my opinion, to create that magical product market fit in the first place. Then I think as you scale up, the default guidance that you often get from operational experts and larger scale company investors is like, okay, you got to industrialize the process of all of this stuff, right? It's kind of like going from a bespoke artisanal, one person made an entire item of clothing to we got to factory produce this thing, right?

**Howie Liu** (01:14:38):
What that means in an organizational context is you then create these different fiefdoms, you hire all these execs and each exec just manages their own swim lane, and there's relatively looser coupling between all of those different groups, right? So then you've got sales executing on its own thing, marketing's executing on its own thing, product's executing on its own thing. Even within product there's different product groups and surface areas that are each executing on their own thing.

**Howie Liu** (01:15:05):
Using the factory metaphor of there's an argument that that's actually kind of an efficient way to scale up production for each of these different swim lanes, right? Each one can operate in a more autonomous and purely scale up focus, wait, how do we produce more of this thing? If the thing happens to be within one product group improving search, that's our main focus. We're just going to go and ship, ship, ship more stuff to improve search. So it's not completely crazy why people give this advice, but I think what you lose is the magical integrative value of holistic thinking and making the bigger picture bets, right?

**Howie Liu** (01:15:48):
I think Brian talked a lot about this on his episode with you, which is like, look, in a company that is really serious about product, first of all, I really liked his point about the CEO has to play a CPO role, you have to care about the product. Ultimately the product is the thing and you can't just coast on scaling up go-to-market around the product forever, you got to keep innovating on the product. By the way, the best way to innovate on the product is not incrementally split over all these different little surface areas, but actually to have a bigger, more step function vision of how this product needs to make a leap, or what's the next big either act of the product or new capability of the product or reinvention of the product, right?

**Howie Liu** (01:16:35):
So I think if you really care about doing that from a product execution standpoint and almost refinding new product market fit on a regular basis, I think it necessitates a completely different operating and leadership model throughout the organization. All of the stuff we just talked about in terms of how to operate in the AI native era I think is actually exactly the same as how you need to operate in this constant product market refinding of fit state.

**Howie Liu** (01:17:02):
So I could not agree more with that concept of you got to think ambitiously and move the organization holistically towards these bigger outcomes, but also ship and learn and experiment a lot more in this era. Then maybe the meta learning I had from all of the above is that the specific advice obviously was like, okay, go scale up in this way or go hire these types of people, experienced operators, et cetera. Now, obviously there's some truth to that, right? The people giving this advice are not incompetent. They had some reason for giving it and in certain contexts that is the right thing to do, but I think my meta learning is it's not enough to just trust the recommendation, like, here's the action you should take from a lot of people, 'cause everybody has different priors and it's almost like we're all our own LLMs, and we all have different training from a different corpus of data informed by our own experiences. Maybe you're trained on the service-

**Howie Liu** (01:18:00):
... experiences, and maybe you're trained on like the kind of ServiceNow or the Oracle training corpus, and this person's trained on the Facebook corpus, and I'm trained on the Airtable one. I think what I've tried to do more and more is not to just ignore advice from smart people. Obviously, that's not the right answer, but to kind of take their... It's almost like in an LLM you can now with a reasoning model actually inspect the chain of thought and see how it's thinking. Why did it come up with this answer? To me, that chain of thought like "Why did you recommend this?", is actually more informative than the actual, "Just do this recommendation."

**Howie Liu** (01:18:44):
The answer might be like, "Hey, at So-and-So company, this is how we eliminated the PM role entirely." For Brian at Airbnb, it made sense. We're no longer having PMs in their traditional form. Now, we have program managers and product marketers, but more than the actual decision because I don't think it's a one-size-fits-all, everybody should do the same, why did you do that? The why actually was very informative, and then be able to take that and say like, "Okay, how would I apply that?" Maybe it yields a different outcome, but the reasoning actually is very informative.

**Lenny Rachitsky** (01:19:19):
It's interesting how this idea founder mode is not so different from this ICCO trend that you're following and it's-

**Howie Liu** (01:19:26):
For sure.

**Lenny Rachitsky** (01:19:26):
... yeah, yeah, it's like being in the weeds, being in the details, trying things yourself, not delegating to execs.

**Howie Liu** (01:19:32):
Yeah, and I think anything taken to an extreme can be problematic. There is a world where you are so in the details and in every detail that you're basically just micromanaging and you're kind of creating like a euphemism for that. That's not really what founder mode is about. That's not like the Brian conception of founder mode is to like micromanage everything and not trust anyone, but I think it's more about finding that right balance of being unabashed about caring about the details that do matter and where the tying together of details across different groups or departments actually is the only way to yield a non-incremental outcome. Otherwise, each person is just optimizing within their own domain, but you'll never get to the global maxima or the global breakthrough. 

**Howie Liu** (01:20:23):
I think the really cool thing about CEOs as I seize it, frankly any leader playing more of an IC-like role and being in the details is I think for the right type of person, it's actually more fun that way. I mean, to be honest, for me, the times where I felt most disintermediated from what I felt was the substance of this company was when I thought that I was almost like forcing myself to step away from the details. I thought that's what a at-scale CEO was supposed to do. I mean, there's some famous CEOs who have talked about, "The less decision I could make the better. The less details I'm exposed to the better. I just want to inspect at the topmost layer how this business is running, and if everything underneath it is going smoothly, then I'm able to do that and everything looks good."

**Howie Liu** (01:21:19):
I just think that's maybe, again, it works in a certain type of very mature type of business. Even then, though, I can't imagine that at a CPG company like a Procter & Gamble. You wouldn't want to have a CEO who still actually goes and tastes the soup and tries the products and sees literally the details of what the new product innovation pipeline looks like, as well as like how it's being experienced on the shelves and so on. I don't know. I guess I'm just more and more skeptical that that hands-off pure delegation and process management role ever works as a CEO. Maybe you go through a long enough period of where the business is coasting that nobody notices, but I got to say, for me it's just much more invigorating to get to play that role. I think for the types of operators and leaders that I most admire, that's what makes the job interesting. They don't want to have a automated away kind of role as a leader.

**Lenny Rachitsky** (01:22:22):
If you could go back in time and whisper something in a decade-ago Howie's ear that would have saved you a lot of pain and suffering over the last decade, what would that be?

**Howie Liu** (01:22:33):
Don't step away from the details that both you love. I mean, first of all, if your passion is building product and product design, even if it feels like at times the company needs to do all this other stuff like scale up, go to market, and operations and just have like a large people organization, that itself creates a lot of need to do things and manage. There becomes a new job invented just to manage a larger group of people, and obviously you're going to have to do some of that. You can't just completely eschew all your responsibility as an at-scale CEO, but don't lose the essence of the thing that you love doing and that really made this product happen and gives this company as many companies that were founded on a magical product market fit finding insight. Don't step too far away from that, and always make sure that is still your number one, even if other stuff has to also add to your plate.

**Lenny Rachitsky** (01:23:45):
I think people don't talk enough about this how if someone starts a company that's an idea they have they're excited about, it takes off and then you're stuck on that for a long time, and then even if things are pushed in a direction you're not as excited about. This point about just remembering what you actually love about it and coming back to that is so important because that's the only way to keep doing this for a long time.

**Howie Liu** (01:24:05):
I think that's so true, and to me that's why there's always been a difference between entrepreneurs who love the act of building a product or the business, too, versus those who saw a just purely business or financial opportunity that they felt like they couldn't pass up exploiting or going after. Look, no knock on people who are more the latter, and there's entire industries where it's all just about alpha generation. You can go into the private equity business and so on, and it's just purely it's rationally about how do I find the alpha? I think that some of the best companies, product central companies, at least in my opinion, are run by those people who actually just love the product. I think you get a feel for that from some of the AI companies like Sam, I think genuinely just loves working on AI.

**Howie Liu** (01:25:03):
If he could spend a hundred percent of his time on just being close to the AI and the research, I mean, he would and he's even said as much. Ranging to like Brian's with Airbnb, it's pretty clear that people like this are not motivated like... Airbnb was not founded because like, "Oh my God, we want to make a lot of money off this arbitrage opportunity against hotels."

**Lenny Rachitsky** (01:25:24):
They just needed to pay their rent.

**Howie Liu** (01:25:26):
Yeah. Well, that and I think they loved the product and I think they also loved the way in which they built the product, the design-centric nature of that product and company and culture. That's what gives you the continued joy of working on what could be the same company for a very long time.

**Lenny Rachitsky** (01:25:45):
Howie, is there anything else that you wanted to touch on or leave listeners with before we get to our very exciting lightning round?

**Howie Liu** (01:25:51):
I just want to reiterate, especially for listeners here who are in an EP or D role and especially in the P role, I really do believe that this is not like you either have or you don't like in terms of the skill set needed to be relevant and AI needed, but I do think it's a call to action to go and bolster your skill sets where they may be less refined right now. I think even programming, I really believe everyone could learn how to be a software engineer if they wanted to. Now, obviously, some people just as with like great writers are never going to be a published author or the Hemingway, but everyone can gain a good enough proficiency of software engineering if they really wanted to. 

**Howie Liu** (01:26:39):
You could take that boot camp. You could do like some coding exercises on the side, et cetera. The point there is that sometimes I think we treat these disciplines like hard, hard skills that if you're already halfway into your career and you're not already an engineer, if you're not already a designer, okay, well, you can never be one. I just think our brains are malleable and there's a lot of great curriculum out there to learn. Lot of it, like I said, just comes down to also like trial and error and building projects, maybe nights and weekends projects even to learn this stuff. Everyone can learn how to be a versatile kind of unicorn product engineer/designer hybrid in the AI-native era. The only thing stopping you is just going out and doing it.

**Lenny Rachitsky** (01:27:30):
That is a really empowering way to end it, and just to double down on that, it's never been easier to learn these things. There are super intelligences that you can talk to that do a lot as they're building can help you learn.

**Howie Liu** (01:27:43):
Yeah. I mean, literally, I go into ChatGPT sometimes and I ask it just like, "Hey, how would you build this app?" I'm just curious. I'm like, "How would you build Manus, the open-ended agent?" Literally, how would you build it? You can ask the questions and it's like having an amazing, brilliant software architect, software engineer, product manager, designer expert tutor that you can literally like there's no dumb question. They have infinite patience. They're literally on and awake 24/7. It is the most incredible time to learn this stuff, to your point. Then, of course, the interactive tools to go and actually build stuff. Anyone can download Cursor and just start asking Composer to generate some code for you, and then looking at the code and trying to figure out what it does. To your point, when I think back to the earliest era that I experienced of building apps, first I learned C++, then I learned PHP and JavaScript and even building kind of JavaScript single-page apps in the early days like '08 through 2010. It was a dark, dark art. I mean, there were some like... You just had to go and like learn some of these things. There wasn't great tutorials for it. You had to reverse engineer certain things. There were just weird things like if you wanted rounded corners in your UI, you literally took Photoshop, opened it up, created like a rounded corner in pixels, and then cut that pixel up into an image that you dropped onto the page at exactly the right position to be at the edge of a box.

**Howie Liu** (01:29:15):
It's like crazy stuff. I mean, everything was so much more arcane at the time, and now it feels so much more fluid and accessible, and the gap between the arcane tech that you have to wade through to build something has just been minimized so much. It's like the effort and abstraction between you and the magical, delightful actual building of the thing that you want has been so minimized. It's never been a more exciting time to be a builder.

**Lenny Rachitsky** (01:29:47):
You remember spacer.gif?

**Howie Liu** (01:29:49):
Oh yeah, yeah.

**Lenny Rachitsky** (01:29:50):
It's like to create. It's that line stuff you just kind of have-

**Howie Liu** (01:29:52):
Yeah, I remember it. Yeah.

**Lenny Rachitsky** (01:29:54):
... the invisible one-pixel thing that you just stick in places.

**Howie Liu** (01:29:57):
Yeah. Yeah, yeah. No.

**Lenny Rachitsky** (01:29:57):
Oh my God, what a time to be alive. Howie, with that, we've reached our very exciting lightning round. I've got five questions for you. Are you ready?

**Howie Liu** (01:30:05):
Yes.

**Lenny Rachitsky** (01:30:05):
Here we go. What are two or three books you find yourself recommending most to other people?

**Howie Liu** (01:30:09):
You know, I've been trying to read fiction more, partly because I think it's just a really nice mental reset. I will say like Three-Body Problem for anyone who hasn't read it, it's a mind-expanding book. I like sci-fi and fiction that kind of opens your brain, so maybe this is my cheat card, but it's a three-book series. Those are three great books.

**Lenny Rachitsky** (01:30:30):
I love that series, and my tip there is it gets good one and a half books in is my tip, so just keep reading. That's where it's like, "Okay, now I'm in."

**Howie Liu** (01:30:40):
I liked even the first one, but I felt like it was inception where every subsequent book was like you dropped into another, like you incepted into another layer, right?

**Lenny Rachitsky** (01:30:53):
Awesome. Okay. What's a favorite recent movie or TV show you've really enjoyed?

**Howie Liu** (01:30:57):
TV show, I just started watching The Studio. It's like the Seth Rogen, Rogen.

**Lenny Rachitsky** (01:31:05):
Yeah, it's so stressful.

**Howie Liu** (01:31:05):
Yep. Yeah, it is pretty stressful, and I mean, Silicon Valley was too close to home when it came out, so I watched it, but it was just cringy. The Studio is kind of fund to watch because it's a little bit about like inside baseball of Hollywood, and yet I'm not in Hollywood, so it's entertaining to watch. It's I thought smart and a funny show because I split time between L.A. and S.F. I also feel like it's very real to me. I see a lot of the literal characters out there in the world that it's characterizing.

**Lenny Rachitsky** (01:31:43):
Do you have a favorite product you recently discovered that you really love? Could be an app, could be gadget, could be clothing.

**Howie Liu** (01:31:48):
Okay, so I'll give two because I feel like I have to say some kind of software product. I mean, I'm a really big fan of Runway, the product and the company. I just think every new model they come out with, they just came out with a new one just I think like two days ago that gives even more controls and refinement on creating exactly the video scene that you want. I think just the photorealism in what you can generate now, and they also built this cool demo thing that's an immersive world generator I mentioned before. I think it's just cool to see. I also like the underdog story. I'm clearly like Google's gunning in the space, has VO3 and so on and has its OpenAI, but I love the underdog story of this sub-hundred-person company still punching above their weight and building really awesome video experiences. That's the software one.

**Howie Liu** (01:32:45):
Then, a very, very kind of nerdy real-world answer on product is I kind of just recently got into this whole cottage industry of artisanally produced basically clothing by small-scale Japanese manufacturers that use literally like hundred-year-old looms to make clothes the old-fashioned way or the old-fashioned industrial way. They have these loop wheeler machines and they spin the cloth in a very slow pace, so it's completely impractical from a production-scale standpoint, but I've gotten some of these t-shirts and I just love the... I guess in a world where it feels like everything is becoming so much faster moving and even tech from five years ago is obsolete, I love a little bit of the throwback to like old things sometimes can be even more cherishable in this new era. Maybe that makes me a hipster, but I love the vintage, the retro increasingly these days.  

**Lenny Rachitsky** (01:33:56):
I feel like anything that starts with artisanal small batch Japanese is going to be really good stuff. Is there a brand you want to share that is that? Or is this like you want to keep it-

**Howie Liu** (01:33:56):
Yeah. No, I mean-

**Lenny Rachitsky** (01:34:06):
... under the radar.

**Howie Liu** (01:34:07):
... actually, so Self Edge, which actually has a storefront, the main storefront is on Valencia Street in S.F. They carry a lot of these items. That's kind of their whole MO and they have like jeans and t-shirts. I've gotten a lot. I mean, they basically curate a really good selection of different actual makers. One of them is called Studio D'Artisan, another one's called... Actually, it's cool. There's this company called... I think the umbrella company is actually just Toyo, T-O-Y-O, Manufacturing, which sounds like it's a big kind of like large-scale conglomerate, but it's anything but. It's like a really small-scale Japanese vintage manufacturer of clothing, but they have a few sub-brands.

**Howie Liu** (01:34:51):
They actually bought the rights to this American post-war brand that was kind of like Hanes, one of the like big four or five menswear, kind of undershirts and athletic wear brands called Whitesville. I don't know where the name came from, but basically it's a bunch of like basic clothing, like t-shirts, et cetera, and this Japanese indie company, they bought the defunct basically name and now is reproducing clothes almost made to the exact shape and stack, and even with the exact recreation of the graphic packaging on these tees, but like today. I just think there's something really funny and ironic about they've taken an American post-war aesthetic and literal brand, but it's actually a indie small-scale Japanese manufacturing approach to making those clothes.

**Lenny Rachitsky** (01:35:51):
I feel like we just tapped into what could be a whole other podcast conversation about clothing and-

**Howie Liu** (01:35:56):
Yeah-

**Lenny Rachitsky** (01:35:56):
... craftsmanship-

**Howie Liu** (01:35:57):
... [inaudible 01:35:57].

**Lenny Rachitsky** (01:35:58):
... but I'm going to pull us out of that.

**Howie Liu** (01:35:59):
The next podcast franchise.

**Lenny Rachitsky** (01:36:02):
Or just Howie and Lenny talking about clothing.

**Howie Liu** (01:36:04):
That's great.

**Lenny Rachitsky** (01:36:05):
Okay, two more questions.

**Howie Liu** (01:36:06):
Yeah.

**Lenny Rachitsky** (01:36:06):
Do you have a life motto that you often find useful in working or you like to share with friends or family?

**Howie Liu** (01:36:12):
I stumbled on this guy Paul Conti, who I think he's an MD, but also a psychologist, and he has a book, but also he did this long-form podcast with Andrew Huberman, and he actually ends up talking a lot about just how to think about your life outlook and kind of your framework for thinking about life, but grounded in a kind of like scientific and neurological and cognitive science basis. I found one particular point really, really powerful it took with me, which is if you live your life in a way that's foundationally built around humility and gratitude. Look, everybody has different circumstances.

**Howie Liu** (01:37:06):
I think I fully own that even though I didn't come from money, my family was very, very financially modest growing up. I still had incredible resources and opportunities afforded to me even just by virtue of growing up in the U.S., being born in and growing up in the U.S., but also having access to a computer and the internet and even all the free resources I could then access and learn about from there. I still feel like whatever you have or don't have to start with, if you kind of approach the world and kind of the future with a spirit of humility and gratitude rather than, I guess, the opposite of that, I think I've felt like it kind of becomes a self-fulfilling prophecy. You're open-minded, you're kind of grateful, and then more opportunities actually come your way, and maybe it's because the energy you're putting out into the world and other people.

**Howie Liu** (01:38:07):
You're kind of attracting good opportunities and good people and good things, but I think there's a lot of other parts of his framework, but the one that is easiest to remember is like, how do I approach each day? Even if I'm going through a tough moment and I had to fire somebody today, or maybe I get disappointed because we lost a customer deal or something broke or whatever, but to still try to look at the entire situation from overall a feeling of humility and gratitude I think just really does shift your like... It spills over into everything else for that day and maybe even for the whole lifetime. 

**Lenny Rachitsky** (01:38:52):
That super resonates. That is really powerful advice that's hard to internalize, but important.

**Howie Liu** (01:38:57):
Yeah, it's easily said, hard to practice.

**Lenny Rachitsky** (01:38:59):
Yeah. Where can folks find you? What should they know about Airtable and how can listeners be useful to you?

**Howie Liu** (01:39:05):
Okay, so I am on Twitter, howietl. I don't post that much, but I'm a lurker, so I listen and watch, and you can always DM me there. You can also email me directly, howie@airtable.com, anytime you have ideas, feedback, et cetera. On Airtable, just go try it. The whole point is we want to make this an experiential product. That's why we're really leaning into the PLG roots. We talked about the homepage literally says like, "Just start building right now. What do you want to build? Go." 

**Howie Liu** (01:39:36):
It starts building, and so use the product, give me feedback, and if you have ideas of your own and you want to rip on them, I love because my passion is thinking about product and product UX, especially in the AI era if you're working on or thinking about something interesting in that space. Even if it's just purely to riff on a concept, that's something I enjoy doing, and maybe I get to learn and sharpen my own skill set from. Feel free to reach out and, yeah, I mean, tell your friends and family to try Airtable as well. That's the main thing.

**Lenny Rachitsky** (01:40:08):
Sounds like you're looking for people to nerd snipe you and-

**Howie Liu** (01:40:10):
Yes. Yeah.

**Lenny Rachitsky** (01:40:12):
... Howie, thank you so much for being here.

**Howie Liu** (01:40:14):
Awesome. Thank you, Lenny. 

**Lenny Rachitsky** (01:40:15):
Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.

---

## [13/18] How Devin replaces your junior engineers with infinite AI interns that never sleep | Scott Wu (Cognition CEO)

**Scott Wu** (00:00:00):
Our whole team is only like 15 engineers a year. We use a ton of Devin when we're building Devin. Most folks on the team are definitely working with up to five Devins at once, and so Devin merges like several hundred pull requests into production in the Devin code bases every month.

**Lenny Rachitsky** (00:00:12):
What percentage of your PRs are Devin versus humans right now?

**Scott Wu** (00:00:16):
It's in the neighborhood of a quarter or so.

**Lenny Rachitsky** (00:00:19):
Where do you think this will be at the end of the year?

**Scott Wu** (00:00:21):
Honestly, we expect it to be a decent bit more than half.

**Lenny Rachitsky** (00:00:24):
You guys are so ahead of how companies work with AI engineers.

**Scott Wu** (00:00:28):
AI is going to be the biggest technology shift of our lives, so most of the big tech revolutions that we've had over the last 50 years, like personal computer, and the internet, and the mobile phone, they all had this big hardware component that was a big part of the distribution. Folks who were building for those industries kind of saw their market grow and grow and grow basically steadily year over year as the number of people with mobile phones increased, right, as the number of people connected to the internet increase. One of the things which is already I'd say different in AI, is just how explosive the technology can be. There's no weight on hardware distribution. It means that the space is just growing so exponentially.

**Lenny Rachitsky** (00:01:02):
How is the act of being an engineer and building changing?

**Scott Wu** (00:01:05):
I think there's going to be way more programmers and way more engineers a few years from now. Pretty quickly. The form factor of what it means to be a programmer obviously is going to change, but at the end of the day, of course the discipline is all about just being able to tell your computer what's do. And so in that lens, I really think that programming is only going to become more and more important as AI gets more powerful.

**Lenny Rachitsky** (00:01:20):
Today my guest is Scott Wu. Scott is the co-founder and CEO of Cognition, which makes a product called Devin, the world's first autonomous AI software engineer. Unlike other AI tools that I've highlighted on this podcast, Devin is designed to act like an actual remote engineer that you chat with like you would with any other human engineer through Slack or through its dedicated website. When Devin launched about a year ago, it was very much a junior engineer. Over the past year, they've made a lot of progress and Devin is now being used by tons of companies in production. We chatted about how their engineering team of 15 uses Devins to build Devin, including how every engineer uses about five Devins each to help them code and move faster. How a quarter of their pull requests today are committed by Devins and that they expect this to be over 50% by the end of the year.

**Lenny Rachitsky** (00:02:05):
We also talk about how Scott imagines software engineering is going to look in the future and how the role of an engineer changes from a coder to an architect. We also get into the eight pivots that they went through before landing on this path, why Scott believes AI tools like this will lead to more engineer hiring versus less. Also where the name Devin comes from and so much more. This episode is going to blow your mind. I highly recommend you listen to it if you're at all interested about where engineering, product building, and AI is going. A huge thank you to Claire Voue for suggesting a bunch of great questions for this conversation.

**Brandon Foo** (00:04:15):
Hey Lenny. Thanks for having me.

**Lenny Rachitsky** (00:04:17):
So integrations have become a big deal for AI products. Why is that?

**Brandon Foo** (00:04:21):
Integrations are mission-critical for AI for two reasons. First, AI products need contacts from their customer's business data such as Google Drive files, Slack messages or CRM records. Second, for AI products to automate work on behalf of users, AI agents need to be able to take action across these different third-party tools.

**Lenny Rachitsky** (00:04:40):
So where does Paragon fit into all this?

**Brandon Foo** (00:04:42):
Well, these integrations are a pain to build and that's why Paragon provides an embedded platform that enables engineers to ship these product integrations in just days instead of months across every use case from RAG data ingestion to agentic actions.

**Lenny Rachitsky** (00:04:57):
And I know from firsthand experience that maintenance is even harder than just building it for the first time.

**Brandon Foo** (00:05:01):
Exactly. We believe product teams should focus engineering efforts and competitive advantages, not integrations. That's why companies like You.com, AI21 and hundreds of others use Paragon to accelerate their integration strategy.

**Lenny Rachitsky** (00:05:15):
If you want to avoid wasting months of engineering on integrations that your customers need, check out paragon@useparagon.com/lenny. Scott, thank you so much for being here and welcome to the podcast.

**Scott Wu** (00:05:29):
Thanks so much for having me. Excited to be on.

**Lenny Rachitsky** (00:05:31):
I'm really excited to have you here because you are building and you've been building something that is very different from what a lot of other AI companies have been doing for a long time, although they are starting to converge to where you guys are now. We're going to talk about that and it's also just such a unique point in the history of AI and just the journey of AI. And so it's really cool to be chatting right now. And I feel like we're going to chat again in a few years and be like, wow, we were so right about so much and so wrong about so much.

**Scott Wu** (00:05:59):
Yeah.

**Lenny Rachitsky** (00:06:00):
And so I'm excited to have you here. Let's start with talking about Devin, giving people an understanding of just what the heck Devin is, is the main product that you guys build. What is the simplest way to understand what is Devin?

**Scott Wu** (00:06:10):
Absolutely. And so Devin is a fully autonomous software engineer that is going to work on tasks end to end, and so there are a lot of great tools for all parts of the stack of the AI code workflow. What Devin does is it is a full asynchronous workflow, and so you can tag Devin on an issue in Slack, you're talking about an issue and you tag Devin, you can tag Devin in Linear, you can have Devin and Devin will make pull requests in your GitHub, and so it's very much built to work with engineering teams as your junior engineer.

**Lenny Rachitsky** (00:06:38):
Amazing, okay. So I remember when you guys launched this, there was this big pitch of this is your new AI engineer and it was really good at a lot of stuff. It wasn't great at other things. It's been a year now about since you guys launched, is that right?

**Scott Wu** (00:06:50):
Yeah, yeah.

**Lenny Rachitsky** (00:06:51):
What's the best way to think about the level of seniority that engineer had back in the day when you guys launched and then the level of seniority of engineer today if that's, I don't know, measure of how to think about Devin?

**Scott Wu** (00:07:02):
Yeah, and it's crazy to think about by the way, because a year ago when we did the initial launch, I mean people didn't really believe that an agent was possible. Right. And it was, I mean, it was a very different time. So like start of 2024, things with model capabilities were definitely quite a bit earlier on, reasoning especially was quite a bit earlier on. And yeah, I mean, in the time since then it's obviously developed a lot. I think in terms of practical skills, there's some comparisons we make. Sometimes we kind of say, well, when we got started it was kind of like a high school CS student and then as time went on, it became more of a college intern and now it's like a junior engineer. But I would say though that those are more rough guidelines because I really like the phrase jagged intelligence for example, because obviously there are certain things that it is much better at than a human. There are certain things that it's much worse at than a human.

**Scott Wu** (00:07:52):
And I think over the last year we've learned a lot especially about not just coding agents but agents in general just really building out how all of us should be working and interacting with agents as part of our flow. And so a lot of the things that we built, I mean, there was no Slack, there was no GitHub integration, there was no Linear, there was no interactive planning phase working back and forth. There was no way to touch up Devin's code. And so a lot of the features that we've built on the product side since then have really been about basically yeah, figuring out how to make working with Devin and handing off tasks to Devin as smooth of an experience as possible.

**Lenny Rachitsky** (00:08:27):
That's so interesting. So a lot of the work has gone not into how do we just make Devin the best possible engineer, but it's how to work with this new type of entity that we haven't ever worked with.

**Scott Wu** (00:08:37):
I think it's a 50/50 of both. I think the capabilities obviously have improved a ton and we've seen these get better and get measurably better. But I think the other side of it is everything to do with yeah, really the product interface and the tools and so on. And I think today folks generally know how to use chatbots and to with chatbots, right, and that's an interface that people are familiar with and obviously with agents it's still like a real curve, I think to learn how to use them and how to get the most out of them. And so it's really exciting to see a lot of others starting to build and do a lot more in the agent space as well. But I think this is the kind of thing that we're all really figuring out together as a space.

**Lenny Rachitsky** (00:09:15):
What can you share about just the scale of Devin at this point, whatever you're comfortable sharing, and then just where do you think the level of Devin's coding abilities will be in a year?

**Scott Wu** (00:09:24):
So we work with companies of all stages and sizes. On the smallest end, it goes to startups of just one or two people who are using Devin to build out a lot of their kind of initial prototype or initial product all the way up to big public companies, Fortune 100 companies or public banks or things like that who are using Devin across their engineering teams. In general, we've seen a huge range of the use cases there. And obviously the kinds of engineering work that you're doing at a one or two person startup, they're very different from the kind of work that you're doing at a public bank.

**Scott Wu** (00:09:55):
But throughout it's all been basically yeah, being that junior buddy of yours that makes you go faster and really multiplies you, I would say. I think it can multiply you as an engineer obviously by just letting you work with your own team of Devins instead of having to be kind of fully synchronous on a single task. And then it's also kind of multiplying your team and multiplying your team's knowledge base because Devin really accumulates a lot of the knowledge from working with every member of your team and is able to bring that into each new session.

**Lenny Rachitsky** (00:10:23):
Awesome. We're going to show people how it actually works later in the podcast. We're going to do a few live demos, but let's actually go to the beginning of the journey. What's just the origin story of Devin? How did this all begin?

**Scott Wu** (00:10:33):
The founding team, I mean most of us have known each other for years and years and years actually. And for almost everyone, this is our first time working together, but we've known each other a long time. And we all actually had our own kind of journeys in AI for the last decade or so. And so for myself, I ran a company before this called Lunchclub, which was an AI for a professional networking product and I ran that for about five years. And one of my co-founders, Steven was one of the first engineers at a company called Scale AI, which has obviously grown a lot and done very well. My other co-founder Walden, was an early engineer at a company called Cursor, which has also obviously grown a lot and done really well. And our whole team was kind of like that. Many of us knew each other from competitive programming and math competitions, but we had stayed very closely in touch in the decade since then and we've all kind of all had our own journeys.

**Scott Wu** (00:11:22):
And so we had one person who was running teams at Neuro, we had one person who was at Waymo, someone who had their own YC tools startup for machine learning, and we were really excited to build something together. And this was around late 2023, so about a year and a half ago at this point. And yeah, when we got started, I mean I think there were a couple of things that we felt really strongly about and one was that reinforcement learning was really working and was going to be the next big paradigm shift in capabilities. Back then it was the initial ChatGPT launch in 2022, and those models were, to first order were what we would call imitation learning in AI, right, which is basically you have the model read all the texts that you can find on the internet and then train it to talk like somebody on the internet would talk. Right. And there are kind of obviously a lot more details on top of that, but that's kind of the first order pass of what was really done.

**Scott Wu** (00:12:16):
And it was amazing. Right. I mean, it passed the churn test, it was able to respond and to have encyclopedic knowledge about a lot of things. And I think this new paradigm which we've gotten into over this last year or year and a half is really high compute RL, which is a very different paradigm, right, which is basically the ability to go and do work on task and put something together and then be evaluated on whether that was correct or incorrect and use that knowledge to decide what to do and to learn from that. Right. And so we felt very strongly that that was going to happen. I think for us, code was the natural thing to work on for a couple reasons. One, because we're all programmer nerds ourselves, and so teaching AI to code is about as cool as it gets for us, but also because code has this whole automated feedback loop, right, where you can run the code and that is the kind of automated feedback that really feeds into the RL, which makes these models so great at coding.

**Scott Wu** (00:13:06):
And then the other thing that we felt very strongly about was that the product experience was going to shift from what I'll call text completion to agents basically. Right. And to first order, I would kind of say there's been a lot of great experiences in text completion. It's been used for marketing, it's been used for customer support, it's been used for education and in Coda obviously as well. The GitHub copilot was kind of really the dominant product of that initial wave. Right.

**Scott Wu** (00:13:34):
But I think that the big shift that we really felt we would see is moving from kind of this text to text model to an actual autonomous system that can make decisions, that can interact with the real world, that can take in feedback, that can iterate and take multiple steps to solve problems. And now we call that agents, but that was what we were really excited about at the time. So it was always coding, it was always agents. And in some ways that kind of feels like it should have been cleared from the start. But even with that, I feel like we've pivoted eight times or something within coding agents over the last year and a half, so.

**Lenny Rachitsky** (00:14:08):
I just noticed recently all the AI, top AI companies sort of, not all, but many of them, the product that is winning is different, has a different name from the company, which is not typical, Cursors, Anysphere, Bolt, StackBlitz, you guys are Cognition Labs, like V0 is Vercel. And it just tells me these all emerge later in the company's journey and they tried a bunch of stuff and like, oh wow, this thing worked and it's so interesting that it's so common amongst these top AI companies.

**Scott Wu** (00:14:34):
Yeah, and there's even, I mean OpenAI, ChatGPT, Anthropic and Claude and Google. Yeah, it's funny. Yeah, yeah, I agree. So when we got started, it wasn't even really a company. I mean, it was more like a project or a hackathon almost. We got a bunch, we booked an Airbnb basically for a couple of weeks. This was around Thanksgiving time and just got a bunch of people together who were just excited to hack on some projects and build something cool. And it's funny, actually the first thing that we were building for actually was more solving these more contest programming problems and using an agentic loop to really do better on that. And so obviously if you run your code on the test cases, you can evaluate, there's a lot of agentic work that you can do there to try and do better, and that we spent some time on that initially. And then we've kind of gone from, I mean the story of the whole company for us in some sense has been going from hacker house to hacker house.

**Scott Wu** (00:15:26):
So after that we had another hacker house and that's where kind of some of the initial ideas for Devin came and really building a software engineering agent and not just a coding agent and having to interact with a lot of these tools, but even then there were so many iterations. And even the idea of talking to Devin for example was like, it was something that we had to come up with. Right. Initially, it was just like you hand off a task and then it works and then it shows you this whole finished code. Right. And now obviously it's like you can jump in at any time, you can get feedback on the plan, you guys can scope out the task together when you're working with Devin. And a lot of these things we had to develop obviously, and certainly we've learned a lot about the use cases, the form factor. We've made a lot of big improvements and step function improvements on the capabilities and Devin's ability to use tools and to bug and make decisions.

**Scott Wu** (00:16:11):
And so it's yeah, it's been a fun journey. I mean, I think I would say that the grounding question for us really, which is one that we think about all the time, is really just what is the future of software engineering and how should we be working with AI to write code? Because I think at the end of the day, of course that's what underlines all of the product decisions that we make, so.

**Lenny Rachitsky** (00:16:34):
I like that you're asking the juicy question I wanted to get to. Before I ask it just for the history books, when did you guys start kind of hacking around and when did Devin launch? How long was that period?

**Scott Wu** (00:16:45):
Yeah, so we started in November of 2023, which was then yeah, just like hackathon mode. We officially made it into a company around the start of 2024, and then our initial launch was in March. And so it was like nonstop, I mean it's been nonstop for the entire last 17 months, but it's getting to the launch and then obviously working with enterprises and developing the product a lot more, building it and getting it to work for a lot of practical use cases and then making it fully available self-serve in December of last year. And now we've rolled out 2.0 obviously just a few weeks ago. And so it's been a very busy time for us.

**Lenny Rachitsky** (00:17:25):
Understatement of the century. Let me ask this question because you touched on it a bit, this whole idea of Devin as a person and this idea of creating a personality for Devin. It's unlike any other, I believe, AI app. No one else has a name and you don't think of it as a person. What made you guys decide to go that approach and just how do you design it to work well that way?

**Scott Wu** (00:17:44):
I would say it's a decision we're pretty proud of, I would say. I mean, I think there's a lot of different product experiences out there, and I think the thing that really makes Devin unique in what it does is that you can really hand off and more and more what we've seen honestly is that I think a lot of kind of explaining the Devin experience to folks is really just explain it as, yeah, this is your junior buddy. And that goes for a lot of the parts of the flow where in the onboarding for example, initially I would say we've definitely had a lot of users come in and just kind of see the blank screen and not really know, or they'd ask, "Hey, I'm going to do this whole big re-architecture, the whole code base."

**Scott Wu** (00:18:21):
And basically what we've learned over time is to basically get folks to think more like whoa, whoa, whoa, let's work on getting the repository set up first. Let's make sure we hand Devin a couple one pointer tasks so it can get familiar with the code base. Let's get it the thing. If Devin needs to be able to test the code or run the linter or CI or things like that. Obviously we want to make sure Devin's got its own virtual machine set up to be able to do that.

**Scott Wu** (00:18:45):
And similarly, I think the usage pattern, I think often it wasn't clear and obviously you can sit and just kind of watch Devin do it action by action and work that way, but we found that the best workflow really as a team building a lot of stuff was to work with multiple Devins and to run them asynchronously and to kick them off and to only jump in basically as you needed to provide feedback or steer the plan or anything like that. And so in many ways, I think Devin as a name really is our attempt to kind of capture the soul of that as a product where it really is treating it like a bit more of an autonomous entity that you can hand off tasks, that you can work with, that you should be teaching and learning with over time.

**Lenny Rachitsky** (00:19:29):
I want to come back to an area you started us down and then I took us away from which is impact on software engineering and how software engineering is going to change. So there's kind of two parts to this. Just like when people are using Devin today, say this year, how is the act of being an engineer and building changing for those companies? What does that look like?

**Scott Wu** (00:19:49):
By the way, we're all software engineers ourselves. It's like I'm a programmer by training and still a programmer at heart certainly. And I think the way that we've always thought about it is there's layers of abstraction and there's tools, and one way I would say it at a high level is kind of I think of AI in general as, yeah, I mean computers are obviously getting more and more intelligence and are able to do more and more and it's possible there may come a day where computers truly do everything that we do and humans are not responsible for any of it.

**Scott Wu** (00:20:20):
I don't expect that to come particularly soon, but I guess what I would say is until that point, for as long as we're still part of the equation, one of the most important things to do obviously is for us as humans is to instruct our computers on what we want and what we want to build and what we want to do. Right. And software engineering is, we think of it today obviously as Python and C++ and JavaScript and all these things, but at the end of the day, of course the discipline is all about just being to tell your computer what to do. And so in that lens, I really think that programming is, if anything, is only going to become more and more important as AI gets more powerful. And I think the thing that's really exciting for us is yeah, it's really seeing that kind of iterative transformation. And so you ask what things look like today, and I would say, yeah, it really is like having a junior buddy or really a team of junior buddies that you can work with. Right.

**Scott Wu** (00:21:12):
And so every engineer on our team, we use a ton of Devin when we're building Devin, and so Devin merges several hundred pull requests into production in the Devin code bases every month, which is, I mean, our whole team is only 15 engineers, and so it's a pretty sizable fraction of all the code that we write. And the way that we use it is basically, yeah, everyone's got their whole team of Devins. If you're going to be looking through various issues, if you're going through feature requests, if you're going through bugs, if you're going through new paradigms that you want to build, then it is naturally the case that there's a lot of handoff points where you just say, "Hey, at Devin, here's what's going on. Can you please take a pass at this?" Right.

**Scott Wu** (00:21:47):
And sometimes Devin will be able to do the task 100% autonomously and just makes the PR and then you merge the PR and that's great. Sometimes you want to be able to jump in for the 10 or 20% that really needs your help. Maybe there's a few details with how exactly you want to scope it or how you're architecting this feature, or maybe you want to go and test the front end at the end yourself to make sure it looks exactly the way that you want and give your one or two lines of feedback after that. Right. But a lot of it is really yeah, is kind of like yeah, learning to work with Devin to be able to just do more in parallel and build more.

**Lenny Rachitsky** (00:22:20):
What percentage of your PRs are Devin versus humans right now?

**Scott Wu** (00:22:25):
Yeah, I'd have to look, but it's in the neighborhood of a quarter or so of all of our-

**Lenny Rachitsky** (00:22:30):
Wow.

**Scott Wu** (00:22:31):
Yeah. Yeah.

**Lenny Rachitsky** (00:22:33):
And then what was it like six months ago?

**Scott Wu** (00:22:35):
Oh yeah, it's grown a ton for, I mean, we've seen it grown exponentially internally ourselves as well. And so it's kind of an interesting one where again, it's always both the capabilities and the product interface. And so I think the intelligence has increased a lot. But the other thing of course is that we've spent a lot of time in figuring out how to build and to really kind of build for an interface where you can get Devin's value on tasks where Devin is able to do the 80 or 90%. And so Devin is obviously not perfect and it'll make mistakes and so on. And a lot of the question is basically, yeah, how do you scope out your initial task with Devin and then just kind of set Devin off and have it go and do the things that you want do? How do you come in at the end and review and give feedback? How do you make sure Devin learns over time? How are you able to kind of just check in as needed and course correct if you want to?

**Lenny Rachitsky** (00:23:24):
Okay, so today about quarter of your PRs are Devins. Where do you think this will be at the end of the year? What would you guess?

**Scott Wu** (00:23:31):
I think by the end of this year, we expect it to be more than half. And I mean, as time goes on, one of the things that we've seen is just you're able to do more and more and more work asynchronously. Right. And you're able to hand off more and more. I think the soul of programming, the soul of software engineering has really been about through all the areas, not just now, but even when it was assembly, right, and even when it was Pascal and even when it was punch cards or whatever, I think the soul of it has really been basically just about defining the problem that you're facing and really thinking through exactly...

**Scott Wu** (00:24:00):
... about defining the problem that you're facing and really thinking through exactly what is the solution that you want to build. Thinking through the architecture, thinking through the details, and really mapping out in your mind exactly what you want to build basically and what you want to have your computer do. I think that's what makes software engineering really great and I think that's the funnest part of software engineering. 

**Scott Wu** (00:24:21):
I think, at the same time, that's probably in the neighborhood of 10% of the average software engineer's time, right? Because 90% of the time is you've got this Kubernetes error, then you've got to debug, and you have to see what went wrong and the system crash, or you left some port open and this is messing up, or there's a bug report that you have to take care of, or you've got to migrate your code, or you got it upgraded to a new version or things like that. A lot more implementation. 

**Scott Wu** (00:24:48):
One of the ways that we've kind of thought about Devin in building Devin is really allowing engineers to go from bricklayer to architect, so to speak. A lot of it is just getting to the point where you can do the high-level directing and you can basically specify things exactly how you want. I think it's very much about still having the human in control and having the human able to do the full specification, but just multiplying the magnitude of what you can do and what you can build in one day or one hour or however long.

**Lenny Rachitsky** (00:25:18):
In the future, say someone is trying to get into software engineering, thinking about becoming an engineer, first of all, do you think people should... Classic question everyone's getting these days. Should you still learn to code? 

**Scott Wu** (00:25:29):
Yeah. 

**Lenny Rachitsky** (00:25:30):
I'd love your perspective there. And then two, for people that are engineers today, what skills do you think will be more and more important and then less important in this discussion of moving from bricklayer to architect? 

**Scott Wu** (00:25:41):
Yeah, for sure. I love this question. First of all, the question of whether you should still learn to code, my answer would be absolutely yes. I think, to a large extent, when you take computer science classes and when you learn these fundamentals, sure you're learning a little bit about how a particular language is, syntax works or something like that. But honestly, most of what you're learning really is about the ability to logically break down problems for number one. And two, I would say is just the model of a computer and a lot of these decisions and a lot of the abstractions that we've built over time. 

**Scott Wu** (00:26:11):
What is a database and how should you think about a database? What is a garbage collection system and how do those work and all of these different pieces? The reason I think that's important is because it's the same with a lot of these other... Arguably we've already gone through these phases in programming and I think this next one is going to be somewhat faster and somewhat bigger, but in many ways a similar flavor, which is when you work with Python today, obviously, a lot of things are already abstracted away from you.

**Scott Wu** (00:26:38):
In some sense, someone from 50 years ago might already call Python. You just get to explain in English what you want and now the computer does it for you. That's great and I think it's really powerful. It's opened it up. I mean, we have far more programmers, obviously, than we ever have before because of that, but I would say certainly as you're building your skills as an engineer, it really helps a lot to understand the abstractions and to be able to peel the layers beneath. Folks will use assembly for example if they're really performance optimizing a piece of code, but also in order to build good systems and to understand these things, you certainly want to understand these abstractions of how does networking work. 

**Scott Wu** (00:27:18):
What is TCP/IP like exactly or what happens with this Python code when it gets interpreted or all of these details. Similarly, I think we will get to a state where, with no experience at all, you're going to be able to build some pretty cool stuff and to do some pretty amazing work just by explaining what it is that you want. But I think that, for quite some time, you really want to be able to think precisely about the details, to peel back the abstractions, to be very precise about what it is that you want to build and how.

**Lenny Rachitsky** (00:27:48):
And then for skills that you think are more and more valuable for engineers, where should engineers today be leaning more and more into versus like, "Forget this. I don't need to think about this anymore"?

**Scott Wu** (00:28:00):
For sure. I think architect... I mean, we already have a term for architect and engineering and I think it is directionally the right term. It's, I think, one thing to just do a routine implementation and write boilerplate code and things like that. I would say that, in many ways, AI coding has already made us much faster at that, but I think a lot of the core questions of understanding very complex systems and working in the context of the whole company and thinking about the product that you're building or the work that you're doing and understanding, "Okay, what are the problems that we want to solve? How do we want to solve those problems? What is exactly the solution that we want to build? What are all of these key decisions and trade-offs that we're going to be making?"

**Scott Wu** (00:28:40):
Basically, I think folks who are able to do that really, really well are just going to be able to leverage themselves more and more. If anything, I think there's going to be way more programmers and way more engineers a few years from now than there are today. I think pretty quickly the form factor of what it means to be a programmer, obviously, is going to change. And in some sense, it already has, but I think there's just going to be so much more for us to build. Folks talk about Jevons Paradox all the time. I mean, software is truly the shining example of Jevons Paradox, where we have always managed as a society to find more and more things that we want to build software for and build more code for. I really think there's a lot more out there to do.

**Lenny Rachitsky** (00:29:23):
For people that don't know Jevons Paradox, can you briefly explain it?

**Scott Wu** (00:29:26):
Absolutely. Yeah. Jevons Paradox just says that as the price of something goes down, it can still be the case that the total spend on it actually goes up. You can think about this with money, you can think about this with time or resources, but the direct version here is, I think, as it becomes easier and easier to program and as programming becomes more and more effective, I think we're going to have a lot more programmers. I think in a kind of zero-sum view, you might say, "Well, we're going to be 10 times faster at software engineering and it means that we're going to need 10 times fewer software engineers." But I think in practice, what really is going to happen is actually we're going to build even more than 10 times as much code. And because all of the work that we do is so capped, obviously, on our ability to actually build and execute and iterate, we're going to have so many great ideas out there, we're going to have so many great products out there. People are going to build a lot more personalized experiences, for example, and there's going to be a lot to do.

**Lenny Rachitsky** (00:30:22):
Going back to the way you guys use Devin, you said that every engineer has this fleet of Devins. How many Devins per engineer do you find most people are working with these days at your company?

**Scott Wu** (00:30:33):
Yeah. It's very asynchronous. Obviously, you can kick them up and start them up and shut them down basically as you see fit, but most folks on the team are often working with up to five Devins at once, I would say. It's a nice flow where you think through, "All right, what are the five things that we want to get done today?" You have Devin one do number one, you have Devin two do number two, Devin three... For what it's worth, I think it's taken us some time to adjust to it and get to the point where it's really intuitive for us, but I think it's definitely a different experience where you're handing off most things asynchronously. 

**Scott Wu** (00:31:14):
The goal for each of your tasks is to be there for the parts that really need your expertise, either you really, really need to define exactly what it is that you're solving for and what you're building or maybe some of the more complex parts where you want to steer Devin towards, particularly what kinds of changes you want to make. "I want the class to be set up this way and we should go and change all the downstream references to this as well or whatever." But basically having Devin do the bulk of the work asynchronously with you.

**Lenny Rachitsky** (00:31:43):
And then how many engineers do you guys have roughly?

**Scott Wu** (00:31:46):
Yeah, our engineering team today is about 15 people.

**Lenny Rachitsky** (00:31:48):
15. 15.

**Scott Wu** (00:31:50):
15, yeah.

**Lenny Rachitsky** (00:31:50):
Holy moly. Okay. And then each one has five-ish Devins.

**Scott Wu** (00:31:54):
Yeah.

**Lenny Rachitsky** (00:31:54):
So there's five times the number of Devins as engineers. What I love about this is just a glimpse into where the future is going. You guys are so ahead of how companies work with AI engineers. Seeing how you operate is going to be, as a sense, essentially how most companies will end up operating.

**Scott Wu** (00:32:11):
Yeah. For what it's worth, we've already seen this shift, I would say, ourselves where... In terms of the team, obviously, folks don't spend that much of their time just writing out boilerplate or just doing pure implementation of features. People get to spend much more of their time focused on really just thinking about the core questions of, "How do we make Devin better? What is the right interface for Devin? What is the right flow or the right set of features that's really going to make this as great of an experience as possible?" That's how we like things.

**Lenny Rachitsky** (00:32:46):
When is the point you reach, where there's takeoff of this being the by... Your Devin starts moving so much further ahead of everyone else. Once you have enough Devins doing all these things, they're just like wherever and you're 10 years, 20 years, 30 years, 100 years ahead.

**Scott Wu** (00:33:00):
Honestly, as a community, I think all of us as engineers around the world, I think, are going to have to think about this and build for this and adapt to these new technologies. But what I would say is, I think more and more... Especially as capabilities get better, but certainly even in studies say today, I think more and more things are going to shift towards this asynchronous flow. And one of the reasons I would say for that is in the real world, you're just capped by real world constraints. I think that one way to put it is kind of like... Don't take these numbers exactly, but it's kind of like the first order math of it is, of course, being able to write files or to complete this function or complete this line or things like that.

**Scott Wu** (00:33:44):
It helps a ton. It's a really great experience. There's a lot of parts of building software that obviously are almost not that at all, right? If you have a bug that you're trying to fix and so you spin up the local server, you click around on your own product on the front end and try to reproduce the bug yourself. Once you have the error, you take a look at Datadog and you see what happened and you try to find other errors in the logs. You look at those files and you see what went wrong, you make some edits, maybe you go and rerun the whole process again now that you make sure your change looks right. That's a lot of what it means to be a software engineer. These are processes that take real time. I think we're going to shift more and more towards this agentic workflow, because that's in some ways the way to really get to that 200%, 500%, 1000% gains that we'll be getting to with software engineering over the next few years.

**Lenny Rachitsky** (00:34:39):
Enough talk. Let's show people what the heck this actually looks like. You've got a couple demos prepped that show a few use cases that you found helpful. You're going to pull up your screen and then we'll kick it off and then we'll talk as it's happening.

**Scott Wu** (00:34:52):
Yeah. The whole process obviously of working with Devin is working asynchronously. I thought it'd be cool for us to actually just watch Devin a little bit in action and then we can go through some other examples of work that Devin's done or things that Devin does for us, even on our team for example. But then we can check back in asynchronously with our Devin after. 

**Lenny Rachitsky** (00:35:12):
Let's do it. 

**Scott Wu** (00:35:13):
I'll share this real quick. The key thing that I would just emphasize here is a lot of it obviously is really just about thinking about as a software engineer or as engineers ourselves or engineering teams, PMs, and so on. What are the things that we would want to build that we would want to hand off? We have Devin set up with our own Devin code base, for example, and so I'll go ahead and kick off with Devin for that. I'll just say, "Devin, I'm on with my friend, Lenny."

**Lenny Rachitsky** (00:35:45):
Hi, Devin.

**Scott Wu** (00:35:48):
"Can you modify Devin web app two?" Let's feature your newsletter as part of the Devin website. 

**Lenny Rachitsky** (00:35:59):
Let's do it. On the real Devin website. 

**Scott Wu** (00:36:02):
"Feature Lenny's site." 

**Lenny Rachitsky** (00:36:04):
Yeah, lose all your features. 

**Scott Wu** (00:36:06):
We're going to kick this off. As you can see, Devin gets started instantly and goes ahead and responds. Again, you can work with this asynchronously, you can work with it synchronously as well. For this, we'll just kind of go in a little bit and see exactly what's going on. But as you can see here, Devin's going through files and taking a look through a lot of stuff. We can follow here basically as we need to and see what makes sense. You can see Devin's already called out a few particular pieces where there's the sidebar, which we have implemented on the front end, and there's pieces there. We're going to have a new component and that component's going to link to Lenny's website. That all sounds good. Devin's asking us any questions if there's anything that we have here. Same story here where you can let Devin make its own decisions and hand off, or you can go ahead and give some more thoughts. Should the button open in a new tab or within an application? I'll say let's open it in a new tab.

**Lenny Rachitsky** (00:37:03):
And you could answer these at any point. Is it waiting for the answer? 

**Scott Wu** (00:37:06):
You answer these at any point, you can hand off, hand back off. 

**Lenny Rachitsky** (00:37:07):
It's not going to be like, "Just god damn it. I just wrote it this way. Why didn't you tell me earlier?" 

**Scott Wu** (00:37:11):
That's right. One of the big pieces with Devin is Devin will always be enthusiastic, will always be ready to put in the hours. 

**Lenny Rachitsky** (00:37:21):
Thanks for changing scope. Thanks, Scott.

**Scott Wu** (00:37:24):
We'll give Devin a chance to work and it's going to go through these files and it'll make a pull request for us and we'll see and go from there. But I thought it'd be fun to show some other examples of Devin in action as well. One of the examples actually this morning, which I just used Devin for, is I asked Devin to help me brush my own facts up for this podcast. Obviously, a huge fan of the podcast and the newsletter. I asked Devin, "Hey, Devin. I'm to be on the podcast. Could you please research everything you can about him and make a nice website quiz for me so that I can make sure I know my facts?" This was just this morning, I asked Devin to do this and I'll just show what Devin did, it looks like. Went to Wikipedia first. Unfortunately, it's not a page in Wikipedia, which is... Lenny, we will work on that. 

**Lenny Rachitsky** (00:38:09):
I'm not a big enough deal yet. 

**Scott Wu** (00:38:12):
They did you dirty. I mean, we need a page list. And so then it went and found it on Spotify.

**Lenny Rachitsky** (00:38:19):
So you're watching what it's researching live?

**Scott Wu** (00:38:21):
Yeah, yeah, yeah. This was this morning, obviously. 

**Lenny Rachitsky** (00:38:24):
This is a playback of what Devin did. This is part of Devin you could just watch what it did.

**Scott Wu** (00:38:29):
Yeah, yeah. Especially when you're building engineering products or something like that, you can see each of the steps that Devin was doing, or if Devin tested the code locally, obviously you want to be able to go and look and see what Devin was cooking around with and testing or things like that. It found the newsletter. It's going and looking at this and it's going and reading all of this. And then it says, "Okay, let's get started with putting the code together." It says, "Hey, I've researched." It's going through and writing all of this, putting the app together. It plays its own quiz itself. Actually, we should just play this quiz actually. Let's see how much I know. 

**Lenny Rachitsky** (00:39:05):
[inaudible 00:39:05]

**Scott Wu** (00:39:06):
What is the name?

**Lenny Rachitsky** (00:39:08):
What is the name of the podcast? Lenny's Podcast. For people not watching, so to say, approximately how many subscribers? A million. Very good.

**Scott Wu** (00:39:17):
Yeah. Yeah. 

**Lenny Rachitsky** (00:39:18):
What are three main topics Lenny is focused on? Oh, product, growth, and career. Very good. It's a good quiz [inaudible 00:39:25] of people. 

**Scott Wu** (00:39:25):
What does [inaudible 00:39:26] besides podcasting?

**Lenny Rachitsky** (00:39:27):
What does Lenny do besides podcasting? 

**Scott Wu** (00:39:30):
I'd say writing, investing, and advising. How often does Lenny-

**Lenny Rachitsky** (00:39:35):
[inaudible 00:39:35]

**Scott Wu** (00:39:35):
Once a week, right?

**Lenny Rachitsky** (00:39:36):
Once a week.

**Scott Wu** (00:39:37):
Yeah. We can go through all these and do all these. I took this quiz, by the way, obviously, to make sure that I was well-prepped, but this is kind of one of the more fun examples, obviously, of [inaudible 00:39:46]

**Lenny Rachitsky** (00:39:46):
Scott, how many subscribers do I have at my newsletter?

**Scott Wu** (00:39:50):
Over a million, actually. 

**Lenny Rachitsky** (00:39:52):
[inaudible 00:39:52]

**Scott Wu** (00:39:53):
And then one last one I'll show and then maybe we can come back to our initial run after. Like I was saying, a lot of this is really built to work with all of the existing code workflows out there. For example, we were doing some exploration with the DeepSeek repository on GitHub and we imported it into Devin and we got our own fork of it set up in Devin. A couple of things I just wanted to show here. One is Devin sets up its whole wiki with all of its internal understanding. When Devin indexes the code base, obviously, building a representation of the code base and learning it and improving it over time is one of the big things that Devin does. Funnily enough, we found that naturally humans really are interested to understand this code base representation as well.

Devin Wiki is something that we built here and you can take a look at all these different pieces and see each of these different things. Here are the FP8 operations, here's an SG [inaudible 00:40:46] integration. There's diagrams of how the different layers are built and put together. There's deployment operations. There's a lot of details about the architecture as well. You can ask questions about it as well. For example, you can say, "How does DeepSeek handle multi-token prediction designed for spec deck?" It'll go through and it's able to search through the entire code base and give you an informed answer based off of that. 

**Scott Wu** (00:41:10):
We use this a lot. It helps when you're scoping out a task for Devin and doing an initial prompt. It also helps, obviously just in a vacuum, you often have questions about your code base that are really nice.

**Lenny Rachitsky** (00:41:20):
**Scott Wu** (00:42:41):
Yeah. We go all the way to the biggest code base as possible. One way I'd kind of put it is how... The way that we as engineers would think about a large code base is certainly when you're making changes or when you're thinking about a particular task. You're not bringing in every single line of the code base at once. You have a high level of traction that you're able to think about and look into and then you're obviously able to zoom in and get to higher resolution on each of these different things. Devin works in much the same way, where the first thing it'll do is it's going to figure out the high level architecture of what's going on here and what this is built for and so on. But within each of the components, it's obviously also going to be able to zoom in and give some more detail about each of these. Here's FP8 to be float 16 and how exactly a lot of that is set up. Here's each of the different parts of the code base. Similarly, we built this to be scalable.

**Lenny Rachitsky** (00:43:38):
It's essentially coming back to the engineer as architect. Now, it's helping you understand the architecture, kind of circling back to that.

**Scott Wu** (00:43:46):
Yeah. Yeah, exactly. One of the fun use cases that we've seen actually with folks is they'll often actually get Devin's help to onboard new engineers on the team. When you're new and you're joining, there's obviously a lot of questions that you have about the code base or about how things are set up. It also sometimes can be a little bit awkward to ask your mentor or your manager the questions and if you're worried that they're going to be really dumb questions. It's nice to just be able to ask Devin and to go through Devin's wiki and to understand these internal representations.

**Lenny Rachitsky** (00:44:15):
I think that's really interesting, because it comes back to your point that Devin is not just a junior engineer. It's what you call a jagged engineer.

**Scott Wu** (00:44:21):
A jagged intelligence.

**Lenny Rachitsky** (00:44:22):
A jagged intelligence, where it's almost like a staff engineer at understanding the code base. Usually, you have to ask an engineer that's been there a long time, " What does this do? Where is this thing? How does this work?" It feels like Devin's very good at that.

**Scott Wu** (00:44:34):
Yeah. Yeah. Obviously, the retrieval and processing a lot of code and a lot of tokens at once is something that language models are really great at. Basically being able to get those gains in the places that you need them is really great. Yeah. Sweet, cool. 

**Lenny Rachitsky** (00:44:50):
All right. You got a couple more use?

**Scott Wu** (00:44:51):
Yeah. One last thought I'll show is just... We just rolled this out last week actually, but it's a full Devin automation setup with Linear. If you have tasks that you're doing on the DeepSeek repository, for example, and it's all set up, all you have to do is you just add the Devin label and Devin will come through and it'll give you this. It's going to give you its thoughts on what the tasks looks like and you can take a look at each of the particular files that you see, or it'll point out snippets that it thinks are important. From there, if you feel good about what was built or the conclusions that we came to, then you can just start off the Devin session that will go and actually do that work.

**Lenny Rachitsky** (00:45:30):
That is insane. That sounds like such a simple idea, but essentially what you're saying is there are tasks in Linear that are fixes and features and now Devin just goes off and can just do them for you.

**Scott Wu** (00:45:44):
Yeah. Definitely it's a hands-on process. You certainly want to be involved when Devin is scoping out the task or giving you its thoughts. The nice thing, too, by the way, is Devin will give you its confidence level. Here's how likely I am to really understand this piece or that piece or whatever, but it helps make things a lot faster. To your point, a lot of product managers, for example, obviously love to be able to use Devin in Linear to understand the code base better or things like that. Claire Vo, for example, from LaunchDarkly is a big Devin user and she loves basically going and scoping out tasks or asking data questions or asking, "Hey, what's going on? Or is this merged into production yet? Or is this a feature flag right now? Or what percent of people are getting this or that feature?" It's a clean way basically to make that intelligence much more accessible.

**Lenny Rachitsky** (00:46:40):
I love, just with the integration with Linear, that you can still keep it really simple. You add a little ticket like, "Hey, this link to this home page, do this," and Devin will be really good at understanding what you mean and then show you, "Here's what I'm thinking." Is this right?

**Scott Wu** (00:46:53):
Yeah. Yeah, cool. Okay. Yeah, Devin did finish working. It seems like there's something going on with the CI and it's debugging that right now, but it went ahead and put up the initial first pass pull request and we can take a look. 

**Lenny Rachitsky** (00:47:04):
Let's do it.

**Scott Wu** (00:47:06):
This is the Devin website, obviously, in this custom deploy and we have Lenny's newsletter right here.

**Lenny Rachitsky** (00:47:11):
Let's ship this to production. We won't be so confused.

**Scott Wu** (00:47:15):
Yeah.

**Lenny Rachitsky** (00:47:16):
That's amazing. Okay. Show it again real quick. Just added it to the home page of Devin.

**Scott Wu** (00:47:22):
Yeah. Devin, obviously, has access to our Devin code base. It does a lot here and so it's super familiar with all the pieces here.

**Lenny Rachitsky** (00:47:28):
Beautiful.

**Scott Wu** (00:47:29):
Yeah, I like how that looks. We've got Devin search, we got Devin [inaudible 00:47:33], and we've got Lenny's newsletter. 

**Lenny Rachitsky** (00:47:34):
[inaudible 00:47:34]. You link to my site. We'll get some PageRank going. 

**Scott Wu** (00:47:37):
Yeah, yeah, yeah. 

**Lenny Rachitsky** (00:47:39):
Okay. Is that a good example? Oh, there it is. What a beautiful website for my newsletter. Is that just a good example of the kind of thing Devin is very good at like, "Here's a very specific thing to change on the website"? How does your people think about what Devin is very good at and maybe where it starts to fall apart?

**Scott Wu** (00:47:55):
The way that we often describe it is, I think, Devin is best when it is working on tasks that are well-defined. One way to put it, you might...

**Scott Wu** (00:48:00):
On tasks that are well-defined. One way to put it is, you want to be giving Devin tasks, not problems. And a lot of these things like what you just saw, which was kind of like a quick front-end feature request or a bug fix or adding testing and documentation or things like that.

**Scott Wu** (00:48:16):
One of the things that makes a loop really nice obviously is a quick way to iterate and test. And so with something like this, obviously super easy for us, for example, to just go pull up the preview and see that the link worked. Obviously it would be easy for Devin to do as well. Devin will often go and log in to Devin and start a Devin session and make sure when it's working on our code base, which is kind of hilarious. But yeah, you generally want something that is kind of easy to verify and easy to test is the main thing. And you can work on bigger projects or bigger asks as well, obviously. But in that case you should certainly expect to need to steer Devin more to make sure it's going the right direction.

**Lenny Rachitsky** (00:48:55):
It's interesting because that's very similar to the way people talk about synthetic data and reinforcement learning, creating data that's very easy. There's a very definitive answer, yes and no.

**Scott Wu** (00:49:04):
Yeah.

**Lenny Rachitsky** (00:49:04):
It's very clear.

**Scott Wu** (00:49:06):
Yeah.

**Lenny Rachitsky** (00:49:07):
Okay. Let me ask you this question. What's something that you guys debated a lot as you were designing and building Devin?

**Scott Wu** (00:49:15):
I'll give a couple that comes to mind. One I would say is a question of, I'll call it how opinionated we should be. We had the workflows that we used to Devin for, which was very much as you can see for basically integrating to our Slack and GitHub making pull requests for us in our repos, responding to issue reports or things like that. And naturally we've had certainly a lot of other different things that have come up that folks have tried. I mean, we have folks who order their DoorDash with Devin, for example. Even we have folks, certainly a lot of people who are building cool websites from scratch or working on things like that.

**Scott Wu** (00:49:53):
Yeah, I mean it is been an interesting trade off for us where I think the way that I would describe it is in our product, certainly the large bulk of the features that we build are for this kind of making pull requests and engineering teams use case. But I think basically our kind of general stance with all the others is obviously if folks want to use Devin for that, that's great and we want to just make sure that they're fully aware about the limitations and about where things can get caught up.

**Scott Wu** (00:50:20):
It is funny with AI and especially because I would say one of, I would say the most common pieces of advice out there I would say is focus on a really niche cohort. Do things that don't scale, make one use case that's really great and then you grow from there. And I think that's great advice across the board. But yeah, it's kind of interesting because I think with generative AI, you naturally see this where a lot of product experiences can turn out to be more general. And so it's an interesting trade-off for us. This is something that we still always go back and forth on and how much do we want to do more to support all the other kind of use cases out there to handle other things that folks might want to do with Devin.

**Scott Wu** (00:51:03):
I think another one that comes to mind is how much Devin should be, let's say a single comprehensive project experience versus a suite of tools. And as you can see here, we have Devin search, we have Devin Wiki, we have the linear ticket scooping, and certainly these tools interact with each other, but I think as time has gone on, we've seen it more and more as really building this suite of tools. And I think the core agent experience and the core kind of agent that will go off and build each of these build things for you, for example, is always of course going to be that's Devin, and that is the core piece. I think that will always be what's really special about our work. But I think that all of the other features out there, there is a complex suite of work that's required for real-world software sharing and engineering is just messy at the end of the day.

**Scott Wu** (00:51:55):
And so I think there are a lot of different flows and a lot of different use cases that makes sense. And an obvious thing to point out is you could ask the same questions to Devin search as you could to Devin and Devin will go through and it'll do the same thing. It'll go through and look through the files and give you an answer and stuff.

**Scott Wu** (00:52:10):
But with that said, on the one hand, I think on the capability side, there's certainly a lot you can do to really optimize things for very specifically question answer about this repository and that made sense to really build into a specific kind feature.

**Scott Wu** (00:52:23):
And then on the other side, I would say we found that users actually really, really like having this access of control. Sometimes you have a task that you're thinking about, but you actually don't want Devin to get started on the task just yet. You want to ask Devin and understand what parts of the code base might be relevant. And so you want to be very direct about saying, "This is just an ask and I just want to see the snippets of the code base that relate," or, "I just want to look at the Wiki and understand the existing representation."

**Scott Wu** (00:52:50):
And so it's on both the capabilities and on the UX side, we've found that that's kind of what's naturally made sense over time.

**Lenny Rachitsky** (00:52:58):
Well, let's talk about the landscape then of just other companies in the space, which is something a lot of people are always thinking about. There's all these different approaches. You guys are going full on AI engineer, there's obviously ID companies. There's also just models being built that are really good at engineering. Everyone's kind of starting to build agents now. You guys are ahead on this in a lot of ways. OpenAI just recently said they're going to build a software engineering agent. Anthropics got something there. Cursor and Windsurf have their only agents and Replit. Thoughts on just where you guys fit in the landscape and then how do you think you win long term. How do you think about that?

**Scott Wu** (00:53:31):
Yeah, and for what it's worth, I think all of these are incredible teams. I think really smart and really forward-thinking folks who are building a lot of great products out there. And I think there's a lot to do honestly over the next few years with the advent of AGI or whatever you want to call it. I think one of the quotes that I love is in 2017, if you asked if we had AGI, the answer is no. And in 2025, if you ask if we have AGI, the answer is, "Well, you have to define AGI. And it depends on your subject." And I think it does get to the point of, I mean there is a lot of really amazing stuff happening. I think that it's easy to underrate, I would say just how big of a shift it is that we're seeing where I think there are a lot of great products out there, for example, over the last 10 years, 20 years, 30 years, that have made each of these different niches of the life cycle of building a product a little bit easier, for example, right?

**Scott Wu** (00:54:28):
There's great products out there for instant response, there's great products out there for logging, there's great products out there for billing, there's all of these different tools. And the obvious thing is what we're seeing with AI is all of these spaces are going to be moving multiple times faster and it's going to be like an order of magnitude shift, if anything.

**Scott Wu** (00:54:47):
And so I think from our perspective, we've obviously had a very specific lens that we've bet on this whole time, and that is autonomous coding agents. And there's a lot of problems to solve there, to be honest, right? There's still a ton to do on the core capabilities, certainly, and we see cases all the time where it's like, wow, why did Devin make that decision? That seems, no human engineer would've ever done that. There's all sorts of spots where, with the product interface, there's obviously a lot to think about.

**Scott Wu** (00:55:16):
And I think it's by the way, not just a single thing that we're working towards, but something that will change with every edition of capabilities. I kind of think of it as there's 20 generations of agent product, agent coding experiences to come. I think the one that we'll get to over the course of several years is probably something where you don't even look at the code at all, right? And you're actually just looking at your own product and you're just able to look and specify and say, "Hey, this button should be a little bit rounder. Let's do that. And by the way, let's add a new tab here and maybe we should save this information. Let's start up a database table and let's index it on X, Y, and Z columns." And you're just basically working with your products in real time and having your agent build out those things for you. Obviously there's going to be a lot of generations in between here and there, but I think the product experience itself is going to change every single time.

**Scott Wu** (00:56:04):
And then obviously there, there's all of the practicality of just getting it out there in the world. And so folks obviously need to learn how to use the new technology. There's a lot to do to deploy into all of the messiness of real world software. There's a lot of COBOL out there still. There's a lot of FORTRAN out there still. There's lots of kind of abstractions and details that folks have done.

**Scott Wu** (00:56:27):
And so I think from our perspective, we have since the beginning have been laser-focused on agentic coding, and that is the one thing that we've really believed in. It's the one thing that we've designed for and that goes all the way to even the revenue model with ACUs and having the usage-based setup. It does into obviously all the product experiences of thinking how, okay, where do you want to talk to Devin? You want to be able to talk to Devin in Slack, you want to be able to spin this up from your issue. You want to be able to all of these things and then of course the capabilities.

**Scott Wu** (00:56:58):
And so I don't think there's any one easy answer. I think it's obviously a combination of things, but this has been the space that we've lived in and spent all of our time in for the last year and a half, and it's going to be that way for the next five or 10 years too.

**Lenny Rachitsky** (00:57:14):
Along these lines, a big question everyone always has in AI's moats and defensibility, it's a question I've been asking every founder that comes on. How do you just think about how to build a moat in this space when it's so much easier to build and so much is built on these models that are themselves advancing so quickly?

**Scott Wu** (00:57:30):
I'd give one slight tweak on that, which is I think it's often less about moats and more about stickiness. And what I mean by that is moats are in some sense, typically what folks mean by moats is something that means that a competitor couldn't even enter the market. And I agree that at a high level, a lot of different folks at different layers of the AI spectrum, the foundation labs or the application layer or so on, I don't think there's any kind of hard barrier that would prevent others from entering. I think what does exist is stickiness, which I would kind of define as once you have a product experience that you really like, are you excited to keep using that experience or is there an effect where it is just as easy from now on to just switch on to a new one and learn a new one and so on.

**Scott Wu** (00:58:15):
And I think from that perspective, I think there's a few things that are really great about coding agents in particular. One I would say is there is a lot of just inherent stickiness and learning and buildup over time, which is that as you use Devin and as your whole team uses Devin, it's the same thing with an engineer. If you're joining on day one versus you've been at the company for five years, you wrote half the code yourself, you've touched every file you've built every single piece, you know all the engineers. And so similarly, it's like Devin will really learn and build its representation of your code base and of your stack and of your process over time and will be able to do a lot more with that.

**Scott Wu** (00:58:51):
And then the other piece of it, which I think is really exciting I'd say is there really is a lot to do of what I would call a multiplayer aspect of code, which if you think about it, is how a lot of things get done in the real world, certainly. And so it's one thing to have your own experience, which you use yourself as just an engineer, but for example, ourselves, we see this all the time where some engineers are working with Devin and teaching Devin things and as I mentioned, folks will have Devin on board their new engineers and kind of convey that knowledge to them.

**Scott Wu** (00:59:24):
Or similarly, it's like I'll start a session with Devin in Slack and I'll say, "Hey, it'd be cool if we could do this thing." And some other engineer will chime in and say, "Oh, by the way, the reason we did it initially was X and Y." And so Devin, just make sure when you do this change that you still support that workflow. And Devin will say, "Okay, sounds great." Or Devin will make a PR, I'll be working with Devin, we'll make pull requests and GitHub and somebody else will be reviewing that PR or give some comments and Devin will work on that too. You'll be in linear.

**Scott Wu** (00:59:52):
So all these kind of spaces, it really does just kind of set up for an experience basically where Devin can just grow in the value that it can provide for your whole work over time. And so I think from that perspective, if anything, we want there to be a lot of innovation and a lot of new products and so on. I don't think that the goal is to try to lock other people out of building. There's a lot of stuff to build, and I think there's going to be a lot of different experiences. I think from our perspective, what we think about is more like how can we make Devin more and more and more useful as you're using it more?

**Lenny Rachitsky** (01:00:27):
It's very similar. We had Michael from Cursor, the CF Cursor on the podcast, and he had a similar point of just he thinks moats are just kind of like consumer, like Google is the way, he thinks it's like Google where people can easily switch. You just have to stay the best, and that's the answer. And it feels like you're adding to that of just like, but also if you can create some stickiness where it is very hard to leave because it's so good at what it's doing and it's built knowledge and integrated to your workflows and builds on that on that stickiness.

**Scott Wu** (01:00:58):
Yeah, and I think one of the things that's nice about our space too is, software engineering for better for worse has a very clear tie to value. And what it means is, I guess one way to put it is there is always kind of a clear next level, at least for the next while. I think there could be some point where you're just like, "All right, just build the entirety of YouTube for me." And Devin just does the whole, it's like there's probably been a hundred million hours of human engineering time building YouTube, building the algorithm, building all the infrastructure, all of the, everything, every little detail. And maybe there's some time where Devin just does that out of the box. That's obviously going to be a long time from now.

**Scott Wu** (01:01:36):
I think on the interim, on every level in between, obviously it makes a difference the quality of software engineering. And I think one of the cool things with developers obviously is developers are really willing to learn new experiences and to put in effort if it means that they're able to have a higher and higher quality experience.

**Lenny Rachitsky** (01:01:57):
Awesome. I'm going to spend a little time on the tech that enables Devin. Without divulging trade secrets, just what allowed you to make Devin so good? Was there an unlock with a certain model? Some folks have shared three points on a 3.5 was a huge unlock for a lot of their products. Just what's kind of the key to the way you've architected or built Devin that makes it work so well?

**Scott Wu** (01:02:20):
We obviously, we've been betting on agents for a long time. I think that agents were doable and workable a lot earlier than most folks might've thought. But certainly I think as the community has really rallied around it, I mean you see the impacts of that in the pre-training, you see the impacts of that in a lot of the work that's done with these models. I actually don't think there's been any, from our perspective, I don't think there's been any single step function based model shift or anything that has been kind of like a night and day difference in Devin, but I certainly think that the curve of every point on the chart, I mean there's a new model that comes out every week now has obviously made a big difference in terms of what we've been able to do. And then obviously on top of that, we work with the research teams at all these foundation labs to do a lot of our work on top.

**Scott Wu** (01:03:11):
And so I think that my hot take here at which I would give is, I think in terms of base intelligence, we're honestly basically already there. And I think a lot of what we see actually and what we spend our time on is less so, obviously, we don't our own models or things like that. It's less so increasing the base IQ of a model, for example, and more about teaching it all of the idiosyncrasies of real-world engineering and thinking about here's how you use Datadog and do this, and here's how you might diagnose this error and here are the different things that you could run into and here's how you handle each of those. And when you're ready, here's how you make GitHub PR.

**Scott Wu** (01:03:50):
And this is true in engineering. It's true in every other space as well. I mean, there's so much detail and idiosyncrasy to the work that we all do obviously day to day. And a lot of it is kind of like teaching the model to mirror the complexity of the real world, I would say, rather than getting it to some higher fundamental level of problem solving, which I think the foundation labs are doing a really great job.

**Lenny Rachitsky** (01:04:15):
There's something you shared when we were chatting before we started recording around the growth of previous transformative technologies were very hardware oriented and there was a limiting factor to their growth and AI is not that. Can you just share that insight?

**Scott Wu** (01:04:30):
For a number of reasons? I think AI is going to be the biggest technology shift of our lives, but I think one thing, which is what we were just talking about before this, which is most of the big tech revolutions that we've had over the last 50 years, I mean, I'm thinking about personal computer and the internet and the mobile phone and stuff, they all had this big hardware component that was a big part of the distribution. And so you had the internet, and initially it was just these universities that were talking with one another, but obviously over time we got the whole world plugged into the internet and it took years and years and years. Same thing was true with mobile phones. Same thing was true with PC.

**Scott Wu** (01:05:06):
And the thing that's interesting about that in particular, which is I would say we're already seeing the effects of that, is in these hardware distribution machines, obviously there's a lot that depends on real time. And so folks who were building for those industries saw their market grow and grow and grow basically steadily year over year as the number of people with mobile phones increased, as the number of people connected to the internet increased. And many of those businesses, it's still crazy to think, but many of those businesses got started right in the beginning. I mean, Apple and Microsoft were started right around the same time. And same is true for a lot of the great internet businesses or wherever, but certainly it was something that touched whole world with time or a large fraction of the whole world. And it had a really massive impact, but it took place over several years because of the time that it took. And I think one of the things which is already I'd say different in AI, is just how explosive the technology can be. Once AI could, and I think we're firmly past the inflection point in AI code where it's, as an engineer, if you're not using AI at all to write code, I mean you're falling behind honestly. And it is a technology that everyone should have and should be using, and there's no kind of weight on hardware distribution that is causing that. It means that the space is just growing so exponentially, basically.

**Lenny Rachitsky** (01:06:36):
Michael Ballin has this interesting point that cliches are cliches because they're so true and that's why they're cliches. I heard that a million times and I think it's like people hear this like, "Yeah, yeah, I know," but it's actually insane what is happening. That's why you're here to help us through this transition.

**Scott Wu** (01:06:52):
Yeah, no, I mean, it is a fun time and I think there will be real investment and real work that it takes. But I think from the perspective of us as engineers, for example, I think it just means it's so important to stay in the loop with everything that's happening. And as we're seeing it's not only because of your learning and your ability to work these technologies, but it's also about basically teaching the AI what there is to know about your code base in order to make it really effective at building with you and doing more of the things that you would want it to do.

**Lenny Rachitsky** (01:07:27):
So along those lines, for people listening that they're like, "Hey, we should be using Devin at our company," what are things you've found to be helpful in helping an engineer at a company get adoption and be able to use Devin either culturally or logistically?

**Scott Wu** (01:07:42):
So a pattern which we often see with folks is there will be a few folks at the team who are really excited and want to try out the new thing, and they want to put in the investment and are really excited to get it going. And they'll go through all the setup. They'll give Devin the repos, they'll teach Devin how to run the lint and the CI and all of those details. And they'll start off by giving it those initial tasks and help Devin build a foothold basically. And as time goes on, eventually folks will see, "Wow, Devin's writing all these PRs, Devin's doing this [inaudible 01:08:14].

**Lenny Rachitsky** (01:08:13):
Who's this Devin person that just joined the company's just knocking out PRs?

**Scott Wu** (01:08:17):
Yeah. And they'll see that, and then naturally they'll get on and they'll get an account. And one of the cool things of course is by the time they join, Devin already knows a good amount of detail about the repositories that they're already working in and they're working with that. And so one of the really cool things which we often see is that the early adopters themselves can really pave the way I think, for everyone else on the team.

**Scott Wu** (01:08:37):
But yeah, I think the main thing I would just kind of call out is it really does take, it's a very different product experience, and I think for what it's worth, I think there's still a lot more that we can do to make it as intuitive and as clear as possible to folks like how to use Devin and what the right steps are and how to really maximize value out of Devin. But I think that, yeah, it's the kind of thing where if you put in the investment and understand exactly what it takes to get Devin to be successful, we've found ourselves that as time has gone on, we just use Devin more and more and more with every next update.

**Lenny Rachitsky** (01:09:15):
So let me follow that thread. There is a question I ask every AI app building founder, which is, if you could sit next to every new user of Devin and whisper something in their ear to help them be successful with Devin, one or two tips, what would those tips be?

**Scott Wu** (01:09:31):
I think the biggest thing I would say is it really is just treat Devin like your new junior engineer. And I think that's the biggest thing. I think folks come in and they see the blank page and they think of all sorts of various things that they want to try out. They think of lots, where I think typically the flow that we see that works best is obviously you can try demos and you can do things, but a lot of it is just like, "Yeah, let's figure out what tickets we want to get done today or this week and let's have Devin get started on those and let's start with the easier ones and then work with Devin and understand what things Devin needs to get set up to be able to test its own code and do this well. And then let's scale up over time." And then obviously as you work with your engineer, you understand better how to communicate with them or what are the right tasks or projects to bring them in on. But I think that really is the one-liner for us.

**Lenny Rachitsky** (01:10:31):
Okay. There's a question I'd be meeting task. I just want to get back to this because it's something I think a lot about with Devins. Everyone's going to have five Devins, let's say 10 Devins. Everyone's kind of turning into basically an engine manager with a bunch of junior engineers, which isn't necessarily the best job in the world because it's just a bunch of, at least you don't have to do performance reviews and one-on-ones, but it's sitting around, checking a lot of PRs all day. There's a sense of you become an architect, which is kind of what every engineer wants to become eventually, right? They're all, "I just want to think about the architecture. I don't want to code all these stupid, fix bugs."

**Lenny Rachitsky** (01:11:06):
So I get that there's a good side to that, but just I imagine you're thinking a lot about this, just like how do you make life pleasant and fun and enjoyable as basically an engine manager of say, 500 Devins in the future?

**Scott Wu** (01:11:19):
Yeah, I can just imagine the performance, "Devin, you've done a really great job on your task, but I really would like you to be more proactive in the team meetings." So what I'd say-

**Lenny Rachitsky** (01:11:29):
[inaudible 01:11:29].

**Scott Wu** (01:11:29):
It's funny actually because something that in terms of the wording that we thought a lot about as well is just, we've used the term manager of Devins in the past, which of course I think is a big part of it. But I think that the only thing I would point out here is I think that the bricklayer versus architect is closer to the experience than being a manager. Because I think a lot of the difficulty of management or the reason that people shy away from it is more because of a lot of the various.

**Scott Wu** (01:12:00):
... [inaudible 01:12:00] because of a lot of the various, let's say... There's all of the context, and the ownership, and the responsibility and stuff, and then there's also all of the emotional aspects of it. Where, I think working with Devin is a little more like just being, more as having an interface to hand off tasks and build tasks. And so, the parallel that I would draw is when we invented Python, obviously... It's like, in many ways the description and the outlining of tasks, obviously it was a different paradigm, but I think certainly it was nowhere near what folks typically think of as management bureaucracy today.

**Scott Wu** (01:12:50):
And I think that with Devin, a lot of it is just like, it's more like finding the right levels of abstraction that you could work with Devin on, and just finding the workflows that work really well, and the obvious thing to say here is, it's like you can always have Devin take a first pass at things. And so you have Devin take the first pass, if it's great, you merge it right away, if it needs some touch up, you can obviously give that feedback, for example, but a lot of it is like, it's more about, basically making Devin part of your flow than it losing control, which I think is the main thing that folks are scared of with management.

**Lenny Rachitsky** (01:13:34):
Are you thinking about a manager Devin? Like a Devin that manages other Devins?

**Scott Wu** (01:13:37):
Yeah, yeah. So, for what it's worth, Devin can start other Devins through the API, right? And so, we've seen this happen quite a bit of times, where naturally if you have some big tasks that you want to do, Devin will do this all the time, it'll chunk up and then [inaudible 01:13:53] into smaller Devins. And so, it's the kind of thing that you need to give Devin the credentials to be able to do that, it's not currently something that is default enabled, but I can certainly imagine as time goes on that there's more and more of that

**Lenny Rachitsky** (01:14:03):
Devins all the way down.

**Scott Wu** (01:14:05):
Yeah, yeah. I think the thing that's kind interesting too is, with humans, the way I almost say it, in technical terms, it's like there's this coupling of a context and a thread, and what I mean by that is basically, each human can only operate a single threaded on the work that they do, and they have their set of context, and then there's other humans obviously who can do other stuff at the same time, but they have their own context. With agents, one of the cool things is you can have an agent that's doing multiple lines of exploration at once, but is sharing all of the context of everything that they find, and so I think that this is very early, and I think we'll see this, but folks obviously love to talk about basically systems of agents communicating with one another, and I think that there will be a lot of new paradigms to build for, once we get there.

**Lenny Rachitsky** (01:14:55):
And it's so interesting what you said about the decision between having one Devin and only one Devin do all the things, and you just tell them things and they fire off jobs versus you have five Devins, and they're each doing individual things, it's such an interesting decision to make.

**Scott Wu** (01:15:10):
Yeah, yeah, yeah, for sure.

**Lenny Rachitsky** (01:15:12):
Okay, two more questions. Maybe the most counterintuitive thing you've learned so far building Devin, that maybe goes against startup wisdom, common startup wisdom?

**Scott Wu** (01:15:21):
Something I've thought about a lot lately as we've built this is, this is not my first company, actually for a lot of us, it's not our first company, I think of our 26 or 27 people total on the team, I think 18 of us have started our own company before this. And yeah, one of the things I think about is, there's actually your point about cliches I think really spoke to me as well, which is, there's the really common things which you hear all the time in startups, where you're like, you got to move fast, or you got to hire great people, it's like, okay, well, obviously you do, I wasn't planning on not hiring great people, I wasn't planning on going slow. And similarly it's like, yeah, you really got to build something that people want. And there's these three to five things which are always repeated, and they're always the common wisdom in startups.

**Scott Wu** (01:16:11):
And I definitely had this idea as a founder, when I was starting initially, that, all right, so those are the three to five basic things, but as you get really deep into it, you spend a lot of years into it, you learn all of the thousands of other things that you have to learn to build a company. And I think to some extent that's, of course, true, and there's lots of little details that you'll get into with all these different things, including team building, and product, and strategy, and engineering decision-making, and fundraising, and sales, and every other component. But I also realized that as time has gone on, more and more, I felt like building companies well sometimes just comes down to doing those three to five things just even more than you could possibly expect.

**Scott Wu** (01:16:58):
And so, with us, it's like... And everyone says we go fast, but it's like, yeah, we had a hackathon in November, we had another hackathon in December, we started the company officially in January, we got the prototypes out to initial users in February, we did a launch in March, we got our first customers in April... It's just like, basically truly pushing the pace in every spot where we possibly could has really made a difference for us. And similarly, it's like, yeah, everyone always says you should hire great people, but I think that the truth within that truth is basically you should fight to all ends basically, to get the folks that you really want to bring in. And one of my favorite stories to share is we had a candidate who came and interviewed, he was a junior at MIT, so he was very, very young, and we gave him an interview, and he did way better than almost any of the full-time candidates that we had ever talked to. And so, we said, hey, what do you think about taking some time off of school and working with us and building out Devin?

**Scott Wu** (01:17:57):
We really think you're just going to be able to come in and just have a ton of impact already from day one. And he thought about it for a while, and he came back and he said, you know what? I'm down, I want to do it, but my parents really want me to graduate from school, and I'm just not sure there's a way to make it work. And so, we talked him more and just understood the situation, and then we flew to North Carolina, went straight from the airport to his parents' house, had dinner with him and his parents, we talked a lot, a really nice Gujarati family, we gave them some gifts and just talked to them about it, and try to understand what would it take, and what would we need to make work. And they just said, it sounds like a great opportunity, but we really want our son to be able to graduate.

**Scott Wu** (01:18:45):
And we talked that through and we figured out a setup basically, where he could work for us essentially full time, but then come in for his required classes, and do what he needed to do to get the diploma basically, but no more than that. And we talked that through, and then we got to a point where everyone was happy with that, and then went straight back to the airport and flew right back, basically. And that was the first and only time that I've ever been in North Carolina, and it was just a great trip. And it is the kind of thing where it's like, hiring great people is one thing, but truly just never giving up, and really giving it everything that you can to make it work for people who really makes sense to be on the team. And he's been with us on the team for over a year now, and he's been an incredible, incredible engineer, and we wouldn't be here without him.

**Scott Wu** (01:19:31):
And similarly, we had someone else who was again, really, really talented candidate, did amazingly well, very young, and had a lot of great offers at a lot of other companies, and we were talking to them about, he wanted to start his own company someday as well, and we were talking to him about certainly a lot of the obvious things, which are having him meet our investors, or get to do work with customers, or see a lot of these other components so that when the time came that he would have all the experience he needed to start his own company. But one of the other things that was big is he was talking with a lot of great companies already, he didn't want to burn any bridges, and so we actually worked with him and basically hand wrote all of his rejection responses to each of the other companies, and worked with him on it to say, here's how you should say it in a way that's going to come off as that you really did appreciate the time with them, and that obviously you want to remain close with them and stay in touch. And it was the kind of thing obviously where it's like, look, obviously our job is to make sure that he's happy enough that he doesn't want to leave at any time in the near future, but I think it's the kind of thing where the way that you put together a really, really great team is by fighting for what's right for them too.

**Lenny Rachitsky** (01:20:45):
Wow, those are incredible stories. And it makes so real these, as you say, cliches, hire the best people, this is what it sounds like to hire the best people. This is what it takes.

**Scott Wu** (01:20:56):
Yeah, no, and I was just saying, a lot of things we've fought very hard to just reimagine things from the ground up because a lot of it really is just thinking about where do we think the technology is going over the next five, 10 years, and what is the place that we want to have in that future?

**Lenny Rachitsky** (01:21:16):
I wonder if people are going to be fighting for the best Devin someday. They're just going to [inaudible 01:21:19] Devins.

**Scott Wu** (01:21:18):
Yeah.

**Lenny Rachitsky** (01:21:19):
They're [inaudible 01:21:22] so smart.

**Scott Wu** (01:21:22):
I'll give you overtime pay, benefits, free healthcare, and everything, and then the Devin's like...

**Lenny Rachitsky** (01:21:29):
Three Devins like Magic: The Gathering cards.

**Scott Wu** (01:21:31):
Yeah.

**Lenny Rachitsky** (01:21:32):
And then just going back to your three to five things. So, essentially this is incredible advice, essentially, it's like you always hear hire the best people, move fast, build things people want.

**Scott Wu** (01:21:41):
Yeah, build something people want, stay as close as possible to your customers, and then I think the other thing is just always think about where things are going, not where they are today. I feel like those are the five things, which is... Especially in AI, with things moving so fast, and there's so much great talent, I feel like a lot of these are even more true, where it's like it's not just thinking about where things are going to be in 10 years, it's like thinking about what's going to happen next week. And obviously, things are moving very quickly, and it is very hard to predict, but you really have to be very rigorous with yourself, I'd say, about thinking through those things and evaluating all of the decisions that you make in that lens.

**Lenny Rachitsky** (01:22:25):
And staying focused is the big takeaway to me here, is it ends up feeling like there's 1000 things you should do, but it's always these five things.

**Scott Wu** (01:22:31):
Yeah.

**Lenny Rachitsky** (01:22:33):
Scott, we covered a lot of ground. We went through every question I had, which is great, is there anything else that you want to share? Anything else you want to leave your listeners with, maybe a final nugget or something really you want to double down on that we said before we let you go?

**Scott Wu** (01:22:49):
The biggest thing that comes to mind for me is there's a lot of different perceptions about AI, right? I think basically every emotion under the sun right now. There's a lot of fear, for example, there's also a lot of skepticism, and we're very skeptical types as well, and we always wanted to try it ourselves to really see it and believe it. And I think the main thing that comes to mind for me, is I'm honestly really optimistic about what we're building here with AI, not just with code and with Devin, but the whole space, and everything that's getting done. And I think one of the cool things that is really actively happening is just the ability for everyone to multiply themselves, and that's how we've always thought about it, it's how we've thought about what we're building. And I think there is a lot more to do out there in the world, I'm not too worried about us running out of things to do, and from that lens, I think that the thing that we've always been most excited about is, how can we all do more?

**Lenny Rachitsky** (01:23:52):
I hear you Scott. Well, with that optimism, we've reached our very exciting lightning round. Are you ready?

**Scott Wu** (01:23:59):
Yeah, let's do it.

**Lenny Rachitsky** (01:24:00):
Okay, here we go. First question, what are two or three books that you find yourself recommending most to other people?

**Scott Wu** (01:24:07):
In terms of nonfiction, I think for folks in startups, I think one of the things that I've really enjoyed is just learning and understanding the history of Silicon Valley. And it's all of these things that we think about, somebody invented them. It's one of the great realizations I feel like, is somebody invented the idea of a seed route, somebody invented the idea of venture capital, somebody invented the idea of product-market fit, and all of these different principles that we talk about. And so, for that, there's a book called The Power Law by Sebastian Mallaby, which I really like. And basically, it's just a tour of many of the great businesses and the great products that have been built over the last 60, 70 years in Silicon Valley, which I really love. I think in terms of fiction, I actually have always really liked The Great Gatsby by F. Scott Fitzgerald, it's one of my personal favorites as a fiction book.

**Lenny Rachitsky** (01:24:57):
Do you have a favorite recent movie or TV show that you've really enjoyed?

**Scott Wu** (01:25:01):
I have to admit, I have not watched... I can't think of a single movie or TV show that I have watched in the last while, so I'm sure, I'm looking forward to watching a lot of great ones post-AGI.

**Lenny Rachitsky** (01:25:17):
That's got to be in the trailer, that's great, I like that. And that just shows how hard you're working, just how much shit is going on, and how fast everything's moving. Do you have a favorite product you've recently discovered that you really love? Could be an app, could be something physical, could be a toothbrush.

**Scott Wu** (01:25:32):
One I would say is I got an Aura frame recently, it's just like a frame that shows photos, and you can show a new photo every day, or every hour, or every 15 minutes, or whatever you like. I've actually, I've really enjoyed it a lot, I think it's a nice way to basically just have a picture frame memories that come up. And then, the other thing I would say as a general purpose thing, it's not particularly new, but I would say, I think AirPods are extremely well-built and well-designed. And I realize now that it's like, I basically use them for all... I'm taking calls on a walk and I'm using AirPods, obviously I'm doing work at my computer, at my desk, I'm plugged into AirPods, and it works quite well, honestly, for a lot of different situations, and they're very comfortable, they're very consistent.

**Lenny Rachitsky** (01:26:21):
I'm going to double down on the Aura frame, I also, I got one of these for my mom and my mother-in-law, and they're so great for just sharing photos of your kids with your family. And people have, they've heard of digital picture frames, but the Aura just does it really well, and it's really easy to add photos, and they're just really nice looking.

**Scott Wu** (01:26:39):
You can imagine not that long from now, we'll have the Aura Frame except Studio Ghiblifies every photo that you have in it, and then it's... Yeah.

**Lenny Rachitsky** (01:26:47):
Or just imagines things you've done that are really cool. Look at my sweet life. Yeah. Cool. And it's Aura, it's A-U-R-A, I believe is how you spell it. Folks who want to check it out, we'll link to it. Not affiliated. Okay, two more questions. Do you have a favorite life motto that you often come back to and find useful in work or in life?

**Scott Wu** (01:27:06):
Yeah, something I've thought about a lot is a lot of the proverbs out there are actually contradictions, right? It's like birds of a feather, and then you also have opposites attract, and you have all... And it's kind of funny, because you feel that both of them are true, and often they both are true, and a lot of it is about understanding why. And one of those that I feel like, especially in the world of startups that I think about all the time is, I think it is very important to be focused and driven, and to really maximize your potential, and then at the same time, it's also very important to not let your own personal emotion get tied up in your success or failure, and I think especially with startups, because there's always ups and downs, honestly, even in the most successful companies ever, it's just like, it's a rocky road, there's a lot that happens, and a lot that goes down.

**Scott Wu** (01:28:03):
And I think one of the things which I've thought a lot about is that somehow you really want to do your best, and put everything you can into it, and do everything you can to... Basically, you want to put it all out on the field. But at the same time, you want to be okay with both wins and losses, and you want to be able to move on, and go into the next one each time. And something, yeah, it's funny, but what I've found personally is that obviously it's really important for your own emotional state and mental state to be able to do that, and we've had lots of mistakes, and I've had a lot.

**Scott Wu** (01:28:49):
I had my first company, which obviously, which was cool, but there are a lot of tricky spots there, and then over the course of Cognition, it feels like it's been already eight years compressed into one year, and it's still going at that pace. But somehow it also actually makes you more successful, I think, too. It's like, you are just more able to give it your best, and to do the things that will lead to success if you're not tying it up in your own personal worth.

**Lenny Rachitsky** (01:29:19):
That is so interesting, I just had a podcast recording recently where with an executive coach, Jerry Kelowna, that I think will come out before this, might be after this, that's one of his big pieces of advice, and it's a very Buddhist approach of just [inaudible 01:29:33] and attaching to a certain outcome.

**Scott Wu** (01:29:35):
Yeah.

**Lenny Rachitsky** (01:29:36):
Okay. Final question, I'm curious if there's a story here, but we could keep it short, is there a story behind Devin as the name, or is there another contender for Devin being the [inaudible 01:29:48]?

**Scott Wu** (01:29:47):
Devin was the name from pretty early on, we were interested, we were working on coding agents from the beginning, and my co-founders are Steven and Walden, for example, and we had this idea, all right, let's get started, and let's try to expand the box as much as we can. So, have everyone think out of the box and do their own thing, and let's have everyone do their own thing first for a bit, and then we'll consolidate and take everything that we've learned. And so, Walden made a virtual developer version of him, which was called DevWalden, and then Steven made one of him, which is called DevSteven, and we had all these... And then we were kind of combining it all into one thing, and we're like, okay, you know? It's Devin. And that was the thing. And so Devin was, yeah, Devin stuck for us quite early on, I would say.

**Scott Wu** (01:30:31):
One thing which we did have a big decision on though actually is what the image of Devin would be. And so, as folks know, there's the hexagons and then people have seen this more recently, but there's actually also an otter, a little otter with a laptop in its lap, and that is Devin as well. And we had this debate over what to go with, and what not to go with and stuff, and it's been a while now, but somehow we still have both the hexagons and the otter.

**Lenny Rachitsky** (01:31:01):
You skipped over where the Devin, did you have just have, it just came to you?

**Scott Wu** (01:31:06):
Oh, so Devin is, it's a dev. Yeah.

**Lenny Rachitsky** (01:31:10):
[inaudible 01:31:10].

**Scott Wu** (01:31:09):
And so, it was kind of like, when we were consolidating all the names, it just seemed clear then that this would be the universal dev that we all liked to work with.

**Lenny Rachitsky** (01:31:17):
Wow.

**Scott Wu** (01:31:18):
Yeah.

**Lenny Rachitsky** (01:31:18):
Incredible. Scott, this was so much fun. Oh my God, I learned a ton, which is always a really good sign. Two final questions, where can folks find you/Devin/anything else you want to point them to, and how can listeners be useful to you?

**Scott Wu** (01:31:29):
Awesome. Yeah, no, we're at App.devin.ai, and you can find us as well on Twitter or a lot of other social media. We'd obviously love to hear any feedback you have about the Devin product, there's so much to figure out, and I think the, like I said, I think we're all still 20 steps away from really the future of software engineering, and so it really means a lot to hear what folks think about the product as they're trying it out. And so, please let us know anytime if there's things that we can do to make it better.

**Lenny Rachitsky** (01:32:00):
Scott, thank you so much for being here.

**Scott Wu** (01:32:02):
Thank you so much for having me. I had a great time.

**Lenny Rachitsky** (01:32:04):
Me too. Bye everyone.

**Lenny Rachitsky** (01:32:07):
Thank you so much for listening, if you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at LennysPodcast.com. See you in the next episode.

---

## [14/18] $46B of hard truths from Ben Horowitz: Why founders fail and why you need to run toward fear (a16z co-founder)

**Ben Horowitz** (00:00:00):
The worst thing that you do as a leader is you hesitate on the next decision. The thing that causes you to hesitate is both decisions are horrible. Probably one of my bigger ones on that was we went public with $2 million in trailing 12 months revenue at 18 months old. That's obviously a bad idea. But the truth of it was the alternative was going bankrupt, and that's a worse idea.

**Lenny Rachitsky** (00:00:23):
It's a very difficult and painful to be a CEO, to be a founder. In spite of that. So many people want to start companies.

**Ben Horowitz** (00:00:29):
The psychological muscle you have to build to be a great leader is to be able to click in the abyss and go, "Okay, that way's slightly better. We're going to go that way. If everybody agrees with the decision, then you didn't add any value because they would've done that without you." So the only value you ever add is when you make a decision that most people don't like.

**Lenny Rachitsky** (00:00:47):
You are famous for writing one of the most popular pieces of literature for product managers.

**Ben Horowitz** (00:00:52):
What I was trying to get out in Good Product Manager, Bad Product Manager, was the job is fundamentally a leadership job. And it's a tricky leadership job because nobody is actually reporting to you.

**Lenny Rachitsky** (00:01:06):
There's always this kind of sense that the PM is not the mini CEO. How dare you call yourself that? I actually think that's exactly what the PM is.

**Ben Horowitz** (00:01:12):
It doesn't matter if you write a good, spec or you have a good interview or you do this or do that. What matters is that the product works.

**Lenny Rachitsky** (00:01:21):
Today my guest is Ben Horowitz. Ben is the Z in A16Z, the world's largest venture capital firm with over 46 billion in committed capital. They're investors in OpenAI, Cursor, Andrel, Databricks, Figma, basically every generational tech company. He's also the author of two New York Times bestselling books, the Hard Thing About Hard Things and What You Do is Who you are.

**Ben Horowitz** (00:04:17):
All right, thank you Lenny. Excited to be here.

**Lenny Rachitsky** (00:04:19):
I'm even more excited to have you here.

**Lenny Rachitsky** (00:04:21):
I want to start with a question that a close friend of yours suggested I ask you, Shaka Senghor. So Shaka, he's, we could do an hour just on how interesting this guy and the things he's done.

**Ben Horowitz** (00:04:32):
Three hours on Joe Rogan that day. He is very-

**Lenny Rachitsky** (00:04:35):
So we're not going to do that. Just to give a glimpse, he was in prison for 19 years. He was in solitary for seven years. He led a huge prison gang. You wrote about him in your book as a great exemplar of great culture in the prison gang that he ran. So interesting. But something that he learned from you that he told me I need to ask you about is about success and how to be successful and how it's not what people think. And he said that you learned this lesson from a pilot. What is that story? What is that lesson?

**Ben Horowitz** (00:05:08):
I mean, I would say it's a long life lesson. But the pilot story is I actually, I ask people silly questions sometimes when they meet them. And so I met this gentleman who was a pilot and it was right around the time JFK Jr. crashed his airplane and ultimately died. And I asked him, I was like, "What happened?" Because there's always the story in the press, and I know this from them writing about me or anything, is it's always what's the best narrative not what's true. So you can never actually find out what happened, you just find the best story version of what happened.

**Ben Horowitz** (00:05:53):
And the story in the press was all about, "Oh, he wasn't trained on instrumentation was flying at night." And I wanted to know is that right? And the pilot said, "Well," he said, "really, it's like all plane crashes are a series of bad decisions. And none of the decisions by themselves is that bad, but when you add them up, it's bad."

**Ben Horowitz** (00:06:14):
So the first decision was he needed to get wherever he was going and that was the priority. And in flying, that can't ever be the priority because there are conditions, there are things that happen. And then the second one was, "Well, his timing of when the sun would go down was wrong." So he thought he'd be flying in sunlight and he wasn't. And then once he got up there, it was when the plane was going down making it go up was a bad decision because he was upside down. And so it was like, I can't remember all the things, but this guy had 17 bad decisions in a row. And the big thing for me that I felt was really true is it's one decision leads to another. And so if you can break psychologically, you can take the sunk cost, then that gets you out of a lot of bad paths. And then a little good decision may be difficult, but you have to believe it's going to lead to the next one. And a lot of success is about that. It's a small thing, a small thing that's hard to do that doesn't seem to have a high impact, but it leads to the next small hard to do thing and then eventually you get an outcome, so that was kind of the concept.

**Lenny Rachitsky** (00:07:46):
So the lesson there is just success is just a bunch of little things. It's not this, "Cool, I got here in a big thing."

**Ben Horowitz** (00:07:52):
If somebody were to write a story about me, they would be like, "Then Ben this really smart thing and blah, blah, blah, happy ending." But it really wasn't like that and I don't think it's like that for you or anybody. And I spent a lot of time with Shaka on how, because it's always your own psychology that gets you. And one of the most insightful things he said to me is, because most people who are in solitary for seven years, that's it. You're insane. You're never coming back from that. It's just an impossible thing. But if you study his story, he actually really was massively self-improved coming out of solitary, and he wouldn't recommend that for anybody, just to be clear. It wasn't solitary. But what it was was he changed, in solitary he was able to change a big set of beliefs that he had about himself that got him out of that.

**Ben Horowitz** (00:08:50):
And the thing that his conclusion from it, which I thought was really interesting, he's like, "Look, was in prison for 19 years. I was in solitary for seven. I come out, I can't rent an apartment, I can't vote, I can't get a gun, I can't do, no rights. None of that was anything compared to what I did to myself."

**Ben Horowitz** (00:09:08):
And I think that's very true for CEOs in general and people in general is all the things that you perceive that are happening to you that are bad, be it the systems against you or somebody undercuts you or racism or sexism or this or that or the other is very small compared to ... It means a lot if you believe it. If you believe what people say about you, if you believe what they did to you, then that destroys you. But if you go, "That's not me," you can overcome almost anything. And he's got a new book out on that anyway that I think is very good because that's the, I'd say more than anything, that's the key to success.

**Lenny Rachitsky** (00:09:52):
If you look at all the writing you've done, it's essentially about the struggle and pain and suffering of being a CEO, your first [inaudible 00:09:59] to Hard Thing About Hard Things. There's a lot of talk these days about just how important struggle is and how valuable it is to go through struggle. Jensen's big on this. He talks a lot about just you have to go through pain and suffering to be a great leader.

**Ben Horowitz** (00:10:12):
You don't really have a choice. That's true.

**Lenny Rachitsky** (00:10:15):
There's something that I saw you share that I love, which is running towards fear versus running away from fear. Something that you tell all your leaders to work on. Easy to hear, hard to do. We don't like doing things that are scary, running towards things that are scary. Why is this so important? Why is this something people need to learn to do?

**Ben Horowitz** (00:10:33):
Well, so the biggest mistake that you make, the worst thing that you do as a leader, there's things in your control and there's things out of your control and hesitation, that's generally the most destructive. And I go through all the ways that it's destructive, but it's extremely bad. And the thing that causes you to hesitate is both decisions are horrible. It's that business school where you're going through a case study, "And if you had done that, then the company would have gone this way. But if you had done that, it's a great success." That's not actually what happens to you as CEO.

**Ben Horowitz** (00:11:16):
What happens to as CEO, it's like, "Okay, if we rearchitect, this product, the architecture is not actually get us to where we need to go. I kind of know that. But if we rearchitect it, we're going to probably miss all the features, miss the quarter, have trouble raising money, shudder, et cetera. So that's really bad. And then not rearchitecting is really bad, and so I'm just going to try to and avoid this subject because I don't even want to deal with either of those." And that's the worst thing, because if action is the better choice and that's good. And then if you don't make an explicit decision, then the whole company's going to get nervous because they know that the architecture is whack and you got to fix it.

**Ben Horowitz** (00:12:07):
And probably one of my bigger ones on that was we went public with $2 million in trailing 12 months revenue at 18 months old. That's obviously a bad idea. I mean, there's no question that wasn't a bad idea. But the truth of it was the alternative because of where the private markets were was going bankrupt. And that's a worse idea.

**Ben Horowitz** (00:12:34):
And if you look at that time, March of 2001 when we went public, you just look at the number of CEOs that hesitated on that and didn't do it and went bankrupt. It's a lot. And so that getting good at making a decision that everybody's going to go, "Wow, that was insane, Ben." The Wall Street Journal wrote a whole long story about how stupid I was. And then Businessweek wrote a story called the IPO From Hell. That was the name of our idea, the IPO from hell, which was accurate in a sense.

**Ben Horowitz** (00:13:13):
So that's really bad, but it wasn't as bad, and this is why it's so scary you make that decision that's going to happen. I knew those stories would get written, there was no question. And yeah, that's the kind of muscle. So if you think about the psychological muscle you have to build to be a great leader is to be able to look in the abyss and go, "Okay, that way slightly better. We're going to go that way. And it's very hard to do." I would say it's a thing people struggle with. And it began something, should I fire the head of sales? So I don't want to have that conversation. And then I'll have to replace them. And then there's going to be a bad PR story. And you can kind of quickly calculate all the bad stuff that's going to happen if you do it. But if you don't do it, that's probably going to be much worse and that's why you have to run towards the pain in darkness.

**Lenny Rachitsky** (00:14:13):
What is the advice you share with founders? Because as you said, it's very hard to do this, just what helps them actually get better at this? Is it just Ben being by their side telling them this is how it is? Is there anything else you can share?

**Ben Horowitz** (00:14:23):
No, no. I would say this is one where I can't really coach you to be good at this. I can point it out so that you recognize that you were slow or whatever. But it's kind of like I always liken when I talk to them, it's like football. You can have a really fast great athlete, but if they don't trust their eyes, if they don't run to the ball when they see it, if they think, "Oh, maybe that was a fake," then they're that step slower and then they'll never be as good. And CEOs are like that. If you don't trust what you see and you don't run at it, then you're just not going to be good. And it's hard to get CEOs not to hesitate. But look, the thing that does help is they look at it and I look at it and I confirm, "No, that is as it appears."

**Ben Horowitz** (00:15:23):
And sometimes they're afraid of the conversation. So that one I can help with. So A CEO might be afraid, like they want to do something but they don't know how to say it. They don't know how to have the conversation with the employee so I can walk them through that. I had an instance where the CEO said, "Hey, I need your help, Ben. My CTO, he's an asshole." And I was like, "Okay, great." I said, "But you're not going to fire him because I know he's a good CTO, or are you asking me should you fire him?" He said, "No, no, I don't want to fire him." And I was like, "So you're asking me what to do? You don't know how have that conversation with him about being an without him quitting. That's what you're saying?" And he goes, "Yeah, that's the problem."

**Ben Horowitz** (00:16:09):
And so I go, "Well, why is he an asshole?" And he says, "Well, he's an asshole because the other day he made a very junior young woman in our finance organization cry." And I was like, "Yeah, I got you." I said, "Look, this is what I would say to him. I'd say I just sit them down and I would say, 'Look, you're a really good director of engineering because you do a great job at managing the team, get the products out, all that. But you're not really a CTO because to be a CTO, you have to be effective with other parts of the organization. You can't just be effective only with engineering. And making somebody cry, she's never going to do anything you want. You lost all effectiveness with all of finance by doing that. And so if you want to get good at that, I'll help you. I'll work with you on it, but if you don't, I'm going to have to hire a CTO at some point because obviously I need that.'" And then he was like, "Oh, okay, I can have that conversation. I can't have the conversation that, 'Hey, you're an asshole,' because I don't want them to quit, but I can have the conversation that's more specific." And a lot of getting people not to hesitate is just getting them over that.

**Ben Horowitz** (00:17:20):
And so often, and I would say early in a CEO's career, a lot of it is just not knowing how to have the conversation.

**Lenny Rachitsky** (00:17:29):
There's also I imagine an element of I just want to be liked. I don't want people to hate me. You have this great line that you want to be liked and respected in the long run, not the short run.

**Ben Horowitz** (00:17:37):
Yeah, that's tricky. By the way, I have to deal with this in the firm too, and people want to be entrepreneur friendly. I'm like, "No, it's not friendly. Respectful. But you've got to be able to tell them the truth in a way that you probably don't tell most of your friends the truth." Because your friend, look anthropologically, we want people to like us. It's just so they don't throw us to the lion or whatever. That's just kind of a thing. So you say tell people what they want to hear, but in dealing at a company level in a context of you're on the board of somebody's company, you've got to be able to tell them what they don't want to hear. That's the most important thing you're going to say.

**Ben Horowitz** (00:18:22):
And yes, they're not going to like it when you say it, there's no question. But over time it could save the company. And all the most important things I've said are things that I've said to CEOs that they did not want to hear. And that's what the leadership is about. If everybody agrees with the decision, then you didn't add any value because they would've done that without you. So the only value you ever add is when you make a decision that most people don't like and that's where leadership comes in because you know that's where it's got to get to. And that's the thing that takes practice.

**Ben Horowitz** (00:19:05):
I think when Jensen talks about luck, you've got to get to near death to get yourself to do that. That's true. It's hard to build that if everything's going great. And I would say the CEOs who had an easy run of it for their, let's say, they just launched a product that's an instant hit, it's very hard for them to develop that muscle compared to the ones that built a company like Jensen where he gutted it out for multiple decades before they had big success.

**Lenny Rachitsky** (00:19:36):
Clearly, it's very difficult and painful to be a CEO, to be a founder. In spite of that so many people want to start companies. So many people dream of having their own company. Who is not right to start a company? What advice do you share with folks that are thinking about starting a company that may not understand just what they're about to get into?

**Ben Horowitz** (00:19:57):
Yeah, so it's funny. So there's a couple of things. John Reed, who was the CEO of Citigroup when I started as CEO, said to me something I never forget. He said, "Ben, the only reason to start a company is because you have an irrational desire to do so, because it's not worth the money." And I was like, "Wow, he doesn't even quantify how much money and this guy's running Citigroup." So he is a very numbers, banking guy and he didn't quantify it. And I remember when we sold LoudCloud for $1.6 billion, I remember thinking, "Wow, that wasn't worth the money whatsoever."

**Ben Horowitz** (00:20:37):
So I think if you're doing it for the money, that's a very bad reason and it will be extremely difficult to get you an outcome. You really have to have an irrational desire to do something larger than yourself to improve the world in some way that somehow that is your purpose. And if you don't feel that, then you'll never get through it. It's just too many bad things happen along the way.

**Lenny Rachitsky** (00:21:08):
So then how do you think of founders that are looking around for ideas that brainstorm, that look for market opportunities versus come from, "I have this a mission, I've got to do this thing in the world."

**Ben Horowitz** (00:21:19):
My business partner Mark always talks about that. So if you have a product that forces you to build a company, that is a great case of it, right? Okay, you built something and the world wants it and you need a company to deliver it, you know, already have the right product. And so that's very helpful.

**Ben Horowitz** (00:21:37):
I think there are cases of people, I think Hewlett-Packard was built that way, that they're like, "Okay, we've got to build technology," it was that abstract, "We've got built in technology for the world." And then they started with, "Well, what do you need?" They called it the next bench thing. What does the engineer sitting next to me need? The next engineer on the bench? So how they define the first set of products.

**Ben Horowitz** (00:22:03):
So it can work the other way, but I think the thing that is in common is it's just a very abstract idea that you have to build something that's going to be important that people are going to like working there, people are going to benefit from the products. You have to have some weird concept other than, "Oh, this is going to be successful and I'm going to make a lot of money." I think way better off taking Zuck's offer at Meta and just doing that. That's a way better deal.

**Lenny Rachitsky** (00:22:37):
Along these lines, something else Shaka suggested I ask you about, apparently there's a story where the CEO of Databricks asked you for $200,000 in the early days and you said no. And it's not because you didn't want to invest and it was more about helping them think bigger. What happened there?

**Ben Horowitz** (00:22:55):
So there were six of them, they were six HD student, well, and Jan Stoicka (phonetic) who was their professor. And Jan was this super genius, but when I met with them, they were like, "We need to raise $200,000." And I knew at the time that what they had was this thing called Spark and the competitor was something called Hadoop. And Hadoop had very well-funded companies already running towards it and Spark was open source, so the clock was ticking.

**Ben Horowitz** (00:23:39):
And I think they didn't quite know what they had. And then there's also a thing always, although I wouldn't say Jan has this mentality, but professors in general it's a pretty big win if you start a company and you make $50 million. You're a hero on campus. That's a pretty cool thing to have done. And so I'm always a little nervous about a company that comes out of academia thinking too small anyway.

**Ben Horowitz** (00:24:09):
And so I said, "Look, I'm not going to write you a check for $200,000. I'll write you a check for $10 million because this company, you need to build a company. You need to really go for it if you're going to do this, otherwise you guys should stay in school." And they were all graduating right then, so that was kind of that. And Ali actually was VP of engineering at the time, and it was a while before I made him CEO and that was very good luck on my part because I had no idea that they had a guide that good inside the company who could become CEO when I invested. That was just, God smiled on me and gave me that one.

**Lenny Rachitsky** (00:24:54):
So speaking of Ali, I actually asked him what to ask you about, and he immediately shared this story. I don't know if you remember this. In your first one-on-one with him, after you-

**Lenny Rachitsky** (00:25:00):
I don't know if you remember this. In your first one-on-one with him, after you made him CEO, he was struggling with a bunch of low performers because he was coming in to lead the company and he was trying to turn things around, trying to coach them, trying to level them up. And your advice to him was quote, "You don't make people great. You find people that make you great, that make the company great, that you learned from, not the other way around." And there's something that he called managerial leverage. What is that all about? What's the lesson there?

**Ben Horowitz** (00:25:27):
Oh yeah, yeah. So understand, he had just become CEO. So I was teaching him, he had been VP of Engineering and CEO is different. And I'll get into why and what I mean by leverage.

**Ben Horowitz** (00:25:40):
So I actually wrote a post about this with a little Wang quote where I think the quote was, "The truth is hard to swallow and hard to say too but I graduated from that bullshit, now I hate school." And that was always my feeling about this particular idea was, look, if you're VP of engineering, you can develop people. You can teach them to be better engineers. You can teach them, be better engineering managers. That's very doable.

**Ben Horowitz** (00:26:08):
But if you're a CEO, what do you know about being CFO? Like what do you know about being VP, HR? What do you know about any of these jobs except maybe VP of engineering?

**Ben Horowitz** (00:26:22):
And so the idea that you're going to take somebody who isn't world-class at marketing and make and you them world class and you don't know anything about marketing, is a dumb idea. It just doesn't work.

**Ben Horowitz** (00:26:34):
And then the company can't afford for you to be spending time on that because they need you to make very high quality, fast decisions. They need you to set the direction for the company and they need you to have a world-class team. And so it's a very hard lesson if you've been VP of engineering because if you're a good VP of engineering, you do develop your people.

**Ben Horowitz** (00:26:55):
But as a CEO, it's not like you don't do any of it, but it is very, very small compared to it. So I like to make things just very stark. So you get what I'm saying? I don't like to hedge it.

**Ben Horowitz** (00:27:08):
And then managerial leverage means it is very simple. It's okay. If I have the ideas about what your department should do next. If I am kind of pushing you to kind of move your organization forward, then that's no leverage. What's leverage is if you're telling me what you should do and how you can push the company forward, that's leverage, then I'm getting more than I'd have if you weren't there otherwise I could just manage a team.

**Ben Horowitz** (00:27:39):
And that's the point when you feel like you're not getting leverage. When you got to go say, "Hey, why aren't we doing this? Why aren't we doing that?" That's when you got to make a change. And by the way, he's unbelievable at that, as good as anybody I've seen as a guy who's not callous as a CEO. He really cares about the people who work for him. He really wants him to have great careers and all that, but he does not hesitate. If he's losing leverage, he'll make a move.

**Lenny Rachitsky** (00:28:06):
Kind of going back to the origin story of A-Sixteen-Z something you guys were really big on was helping founders stay COs become great COs, not replace them with professional COs.

**Lenny Rachitsky** (00:28:16):
I want to flip this question on you. When does it actually make sense to replace CO? When are people not going to make great COs?

**Ben Horowitz** (00:28:26):
There's a very consistent thing that happens, which is when somebody doesn't make it and it kind of starts with confidence is the way I would put it.

**Ben Horowitz** (00:28:35):
So if you invent a product, you kind of recruit a team so forth, all of a sudden you're CEO, but you don't run a big organization, you don't know how to do that. Most founders are like that. And so, if you don't know what you're doing, you're going to make mistakes and they all make a lot of mistakes. And then when you make those mistakes, they're very expensive. They could cost you to do a down round or they could cost you to lose a company or they could cost you a customer or you scrub the product. They're very high impact and not just on you, but everybody who you talked into joining you. And so that kind of motion can really cause you to lose confidence.

**Ben Horowitz** (00:29:20):
And then if you lose confidence, what happens is you hesitate on the next decision. And as we talked about, hesitation is very dangerous because one, it locks up the company, but even worse what happens is if you have senior people working for you, they get very nervous and they feel like they need to jump into that void and make the decision for you. And that's when it gets political, very political, because people are vying for power inside your little screwed up company.

**Ben Horowitz** (00:29:57):
And so now you've got a political dysfunctional organization and that's generally where, okay, the founder probably can't run this thing anymore. That's how it happens.

**Ben Horowitz** (00:30:11):
So most of what we do as a firm is to try to help people with that confidence problem and there's a whole series of ideas that we have around that, but you kind of have to somehow climb the confidence and the competency curve together. It's very hard to do and particularly if you're an engineer and you're used to getting things right or if you've been a straight A student or something like that, it's very disconcerting. Sometimes it's better to have CEOs who are like C minus students.

**Lenny Rachitsky** (00:30:45):
Why is that?

**Ben Horowitz** (00:30:45):
Yeah, a little facetious. Well, it's just good to be used to failing. So I think I wrote this, but the median on the CEO kind of test is like 18. It's not like 90. And so you got to be comfortable getting a lot of D minuses because the D minus is fine, as long as you don't get the F, as long as you don't run out of cash, as long as you don't lose all thing. Okay, you got through it, keep going. And match, that's a lot of the thing that we try to do CEOs.

**Lenny Rachitsky** (00:31:20):
Yeah, it comes back to your core, I don't know, message through your first book is just how much you'll fail and how much you'll struggle and how much paid you'll go through as CO.

**Ben Horowitz** (00:31:29):
Yeah, yeah, and I mean a lot of why I wrote that book was just to nanalyze it. I think what happens is, particularly when I wrote it, and I think it's come back and been true now, is the way the narrative gets written on all these successes is like, "Ph, they came up with a genius idea and then they built this company and they hired all these smart people and it was all great." But that's not at all how it happens and I've spent enough time with everybody from Mark Zuckerberg to Sam Altman and so forth, that they all go through that same thing that who has your struggling company go through? You screw a lot of things up and they have massive consequences, but you have to maintain your confidence.

**Lenny Rachitsky** (00:32:19):
Actually, I was at a storytelling event last night and I was chatting with someone that I ran into there and told her I was chatting with you today, and she said how meaningful your first book was to her as a founder. Exactly as you said, normalizing that it's very hard and painful and this is just the way it is.

**Ben Horowitz** (00:32:36):
And the feeling, look, if you think about organizational design or goals and objectives or OKRs or whatever management technique, you need a basic eighth grade education to do any of that stuff. It is not that complicated. The difficult part is the feeling that you have when you have to do it is very, the hard thing of matter a reorg is you're redistributing power, so you're going to have people really fricking mad at you because somebody's losing power if you do it correctly. And that person may be a really good employee. Dealing with that is the hard thing. Knowing how the organization should work to make communication better, it's not that complex.

**Lenny Rachitsky** (00:33:18):
Yeah, I think about I was at Airbnb for a long time and just thinking of Brian, who I don't know even know if he had a job before Airbnb now.

**Ben Horowitz** (00:33:25):
Oh yeah. I spent a lot of time with Brian and after COVID, it all kind of clicked for him and then he did that he and that good talk on founder mode and so forth. But the reason that was so articulate is he had screwed every one of those things up and he hired LT and all this stuff, and these are very senior people and he wanted to defer to them, but you can't defer as the CEO because you know what Airbnb should be doing. He may know what finance should do, but you know what Airbnb should do and this kind of thing. And then it gets really wild when you can't defer decisions as the CEO. You got to understand what people are saying and go, "Now we're going to do this."

**Lenny Rachitsky** (00:34:13):
And this again comes back to the point of you have to go through the struggle and pain and failure to learn those lessons.

**Ben Horowitz** (00:34:18):
Yeah, no. They're really hard to learn without doing and often without paying the consequence. Even I, like I make mistakes. I was having conversation with Ali the other day and I was like, he's like, "How's it going Ben?" And I was like, "Well, I'm finally dealing with something that I had put off for a very long time." And he said, "Why'd you put it off?" I said, "Because things were going too good. I didn't have to deal with it. "And he was like, "Yeah." He said, "I know that." I'd say Ali is one of the, if not the kind of best private company CEO out there, and he's making a mistake and I'm making a mistakes. So, it is just tough.

**Lenny Rachitsky** (00:35:01):
You said that one of maybe the main reason founders fail the CEOs is they lose confidence, and you had some ideas that you guys have to help founders work through that. Are there a couple you can share how you help?

**Ben Horowitz** (00:35:12):
Yeah. So we do a lot of things on that. So the kind of design of the firm is about confidence. So the first thing is, well, what would give you? Well if you can get stuff done. So what if I could give you a network that's as good as Bob Iger's network, day one, the day you stepped into the job.

**Ben Horowitz** (00:35:31):
And so we have 600 people at the firm and why is that? Well, most of them are building that network for you, so you can call any CEO or anybody in Washington or any executive or that kind of thing and get them on the phone and they'll talk to you and you can kind of deal with that thing. And then that just makes you feel like a CEO.

**Ben Horowitz** (00:35:55):
And then we have a lot of people in the firm like myself, who you can talk to on a CEO to CEO basis, as opposed to an investor to CEO, and just kind of feel that. Early in the firm days, we used to do this thing. I think I'm going to bring back in some form this thing called the CEO barbecue. And it was like a lot of people have these events where they bring in speakers and this and that and the other. And I always felt like those were one, they were too many days. And then sometimes what they said wasn't really applicable and that kind of thing.

**Ben Horowitz** (00:36:32):
So I said, "Why don't we just have a barbecue?" I would barbecue. We get everybody in my backyard because was 500 people at the peak, which is why at the stuff I couldn't cook that much, that kind of stuff. And then we'd have Larry Page and Mark Zuckerberg and Kanye West, and so you're a CEO in there with portfolio. You're like, "Wow, I must be important. I'm here with all these guys" and we're just hanging out having a drink, eating barbecue.

**Ben Horowitz** (00:37:04):
And so then when I go back to my company, I feel like I am somebody. And okay, I might not be perfect at all this, but I am really a CEO. I was at the CEO barbecue, for crying out loud and that kind of thing. But the whole idea was always like, "Okay, do you feel like you can do it?" Because that's half the battle?

**Ben Horowitz** (00:37:29):
And look, having been in and every CEO has been in a position where they feel like, "Well, maybe I shouldn't be the one running this thing. Maybe it's just too big for me." And that's a bad, you don't want to go there. And because as we've said, founders can get to the next product and that's something that almost no professional CEO is able to do. There've been rare cases, but very rare.

**Lenny Rachitsky** (00:37:57):
So clearly you've worked with a lot of companies, a lot of founders. Let me kind of zoom out a little bit and ask you this question.

**Lenny Rachitsky** (00:38:03):
What's the most counterintuitive lesson you've learned about building companies that goes against common startup wisdom?

**Ben Horowitz** (00:38:09):
Well, the common startup wisdom keeps changing. One of the early ones that was wrong, and Brian articulated it, and then now I think a little bit of what people have gone to is also wrong.

**Ben Horowitz** (00:38:25):
So the first idea that was wrong was like, okay, build a team of senior executives as soon as you get product market fit as fast as possible and they can scale the thing. And I think that you got to build that team slowly and deliberately, kind of pace to your ability to integrate and then manage them. Because if you bring in a bunch of senior people and you don't know really how they match to your company or how that function works or so forth, then you're going to start deferring. And once you start deferring, it's going to get out of control very fast because they're going to build empires, they're going to get political, they're going to do all that kind of thing.

**Ben Horowitz** (00:39:08):
So that was bad advice. You kind of have to do it in a measured way. I think that founder mode, I think a lot of people have taken to never hire anybody with experience. And that's also bad advice in that, look, somebody who knows how to do something can really accelerate your thing. So very early on, one of the founders, great founder Arsalan at Databricks was running sales. And I'm like, "Oli, you're going to have to hire somebody who knows sales because Arsalan's, PhD in computer science. I like that, but that's probably not where you're going to have to start if you're going to catch these guys before they take Spark and use it against you."

**Ben Horowitz** (00:39:53):
And I sat down with Arsalan and I explained why. I said, "Look, a lot of what sales, there's a lot of knowledge in how to build a worldwide sales organization; maybe knowledge of customers, territories, territory splitting, rep profiles. There's just a litany of stuff that you really can only learn by doing trial and error and you don't know anything. And so you're phenomenal. Let's get you. And still, he's a very senior executive in the company now, but we need somebody who knows that. And the idea that there are companies that go, " Okay, we're just not going to hire that in founder mode." That's also a mistake. So there's a lot of, it's more subtle than you think, and it's more complex than you think. And so you kind of have to get all the way to the truth. And these little snippets of advice that he sees good, because they watch some fucking podcasts, are all fucking stupid.

**Ben Horowitz** (00:40:51):
There's a lot of depth to these things. You have to know the answer to the next question, the next question and the next question, and it does drive me crazy. One of the funnier things that happened along these lines, just to show you how little. You know as an investor, about what it means to be CEO.

**Ben Horowitz** (00:41:11):
We were at a board dinner. One of the CEOs says to me, he goes, or one of our CEO says, "Hey Ben, that thing you told me a while ago about don't be CEO at home." He said, "I was doing that and I stopped and it really helped me."

**Ben Horowitz** (00:41:28):
And then the other kind of VC said, " Yeah. You got to unplug some time." And I said, I was like, "What the fuck are you talking about? He's CEO. He's not unplugging. He's getting shit all the fucking time. He's got to deal with that. That was not what I meant. I was like, you can't go home and boss your family around. That's what I meant."

**Ben Horowitz** (00:41:48):
You hear something from somebody who, but if you haven't done it, you don't even know what that means. And so then you then trying to transfer the advice to the next guy, that's not going to work. So anyway, but he was very innocent. I just don't want to kind of speak bad of him, but that's how it sounds, right? But that's not what it is.

**Lenny Rachitsky** (00:42:09):
That's an amazing story. So the advice partly here, is just don't believe everything you see on Twitter and little sound bites of advice.

**Ben Horowitz** (00:42:18):
I think actual CEOs know it. And that's kind of how people in my profession going to get a bad wrap. Because giving advice, that's not something that you know but something that you heard, is very dangerous, I think.

**Lenny Rachitsky** (00:42:31):
So speaking of advice that you've shared that might be out of date now, you are famous for writing one of the most popular pieces of literature for product managers. There's a lot of PMs that listen to this podcast called Good Product Manager, Bad Product Manager. And if you actually go to that post today at the top you say, "This document was written 15 years ago and it's probably not relevant today for PMs. I present this nearly as an example of a useful training document."

**Lenny Rachitsky** (00:42:56):
Still people link to it. I actually just link to it as just like this is something every PM needs to read. What is it that you think people should maybe not take away from it, and what do you think people still should take away from that piece?

**Ben Horowitz** (00:43:07):
Yeah, so the reason I wrote it when I wrote it was that I had a lot of product managers. And one thing about product management is it's a job that's completely different at every company and there is no training for it. So everybody kind of figures it out as they go. And depending on what's being emphasized, they'll get wrapped around the axle on if it's an enterprise company, well pitching to customers or I need to be really good with the press, or I need to be really good at writing the product requirements document or that kind of thing.

**Ben Horowitz** (00:43:49):
And those are all these tasks, but none of those were the job and what I was trying to get out in Good Product Manager, Bad Product Manager was, the job is fundamentally a leadership job and it's a tricky leadership job because nobody is actually reporting to you.

**Ben Horowitz** (00:44:14):
So it's like this influence, how do I get people to do what I want even though I'm not paying them. I can't fire them. I can't promote them, and so forth, which is kind of the essence of real leadership because if you start to rely on promotion and firing and so forth for authority, then you're never going to be good at being CEO or anything.

**Ben Horowitz** (00:44:38):
So I wanted them to get into the mindset of, "Okay, your actual job is to get a product into market that customers love that's better than anything that anybody else in the world puts in market. That's your job." And so to accomplish that job, you need engineering to understand you with clarity. You need to understand engineering with clarity. You need to have a really good view of the market and the competitors and the technology and so forth and you need to put that all together and deliver the thing.

**Ben Horowitz** (00:45:17):
And all the other things are tasks that you may or may not need to do. I don't know if you need to do them, but the thing is, you have to be the leader. You've got to get the thing done. And so what I think it's still good on is that the mindset, be the leader, I think the details of any kind of thing that was kind of task specific was really for my group a Netscape in 1996, whenever the hell I wrote it.

**Ben Horowitz** (00:45:56):
So as a kind of document I wrote out of frustration. But I am glad that the people still like it, and I think leadership in general is undervalued, underestimated. It's the most powerful thing. And most of the great companies, Jensen is a great example. What a phenomenal leader he is not just of Nvidia, but of the whole industry. And he doesn't have authority over the industry, but he drives it forth and that's why the Good Product Manager, Bad Product Manager is so important because that thing, if you learn how to do it, that's the thing.

**Lenny Rachitsky** (00:46:39):
I didn't realize you wrote that initially as just an internal document and then you made a-

**Ben Horowitz** (00:46:44):
It was kind of before blogging took off, so it is just an internal thing and I published it later. I was just getting so mad. And by the way, my product management team at the time was very good, very talented people. They just were not getting that concept. So like David Wyden said, Coastal Ventures, Raghu Raghuram, who wanted on to be CEO of VMware, the team was like that team, but they were driving me crazy. And so I was like, "I can't yell at people anymore. I have to explain to myself." And so it's a good thing if you find yourself yelling at people, you probably haven't explained what you want, was the other big takeaway from that.

**Lenny Rachitsky** (00:47:30):
Did you ever think that piece would be so long-lasting? And so, I don't know, popular?

**Ben Horowitz** (00:47:34):
I didn't even know. I thought it was kind of like aggressive when I wrote it. You could tell I was mad. I called a Good Product Manager, Bad Product Manager. It's like bad dog, bad, bad, bad product manager. That was kind of the emotion I had. So, it is kind shocking some of the things that you write.

**Ben Horowitz** (00:47:54):
I would say that that's and kind of creative, and you probably know this. The idea is that you have the things that you write in five minutes end up being much better than things you write in five weeks. And I find in talking to musicians or writers or everybody has that same experience. The thing that you've already synthesized so much that you just have to write it out, that's the best stuff.

**Lenny Rachitsky** (00:48:21):
There's something that you mentioned there in your answer about the PM being the leader. There's always this kind of sense that the PM is not the mini CEO. How dare you call yourself that. I actually think that's exactly what the PM is. They're basically the closest to the CO. Their kind job is to think like the CEO within the team.

**Ben Horowitz** (00:48:39):
People get mad because everybody, this is the whole challenge of management in general. People get jealous over stewardship. But from the perspective of the PM, it doesn't matter if you write a good spec or you have a good interview or you do this or do that. What matters is that the product wrench. And you have to get all the way to there and work backwards from that and you can't do that without leadership because it is about, okay, we want to build that. And you're not necessarily the person who comes up with every idea or this or that, or that. You're just the keeper of the vision.

**Ben Horowitz** (00:49:22):
And that's true for CEOs too. You don't want every idea in a company coming from the CEO. I think it's a misunderstanding of what a CEO is, is why people don't like that. They don't know what a CEO is. But a CEO isn't the one who has every idea it gives, every order does every. That's not the way it works.

**Ben Horowitz** (00:49:40):
The way it works is there's somebody who's got to consolidate, get all the good ideas, prioritize them, decide which good ideas we're going to do, and then get everybody on the same page, so that they have very high fidelity understanding of what that is. And so that it is a CEO, kind of function. Now, it doesn't mean I'm better than you, it just means that...

**Ben Horowitz** (00:50:00):
... kind of function. Now, it doesn't mean like I'm better than you. It just means that that's what I'm doing.

**Lenny Rachitsky** (00:50:05):
**Ben Horowitz** (00:51:35):
Adam is probably the single most controversial investment that we ever made. We got called everything from stupid to sexist to racist to this and that for literally just funding that. And I think it's going to end up being one of the best investments we ever made. He's doing a phenomenal job there. There's an important principle in that which we do as a firm, which I think is not widely done, but I would love it if people copied it, which is, and it's something I learned somewhat from Shaka, which is you don't judge a person by the worst thing that ever happened to them. We've all had bad things happen to us. We've all made bad decisions. Most of them, they don't make a miniseries about.

**Ben Horowitz** (00:52:35):
And so to judge them on that, you want to judge people on what they do well, not what they screwed up, because that's where you see the talent. If you look at what Adam did well, it's truly spectacular. Everybody knows WeWork. Name a more important commercial real estate brand than WeWork. You can't. And so what an accomplishment. And there were so many things that went into that and so many things he did right.

**Ben Horowitz** (00:53:10):
And then if you kind of look at really unravel the things that went wrong, most of it was like a combination of inexperience and nobody around him that would tell him the truth. And, yeah, maybe he wasn't good at listening to the truth either at the time, but to throw away a guy on that, which is by the way, the world was so mad at us for not throwing him away, for believing in him, is just that it's a big mistake. And I credit Mark because Mark is the one who called him up originally, and just said, "Hey, Adam, what are you doing?" Because we watched what you did at WeWork and we thought it was pretty impressive.

**Ben Horowitz** (00:53:54):
And so I think that that's probably the biggest secret there. Judge Al Davis once said, "Coach players on what they can do." And I think that's very true. Judge people on what they can do, coach people on what they can do, help them take their strengths and use them as opposed to over focus on their weaknesses and just hand wringing about the one fucking thing they don't know how to do. Because look, everybody's uneven.

**Lenny Rachitsky** (00:54:31):
What you're describing, essentially, this is the job of an investor, is to find an underappreciated asset and invest or before something people don't see.

**Ben Horowitz** (00:54:42):
Venture capital is really about investing in people. You have ideas as an investor, but what you really are ultimately betting on is the entrepreneur and the entrepreneur's idea because the initial idea isn't where they end up usually. It changes a lot with everybody we invest in. So you kind of have to make the judgment on the person. And how you do that is really, really important. And one of the things we emphasize inside the firm is, look, we're investing in strength, not lack of weakness. I want to know how good, are they world-class? Do they have a world-class strength? And can that beat anybody? And look, everybody's flawed. And so let's help them deal with the flaws and surround them with people who can handle that and put the right person on the board who can talk to them.

**Ben Horowitz** (00:55:43):
You know Mark's on the board, I go to all his board meetings because I'm the one who's good at killing a guy who's that confident when he's like, "Okay, that's not your best idea." That's good role for me. But that's how you deal with that. You don't throw them away and go, "Oh, okay, we don't want to be called names. So we're not going to invest in Adam Neumann after we built WeWork. That's crazy.

**Lenny Rachitsky** (00:56:08):
That's also why you guys invested in Cluely, I imagine similar.

**Ben Horowitz** (00:56:10):
Yeah, yeah, no, that's right. I mean, look, if you look at what those guys did, that was some high level marketing genius too, and that's really something. Plus the product is awesome.

**Lenny Rachitsky** (00:56:23):
I'm going to bring us back to AI. Something that a lot of people are talking about right now while we're recording this is this potential huge bubble we're in with AI. Sam Altman said, "We're in a big bubble," which is, that's saying a lot. I'm curious just how you that it-

**Ben Horowitz** (00:56:36):
What is it saying? First of all, I should qualify this by saying I am an investor and Sam's a CEO. So CEOs have to have much more purpose when they talk. Investors just have to be entertained. So you got to give Sam credit for what is it in his interest to say it? Well, if it's a bubble, then the one thing you should invest in is him and not all these guys chasing after him.

**Ben Horowitz** (00:57:09):
I would say that's very smart. And then the other thing that's smart about it is there's nothing that you can say to the press that will make them love you more than saying, "All investors and entrepreneurs chasing this are idiots." They love them. The press are generally haters, and so it's just red meat for the haters, which was also super clever. So I think whether even he believes that or not, that was a super smart thing to say. So I'll just put it there. Whereas what I'm going to say won't be as smart, but it will...

**Ben Horowitz** (00:57:45):
I don't really have an ax to grind here. I mean, I could have an ax to grind and say, "Okay, let's get all the other investors out." I'll say, "It's a massive bubble." But what I would say about that is, so the first thing, the one thing about bubbles is anytime everybody thinks it's a bubble, it's not a bubble because in order for it to bubble, you need capitulation. In that you need everybody to believe it's not a bubble, because then the prices really go out of control. But as long as there's people who think it's a bubble, then it's hard for that to happen. And it's funny, I had this debate in The Economist, I think with Steve Blank in like 2011 or '12 when everybody thought it was a tech bubble, if you can imagine that, which it absolutely was not. But because there were 1,400 articles saying that we were in a tech bubble, and I mean where prices were then compared to where they are now.

**Ben Horowitz** (00:58:52):
But I knew because everybody was saying it was a bubble, it wasn't a bubble. I knew that. The prices were higher, but the reason the prices were higher was we're getting to a global market. AI prices are higher than prior prices, but if you look at the revenue growth and numbers, we had not seen anything like it. Sam's product worked so amazingly, we'd never seen that before. Not even Google, not anybody. And so that's real.

**Ben Horowitz** (00:59:23):
And we have companies that went from zero to 800 million in a year and that kind of thing. I would say there's a basis for the prices going up, first of all. I think the thing that's right about what Sam is saying is the landscape is early, really early. The technology is very immature. As amazingly as it works, there's a long way to go to improve it. So it's very possible when you have that much technological change, that the positions that these companies have achieved with their high revenue isn't sustainable, and that there'll be a competitive change that either lowers prices or a new number one emerges or that kind of thing.

**Ben Horowitz** (01:00:18):
Yeah, that's possible. But I wouldn't characterize that as being a financial bubble in that if you go back to the great dot-com bubble that everybody is always waiting for it to happen again, which I was CEO during. The thing that happened there was very different, which is it was the internet and every smart investor knew that the internet was a big deal. How could you not fucking know that the internet was, of course it's a big deal. But if you go back to 1996, at Netscape, we had 90% browser share and we had 50 million users. So there were 55 million people on the internet in total, and half of those were on dial-up.

**Ben Horowitz** (01:01:03):
And then to build a product like Evite, the greeting card company, had 300 engineers. That's how hard it was to build this stuff. And so the math didn't work, and the math didn't work on any of those ideas, but the investors kept pouring money in. And then eventually everybody went bankrupt because there was no revenue coming in. And when they figured that out, then nobody would invest in anything. And of course then everybody realized, well, the internet was actually real and Paul Krugman didn't know what he's talking about, and like it was going to be a big thing. And then Facebook and Google and all these things emerged.

**Ben Horowitz** (01:01:44):
But the thing that made it a bubble was the unit economics didn't work, the businesses didn't work. These businesses are all working, and they're being priced appropriately for how they're growing. So that's not in effect. The thing that you could say is, "They're not going to keep growing like that," and so forth. And I'm not sure about that. The products, like I said, are working so much better than any technology product that we've ever built has worked. It's just mind-blowing how good this stuff is. And so, I don't know, if I had to bet, I would bet not a bubble. I think there'll be some dislocation. I think always in venture capital, if you've got a run like this, then the great company and the crap company both get funded. But that's just venture capital. That's not a bubble.

**Lenny Rachitsky** (01:02:44):
Four founders starting companies these days. When you look into the future of the AI industry, say in five, 10 years, how do you think things will play out slash where do you think the biggest opportunities remain? Where are you guys looking to invest most?

**Ben Horowitz** (01:02:57):
In infrastructure. I think that there is obviously a real estate power cooling play. I think that's a little outside of hardcore technology investing that we do. But there's another layer which is take a given open source model, who can run it the cheapest with the lowest latency? And that's going to be extremely valuable, whoever has that. And Google has been historically very good at that and so forth, and Sam is really trying to build that now with Stargate. And so I think that's going to be a very important layer of value. I think that on the foundation model side, you have to be very selective as an investor. So in order to compete in foundational model world, our basic rule of thumb is you have to be able to, without much product progress, raise at least $ 2 billion because that's basically what it's going to cost you to train something that gets you competitive enough to make money.

**Ben Horowitz** (01:04:08):
And there are just very few founders like that. So Ilya is one of those. Mary's one of those, Fei-Fei is one of those, but that's the kind of class of person you need. And there's whatever. There's certainly less than 10 of those in the world. And so that's kind of an important area, but a small area. And then I think the application layer is going to be very, very interesting. And I think that if you look at Sam, he's making most of his money off ChatGPT, almost all his money off ChatGPT now. And Chat GPT, like it or not, it's got a real moat. It's very hard to knock it off. It's perched.

**Ben Horowitz** (01:04:57):
Everybody's taking a shot at it, people, great distribution like Google and Elon and Zuckerberg and everybody. And that thing just keeps going like it is. So I think the applications are both more complex and kind of stickier than people thought they were originally. The thing that people got very wrong is this whole thin wrapper around GPT, that's really wrong. In fact, here's how wrong it is. Back in the '80s, that same phrase was used, but it was thin wrapper around an RDBMS-

**Lenny Rachitsky** (01:05:38):
Database.

**Ben Horowitz** (01:05:39):
Yeah, yeah, yeah. So it meant companies like Salesforce were basically just a thin wrapper. And I think that that's kind of the mistake people made. So we're at this company Cursor, and if you look under the covers in Cursor, they've built 14 different models to really understand how a developer works, a high-end, a real developer. Those models have tons and tons of interactions with how people talk to their friend Cursor about how they should design their programming so forth. And that's real, that's not just a thin layer on a foundation model. And I think there are many, many applications like that. And so I think there's going to be a lot of opportunity at the application layer. There's going to be some opportunity at the foundation model, and of course you can invest in Sam, you can invest in Anthropic and so forth as well.

**Ben Horowitz** (01:06:43):
But there will probably be a very small number of companies at that set, and then a almost unlimited number of companies at the application layer. And then as the technology advances, we'll of course see more things we can body to AI. I mean already autonomous cars are working really well now after a long, long, long, long time. Since I think Sebastian won the challenge in 2006 when he drove the self-driving car across the country. And here we are 20 years later and now they're deployed. So that was a long time. Robots I think is a harder problem than self-driving cars. So we'll see how that goes. But, yeah, there's certainly a lot in that world as well.

**Lenny Rachitsky** (01:07:34):
Wow, okay. There's a lot to this answer. No, that was exactly what I was looking for. The Cursor example, it's something that comes up a lot on this podcast, in the application layer specifically. The thought that the way to win in this space and to build a moat is, as you said, "Build your own model slash have proprietary data that you build through people using your product." Thoughts on that?

**Ben Horowitz** (01:07:58):
Yeah, I mean, I think that ends up just being what's required. So it turns out that the universe is long-tailed, is fat-tailed, and humans are very fat-tailed in terms of human behavior, human conversation and so forth. So to get to the real meaning of it and to get to the kind of essence of the problem, in any domain turns out to be, I think, more complex than we thought. And so the early things and people were running around saying, "Okay, there's going to be one big brain to rule them all on these kinds of things." That's kind of not played out yet.

**Ben Horowitz** (01:08:41):
And in fact, if you look underneath the covers, you have LLMs, which have generalized pretty in fascinating ways, but they've kind of also asymptoted in that we have run out of data for the most part. And so if you look at the GPT-5 LLM compared to the GPT-4.1 and how much more it costs to train and so forth, it's definitely not going linear anymore.

**Ben Horowitz** (01:09:12):
On the other hand, the reinforcement learning side has been linear, but it doesn't generalize. So if you build a great programming model, it may be an idiot at math. And so that I think is just very different than what people would've said three years ago. And I think that there's not something that's both scaling and generalizing yet, and maybe we'll get there, but that certainly opens the door to something that's more user-friendly, that's more effective in any number of domains than just the basic foundation model infrastructure. Now those models are incredibly important, and I think OpenAI is probably 80% of the revenue in AI are something like that now. It's massive, and so that foundation model is really, really important. And then the basic consumer app is really, really important.

**Ben Horowitz** (01:10:12):
That just answers whatever the hell you want to know. Those things are very, very real. But I do think particularly, and then if you get into enterprise stuff and then it's no longer internet data, it's their data that becomes very different. Databricks is having a lot of success there because, okay, well, once you're inside a company, guess what? You care about access control. That's hard with an AI world. It gets trained on some stuff. How does it know who has access to that information and who doesn't, and so forth. You have semantic issues. So if you look at an enterprise, find 10 enterprises, they all have a different definition of what a customer means. You would think customer is a basic thing. Well, is it a department at AT&T? Is it AT&T? Is it a person at AT&T? What the hell is the customer? And it turns out to be very, very meaningful, particularly if you're trying to figure out important things like churn and this and that and third.

**Ben Horowitz** (01:11:19):
So that kind of stuff matters. So I would just say the problem space is a lot bigger than you can just attack with a basic foundation model currently. Maybe that will change, and if that changes, then certain prices will have, in retrospect, look way inflated and others will look too low. But that is TBD.

**Lenny Rachitsky** (01:11:48):
So a big takeaway from this is that there's still tons of opportunity for founders to start companies building AI products.

**Ben Horowitz** (01:11:55):
I think so. Everything that we couldn't solve with software we can solve now, almost. So it's a really big world. And it's funny because we're investors in Waymo, and one of the things when you get into what took so long to make Waymo so safe like they are now, it wasn't the things that everybody reported on the podcast. There wasn't sleet and heavy rain, it was people. It was like the human who was driving 75 in the 25 zone. It was very hard for the AI to anticipate because it was rare but important. And the number of rare, important crazy shit that humans do is very high. And I think that that goes for all of AI. So to make things work really well, you have to understand this very kind of fat tail of human behavior.

**Lenny Rachitsky** (01:12:52):
Along this AI thread, something that is really important to you, clearly something you talk a lot about is the US being successful in AI, in leading the world in AI. Why is this so important? Why is something you spend a lot of time on?

**Ben Horowitz** (01:13:06):
It starts with, I think, my view of the US and its role in the world. My personal view is it's very, very, very important. Not for society to be completely fair because it's not going to be completely fair or completely equal because we've never had one that's been completely equal. But it's important that everybody have a chance at life, and particularly both culturally, but also just you can't advance the world if you can't tap into all of your resources. And so if you kind of take away motivation and these kinds of things, you get into trouble. And if you look at the kind of every country today, and this is by the way, so you want the right amount of decentralized power. You don't want it to be completely concentrated, concentrated power makes it very, very difficult for everybody to have a chance. This is the big lesson of communism over the last hundred years is it turned out right. And we still have politicians selling it this way today, it's like, "Oh, it's power to the people."

**Ben Horowitz** (01:14:23):
No, no, no. It's power to you because you're removing all power from the private sector and installing it into the government, and then you're putting yourself in charge of the government. And so I become extremely powerful. And this is why it didn't matter if it was Mao or Pol Pot or Ceausescu or Stalin, everybody died because when you give anybody that much power, nobody has a chance. There is no incentive, there's no carrot, there's only stick. And so you use that stick, and that's just the nature. It's a system's problem. It's not a person problem. It's not Stalin was evil, Ceausescu is evil. It was that system is evil. And-

**Ben Horowitz** (01:15:00):
It was evil. It was like that system is evil. And that's the saying with fascism. Be it Hitler or Mussolini, it doesn't matter. That level of power is evil.

**Ben Horowitz** (01:15:11):
And the US does the best job. Systematically, it's the best system. It's got all kinds of issues, it's got problems. People always try to defeat it. But one of the things that if you look at the Declaration of Independence or the Constitution, the language is very important. It's, "We hold these truths to be self-evident."

**Ben Horowitz** (01:15:39):
What does that mean? It means it's not my rule, it's not the president's rule, it's God's rule. And so those rules are above the President, and then you work in that context and that distributes the power, because you're under the law, not under the person. We see that now even with Europe, where the leaders are going, "Well, I have a rule. It's you can't say certain things or I'll throw you in jail." And the kind of shield they hide behind as well, we have to keep the kids safe. But if you say something that I don't agree with and the kids hear it, they're not safe. So the kind of transited property of bullshit is going to override.

**Ben Horowitz** (01:16:29):
And so it's really important that we have at least one society. And as flawed as we are, as flawed as the US is, it's still the best. And you can see it by the number of new company creations, the number of new ideas that come out of here and so forth. It's really, really important that the US stays important and powerful in the world.

**Ben Horowitz** (01:16:51):
We know from the last century, if you look at the last century, who were the countries that had economic power, military power, cultural power? They were the ones that industrialized, and the ones that industrialized first and best. And the ones that did became Communists, like Russia, China. They were slow on industrialization and they fell into this very fucking dangerous system.

**Ben Horowitz** (01:17:19):
Looking forward, that's going to happen again, but it's going to be AI. And so it is fundamentally important, not just to America, but to humanity, that America succeed at that. We don't have to be the one winner or this or that, but we do have to be in that tier.

**Ben Horowitz** (01:17:41):
And as I go around the world and travel, I can't tell you, everybody ... and then everybody was getting, by the way, very, very worried about us earlier. And they say, "Look, we need you to succeed. Don't destroy the dollar. Don't fall behind in AI. Don't over-regulate it too early. Don't do these things, please, because we need you to win, because we're all counting on that."

**Ben Horowitz** (01:18:08):
And I think it's the most important work that we do. It's why we're so involved in policy and so forth. I think this is also going to be very, very true with crypto, which ends up being an incredibly important networking technology that complements AI. That work is, I would say, beyond for the money. Although we will end up making a lot of money with the right policies, so I don't want to seem totally philanthropic on this, but it's more important than that. It's certainly more important than us succeeding or anything like that, that the country succeed.

**Lenny Rachitsky** (01:18:53):
Speaking of philanthropic and other passions of yours, something that I don't think most people know about you, and I think will give them another insight into how interesting you are, you run an organization called Paid in Full, which is incredibly cool. Talk about what that's about, why this is so important to you.

**Ben Horowitz** (01:19:14):
Our ethos as a firm is kind of what I'd say is something from nothing. This is the greatness of entrepreneurship. You start with nothing and then you make something really important. That is also how Hip-hop started, where you have a bunch of kids who didn't even have instruments, and they created something out of nothing. And the people in that world always talk about that.

**Ben Horowitz** (01:19:43):
And one of the really unfortunate things that happens is, the people who invent the art form, and certainly in the case of Hip-hop, don't get anywhere near the kind of proportional benefit of their invention. And a lot of the guys, people have forgotten about, or are struggling to make ends meet, and so forth.

**Ben Horowitz** (01:20:06):
So what we created was this thing called the Paid in Full Foundation, named after the Rakim, Eric B. song, which I did call Rakim and ask him for permission to use the name, so we didn't just take it. What we do is, we give essentially pensions to the old rappers, that enable them to kind of continue their work.

**Ben Horowitz** (01:20:28):
And then we have a big event go to paidinfullfoundation.org for tickets, which is amazing, where they get the award and they're celebrated by all their peers and so forth. And it's really phenomenal. Some of the awardees have been Rakim, Scarface from the Geto Boys, Roxanne Shante, Grandmaster Caz, Kool Moe Dee.

**Ben Horowitz** (01:20:56):
This year we're honoring George Clinton for being sampled, Kool G Rap and Grand Puba, and also Jalil from Whodini. I can't even describe how high impact it is on these guys. I think Rakim was touring close to 200 nights a year, and he got his award, and came out with his first album in 15 years and is doing amazingly well. All of a sudden everybody's going, "Oh yeah, that's the greatest rapper of all time." They're finally going back and remembering all the things he did. Roxanne Shante, nobody had mentioned her in years and years and years. I think six months after we gave her the award, the Grammys gave her a lifetime achievement award, which is amazing. It's super high impact. It's a great thing. As a Hip-hop fan, I say dream come true. It's my only guilt.

**Lenny Rachitsky** (01:21:51):
What's the origin story of you and Hip-hop? I imagine many people look at you and wouldn't imagine these records behind you, Nas gifted you. You are so deep into the community, just how did this all begin?

**Ben Horowitz** (01:22:03):
Well, I actually wrote a blog post on it called The Legend of the Blind MC, which I think would be, if you're really interested, it's worth reading. But it is kind of a story of me becoming a rapper, and how that occurred, and how it went, and so forth.

**Ben Horowitz** (01:22:23):
I always say the very short story is, I was in New York at the birth of Hip-hop and when it really became big, '84 through '88. It's just the most exciting thing to see a new art form pop out and the creativity and everything.

**Ben Horowitz** (01:22:43):
Once music becomes mainstream, I would say it's very shaped by business. And in the early days, everybody's just coming out with whatever idea they have and so forth. The early days of rock and roll were like that. The early days of jazz were like that.

**Lenny Rachitsky** (01:23:02):
I see now even more why you guys brought on Erik Torenberg. Beyond his many talents, he's also really big into rap himself.

**Ben Horowitz** (01:23:10):
Yeah, that's a good reminder. I need to make sure he gets to Paid in Full.

**Lenny Rachitsky** (01:23:17):
Ben, this was incredible. Is there anything else that you want to leave listeners with or share before we get to our very exciting, quick lightning round?

**Ben Horowitz** (01:23:28):
Yeah, I would just say, if you're a CEO listening to this, then know that how you feel about yourself is going to end up meaning as much as anything, and take your time on that. Self-evaluation is ... one of my favorite quotes is that when my old manager saw me, as select players know, "From this day on, no credit will be given for predicting rain, only credit for building an ark."

**Ben Horowitz** (01:24:05):
And I think that's more true for CEOs than anybody. You have to build the ark. It doesn't matter if you predict you're going to fail, you've still failed. It gets you nothing. So what you have to do is figure your way out of it and spend all your time on that.

**Lenny Rachitsky** (01:24:23):
Well, with that, we've reached our very exciting lightning round. I've got five questions for you. Are you ready?

**Ben Horowitz** (01:24:27):
Yes, sir.

**Lenny Rachitsky** (01:24:29):
First question, what are two or three books that you find yourself recommending most to other people, not including your own books?

**Ben Horowitz** (01:24:35):
So two that I learned a lot from, one is The Weirdest People in the World, which it's a kind of big history book, but through an anthropological kind of cultural lens. It's really fascinating. It explains an awful lot about how society works and how little changes in the rules completely change the culture.

**Ben Horowitz** (01:25:02):
One of the things that he basically endeavors to explain, why did the West get ahead on the rest of the world? It kind of comes down to a weird anomaly with the Catholic Church, where they enforced monogamous marriage. And it turns out that the natural state of humans is polygamy, but the problem with polygamy is, there's no cooperation among men, which is a problem. And as a result, and then everything is secret, so there's no sharing of knowledge, because it's all kept in the family, in what he calls kin-based culture.

**Ben Horowitz** (01:25:44):
It leads to something called kin-based culture. And even recipes won't be shared if you're in a kin-based culture, because it's a secret. So you can't build science, you can't build cities, you can't build big companies. So because the West kind of enforced broad monogamous marriage early, it was able to evolve into all these things.

**Ben Horowitz** (01:26:07):
And he kind of shows why and how. He does these psychological tests today with people from monogamous marriage culture, Western culture, and kin-based culture. And the psychology is completely different. He gives examples like if a person murders another person, is that still murder if that person who did the murder is your brother? And in Western society, yeah, that's a murder. In kin-based culture, it's not a murder. It's just not.

**Ben Horowitz** (01:26:39):
And so that's really different. And so morality is different, everything comes off of that. So it's just a fascinating, fascinating book. So that's one.

**Ben Horowitz** (01:26:55):
Another book that I recommend a lot is Shaka's book, his first book, Writing My Wrongs. He has a new book that I think is better, which is called How to Be Free. It's not quite out, or maybe it's releasing. It's releasing next month, so I recommend it.

**Ben Horowitz** (01:27:11):
What it does is, it goes through how did he work his way out of prison and solitary confinement to his current psychology, and what are the techniques he used? They're very powerful, very powerful ideas. CEOs always asks me, "How do you deal with it? How do you deal with it?" They'll ask in a work-life balance. It's like, "What are you talking about? No work-life balance for CEOs," or particularly entrepreneur CEOs, come on.

**Ben Horowitz** (01:27:43):
But what do you do? Do you meditate? Do you do this or that? But he kind of goes through the things that you really ought to do. That's what I would recommend, if somebody's wondering, how do you deal with all this pressure?

**Lenny Rachitsky** (01:27:57):
I love this. Just the whole idea of Shaka being this help for CEOs, someone that killed someone, went to prison, led a prison gang. I just love that, how valuable his lessons-

**Ben Horowitz** (01:28:08):
Prison gangs turn out to be really complicated things to manage. Sorry for getting into this, but the problem with running a prison gang is, you're just dealing with people who all come from broken culture.

**Ben Horowitz** (01:28:23):
So in any organization, like the fundamental thing is trust. And so you're bringing in a bunch of no trust people. And so again, it's kind of like a military organization. It's a gang. If you think about a military operation, if you don't have trust, people don't trust the order, then you're completely dysfunctional.

**Ben Horowitz** (01:28:46):
So how do you build trust from zero is a very interesting problem. And he's a genius at that, by the way. And I learned a lot from talking to him about it. So from a management standpoint, I would just say it's an important kind of boundary case of how you build culture. He is very, very smart at that. Like I said, you have to look at people about their greatness, not the worst thing they ever did.

**Lenny Rachitsky** (01:29:23):
I know it's the lightning round, but can you just share one example of something he did that was like, wow, that's a really good lesson, someone trying to create a good culture that worked for him?

**Ben Horowitz** (01:29:32):
Yeah. So one of the things that he did that I thought was really smart is ... Well, he did a couple of things. One is just a simple thing. He just made everybody eat lunch together in the gang, just to build rapport, relationship, trust, like it's all one thing. We're all together on that.

**Ben Horowitz** (01:29:56):
And I think particularly in the remote work world and so forth, people really underestimate how powerful just that idea can be. And then another thing he did is, he made morality ... he had very specific things about, you had to be good to your word internally and externally.

**Ben Horowitz** (01:30:26):
So normally a gang, like I said, it's kind of a kin-based culture thing. But it made it much more powerful when he said, "Look, you can't do devious shit outside the gang either." And he had a bunch of examples of that, that I went through in the book.

**Ben Horowitz** (01:30:50):
Like I said, because you're building it from zero, you really have to take a hard line on things that I think people in companies don't even take a hard line on. Is it okay to lie internally? Probably not. Is it okay to lie to a customer? Well, in some organizations it is, but in Shaka's organization, that's as big a penalty as lying internally. These things, I think, end up being really important.

**Lenny Rachitsky** (01:31:15):
We're going to link to this book. This is, What You Do is Who You Are. This is your second book that fewer people know about. And this is one of the stories you tell and just what the lessons are for building-

**Ben Horowitz** (01:31:23):
Yeah. It's kind of the more advanced book. You kind of have to survive to want to care about dealing with the cultural issues. And so the survival book has a bigger audience.

**Lenny Rachitsky** (01:31:36):
Amazing. We're going to keep going with Lightning Round. Is there a favorite recent movie or TV show you have really enjoyed?

**Ben Horowitz** (01:31:42):
Well on TV, I really like Slow Horses, which is the show about the MI6 cast-off guys. And then I haven't seen a lot of movies lately, but I watched Sinners. I went to theater for Sinners. Just the cinematography is unbelievable, and the story is really original, and the acting is incredible, and the costumes are amazing. It's just a great, comprehensive piece of work. The craftsmanship on that thing is just a lot beyond what most people making movies are doing these days, so I really enjoyed that.

**Lenny Rachitsky** (01:32:24):
I hate scary movies, but I watched it and loved it.

**Ben Horowitz** (01:32:27):
It wasn't that scary.

**Lenny Rachitsky** (01:32:29):
It wasn't that scary, but still, it was zombies popping out of corners. That's not my jam usually.

**Lenny Rachitsky** (01:32:35):
Is there a product you recently discovered that you really love? It could be a gadget, it could be clothes, could be something else.

**Ben Horowitz** (01:32:40):
I bought a coffee machine called the Technivorm Moccamaster, which is freaking incredible. In fact, a friend of mine saw it and was like, "What is that?" And I just bought it for him too, because it's so awesome. This thing makes coffee that it's just perfect. There's no bitterness. It's completely clean. It's amazing. The only problem with it is, I can't drink coffee that it doesn't make anymore. I don't know if that's a good thing or bad thing, but ...

**Lenny Rachitsky** (01:33:13):
That might come out someday in the AI future.

**Ben Horowitz** (01:33:16):
Yeah.

**Lenny Rachitsky** (01:33:17):
Two more questions. One, is there a life motto that you often come back to, find really useful in work or in life?

**Ben Horowitz** (01:33:24):
The thing that I would say has had the biggest effect on me is something my father said to me years ago, which is, "Life isn't fair." That seems really, really simple, but I think that the thing that defeats people more than any other thing that I've seen, just in life, is the expectation of some fairness. It's just not fair.

**Ben Horowitz** (01:33:54):
There are all kinds of stuff that are going to happen to you, and happen to everybody, that don't happen to other people, that are completely unfair. But it doesn't matter, because that's the way it is. As soon as you get that idea out of your mind, then you can just deal with it, like, oh yeah, of course it's not fair, but what should I do now, which is the real question, not how do I go back and get people to be fair? Nobody's going to be fair. It's not fair.

**Ben Horowitz** (01:34:25):
It's the nature of it. If you think about it for more than five seconds, you'll realize that. It's as an individual, if you want to make the world a better place, whatever, but as an individual, do not expect anything to be fair, it'll only defeat.

**Lenny Rachitsky** (01:34:43):
Final question. This comes from Shaka, actually. He gave me so many great suggestions. I hate to save this one for last. So the question is, if you had to build a business curriculum from two Hip-hop albums and one Funk album, what would they be and why?

**Ben Horowitz** (01:34:58):
I think probably Follow the Leader by Rakim. And the reason is what we had kind of gotten into earlier, which is leadership. When he came out with that song, which was maybe the greatest Hip-hop song ever written, he's telling people to follow him, follow the leader. And just to have the idea that he was the leader of the entire art form, not just his band, it is amazing idea. And then the way he expressed it was incredible. And then he's got other great concepts in there that would give you ... It's hard to listen to that record and not have confidence.

**Ben Horowitz** (01:36:02):
I think from a competitive, purely competitive standpoint, Stillmatic from Nas, that's the one with Get Ur Self a Gun, that's the one with You're da Man. It's like all of the idea of competition is encapsulated in that album, so that would be the other one.

And then Funk, One Nation Under a Groove, for sure. Because it's like, how do you initiate people into a concept or an idea, and how do you infuse them? One Nation Under a Groove is all about joining the nation, and it's so musically interesting and getting people to be part of that. [inaudible 01:36:58] If you asked me tomorrow, I'd probably have three other ones.

**Lenny Rachitsky** (01:37:03):
Incredible. I'm going to go listen to these. Ben, final question, how can listeners be useful to you?

**Ben Horowitz** (01:37:07):
If you get something that makes you better, please take it. If you need more advice on it, let me know. Look, my job is to help everybody build something great, so if you're an entrepreneur, thank you for that.

**Lenny Rachitsky** (01:37:22):
Also, check out Paid in Full, paidinfullfoundation.org, if you want to learn more about that nonprofit.

**Ben Horowitz** (01:37:26):
Yes, definitely, we would love to have you.

**Lenny Rachitsky** (01:37:29):
Amazing. Ben, thank you so much for being here.

**Ben Horowitz** (01:37:31):
All right, awesome. Thank you, Lenny.

**Lenny Rachitsky** (01:37:33):
Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.

---

## [15/18] The ultimate guide to AEO: How to get ChatGPT to recommend your product | Ethan Smith (Graphite)

**Lenny Rachitsky** (00:00:00):
There's this term everyone's hearing about, AEO.

**Ethan Smith** (00:00:02):
Answer Engine Optimization is how do I show up in LLMs as an answer?

**Lenny Rachitsky** (00:00:06):
It feels like such a big deal to win at AEO.

**Ethan Smith** (00:00:09):
In order to win something like what's the best website builder? At Google, they would win if their blue link showed up first.

**Ethan Smith** (00:00:15):
But that's not the case in the LLM, because the LLM is summarizing many citations, and so you need to get mentioned as many times as possible.

**Lenny Rachitsky** (00:00:21):
ChatGPT is driving more traffic to my newsletter than Twitter.

**Ethan Smith** (00:00:25):
You can get mentioned by a citation tomorrow and start showing up immediately. You can have a Reddit thread, you can have a YouTube video.

**Ethan Smith** (00:00:31):
You can be mentioned on a blog. So early-stage companies can win, they can win quickly.

**Lenny Rachitsky** (00:00:36):
Are the leads that these answer engines are driving to companies actually valuable?

**Ethan Smith** (00:00:41):
Significantly more valuable. Webflow saw a 6X conversion rate difference between LLM traffic and Google Search traffic.

**Lenny Rachitsky** (00:00:48):
A lot of people are seeing this as everything is different. Nothing we've done before is going to work. We have to rethink everything.

**Ethan Smith** (00:00:53):
There's significant misinformation on AEO. There's news articles about how Google Search is going to die because there's a new thing.

**Ethan Smith** (00:01:00):
Google's slice of the pie stays the same. The pie gets bigger.

**Lenny Rachitsky** (00:01:05):
Today my guest is Ethan Smith. Ethan is the CEO of Graphite and my go-to expert for all things SEO. SEO is going through a major transition right now. Everyone used to go to Google anytime they had a question, or were looking for a product or doing research. These days, a lot of people are moving to ChatGPT and Claude, and Gemini and Perplexity to get answers to their questions, and this will only be accelerating over time.

**Lenny Rachitsky** (00:01:29):
And even Google is changing the search experience in a pretty radical way with AI Overviews at the top, and their newly introduced AI Mode, which is basically their own version of ChatGPT. This means that the world of SEO is going through a big change, including the rise of AEO, which stands for Answer Engine Optimization. Basically, SEO for ChatGPT, getting your product to show up in the answers that people get.

**Lenny Rachitsky** (00:01:51):
Ethan has been at the forefront of this new skill and channel. And in this conversation, he shares everything that he's learned about how to get your product to show up more often inside of the answers that people get. The advice that Ethan shares in this conversation is incredibly tactical and worth a lot of money. So please slurp it up and use it for your own products.

**Lenny Rachitsky** (00:02:10):
If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube, it helps tremendously. And if you become an annual subscriber of my newsletter, you get a year free of 15 incredible products, including Lovable, Replit, Bolt, n8n, Linear, Superhuman, Descript, Wispr Flow, Gamma, Perplexity, Warp, Granola, Magic Patterns, Raycast, ChatPRD and Mobbin.

**Ethan Smith** (00:04:41):
Excited to be back.

**Lenny Rachitsky** (00:04:42):
We did a podcast episode just over two and a half years ago. I think of it as the definitive guide on how to win at SEO. People have been referencing it ever since. I'm really proud of what we did there, but things have changed.

**Lenny Rachitsky** (00:04:54):
Things are changing in the world of SEO. And so I'm excited to talk to you again about how to be successful in this new-emerging world where AI is changing how SEO works, the rise of AEO and GEO.

**Lenny Rachitsky** (00:05:08):
Let me start with just this question. How long have you been working on SEO at this point? And has anything come close to being this significant in changing the skill of SEO?

**Ethan Smith** (00:05:19):
Yes. So I got started in SEO in 2007, so it's been 18 years. Actually, the largest change when I got started in SEO, I got started in programmatic SEO and commerce SEO, like NexTag and Shopping.com and PriceGrabber. And that was when you could do mass, auto-generated landing pages.

**Ethan Smith** (00:05:41):
And that was probably the biggest shift, which is Google introduced a bunch of algorithms, Panda and similar things, to prevent you from doing spam. So essentially, you went from the SEO being spam to not spam. That was probably the biggest change, and then this is probably the second-biggest change.

**Ethan Smith** (00:05:56):
I think that the main thing here is it is related to search, but it's a summarization of search and there's new inputs. So it's probably the second-biggest change.

**Lenny Rachitsky** (00:06:03):
Okay, that is really interesting, because I think a lot of people are seeing this as everything is different. Nothing we've done before is going to work.

**Lenny Rachitsky** (00:06:10):
We have to rethink everything. You're saying this is actually the second-biggest change, and just like Google's update back in the day was actually even more significant?

**Ethan Smith** (00:06:17):
Yep.

**Lenny Rachitsky** (00:06:18):
Very cool. Okay, let's set a little context for folks. Let's define some terms. There's this term everyone's hearing about.

**Lenny Rachitsky** (00:06:25):
There's actually two, AEO and GEO. What do they stand for? Are they different? What are they referring to specifically?

**Ethan Smith** (00:06:33):
They, I think, are the same. Ultimately, the definition of a word is whatever a group of people agree is the definition of a word. So I think we'll see what people decide is the definition of the word. I'll put forward my definition. So AEO and GEO are essentially trying to describe the same thing, which is how do I show up in LLMs as an answer?

**Ethan Smith** (00:06:52):
And I personally prefer Answer Engine Optimization versus Generative Engine Optimization, because generative, you can generate images and videos and things other than an answer. Whereas answer is more narrowly defined, so my personal preference is we're talking about optimizing LLMs.

**Ethan Smith** (00:07:09):
So an answer is more narrow of a definition than generative, but ultimately, it's whatever we decide is the name and the definition is what it will be.

**Lenny Rachitsky** (00:07:19):
Okay. Yeah, yeah. Answer Engine Optimization sounds a lot cleaner to me if you had to pick one. So it's good to know they're the same thing. Some people just prefer the latter one for some reason.

**Lenny Rachitsky** (00:07:28):
It's interesting because recently, I don't know if I told you this, but I was looking at my referral traffic. And I found that ChatGPT is driving more traffic to my newsletter than Twitter, which I did not see coming.

**Lenny Rachitsky** (00:07:40):
So somehow it's already happening. I'm excited to learn just how to lean into that potentially and optimize it further.

**Ethan Smith** (00:07:47):
And when did you see the spike? Did you see when it started growing dramatically?

**Lenny Rachitsky** (00:07:50):
Unfortunately, the dashboard I have doesn't give me great peripheral traffic optimization. When do you think I probably saw it?

**Ethan Smith** (00:07:57):
Companies that we work with started in January and it started, one, because of more adoption, but two is because the answers became a bit more clickable.

**Ethan Smith** (00:08:05):
You have maps, you have shopping carousels, you have clickable cards. So I think the clickability of the answer is increased, and then the adoption increased and that was around January.

**Lenny Rachitsky** (00:08:14):
Okay. I want to come back to this question of, "Is this good that ChatGPT is sucking all my content and giving people answers, and then sending me some percentage of that?" But let's not get into that yet. I want to talk about just what kind of impact you can have on having your stuff show up in ChatGPT.

**Lenny Rachitsky** (00:08:32):
So I had the head of ChatGPT, Nick Turley, on the podcast recently. I asked him, "What do you think of all this stuff, AEO, GEO?" He's like, "Don't worry about any of that. Just write awesome stuff, great quality content. It'll figure it out. It'll find the best stuff." I imagine you very much disagree.

**Lenny Rachitsky** (00:08:47):
I imagine you have seen real impact getting your stuff proactively into these answer engines. Talk about just the kind of impact you've seen and just your reaction to that?

**Ethan Smith** (00:08:56):
Yeah. I agree and disagree, but the way that I think about it is anything can be optimized. You just need to understand the underlying systems and the rules of the game, and if you do that, then you can optimize anything. You can optimize algorithms, you can optimize people, anything could be optimized.

**Ethan Smith** (00:09:11):
What I think he probably meant by that, he probably meant two things. One is, "Please don't spam my product." And two is, "If you do, I will see it and I will stop you from doing that." So it's not a long-term, robust strategy to create spam, just like it wasn't a long-term, robust strategy to create spam on Google.

**Ethan Smith** (00:09:29):
Eventually, Google was going to say, "Huge shopping comparison sites are making 100 million auto-generated search pages and I don't like it, and I'm going to get rid of the whole category." So same thing with ChatGPT, anything can be optimized, but if you're spamming it, they'll see that.

**Ethan Smith** (00:09:43):
And they'll have a whole team looking at that and then they'll change your algorithm to prevent you from doing that.

**Lenny Rachitsky** (00:09:48):
What kind of impact have you seen? You've done work with a lot of companies, we'll talk through a few examples.

**Lenny Rachitsky** (00:09:53):
Maybe share one to give us context just like how much can you impact this sort of thing where you show up in, say, ChatGPT more often?

**Ethan Smith** (00:10:01):
You can affect it a lot. So a specific example with Webflow is we are working with Webflow on their SEO. We're working on their content, and we're seeing a lot of wins on the Answer Engine Optimization side.

**Ethan Smith** (00:10:16):
So the specific things that we've done there, one is just traditional SEO. So make landing pages for high-search volume keywords, like best no-code website designer.

**Ethan Smith** (00:10:28):
And then for free, you'll get Answer Engine Optimization impact from that. So that's just traditional SEO, which works very well for AEO.

**Lenny Rachitsky** (00:10:35):
I was just going to say, that sounds exactly the same as regular SEO.

**Ethan Smith** (00:10:38):
Yeah. So I would say everything that works in SEO works in AEO, but there are additional things beyond SEO that also work in AEO. So second thing, and the way that I think about AEO versus SEO is that the head and the tail are different. So the head is different in that in order to win something like what's the best website builder?

**Ethan Smith** (00:10:59):
Even if Webflow's URL shows up number one on the citations, they're not going to win the answer because their URL showed up number one, but at Google they would win. If their blue link showed up first they would win, but that's not the case in the LLM. Because the LLM is summarizing many citations and so you need to get mentioned as many times as possible.

**Ethan Smith** (00:11:18):
So usually when you ask something like, "What's the best tool for X?" The first answer will be mentioned the most in the citations, because that's very different from Google. And so for Webflow, we work with them on YouTube videos, Vimeo videos, getting mentioned in Reddit, getting mentioned in other blogs, affiliates, stuff like that.

**Ethan Smith** (00:11:39):
So tried a bunch of stuff. Stuff that worked especially well was just straight SEO, number one. Number two is YouTube videos, and then the third is Reddit optimization.

**Lenny Rachitsky** (00:11:47):
Okay, wow. So you're saying if you can get to number one, when you ask ChatGPT, "What's the best website builder?"

**Lenny Rachitsky** (00:11:54):
And Webflow's at the top, that doesn't actually drive them as much traffic as simply being mentioned most often across the summary?

**Ethan Smith** (00:12:01):
Yes. And part of why that's interesting is because when startups come to me and ask me for SEO help, my first response is, "Don't do it at all. Spend your time on something else because you're not going to be able to grow SEO early on in search." Because you don't have enough domain authority and it takes a while to get domain authority, and only once you have domain authority can you rank.

**Ethan Smith** (00:12:23):
And so for Google, it's usually something that you do Series A, Series B or later. You don't do it as soon as you start because can't win early on. That's not the case for Answer Engine Optimization, because you can get mentioned by a citation tomorrow and start showing up immediately. You can have a Reddit thread, you can have a YouTube video.

**Ethan Smith** (00:12:42):
You can be mentioned on a blog, like a brand-new YC company launches, everyone's talking about them. They could show up in an answer tomorrow as a result of that. So early-stage companies can win, they can win quickly. And they can win quickly and anyone can win quickly, by getting mentioned as many times as possible by the citations. So that's what's different about the head.

**Ethan Smith** (00:13:04):
What's different about the tail is that the tail is larger in chat than in search. So the average number of words, I think, Perplexity said this to somebody else, said that it was around 25 words, where versus Google words it's around six words. So the tail is just much, much larger. People are asking lots of follow-up questions.

**Lenny Rachitsky** (00:13:19):
The tail, the prompt essentially, the question you're asking?

**Ethan Smith** (00:13:23):
Yes. Meaning that if you map out all of the questions that people ask, kind of like an SEO, long-tail keywords, if you do long-tail questions, the size of the tail is larger. Meaning the amount of questions that are very specific is larger, the share and the volume.

**Ethan Smith** (00:13:42):
And there's probably questions that have never been asked before and questions that have never been searched before, because search can't support lots of really specific, super-specific stuff. Whereas chat is specifically made to ask a bunch of follow-up questions and have a conversation.

**Ethan Smith** (00:13:58):
And so there's all these questions that have never been asked or searched for before that are now being asked, and then you can win that. And when I got started in SEO, it was long-tail SEO where you have a page for every single keyword, which doesn't work anymore, but now the long tail is back in chat.

**Ethan Smith** (00:14:13):
And if you know all those really specific questions that people are asking, you can also win that, and you can probably also win that early. And I've seen examples of early-stage companies who just launched some really specific AI-enabled payment processing API thing, and they will show up. And they'll show up because they're answering questions that's never been answered before.

**Lenny Rachitsky** (00:14:35):
Are the leads that these answer engines are driving to companies actually valuable? Are these good-quality leads for B2B SaaS especially?

**Ethan Smith** (00:14:44):
They are significantly more valuable. So Webflow, we saw a 6X conversion rate difference between LLM traffic and Google Search traffic.

**Lenny Rachitsky** (00:14:55):
Six times?

**Ethan Smith** (00:14:56):
Six times so significantly more qualified. I think that's probably for a couple of reasons. Probably it's because you're so primed because you're having a conversation with multiple follow-ups, and so there's so much intent that you've built.

**Ethan Smith** (00:15:08):
And you've probably really narrowed in on what you want, so when you're going somewhere, it's probably highly qualified. And so we're seeing that it's just a much higher conversion rate.

**Lenny Rachitsky** (00:15:18):
Wow, this is so interesting, and it makes sense. People trust ChatGPT to tell them the answer, and if you are the answer, you have so much advantage. Like that is what people want to know, and then, "Okay. Cool, thank you. I'm going to go check this out."

**Lenny Rachitsky** (00:15:35):
This all just makes sense. Going back to the three levers you shared, essentially it's the things that you see work in driving you showing up more in these answer engines, landing pages, YouTube videos and Reddit. Is that right?

**Ethan Smith** (00:15:49):
Those are some of them.

**Lenny Rachitsky** (00:15:49):
Okay.

**Ethan Smith** (00:15:51):
The other things, so I would break it up into stuff on your site, onsite and offsite. So onsite would be traditional SEO. The difference would be this long tail. I would also say that the difference is lots of follow-up questions about does your product do this thing? What are the use cases, features, integrations, languages?

**Ethan Smith** (00:16:07):
Tell me about your product and really specific details about that and that's on your site. And then the second group would be offsite, which is show up in all the citations. Citations are comprised of video, UGC like Reddit and Quora, affiliates.

**Ethan Smith** (00:16:23):
Dotdash Meredith is showing up all over the place, Glamour, Good Housekeeping, it's like getting mentioned there, blogs, so it's those two groups.

**Lenny Rachitsky** (00:16:33):
And that all sounds very similar to SEO showing up on other people's pages. Showing links from, say, Reddit is always great.

**Lenny Rachitsky** (00:16:41):
It's interesting that Reddit is such a big deal. What's going on there do you think?

**Ethan Smith** (00:16:45):
Okay, Reddit is one of the most interesting things. It's hugely cited in LLMs. And it's probably the number one thing people are asking, customers are asking me is, "How do we optimize for Reddit?" And this goes back to the head of ChatGPT's question about, "Please don't spam my product."

**Ethan Smith** (00:17:06):
And so Reddit is a community where it's real opinions from people, authentic, and it's heavily managed by the community and the community is very good at managing it. And so the obvious strategy for a growth person is, "Let's make a bunch of automated spam and spam Reddit all over the place and get my product to show up everywhere."

**Ethan Smith** (00:17:27):
That's the growth mindset, which makes sense, the hustle mindset. So what are people looking at? They're looking at creating hundreds of fake Reddit accounts pretending to be someone that you're not. I have a single person, I'm going to make 100 Reddit accounts. I'm going to autopost comments and then like my own comments.

**Ethan Smith** (00:17:46):
And then build a trust score, and then shout, say everywhere that my product is the best product. Fortunately, that doesn't work very well, but that's the obvious strategy. And so we're seeing people trying to do that and then we're also seeing those accounts get banned, those comments get deleted. And so we're seeing people trying to spam and being unsuccessful, so that's one strategy.

**Ethan Smith** (00:18:05):
The other strategy is the whole purpose of Reddit is to post useful, high-quality, authentic comments from real people. So at Webflow, we have a couple of people at Webflow going to comments and saying, "This is my name, this is where I work, and here's a useful piece of information." So the strategy is find a thread that is a part of a citation that you want to show up in.

**Ethan Smith** (00:18:30):
Say who you are, say where you work, and then give a useful piece of information, and that works really well. And that sounds simple if you're not in the growth mindset of, "I need to scale this to hundreds of comments." But you don't actually need 10,000 comments, even five could be great and that scales perfectly well.

**Ethan Smith** (00:18:47):
So the Reddit strategy is the obvious strategy, which is just to be an actual user of Reddit. Make an account, say who you are, say where you work, and give a useful answer.

**Lenny Rachitsky** (00:18:56):
We had the early-growth leader from Deel, D-E-E-L, on the podcast a while ago. And this is how they grew initially, before AI even came around, just going big on Reddit and answering people's questions.

**Lenny Rachitsky** (00:19:07):
And like, "Hey, happens to be Deel. Can I help you with this problem?" So that's interesting. It's so interesting that Reddit is what is keeping ChatGPT from being spammed with stuff. It's not that ChatGPT is stopping the spam, its Reddit is just really good at that.

**Ethan Smith** (00:19:23):
I think that in a sense, ChatGPT is policing because ChatGPT is running a search, it's finding citations. There's a search algorithm that's trying to select which citations are useful. There are people at ChatGPT who are tuning their search algorithm to select which sources they trust.

**Ethan Smith** (00:19:40):
I'm sure that there's a search evaluation team saying, "Do I like these citations, yes, no? Is Reddit showing up? I want it to show up." So I think that there are actual people at ChatGPT who are intentionally configuring their algorithm to use Reddit because it's trusted. And if it wasn't trusted, they wouldn't use it.

**Ethan Smith** (00:19:57):
Same with Google. Google has specifically configured their search algorithm to rank Reddit and Twitter and Quora, because they want user-generated content. And if it wasn't good content, then they would change the algorithm and they wouldn't rank it. So I think that they are policing it in a sense.

**Lenny Rachitsky** (00:20:13):
Got it. And all of this is post-training, search-oriented features of these models. It's not data they are trained on, is that right?

**Ethan Smith** (00:20:24):
I would assume that so there's the core model and then there's RAG. So the core model is I'm looking at common crawl on billions of web pages, and then I'm retraining the model. And if you ask something like, "What's the capital of California?" It predicts the next word, which is Sacramento. And that's based on the core algorithm, which is next-word prediction.

**Ethan Smith** (00:20:44):
Then there's RAG and RAG basically means search, retrieval-augmented generation. So I'm going to do a search and then I'm going to summarize the search. There are these two different things. And so most of what I'm describing is about the RAG piece, not the core model piece. To influence the core model is probably extremely hard and maybe you'll see the impact a year later.

**Ethan Smith** (00:21:03):
And it's probably something, some sort of obscure thing that nobody would want to do, like make a million pages that say, "Best product for X is brand." Which I don't think most people want to spend their time on. So I'm mostly focused on the RAG side, because that's the main thing that's controllable.

**Ethan Smith** (00:21:18):
And I think also the LLM is probably not going to say your product if it didn't show up anywhere on the RAG. So I think that's where most of the interesting stuff is from an optimization perspective.

**Lenny Rachitsky** (00:21:27):
Cool. Yeah. I didn't even think about this side of it when we started talking about this, but I think that's an important thing to note, is just this has nothing to do with the training data.

**Lenny Rachitsky** (00:21:34):
This is post-training, once the model's live, what it can do to find recent information using RAG, web search, things like that. Okay. Before we get into how to actually do this step-by-step, how to win at AEO.

**Lenny Rachitsky** (00:21:48):
What are two or three things that you think are important for people to understand to be successful in this world just broadly?

**Ethan Smith** (00:21:54):
First thing is just recognizing that this is related to search. So it's LLM plus RAG, it's summarizing a set of search results usually. So LLM plus RAG, number one. Number two is topics. So in search, a landing page is targeting hundreds of keywords, which we talked about on the last podcast.

**Ethan Smith** (00:22:12):
So I'm not targeting one keyword like I was in 2007, I'm targeting 1,000 keywords, and each landing page needs to target that set of 1,000 keywords, and that's a topic. Same thing is true for Answer Engine Optimization. Each page is targeting hundreds, thousands, maybe tens of thousands of questions.

**Ethan Smith** (00:22:28):
And so I want to group all those questions, which then brings us into content, so how would I rank? How would I get my URL to rank? Or how are other URLs being decided whether or not they rank? Then answer all the questions. The more of the questions that I answer, the better.

**Ethan Smith** (00:22:42):
So in Google Search, if I have a landing page about website builders, the more that my page answers all of the subtopic, follow-up questions, the more likely I am to show up in Google Search. Same with chat, the more you answer all the questions, the better. If you don't answer a question, then you're probably not going to show up.

**Ethan Smith** (00:22:57):
And if you answer a follow-up question and subtopic somebody else is not answering, you're going to be more likely to show up. So topics, number two. The third is question research, so how do I know which questions people are asking? And that's actually pretty hard, because in search, Google just tells you what their ads API.

**Ethan Smith** (00:23:15):
They say, "This is the search volume for this keyword." There's a truth set from Google and ChatGPT is not giving us that, at least not yet. Maybe when they do ads, they'll give us more access to search volume, but there's no truth set. So how do we know the questions that people are asking?

**Ethan Smith** (00:23:31):
One way would just be to take all my search terms and change them into questions. So website builder, you can assume that what's the best website builder is probably a question that's probably asked proportional to the search volume for that keyword, so that's one.

**Ethan Smith** (00:23:44):
But then I mentioned that the tail is larger, and there's parts of the tail that don't exist in search. So how do we know what the tail looks like? And one strategy that you can use, is what are all the questions people are asking you on your sales calls, customer support on Reddit?

**Ethan Smith** (00:24:02):
Mine all those questions that exist somewhere else. Probably those same questions are being asked in chat, so that's another way to find questions. The last is citation optimization or offsite. So again, the LLM is summarizing RAG. So how do we show up with as many citations as possible?

**Ethan Smith** (00:24:21):
And you can break up the citations into different groups, my site, video, YouTube, Vimeo, UGC, Quora, Reddit. Tier-one affiliates like Dotdash, tier-two affiliates, blogs. So it's breaking up all those different citations and having specific strategies for each group.

**Lenny Rachitsky** (00:24:40):
What is Dotdash exactly?

**Ethan Smith** (00:24:42):
Dotdash Meredith is a large media conglomerate with Good Housekeeping, Allrecipes, Investopedia. It's probably the most successful SEO company of all time.

**Ethan Smith** (00:24:53):
And it's also one of the most cited, probably the most cited in LLMs as well.

**Lenny Rachitsky** (00:24:59):
Wow, did not know this. As you talk, I think about if you go to Google, no offense, Mr. SEO, but if you go to Google these days, it's just like a bunch of unuseful stuff, just like this hyper SEO'ed content.

**Lenny Rachitsky** (00:25:12):
Do you think ChatGPT will be able to avoid that fate where it's just a bunch of hyper SEO'ed content that is not what you actually want?

**Ethan Smith** (00:25:19):
Probably. And what you're saying with SEO is that everyone's rewriting each other's content, nonexperts rewriting each other's content. So I get a content-scoring tool, which then looks at all the results in Google and it says, "These are all the things that the other articles are saying. And then this is what you haven't said yet, so here are recommendations for how to be more typical."

**Ethan Smith** (00:25:40):
And then everyone rewrites each other's article. And then one other interesting thing is that the majority of landing pages drive no impact. So we did an analysis where one out of 20 landing pages drive roughly 85% of all your traffic. So 19 out of 20 landing pages drive little to no traffic, which means if I want to get ROI, I need to spend a small amount of money on a large number of pages.

**Ethan Smith** (00:26:06):
And so then you get a nonexpert to say, "Rewrite this other person's article," because that's cheaper than hiring someone from The New York Times to write your article about what's the best payroll management software? But if you knew the few things that would work, the few landing pages that would work, and you wrote them really well, then you could push all that money to that one page, which is what we try to do.

**Ethan Smith** (00:26:28):
But right now it's people rewriting each other's content, so Google has not solved that yet. That's probably a very hard problem to solve. Will they ever solve that? Probably. Will ChatGPT ever solve that? Probably. How I would solve that would be, one concept would be information gain. So did you say something that somebody else didn't say? Two is how typical are you?

**Ethan Smith** (00:26:50):
Are you so typical that I think that you're a rewritten version of somebody else's content? Potentially, Google has EEAT, expertise, authority, trustworthiness, which actually I don't see having an effect unfortunately, but it could. And I could say, "Well, this person's an expert, this person's a certified financial advisor, rank them higher."

**Ethan Smith** (00:27:08):
And I'm actually not seeing that, but they could increase the weight of that. So these are all potential solutions, but I'm sure that the reason why it has not been solved yet and why everyone's rewriting each other's articles. It's probably just hard to build an algorithm to solve that, but will they ever solve that? Probably.

**Lenny Rachitsky** (00:27:23):
This algorithm or heuristic you just shared is so interesting, because it's helpful for just what is good content, say, with a newsletter or a podcast? Info gain, and is it typical?

**Lenny Rachitsky** (00:27:34):
Are you adding something new to the conversation and is this unique? I think it's a really good strategy for just producing great newsletters and podcasts and all the content in the world.

**Ethan Smith** (00:27:45):
Yes. And ideally, did you do original research and do you have some domain expertise? And did you mention that in the content?

**Lenny Rachitsky** (00:27:51):
This is a great heuristic for just content in general, which is exactly what you want these algorithms to be looking for, so the alignment is there.

**Ethan Smith** (00:29:12):
First, I would figure out which questions I want to rank for. How I would figure out which questions I want to rank for, I would take my search data. I would maybe take my paid search data, like, "What are my money terms? What are my competitors' money terms?" So if I'm rippling, what is deal.com bidding all their paid search on?

**Ethan Smith** (00:29:30):
Then I would transform those into questions. And actually you can just give those keywords to ChatGPT and say, "Make these into questions," and it does a pretty good job. So take your competitors' paid search data or mine or your own, put it in ChatGPT, get the questions. That's step one. Step two is then track them, so put them in an AEO tracker, in an answer tracker.

**Ethan Smith** (00:29:51):
Third thing would be who is showing up as citations? And then have a strategy for each of those different groups of citations. The third would be make your own landing pages. So what are the kinds of landing pages that are appearing? Is it a listicle? Is it a category page? Is it an article, tool page? Figure out what page type that seem to be showing up the most, and then you make your own page for that.

**Ethan Smith** (00:30:14):
How do you have your page rank? Answer all the follow-up questions. So what are all the follow-up questions that someone might ask? You could go back to your search data and look for groups and themes of your keywords that are in your SEO topic. Same thing for AEO topic. Then on the offsite, so different strategies for each of those groups.

**Ethan Smith** (00:30:37):
And I would say that depending on the company, paying an affiliate to mention you, that's pretty easy if you have the money. So if you want to be the best credit card, you pay Forbes and then you're the best credit card. So that's strategy one, expensive, easy, controllable. The YouTube, Vimeo strategy is also actually pretty easy because there's no community saying, "I don't like your YouTube video."

**Ethan Smith** (00:30:59):
You make a YouTube video, you do whatever you want. Maybe people view it, maybe they don't, but you can make a YouTube video or a Vimeo video. And the interesting thing with this, especially for B2B, is that YouTube, Vimeo, other video sites, the kinds of things people make videos for are food, traveling, fun, beauty.

**Ethan Smith** (00:31:18):
There's not that many videos about AI-powered payment processing APIs, as interesting as that is, but it's a great money turn. So if you make a video for these really specific, high-LTV, maybe nonglamorous keywords, questions, topics, that's actually a big opportunity. Then Reddit, so I mentioned with Webflow what we did, which is just make a Reddit account, say who you are, say where you work and give a useful answer.

**Ethan Smith** (00:31:48):
That one is a little bit trickier because the community might say, "I don't like your answer." So you can't guarantee that your comment is there, but it is easy, so I would do that group. Oh, and then experiment design, experiment design and seeing what works. So SEO and AEO are both interesting in that the majority of the information and best practices are not correct.

**Ethan Smith** (00:32:13):
And the reason why is because people don't do analysis. Somebody will say something and then it will get repeated, and then it becomes best practice and no one ever did an analysis. So you did all the stuff that I just mentioned. Do an experiment and see if it worked. Maybe half the stuff I said works, maybe half it doesn't. Do your own experiment.

**Ethan Smith** (00:32:32):
Most best practices, most blog posts are not correct. So how do you set up an experiment? You get your questions, you turn tracking on, give it a couple of weeks. Make your changes, have a test group, have a control group. Intervene on the test group, make your changes, see if the chart went up, see if the control group did not, and now you know your particular strategy worked.

**Ethan Smith** (00:32:55):
So I would definitely do experiments and I would not assume that stuff you read online is correct. And then you need a team, so who's your team? Probably your team is your SEO team, or your SEO agency or your SEO consultant. Probably, hopefully they can do this stuff, and then however, what I think is hard to hire for is the offsite stuff.

**Ethan Smith** (00:33:16):
So most SEO people are not going to be amazing at creating YouTube videos and Reddit strategy, so you might need a different person for that. That might be a community generalist marketing person. So it would basically be your SEO team, "Please now do Answer Engine Optimization." And then marketing community team, "Please help me show up in more citations."

**Lenny Rachitsky** (00:33:35):
Wow, okay. That is incredibly valuable. Thank you for sharing all that. I imagine some of this is you're just giving away a lot of amazing advice for free here. Thank you. First of all, I imagine there's a layer, there's only so far you can go on your own.

**Lenny Rachitsky** (00:33:49):
And so eventually it's like, "Okay, we really need help." And that's where a team like yours comes in. Let me ask a few questions here to follow up. One is this tracker concept. So what is this tracker, it can track how often you show up? Say Lenny's Newsletter shows up and answers for the questions that I'm targeting?

**Ethan Smith** (00:34:03):
Yeah, so there's answer tracking, which is like keyword tracking. So keyword tracking would be best growth podcast, and you put that in a keyword tracking tool. There's 100 of them, they're all the same, and you see whether or not what you rank. Maybe you rank, hopefully you rank number one. Now in answers it's very different, but it's related.

**Ethan Smith** (00:34:25):
So if you ask the same question, you will have different answers each time. If you ask a question, there's different answers per run. And so ChatGPT is basically calculating a distribution of all the potential answers it would give. And depending on when you ask it, it's basically like a weighted, random sample, and so you're going to get different answers.

**Ethan Smith** (00:34:46):
You also have question variants, so you can ask different versions of the same question, and you might show up in one and you might not show up in another. Then there's different surfaces, there's Perplexity, there's Gemini, there's ChatGPT, there's Meta AI, and so these surfaces have different answers.

**Ethan Smith** (00:35:00):
And so you essentially need to create a share of voice across all these different things like a distribution. So how often am I showing up? What's my average rank? And that's answer tracking. So then where do you get answer tracking? And answer tracking is essentially an evolution of keyword tracking. So we have a page with 60 different answer tracking tools.

**Ethan Smith** (00:35:22):
But it's ultimately just like keyword tracking, it's all the same thing roughly. And so pick one of the 60, we have answer tracking, we're building answer tracking. There's 59 other options, probably all pretty good, probably all pretty similar, but pick one. My general suggestion is pick the cheapest one that does what you need.

**Ethan Smith** (00:35:41):
Just like keyword tracking, you can only, there's not a premium version of keyword tracking. You rank number three or you don't. So pick the keyword tracker that is the cheapest that does what you want. Same with the answer tracking. And so then when I'm doing the experiment, put your answers in, track them, see a chart over time, see your average rank.

**Ethan Smith** (00:35:58):
How often are you showing up and what's your average rank? And then you make a change, and then hopefully you go up.

**Lenny Rachitsky** (00:36:03):
Amazing. I love this term voice share. I never heard that before, it makes sense. Like percentage of time you're showing up in LLMs, is there an LLM, is it just like ChatGPT?

**Lenny Rachitsky** (00:36:13):
Is Google equivalent now to ChatGPT? How do you recommend people think about, say, Gemini or Claude, or Perplexity and others?

**Ethan Smith** (00:36:21):
So interestingly, there are similar, foundational algorithms across all of these. They're all using search, they're all using search, and they're all using LLMs, which foundational algorithms are all the same. The results are actually pretty different. So we're doing a study, we're seeing that Google and Bing are not that similar search engines.

**Ethan Smith** (00:36:41):
We're seeing that ChatGPT citations and Google Search results are actually not that similar. Perplexity is interestingly more similar to Google than ChatGPT. We did a study looking at thousands of questions and saw the citation overlap with Google Search results was around 35% for ChatGPT and Google, so not that much.

**Ethan Smith** (00:37:02):
Perplexity was around 70%, but essentially they're all similar algorithms, but with very different citations and results. So then look at which surfaces have the most traffic and then track those. You probably don't need to track all of them, but look across all those.

**Ethan Smith** (00:37:17):
But you do need to look at your share of voice or the percent of time you show up across all these surfaces. You need to ask the question multiple times, and you need to ask the variance of the question to truly know how frequently you're showing up.

**Lenny Rachitsky** (00:37:28):
Considering that ChatGPT, they're going to hit something like a billion weekly active users in the near future, do you need to worry about Claude and Gemini and Perplexity?

**Lenny Rachitsky** (00:37:39):
Is the traffic there meaningful? I know it is a lot of people, but how important is it to focus on those other LLMs?

**Ethan Smith** (00:37:45):
Well, the way that I would answer that is I believe AOL was one of the largest search engines early on and Google was not. And so we could ask in 1999 or whatever, "Should we just focus on AOL search and Yahoo search? Do we really need to worry about Google?" And the answer is we don't actually know.

**Ethan Smith** (00:38:04):
It's very early, we don't know who's going to win. I do think that ChatGPT for sure is going to be large. Will Perplexity or Claude or these others compete with them? Probably. Just like search, I think that there will probably be multiple winners and probably you'll need to optimize for several.

**Ethan Smith** (00:38:18):
I don't think that you'll need to optimize for 10, but there'll probably be around three or so that will win that you want to optimize for.

**Lenny Rachitsky** (00:38:26):
Okay. By the way, I want to make it clear, I love Claude. I use Claude and ChatGPT equally, roughly. I didn't want to make it sound like ChatGPT is the only product people use.

**Lenny Rachitsky** (00:38:34):
Okay. How does this strategy change depending on the kind of company you are? Say you're a B2B SaaS company or a consumer product, does anything in these seven steps change significantly?

**Ethan Smith** (00:38:44):
Let's take B2B, for example. The first thing is that the citations that are being mentioned are going to be quite different. So citation optimization will vary quite a bit.

**Lenny Rachitsky** (00:38:53):
Just to clarify what you just said, what do you mean when you say citation strategy is different?

**Ethan Smith** (00:38:57):
Meaning the citations that show up for B2B versus marketplaces are different kinds of citations. So for B2B, it might be like TechRadar shows up a ton when I ask questions. I've never read TechRadar, but for some reason it shows up all the time. I'm sure it's great. But TechRadar is showing up a ton for B2B for whatever reason.

**Ethan Smith** (00:39:19):
In commerce, it's not going to be that, it's going to be Glamour and Cosmopolitan. For marketplaces, it'll be Eater and Yelp, TripAdvisor, places like that, so the kinds of citations that show up are different. Most of the stuff that I've been talking about is specific to B2B stuff that's different for commerce.

**Ethan Smith** (00:39:38):
So for most B2B questions, the answers are not clickable. There's nothing to click on. And so if you actually want to measure the impact, you cannot just look at last-touch referral traffic. You have to see whether or not you showed up in the answer with tracking. And then you also need to ask the user, "How did you hear about us post-conversion to actually know the impact?"

**Ethan Smith** (00:40:00):
So it's harder to track for B2B. Also for B2B, you're probably deciding which payroll management software to use after 50 touchpoints. With a brand, it's not going to be you just search for something, you suddenly spent $100,000 on payroll management software. So that's B2B. Commerce is different, so Commerce actually now has more clickable cards like you would in a Google.

**Ethan Smith** (00:40:21):
So if you ask, "What's the best TV for apartments?" There are actual shoppable cards. Those shoppable cards are showing multiple sellers. Those sellers have rich snippets. Schema is important, the number of reviews are important, so it's actually quite different. You can look at last-touch referral traffic to get a good sense about the number of conversions that you're getting.

**Ethan Smith** (00:40:44):
For commerce, similar with restaurants and hotels and local marketplaces, similar there. And then I would say early stage is also different. So I mentioned earlier, early stage my recommendation is don't do SEO at all. For Answer Engine Optimization, definitely do AEO, and only do citation optimization and long tail. Don't do any of the mid-SEO stuff, just get cited and answer really specific questions.

**Lenny Rachitsky** (00:41:12):
It's so interesting that so much of this is just showing up as the little tag/pill in the answer, because it's obvious now that I think about it.

**Lenny Rachitsky** (00:41:21):
That's the only way someone will get to your site from an LLM is just clicking that, "Okay, let me go read this article."

**Ethan Smith** (00:41:27):
Yes. But what they will do is they will open a new tab, and they will type in the brand name and they'll go to Google.

**Ethan Smith** (00:41:33):
And then they'll click on your domain, and you will think that it was a branded Google Search when it wasn't.

**Ethan Smith** (00:41:38):
Or they'll open up a new tab and they will type in your domain, and they'll go directly to your domain and you'll falsely think that it was direct traffic.

**Lenny Rachitsky** (00:41:46):
Coming back to a question you raised at the beginning. So for my newsletter, the fact that they're sucking up all this content, I don't even know how much, and sending me some percent of traffic.

**Lenny Rachitsky** (00:41:55):
Do you have any, I don't know, just sense of is this good? If you were running my newsletter, would you encourage all these outlets to suck up my stuff? And then be like, "Oh yeah, you could check it out in Lenny's Newsletter if you want"?

**Ethan Smith** (00:42:08):
Yes. And I would give the same answer that Brian Balfour gave on your previous episode on this, which is that it's not your choice whether to play the game. You are playing the game whether you want to or not, so you might as well try to show up. If you just say, "Don't look at any of my data," then you cannot show up and your competitors will.

**Ethan Smith** (00:42:27):
Now, what you can do is you can say, "I don't want you to train on my data, so you can index my site, but please don't train on my data." And they have different user agents for that and different bots, so you can just say, "And we're building a Webflow app to block training but not indexation."

**Ethan Smith** (00:42:43):
Or you can just put it in your robots.txt, "This training bot not allowed. Index bot, you are allowed." So if you're concerned about that, I would suggest that, and I think probably a lot of people will do that. But saying, "You can't index my site at all," that doesn't make sense to me.

**Lenny Rachitsky** (00:42:57):
Such a good point, because I don't know if I have competitors in this exact space, but basically they would show up instead and then I lose all that traffic.

**Ethan Smith** (00:43:05):
Yes.

**Lenny Rachitsky** (00:43:06):
Such a good point. Okay. Let me come back to the steps you shared just to see if there's something here that's worth diving into a little further. So this is essentially how to be more successful showing up in LLM responses. One is figure out what questions you want to rank for.

**Lenny Rachitsky** (00:43:19):
And you could do this by looking at what your competitors are advertising and their paid ads and things like that. Just look at the terms, ask almost ChatGPT or Claude, "Turn these into questions people would ask to find these terms." Then set up a tracker to see just how you're doing today. How often are you showing up?

**Lenny Rachitsky** (00:43:36):
There's a million trackers, you have a link willing to check these out. Then you look at who is showing up today? Where are they being taken today? Use that to inform landing pages that you create to answer those questions better. And you make it very clear that it's very important not to just answer that main question, but also follow-up questions. Then there's offsite stuff.

**Lenny Rachitsky** (00:43:57):
So get into affiliates like Dotdash, YouTube, Reddit, Quora sounds like are the core, and then run an experiment. So you look at this tracker, and let me actually ask this, and the next step is just set up a team. But just to come back to this step, how do you set up an experiment that isn't just like a before, after? How do you do a control group situation?

**Ethan Smith** (00:44:19):
Yeah. So what I would do is I would take 100 different questions, half of them I will intervene, half of them I won't. Or let's say, let's take 200 questions. So 100 of the questions, I'm not going to do anything, so that's my control group. And we are seeing a fair amount of variance and answers just without doing anything at all, so you definitely want a control group.

**Ethan Smith** (00:44:40):
And also we're seeing people are using LLMs more and LLM traffic is going up. So you definitely need a control group, especially in Answer Engine Optimization. So control group is, "Don't touch it at all, leave it as it is." That's the control group. Test group would be, "I'm going to now comment on Reddit threads, so let's test that."

**Ethan Smith** (00:44:57):
Or I'm now going to make a YouTube, Vimeo video, or I'm now going to pay Forbes advisor to say that I'm the best credit card. Maybe break those up into a few different buckets, track them. Have a couple of weeks before, a couple of weeks after, compare against the control group.

**Ethan Smith** (00:45:11):
And then the stuff that went up when the control group did not worked, and the stuff that didn't did not, and then reproduce it. So reproducibility is very important. And my background's in academic research, and it's common to do a study that cannot be reproduced. And so for something to truly be accepted with an academian, it needs to be reproducible.

**Ethan Smith** (00:45:34):
Meaning multiple people have done this study and reproduced that thing over and over again. And especially in SEO, it's common for something to change. And you think that it was this thing that caused it and it's actually not, and you just assume forever that that works. So reproducibility is very important.

**Ethan Smith** (00:45:49):
Try to do that study multiple times, try to get studies from other people, and if it works 10 times, then it probably works. And this comes back to the waste problem, most work is wasted in SEO. Most work is wasted in AEO, so how do you know what's not wasted? You do an experiment, you don't assume that what you read online is true.

**Ethan Smith** (00:46:07):
You do your own experiment, and then you reproduce it multiple times, and keep doing the stuff that works and don't do the stuff that doesn't.

**Lenny Rachitsky** (00:46:15):
It feels like such a big deal to win at AEO. Just coming back to this idea that people are coming to ChatGPT, Claude, Gemini looking for an answer.

**Lenny Rachitsky** (00:46:25):
If you're that answer, I feel like that could just make or break your company. It feels like even more important than SEO, just getting this right.

**Ethan Smith** (00:46:33):
I would say that where I want to get the most conversions possible, how big is the channel? The channel is not as big as search. The search is definitely larger, but it is a substantial channel now. And Webflow, they get 8% of those signups from LLMs.

**Ethan Smith** (00:46:50):
It's now one of your top channels so it's large. It's not the largest channel, it's not the number one channel. Paid is probably the number one channel, but it's definitely a substantially large channel and one worth optimizing for.

**Lenny Rachitsky** (00:47:02):
And as you said, probably growing over time.

**Ethan Smith** (00:47:04):
Yes.

**Lenny Rachitsky** (00:47:05):
Okay. Let me zoom out a little bit, and let me just ask you this.

**Lenny Rachitsky** (00:47:08):
What do you think are maybe the most surprising or underdiscussed topics when it comes to AI and SEO and AEO that we haven't already talked about?

**Ethan Smith** (00:47:18):
The first thing is that there's significant misinformation on AI and on AEO, and it's pretty extreme. It's unusually the percent of misinformation to correct information is pretty substantial. So one example is every two years there's news articles about how Google Search is going to die or it is dying because there's a new thing.

**Ethan Smith** (00:47:42):
So that's happening right now with AI Overviews and with AEO, Google's going down, which is not true. Before that it was TikTok search, so everyone is using TikTok now. Gen Z is using TikTok, they're never going to use SEO. SEO's going to be dead, and so you really need to focus on TikTok search, which is not false. It's not untrue, but it's not taking share away from Google, it's just a new surface.

**Ethan Smith** (00:48:05):
And then before that it was Instagram, and then before that it was Facebook and it was YouTube. And people do search and discover on Instagram, TikTok, YouTube, but it doesn't take away from Google Search. It adds on top of it. These are all new channels, so Google's slice of the pie stays the same, the pie gets bigger.

**Ethan Smith** (00:48:23):
And so misinformation about Google going down, Google is not going down. Google published something recently, their VP of search explicitly said, "I looked at the traffic that we're sending to publishers, and it is not down, it's up slightly." So it is not true that Google Search is going down.

**Ethan Smith** (00:48:37):
And most of the news information about that is saying that it's going down, so that's the first surprising thing. The second surprising thing is tooling. And I've never seen a channel where these extremely expensive tools that essentially do commodity tasks. So imagine if I said, "I'm going to charge you $50,000 for keyword tracking."

**Ethan Smith** (00:48:59):
You would say, "Well, of course, that's absurd. It's keyword tracking, I could write this in a day." No one would do that. But for answer engines, it's mysterious and people don't really know how it's working. Also, the slope of the growth curve is so significant, that I'm seeing people spend huge amounts of money on what are essentially keyword tracking commodities.

**Ethan Smith** (00:49:19):
So that's the second thing. The third thing is the growth curve of the channel. And we did a Reforge AEO webinar a year ago, and there was excitement and then it died and there was very little excitement about it. This was in June, and then people didn't really care. They were intrigued intellectually by it, but they didn't care because they didn't see the impact from that.

**Ethan Smith** (00:49:40):
So there was essentially very little interest between July and January, and then suddenly in January it's just skyrocketing. So it's ChatGPT launches, people are very interested, and then it's not that interesting for growth people. And then there's this little spike in June, and then it's like this, which is usually not what you see with a new channel.

**Ethan Smith** (00:50:01):
So the slope of the curve is unusually steep, and the shape of the curve is also very unusual. The last is that a lot of people do think that SEO and AEO are different and they're not different. I think probably part of that is because it sounds great to say that there's this new channel, it's completely different.

**Ethan Smith** (00:50:23):
And I'm an expert and I have a tool to sell you, and it's totally unique and all these other tools are not relevant. In reality, it's actually there's quite a bit of overlap. There is the difference of the citation optimization. The head is different and the tail is different, but the core technology is pretty similar. So those are probably the most surprising things.

**Lenny Rachitsky** (00:50:43):
This piece about January being the inflection point, you mentioned that it was because references started showing it more prominently. Is that the big change?

**Ethan Smith** (00:50:51):
I think it's increase of adoption of LLMs by people, so it's just actually growing more and then the clickability. And I am seeing, you are seeing now this large increase of actual clicks.

**Ethan Smith** (00:51:02):
Probably before you got no clicks, even if you showed up an answer, so the clickability of the answer has increased.

**Ethan Smith** (00:51:08):
Especially for things like commerce and local and hotels, because they have these rich modules where you can click on stuff and go somewhere, which was not true before. That and I think people are just using LLMs more.

**Lenny Rachitsky** (00:51:20):
Ethan, let me just say, I'm learning so much from this conversation, what a fun thing. I could see, it's just clear how much you love this stuff, and just how nerdy and deep you get into it. And it's just fun to talk to someone that's so deep and knowledgeable about all these things, so thank you for sharing all this with us.

**Lenny Rachitsky** (00:51:36):
I'm going to go in a slightly different direction. There's this whole world of AI content, people generating content with AI, generating landing pages. Just like, "Oh my God, SEO is never going to just generate all this stuff. AI is going to make all this stuff easier."

**Lenny Rachitsky** (00:51:48):
You guys did a really big study on how that works, whether it's a good idea to generate content with AI. Can you just talk about what you learned from that, and how people should think about AI in generating content?

**Ethan Smith** (00:51:59):
Yes. So I remember when ChatGPT launched and Brian Balfour posted on LinkedIn, "What do you people think that is going to happen from ChatGPT and AI?" And my immediate response is spam, so just lots and lots of spam, especially SEO spam. And then there was a whole industry around AI-generated content, and I knew immediately that it wouldn't work.

**Ethan Smith** (00:52:20):
And the reason why I knew it wouldn't work, and when I say AI-generated content, I mean automated content with no human-in-the-loop. So I think that the future of content is clearly AI-assisted. Clearly, you and I will be using AI to help us write, so it's not no AI at all, but it's not 100% generated with AI. I immediately knew that it wouldn't work.

**Ethan Smith** (00:52:38):
Why did I know that? I knew that because I created spam in 2007, and I knew what Google did about it and how, and I knew the exact same thing was going to happen. So what I did in 2007 is I and all the other shopping comparison people scraped all each other's content, reviews, chopped it up, scraped content, 100 million search pages, snippets, and it worked really well.

**Ethan Smith** (00:53:02):
And then it stopped working, and then all those companies disappeared. I knew that was exactly what's going to happen with AI-generated content. And so from the beginning, I've not focused on AI-generated content. Many people have, but I don't know, so maybe it does work. There's lots of case studies about it working.

**Ethan Smith** (00:53:20):
So let's do the study, let's do an analysis. So we took, we looked at both Google and at ChatGPT where we took thousands of searches and thousands of questions, and we put those searches into Google Search. We put those questions into chat and the ChatGPT, and then we looked at the citations or the Google Search results. Then we looked at an AI detector.

**Ethan Smith** (00:53:43):
So we used Surfer SEO's AI detector. Now, when I tell people this, they say, "Well, you can't detect AI." So then we evaluated the efficacy and the accuracy of the AI detector. So we did that by generating thousands of AI-generated articles and it was very predictive. And then we looked at real articles, we did that two different ways.

**Ethan Smith** (00:54:05):
One way is we write real articles, and the other is we took a random sample of 100,000 URLs from Common Crawl over the last five years. And then we looked at the AI detector before ChatGPT was launched, so it necessarily was content not created by a human. And then the false positive rate was around 8%, so basically the AI detector is very accurate.

**Ethan Smith** (00:54:29):
So we took that, then we ran it on the content. So then what we saw was around 10% to 12% of content in Google Search, and then ChatGPT or AI-generated, 90% are not. And we ran a correlation analysis showing the exact same thing. So we essentially did a very rigorous study showing that AI content does not work. AI-assisted content edited is great.

**Ethan Smith** (00:54:52):
We do that sometimes, other people do that, that is clearly the future of content. So that does work and should work and that's good, but purely 100% AI-generated does not work. So then the second thing that we did was we found that, this was unexpected, but we found that there's more AI-generated content on the internet than human-generated content.

**Ethan Smith** (00:55:13):
So back to the Common Crawl study, we looked at 100,000 different URLs over the past five years. And then you can see this curve where AI-generated is now higher than human-created. So there's more AI-generated content on the internet than human-generated content, which is disturbing. So then let's say that AI-generated content did work.

**Ethan Smith** (00:55:32):
If AI-generated content worked, then everyone would do it. Just like in 2007, shopping comparison sites, if I can scrape my content, why would I pay anyone to write it? I'll just scrape it from you and I'll chop it up. So then everyone will do that, and then it will go from most content is AI-generated to almost all of the content is AI-generated.

**Ethan Smith** (00:55:51):
Then what will happen if that works, is that Google now becomes a search engine for ChatGPT responses. So if Google's a search engine for ChatGPT responses, there's no reason for Google to exist. Just go to ChatGPT, which is the exact same thing that happened in 2007.

**Ethan Smith** (00:56:04):
Google said, "I see all these shopping comparison search engines showing up in my search results. So I'm essentially a search engine for search engines." I should be showing the TV in my results. I shouldn't be showing other vertical search engines, so I'm going to get rid of them and I'm just going to go straight to the product.

**Ethan Smith** (00:56:23):
The same thing will be true for ChatGPT. Now for ChatGPT, let's say that ChatGPT ranks its own derivatives in its citations, so then you have this infinite loop of derivatives. So I go to ChatGPT, I say, "Generate 10 articles." I put those articles into the citations and then I say, "Summarize these citations that were derivative."

**Ethan Smith** (00:56:41):
And then I keep on doing derivatives of derivatives, and then you have an infinite loop of derivatives, and now AI is summarizing itself. There's a paper about this called Model Collapse. So again, there's the core algorithm and then there's the RAG piece. So the core algorithm, a group did a study showing model collapse, which was what if you feed in AI derivatives into the model and train the core model on the derivatives?

**Ethan Smith** (00:57:06):
And then what happened was you had all these problems, hallucinations, things break very quickly. Okay. So then we did a study on what if you feed derivatives into the RAG piece? So generate 10 derivatives, put that into RAG, summarize that. And then generate 10 more, and then summarize my summarizations, infinite loop of derivatives. What happens?

**Ethan Smith** (00:57:25):
And so what happens is there's a wisdom of the crowd. The LLM is summarizing the opinion of many people. So if you ask a question like, "What's the best flavor of ice cream?" There's not one answer, there's thousands of opinions. So the LLM is summarizing these many, many opinions in this wisdom of the crowd.

**Ethan Smith** (00:57:40):
And the wisdom of the crowd basically says that, "If you take the average of a large group of people, their average response will be better than the best single individual in the group." And so it's better to have more diversity of opinions, wisdom of the crowd. So what happens to the infinite loop of derivatives? You essentially converge on one opinion.

**Ethan Smith** (00:58:00):
So if you ask, "What's the best flavor of ice cream?" It will eventually say, "It's vanilla and it's only vanilla, and there's no other flavor of ice cream." And so that's a simple example, but if you feed in derivatives of derivatives into the model, you'll basically take the wisdom of the crowd.

**Ethan Smith** (00:58:15):
And that will shrink and you'll have a single opinion on everything, which is really bad. So that's what happens if AI content, 100% unassisted AI content works.

**Lenny Rachitsky** (00:58:25):
I'm afraid of this world where everything is trained on AI, and AI is trained on AI and generating AI, and just like nothing is trusted. And I love how it's interesting just how much of these incentives are driving this.

**Lenny Rachitsky** (00:58:36):
If ChatGPT was finding this valuable, this is what people do and then just goes off the rails. So there's just some team there that is keeping this from happening. How do you think this evolves?

**Lenny Rachitsky** (00:58:46):
If you were them, what would you do over the next few years to keep things high quality and not drive these perverse incentives?

**Ethan Smith** (00:58:55):
So I would identify what the perverse incentives might be, and AI-generated content is one of them. The second thing is I think that LLMs and search are going to converge. And so you're seeing that with Google Search where they're having LLM, AI Overviews. You're seeing that with LLMs where they're incorporating maps and shopping carousels, and it's converging on search.

**Ethan Smith** (00:59:14):
I think it'll converge on a single experience, so that's the first thing. Figure out what 2007 Ethan would do not to create spam and make sure that he doesn't do that, like AI-generated content or it's great content. That'd the second thing. And the third thing is there's all these other interesting features, use cases that LLMs can be great for.

**Ethan Smith** (00:59:33):
So LLMs could be great for remembering everything that you've ever asked. It could be good for personalizing stuff specifically to Lenny. One interesting use case that I think will eventually come would be, I say, "Plan a trip to San Francisco," and decisions are made for you without any intervention. I have this wonderful EA named Jen.

**Ethan Smith** (00:59:51):
And I say, "Jen, I'm going to Miami. Please, just do everything for me," and she does everything for me. She knows me, she knows my preferences, she knows that I want a ocean view and I want a restaurant with music. She does all of that and I don't have to intervene. AI can essentially do that eventually, and that would do that because it would deeply understand you.

**Ethan Smith** (01:00:09):
It would remember everything about you. It would have context, it would have a reasoning, and then it would be able to make all these decisions without your intervention, which would be autonomous agents. So I think that that's also another very interesting place for someone like me to optimize for as well.

**Lenny Rachitsky** (01:00:25):
Yeah. I was just going to say, just imagine not even being told this is what you're choosing. Like, "Oh, and go check out, subscribe to the best newsletter out there." And if you're out there, the good things will happen.

**Lenny Rachitsky** (01:00:36):
Wow, what a wild world. Is there anything else that we haven't covered that you think would be helpful to folks that are trying to get better at this stuff? Try to take the first steps down this road of AEO?

**Ethan Smith** (01:00:49):
Yes, the most exciting topic, which is help center optimization and support.

**Lenny Rachitsky** (01:00:53):
Sweet.

**Ethan Smith** (01:00:54):
So I mentioned that people in chat are asking follow-up questions. They're looking for tools. Do you have this feature, this use case, this integration? And that frequently can be answered in help centers. Usually, you would not have an SEO team and say, "We really want you guys to focus on the help center."

**Ethan Smith** (01:01:14):
But in chat, since there's all these questions about can you do this thing, can you fulfill my use case? A help center is actually a great place to do that, and so I think how can you optimize the help center? So number one is it's frequently on a subdomain. For whatever reason, subdomains don't work as well as subdirectories, so move it to a subdirectory, number one.

**Ethan Smith** (01:01:33):
Number two is make sure that you're cross-linking well. So usually you do not have optimized internal links, so link from help center page to help center page, make sure there's lots of cross-linking. The third is you probably have help center content about the head, but the tail you probably don't have any help center content for.

**Ethan Smith** (01:01:51):
So an example of this is I was looking for, I wanted to track our sales calls and look to see who was in the meeting and what the sentiment was. And I wanted to put that into Looker, so I said, "Which meeting transcription tool integrates with Looker?" And the answer is none of them, but you could use Otter because Otter has a Zapier integration.

**Ethan Smith** (01:02:15):
You could send a Zap of the meeting, put it into BigQuery, and then do Looker on top of that. But there wasn't a help center article about that because it's a very obscure use case, but it's not a zero use case. And so the tail, there's going to be a bunch of questions in the tail that you may not have help center articles for.

**Ethan Smith** (01:02:31):
So again, what are the questions in sales calls? What are the questions that you're seeing in customer support? Having pages for that, I might even open up to the community. Anyone can ask anything because the community will then fill on the tail and then answer those.

**Ethan Smith** (01:02:46):
And again, in many cases there might be nobody talking about this at all. So you could be the only citation for this, and then win that tail of questions.

**Lenny Rachitsky** (01:02:55):
Are there any help desk, I don't know, system software that are just making this easier yet? Or do you think that's an opportunity for, say, Zendesk or Intercom?

**Ethan Smith** (01:03:02):
I think probably all of them should work perfectly well. I think that the only thing you need to do is cross-linking and subdirectory rather than subdomain, which probably most of them do. So I think that they should all work for free.

**Ethan Smith** (01:03:12):
That the main thing you would want to do would be, again, open it up to the community and make sure that you fill in the tail. But probably all those tools should be good for this.

**Lenny Rachitsky** (01:03:19):
Well, with that, we've reached our very exciting lightning round. I've got five questions for you, Ethan. Are you ready?

**Ethan Smith** (01:03:24):
I'm ready.

**Lenny Rachitsky** (01:03:25):
What are two or three books that you find yourself recommending most to other people?

**Ethan Smith** (01:03:30):
Number one is Emotional Intelligence, and people talk about the concept of emotional intelligence, but there's actual research and psychology around that. I believe it was published in the '80s, but there's a really good book that summarizes the foundational research around emotional intelligence. And it's very useful when building relationships and communicating with people to understand their emotions. So that's the first one. And doing growth because growth is getting people to use your stuff. And so if you have frameworks to inform how people will use your things, then you can be a more effective growth person. Which brings me to my second book, which is Cialdini's Persuasion book. Robert Cialdini does a bunch of books around persuasion, but again, there's frameworks for how to persuade somebody to sign up, buy something. And so he breaks down his framework for that, and again, it's based on psychology. And I think especially in growth, there's all kinds of psychology research and behavioral economics research to inform tests.

**Ethan Smith** (01:04:25):
And if you just read Thinking, Fast and Slow, Persuasion, Emotional Intelligence, you can basically take those frameworks and apply it to growth in all kinds of different ways. And then the last is How to Measure Anything. So How to Measure Anything is about measuring things that are not immediately obvious to measure.

**Ethan Smith** (01:04:42):
They give this example of they wanted to measure how good an orchestra conductor was and they could survey or they could see the number of standing ovations for each orchestra conductor. And the more standing ovations probably means it's this better one and that you don't need to survey people.

**Ethan Smith** (01:04:58):
But much of growth and business is things that are not immediately obvious for how to measure, but anything could be measured, and so that's my third record.

**Lenny Rachitsky** (01:05:06):
Is there a favorite recent movie or TV show you've really enjoyed?

**Ethan Smith** (01:05:09):
I don't really watch TV, but I watch two different groups of things. I watch really aggressive sports, so I really like Michael Jordan documentary, Last Dance. I like Lance Armstrong documentaries about how aggressive and confrontational he is, and I love watching UFC. I like extreme aggression and intensity. The other group of stuff that I like to watch are climbing documentaries.

**Ethan Smith** (01:05:34):
So anything that Alex Honnold, Jimmy Chan do, I watch all that, which is the exact opposite of aggressive sports. So it's zen, being present, slow-and-steady craftsmanship. But this is how I approach my work, which is extreme intensity and aggressiveness, and then the zen craftsmanship, being present.

**Lenny Rachitsky** (01:05:57):
I love how this explains why people love working with you and why you're good at this, is like this competitiveness and also just the super nerdiness to get really knowledgeable about how this stuff works.

**Lenny Rachitsky** (01:06:09):
And then I didn't think about the zen element of it, just lik staying calm throughout it all.

**Ethan Smith** (01:06:13):
Flow, flow state.

**Lenny Rachitsky** (01:06:15):
Flow, what a funny microcosm of why you're so good at this.

**Ethan Smith** (01:06:19):
Thank you.

**Lenny Rachitsky** (01:06:19):
Okay, I'm going to keep going. Do you have a favorite product you've recently discovered that you really love?

**Ethan Smith** (01:06:24):
This camera and this microphone. So I got a Sony mirrorless SLR, I forget which one. But, sorry, getting a mirrorless SLR with a wide-angle lens really transforms your video calls. And then I have this Shure microphone and I think it's like $180.

**Ethan Smith** (01:06:46):
This dramatically improves the quality of my video call. And I like to design things and you can design your video calls and you can make them amazing. You can have flowers in the background, over here, some sunflowers.

**Lenny Rachitsky** (01:07:00):
Beautiful.

**Ethan Smith** (01:07:00):
So my favorite products are my SLR camera that I use for video calls and my microphone.

**Lenny Rachitsky** (01:07:07):
Your background is quite exquisite and I didn't mention that, but it looks beautiful. Okay, two more questions.

**Lenny Rachitsky** (01:07:13):
Do you have a life motto that you find really useful in work or in life?

**Ethan Smith** (01:07:19):
There's the Outliers book about 10,000 hours. And the themes there are you don't have to be the smartest, you have to be sufficiently smart, number one. Number two is focused practice, so it's not just trying hard, it's doing it in an intentional, focused way. And the third thing is lots of practice, so no one can master anything because they're a genius.

**Ethan Smith** (01:07:44):
They master it because they spent a significant amount of time practicing and they practice in an intentional way. And so my motto is essentially a combination of those things, which is that I'm not going to necessarily win because my brain is the largest brain or that I tried the hardest.

**Ethan Smith** (01:08:00):
It's because I'm going to be the most intentional about my practice, and I'm going to be as intense as I possibly can be about that practice.

**Lenny Rachitsky** (01:08:08):
Okay, final question. I'm curious if there's just like an SEO or even an AEO win, you're just most proud of?

**Lenny Rachitsky** (01:08:15):
That you always think about, "Wow, I can't believe I pulled that off. I can't believe the impact we had there"?

**Ethan Smith** (01:08:19):
I always liked the example of butter lettuce with MasterClass. Because MasterClass, when I was first working with them, they did not have nearly as much authority as Allrecipes and Martha Stewart. And I actually didn't know if I should take the project because I thought it might be too hard.

**Ethan Smith** (01:08:37):
But I did the project and it was hard, but we were able to rank really competitively and way better than I expected. And I think it's probably because of all these specific, little execution details. But butter lettuce was my favorite one, and I like butter lettuce, so I can search for butter lettuce and I can get a recipe on MasterClass.

**Lenny Rachitsky** (01:08:55):
That's amazing. I don't know if butter lettuce has been mentioned on this podcast before. Ethan, this was incredible. This was everything I was hoping it'd be.

**Lenny Rachitsky** (01:09:03):
I feel like we've just leveled up everyone's knowledge on what the hell is happening with SEO and AEO? Forget about GEO.

**Lenny Rachitsky** (01:09:09):
Two final questions, where can folks find you if they want to potentially work with you guys? And how can listeners be useful to you?

**Ethan Smith** (01:09:15):
So where you can find me, number one, is on LinkedIn. I spend lots of time on LinkedIn and I publish original, so we do original research. We have a whole research team hypothesizing and evaluating those hypotheses. So we publish, all the studies that I mentioned, we publish on our site and I publish them on LinkedIn.

**Ethan Smith** (01:09:33):
So follow me on LinkedIn, add me on LinkedIn, send me a message. LinkedIn, number one, and then number two is we have a blog which we call The 5%. So /5%, which stands for 5% of work, 5% of landing pages drive almost all the impact, so that's the theme. This is only useful stuff. So our blog at 5%, you could subscribe to our email and to our studies. And then how can people be useful to me?

**Ethan Smith** (01:09:58):
So I spent time thinking about this and there's two ways people can help me. The first way is that there's not that much research around what works in AEO, and I would love to know what people are testing and what the results are and what works. So people doing studies and publishing that are sending it to me, I would love as much analysis and research as possible, number one.

**Ethan Smith** (01:10:20):
Then the second one is to help me on LinkedIn by commenting on my posts and on my comments. So you posted most recently the Brian Balfour episode, for which I wrote a long, thoughtful comment, and then I got about 25 likes and then I got responses to that. And so I've been commenting on other people's LinkedIn posts and I've been writing these long LinkedIn posts.

**Ethan Smith** (01:10:43):
And when people comment, it boosts the engagement within LinkedIn and then I get mass distribution. So the more people and thoughtful comments, so not this is great, but a long, thoughtful comment that stimulates conversation. So if people comment on my posts, then I'm just going to blow up on LinkedIn and I might be as big as you someday.

**Lenny Rachitsky** (01:11:01):
I love how tactical his ask is. It's something Bryan Johnson I noticed is really good at on Twitter, the longevity guy.

**Lenny Rachitsky** (01:11:08):
He just replies to tweets in a really funny way and feels like that's a big growth channel for him. So I love that you have this in common with Bryan Johnson.

**Ethan Smith** (01:11:16):
Yes.

**Lenny Rachitsky** (01:11:17):
Also, just to point people to your domain, graphite.io, is that the right domain?

**Ethan Smith** (01:11:21):
Yep.

**Lenny Rachitsky** (01:11:21):
Amazing. Ethan, thank you so much for sharing so much with us and for being here.

**Ethan Smith** (01:11:27):
Absolutely. It's good to be here.

**Lenny Rachitsky** (01:11:29):
Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.

**Lenny Rachitsky** (01:11:38):
Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.

---

## [16/18] Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor)

**Brendan Foody** (00:00:00):
The wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities.

**Lenny Rachitsky** (00:00:05):
We are entering the era of evals.

**Brendan Foody** (00:00:07):
We started working with all of the top AI labs. What the labs need is labor marketplace. They actually need extraordinary professionals that can measure model capabilities.

**Lenny Rachitsky** (00:00:17):
They found this pocket, maybe the biggest business opportunity in history.

**Brendan Foody** (00:00:20):
We grew from 1 to 400 million in revenue run rate in 16 months, fastest ascent in history.

**Lenny Rachitsky** (00:00:27):
Why is this so valuable?

**Brendan Foody** (00:00:29):
The market is bound by the amount of things where humans can do something that models can't. The lab's primary bottleneck to improve models is how they can effectively have some way of measuring what success looks like for the model.

**Lenny Rachitsky** (00:00:43):
There's this tweet that you retweeted. "If you really think about it, we were put on Earth to create reinforcement learning training data for labs."

**Brendan Foody** (00:00:49):
It's highly likely that the entire economy will become an aural environment machine, building out all of these worlds and contexts. And I think the narrative in AI over the last three years has almost entirely been one of job displacement, but very few companies and people have talked about this new category of jobs that's being created.

**Lenny Rachitsky** (00:01:08):
I talked to a lot of people about what should I be studying? Where should I be getting better?

**Brendan Foody** (00:01:12):
How can they leverage this technology to do so much more? We'll give people interviews where we say, "Use whatever tools are available to build a website and let's see what product you're able to build in an hour."

**Lenny Rachitsky** (00:01:24):
Today, my guest is Brendan Foody, CEO and co-founder of Mercor. Mercor is the fastest-growing company in history to go from 1 to $500 million in revenue. They did this in 17 months, less than a year and a half. Brendan is also the youngest unicorn founder ever. They just raised $100 million at $2 billion valuation. Mercor, if you haven't heard of them, helps AI labs and AI companies hire experts to help them train their models using AI. They've never had a customer churn, their net retention is over 1,600%, and they're on a nine-figure revenue run rate.

**Lenny Rachitsky** (00:01:59):
In our conversation, we talk about the increasing value and importance of evals, the landscape of AI training companies like Mercor, and why they've become so important and valuable, how Brendan discovered this opportunity, his insights on what product market fit looks like, the core tenets he's instilled within his organization that have allowed him to build the fastest growing company in history, what people writing evals for labs are actually doing day to day, which skills and jobs are going to last the longest with the rise of AI, why he doesn't think we'll see AGI or superintelligence anytime soon, and so much more. This episode is incredible. You need to hear this.

**Brendan Foody** (00:05:45):
Thank you so much for having me, Lenny. I'm a huge fan, and so excited to have a conversation.

**Lenny Rachitsky** (00:05:51):
I'm really excited to have this conversation as well. I'm a huge fan of yours. I'm excited for more people to learn about you and what you're building.

**Lenny Rachitsky** (00:05:57):
I want to start with a tweet that you have pinned at the top of your Twitter feed right now, and here's the tweet. "We are now working with six out of the Magnificent 7, all of the top five AI labs, most of the AI application layer companies. One trend is common across every customer. We are entering the era of evals."

**Lenny Rachitsky** (00:06:19):
The reason this caught my attention is that's one of the most recurring trends on this podcast, people talking about the increasing value of learning how to do evals well and the value of evals for companies. It feels like still people don't know what the hell this is what we're talking about, why this is so important. Talk about just what you think people are still missing, what they need to know, what this era of evals means.

**Brendan Foody** (00:06:39):
If the model is the product, then the eval is the product requirement document. And the way that researchers' day-to-day looks is that they'll run dozens of experiments where they'll make small improvements on an eval set. And reinforcement learning is becoming so effective that once they have an eval, they can help climb it. If you look at just how fast people were able to saturate Olympiad Math once they focused on it, how fast we're even saturating SWE-bench once we focus on it. And so in many ways, the barrier to applying agents the entire economy to automate every workflow is how do we measure success? How do we eval it? And write the PRDs for everything that we want agents to do, which Mercor is obviously a huge part of doing.

**Lenny Rachitsky** (00:07:25):
So people hearing this, they're like, "Oh, yeah. Okay, shit. I got to really pay attention to this eval stuff." Any advice about learning how to do this well? What companies that are doing this well are doing differently? Help people get better at this thing.

**Brendan Foody** (00:07:39):
Yeah. I think that for enterprises especially, the core way to think about it is how can they build a test or systematic way to measure how well AI automates their core value chain? So if it's an architecture firm that's producing these architecture diagrams of what they provide to their end customer, how can they effectively measure that? And each company has its own value chain or maybe a handful of them if it's a multi-product company. And just thinking about how they measure that is the prerequisite to really effectively applying AI throughout their entire business.

**Lenny Rachitsky** (00:08:21):
I saw you talking about this on the No Priors podcast with Sarah and Elad, and I don't know if it was after this or before this, but Sarah tweeted, "Evals equals your new marketing." What does that mean? What do you think she's saying there?

**Brendan Foody** (00:08:32):
Yeah. Well, it ties to what I said earlier about how if the model is the product, evals are the PRD, but also subsequently the sales collateral, right? Because evals are what you give to researchers to show them what they should be building and going on, but they're also the way that you demonstrate the efficacy of capabilities.

**Brendan Foody** (00:08:51):
And historically, everyone's been pointing to these academic evals of PhD level reasoning with GPQA, Humanity's Last Exam, or Olympiad Math, but now it's moving towards the capabilities that people practically care about of how do we get models to automate the way that we build a software platform or automate the way that we do an investment banking analysis. And I think labs as well as application layer companies will increasingly use evals to demonstrate the capabilities of their models and their products.

**Lenny Rachitsky** (00:09:26):
Okay. So let's build on this and zoom out a little bit and talk about the landscape of the market that you're in. And I was just reflecting on this as I was preparing for this conversation. If you think about the companies growing faster than any company's ever grown in history, there's essentially three buckets. There's the foundational model companies, there's vibe coding apps, Cursor and Loveable and Bolt and Replit and all these here, and then there's data labeling data companies like you. So I've had the CEO of Handshake on the podcast. I have the CEO of Scale coming on. There's also Surge. There's you guys. Help us just understand the landscape of what this is all about because I think people don't really know what the hell's going on and see all these companies growing like crazy.

**Brendan Foody** (00:10:06):
Yeah, I'll give a little bit of the origin story, incorporate in that and how it frames the landscape. Because when we started the company, I met my co-founders when we were 14 years old. We started the company together when we were 19 initially, in January 2023, initially hiring people internationally, matching them with our friends and automating all the processes of how we did that. So similar to how a human would review a resume, conduct an interview, and decided to hire. We automated all of those processes with LLMs, bootstrap the company to a million dollar revenue run rate before we dropped out of college.

**Brendan Foody** (00:10:40):
And then a handful of other things happened, but we met OpenAI and we saw that there was this enormous transition in the human data market where it was moving away from this crowdsourcing problem of how do you find low and medium skilled people that can write barely grammatically correct sentences for early versions of LLMs and moving towards this sourcing and vetting problem. How do we source and assess the best professionals, the experienced? Think software engineers, the investment bankers and doctors and lawyers that can actually help to evaluate and interpret all of the capabilities that people want models to have.

**Brendan Foody** (00:11:21):
So from there, we start working with all of the top AI labs. We grew from 1 to 400 million in revenue run rate in 16 months, and it's been an extraordinary journey and super exciting.

**Lenny Rachitsky** (00:11:36):
Okay. First of all, that is out of control. I don't know if people understand. I think this is the first time you're sharing that number. I know we're recording this, you'll have announced it by now, but 1 to 400 million in revenue in 16 months.

**Brendan Foody** (00:11:49):
Exactly. So fastest ascent in history, which is an exciting statistic we're very proud of.

**Lenny Rachitsky** (00:11:57):
Okay. So something big is happening here. Why is this so valuable? What is going on here? So it's just to try to summarize what you guys do simply is you help hire people for labs to help them train their models, and you help them find not just generalist labor, but experts, helping them with very specific gaps in the model's knowledge.

**Brendan Foody** (00:12:21):
Yeah, precisely. And so it really ties to your first question around the era of evals that's framing all of this, which is that the lab's primary bottleneck to being able to improve models is how they can effectively have some way of measuring what success looks like for the model, both to use it as the eval for the tests that they're measuring their progress against, as well as the verifiers in an RL environment to then reward the model, improve capabilities, et cetera. And they need this across every domain for every capability that models don't know how to use. And the wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities where Mercor is sitting at the forefront and the primary bottleneck.

**Lenny Rachitsky** (00:13:12):
Okay, what are these people actually doing? So what's an example of a kind of person that is sought after? And then what are they doing sitting there at the computer?

**Brendan Foody** (00:13:19):
Effectively, the market is bound by the amount of things where humans can do something that models can't. So I'll make that very concrete. Say you have a model that you want to write a red line for a contract in the way that a lawyer would, and it makes a handful of mistakes, misses a bunch of key points in doing so. What you could do is have a lawyer create a rubric similar to how a professor might create a rubric to create a deliverable for what are the things we want the model to be able to do?

**Brendan Foody** (00:13:50):
So it can effectively score that, right? Plus however much of it identifies this or XYZ key point. And that's really the foundation to measuring what does progress look like for models? Is this model achieving the capabilities that these professionals want? As well as how do we use this as training data to reward and to reinforce a lot of the capabilities that people want models to achieve.

**Lenny Rachitsky** (00:14:19):
Okay, so they're essentially writing evals just to connect it back to original conversation.

**Brendan Foody** (00:14:23):
Exactly. Well, that's an interesting thing is everyone talks about RL environment. I feel like the two hot button things are like RL environments and evals, but one thing like Andrej Karpathy's tweeted out about a bunch is there's not actually a nuance. It's in the data type. It's more just a different semantic way of describing what it's being used for. But ultimately, it's just some stasis point for how do you measure what good looks like? And you can use that either as the benchmark to the sales collateral, as Sarah was saying, to say, here is why are models the best model in the world and here's the capabilities that we've been working towards, or you can use it on the post-training side to reward certain model trajectories and achieve those capabilities.

**Lenny Rachitsky** (00:15:08):
Okay. So say this lawyer, this person is writing, "Here's what a great red line contract looks like and here's the rubric of what excellent is." Then are they also providing data, like actual examples of red line documents as a part of that?

**Brendan Foody** (00:15:22):
They may. The data landscape historically has included two kinds of data. The first is supervised fine-tuning data, which is input/output. When people think about fine-tuning in the historical sense, that's what it is. The second is RLHF where the model will generate a couple of examples. We'll choose which is the most popular example.

**Brendan Foody** (00:15:43):
What everyone is generally moving towards is reinforcement learning from AI feedback instead of human feedback where you have instead the human defined some sort of success criteria, some way to measure that. And examples in code, it could be a unit test. We can scalably measure success and other domains that could be a rubric. And then you use that to incentivize model capabilities. And it's far more scalable and data-efficient, and so that's why a lot of the broader trend in the market across the board is moving towards RLHF to both eval models as well as improved capabilities.

**Lenny Rachitsky** (00:16:24):
I had one of the co-founders of Anthropic on. He said exactly the same thing. That's what they've done at Anthropic, is move towards AI-driven reinforcement learning.

**Lenny Rachitsky** (00:16:32):
So essentially, if I can understand this correctly, I'm the lay person here trying to understand this on behalf of the audience. So essentially a lawyer is like, "Here's what correct looks like for redlining," and then it's AI is just on its own almost, just like, "I'm going to try to get this. I'm going to try to improve on this and I know if I'm heading the right direction based on this eval/rubric I've been given."

**Brendan Foody** (00:16:55):
Exactly. Applying all of the criteria of what good looks like similar to how the TA might apply the professor's criteria of does the student's response meet this criteria or this criteria plus however many planes, et cetera.

**Lenny Rachitsky** (00:17:10):
Awesome, okay. Let me shift to talking about the broader labor market here. So there's two parts to this question as we talk about this. One is just how long will we need to do this? You guys grew so incredibly fast. Is there a point of like, "Okay, we don't need humans. We're tapped out." So let's start there and then I'll ask a broader question.

**Brendan Foody** (00:17:29):
So the key question is how long there's going to be things in the economy that humans can do that AI can't do? And I think there's certainly a bucket of people that say we're going to have superintelligence within three years and humans won't play a role in the economy. And that's one school of thought.

**Brendan Foody** (00:17:46):
Our perspective is very different. Our perspective is that these models are extraordinary and automating a lot of things very quickly, but there's a lot of things that they're horrible at. Even still, it can't schedule time on my calendar. It can't draft emails for me. It can't use basic tools. And we need evals for everything. For everything that the models can't do, we need evals for the tool use, evals for the long horizon reasoning.

**Brendan Foody** (00:18:12):
Imagine in 10 years when we want models to be able to go out and build a startup for 30 days. We need evals for that to effectively reward it. And I think that that road to improving models will last for as long as there is anything in the economy that humans can do which models can't and be a huge portion of what the future of work looks like. And so our mission is creating the future of work, and I think that this is a really exciting industry and giving us a glimpse into the direction that everything is headed towards.

**Lenny Rachitsky** (00:18:49):
There's this tweet that you retweeted that I want to ask you about. "If you really think about it, we were put on Earth to create reinforcement learning training data for labs."

**Brendan Foody** (00:18:59):
Yeah.

**Lenny Rachitsky** (00:19:00):
What does that mean to you? What is this person implying? And it's basically what you're saying is we're just helping train models.

**Brendan Foody** (00:19:06):
It speaks to conversations I've had with a lot of researchers and executives at top labs, which is that it's highly likely that the entire economy will become an aural environment machine, building out all of these worlds and contexts for us to then have rubrics or other kinds of verifiers. And that is really exciting in so many ways.

**Brendan Foody** (00:19:32):
Because I think let's draw an analog to other revolutions where when we had the industrial revolution, everyone was freaking out about losing their jobs, but there was this whole new class of jobs of how do we build the machines? How do we have knowledge work? How do we create everything new? And I think that the narrative in AI over the last three years has almost entirely been one of job displacement, right? Sure, there's ChatGPT is growing fast and it's very cool that everyone loves using it, but from an economic standpoint, people talking a lot about job displacement. But very few companies and people have talked about this new category of jobs that's being created and what that's going to mean and how people can prepare and upskill for that. And I think that the most exciting thing possible is creating that future of how do humans fit into the economy and how will that evolve over time?

**Lenny Rachitsky** (00:20:22):
I talk to a lot of people about just what should I be studying? Where should I be getting better? People in school right now are just like, "What is even going to be valuable in the future?" You're at the center of a lot of just what jobs are most in demand, how hiring is evolving. So let me just ask you a very concrete question. What jobs do you think will remain in the future/what skills are still worth investing in for younger people, especially?

**Brendan Foody** (00:20:47):
In terms of jobs, I would respond with a category of things that have very elastic demand are going to be super exciting. Because when we make people 10 times more productive, we'll build 10 times, if not 100 times as much software as an example. And so I think the product managers that can now do so much more are going to be extremely well-positioned. And so far as the skills, I think it's people that can leverage AI to do whatever their day-to-day workflows are.

**Brendan Foody** (00:21:16):
I have had a couple conversations with teachers where they get my thoughts on how they should be assessing their students because we originally started out curating all of these AI interviews and assessments for people and have thought about this immensely. And what we realized is that you don't want to fight against them using the models. It's similar to when the calculator came out, you don't want to give people all of this arithmetic work of how do you get them to do it and not use the calculator. You want to tell them, "Use the tools and let's see what you can do."

**Brendan Foody** (00:21:49):
And so we'll give people interviews where we say, "Use ChatGPT and Kodak. Use Claude code. Use whatever tool cursor and whatever tools are available to build a website and let's see what product you're able to build in an hour." And so I think that I give that an example in so far as talent assessment because I think it pertains also to the skills that people should be honing in on of how can they leverage this technology to do so much more in whatever industry or vertical they're operating in.

**Lenny Rachitsky** (00:22:17):
When you talk about elastic, being elastic, is it generalists being good at just a bunch of different things, or what do you say? What do you mean when you think elastic?

**Brendan Foody** (00:22:25):
So I more mean how much capacity for demand there is in that industry. So I'll give a couple of examples. In accounting, I think realistically we only need so much accounting in the world. Maybe there's areas where we can do more and that'll be good, but it doesn't feel like the world needs 100 times more accounting.

**Brendan Foody** (00:22:46):
On the other hand, in software development, I think we can ship 100 times more features for our products, move 100 times faster, build so much more. There's just it feels like there's unlimited demand for the industry. And I think Mark Andreessen tweeted about this recently, that software is the most elastic industry of all where when we increase productivity, there's so much more that will be built. And it's definitely characteristic of a lot of other domains as well. And so I would focus on those domains where if we make everyone 10 times more productive, that'll increase demand, not reduce it.

**Lenny Rachitsky** (00:23:23):
Okay. So you're in the bucket of learn to code, still useful as a skill. You take computer science. And so in terms of elastic categories of jobs, sounds like engineering, product management is in that bucket. Great. A lot of people listening to this are PMs. What else, like design users? I don't know. What else do you feel is in that bucket from what you've seen?

**Brendan Foody** (00:23:44):
Yeah, I think that there's a lot of things where the whole value chain of building companies has a lot of these variable costs, even large portions of operations or consulting. Imagine if we could have 10 times as many McKinsey consultants, what would be possible in so far as the research we could do, the analysis, et cetera. But I think the companies and people that are going to succeed are those that lean into this narrative of abundance of how do we do so much more rather than fighting back against it of how do we try to stop displacement.

**Lenny Rachitsky** (00:24:20):
So along those lines, I think about your second bucket, which is the people that will be most successful. It's not like a specific skill, but it's being good with AI, using AI to become better at what you're already doing. This reminds me of Elon's whole thing with Neuralink, which I don't know if this is how we put it, but the way I've always heard it is you wanted to build Neuralink because in the future when AGI and superintelligence is around, we need a way to compete and the best way to compete is plug our brains into a superintelligence so we have a chance. And it feels like that's what AI is. Getting good at AI tools is essentially is having this super superpower.

**Brendan Foody** (00:24:58):
Figuring out how to leverage them and incorporate it will definitely be of paramount importance.

**Lenny Rachitsky** (00:25:04):
It just comes back to this almost cliche quote now. It's, "AI won't replace you. People that are really good with AI will replace you."

**Brendan Foody** (00:25:10):
I think it's totally spot on. And I've definitely seen this at the enterprise level as well where there's certain enterprises we talk to that are almost fearful not wanting to engage, not wanting to eval their businesses because that'll provide the evidence that their value chain is being automated. And there's others that... Literally some of the most recognized sophisticated Fortune 500 businesses that have this mentality and there's others that are leaning into it of if we have the ability to do 10 or 100 times more, what will that mean and how do we lean into that future? Because there's so many things that are going to change over the next 10 years, and I think those are the kinds of businesses that are going to be successful.

**Lenny Rachitsky** (00:25:54):
Let's talk about labor markets more broadly. You guys, so it's interesting though. You started not feeding people to AI labs, not training models. It was just like help people find jobs, help companies hire, and then you're like, "Oh wow, this whole opportunity." You have this really interesting view on the future of just labor markets and hiring. Talk about that.

**Brendan Foody** (00:26:14):
Yeah, it's interesting. I remember when we started the company, as I mentioned, we were 19, and just had this gut intuition that it felt so wildly inefficient that labor markets are so disaggregated. And what I mean by that is when we would hire someone internationally, they would apply to a dozen jobs. When we as a company in the Bay Area were considering candidates, we would consider a fraction of a percent of candidates that were available in the market. And the reason for that is that there was this matching problem that everyone's solving manually where they'll manually review resumes, they'll manually conduct interviews, and manually decide who to hire. But when we're able to automate that matching problem at the cost of software, it makes way for this global unified labor market that every candidate applies to and every company hires from facilitating a perfect flow of information in the economy.

**Brendan Foody** (00:27:08):
And I think that that future is undoubtedly what we're heading towards, but what we've realized over time is that the nature of work is also changing dramatically. And part of building that future over a 10-year time horizon is creating that future of work and all of the more tactical things we do and building these incredible data sets across evals and RL environments for our customers.

**Lenny Rachitsky** (00:27:35):
What I've seen in how hiring has changed, I'm doing research on this with a partner, Gnome, it's so much easier to apply for companies that everyone's just applying now, to hundreds of companies. AI is just making it easy to adjust their resumes and cover letters and make it feel like, "Oh, I applied to more of course very specifically, but it was one of 100 places." And then on the flip side, hiring managers are getting flooded with applications and so now they need AI to filter. So even if we didn't want to get to this place, we're almost being pushed into this direction of so much volume on both sides. We need something really smart at filtering and helping us hire and select, and this is exactly what you guys have been building for a long time.

**Brendan Foody** (00:28:13):
Precisely, yeah. And the fascinating thing a lot of people ask, do we think about ourselves as a labor marketplace or do we think about ourselves as a data company? And I think that the reason it's an interesting question is our realization on from what the labs need is that they actually need a labor marketplace. They actually need these exceptionally high caliber people. And of course we'll layer on some project management and some software platform associated with it. But the really core thing that they want is how do they find these extraordinary professionals across all of these different domains that can measure model capabilities and work to build that future work together?

**Lenny Rachitsky** (00:28:56):
**Brendan Foody** (00:30:29):
Exactly. Well, so the way it works, at least what most people's understanding is there's a lot of complexity in how the models work, is that pre-training gets a lot of the knowledge into the model of what are all the different things that see into the world. And then post-training and reinforcement learning is for all of the reasoning of what are the pieces of knowledge that are accurate, what are inaccurate, and what to prioritize at any given time to make a decision. And so behind that, there would've been radiologists that worked on the post-training data set to create some stasis point for here's the diagnosis and rewards and penalties associated with it. And it's really the quality of those people that went into the quality of the decision and recommendation that ChatGPT ultimately made.

**Lenny Rachitsky** (00:31:14):
So let's actually follow that, right, because that's really interesting and I don't know how many people understand it. I understand it. So the work that you do and these experts do is post-training. It's not feeding data into the model that it's trained on. It's, "We have this model GPT-5. Now here's all the things that's missing. Let's add to it."

**Brendan Foody** (00:31:31):
Exactly, yeah. It's really unlocking, allowing the model to focus on all the right tokens, from pre-training all the right things in model context, up weighting the effective reasoning chains to enable the models to reason better in a more generalized way.

**Lenny Rachitsky** (00:31:48):
What's the scale of people just working on the stuff. It's like thousands, tens of thousands, hundreds of thousands?

**Brendan Foody** (00:31:53):
Tens of thousands at any given time, hundreds of thousands more generally. It's huge. And the most exciting thing is that it's growing really quickly. I think that to your question also about the competitive landscape, historically there were all these crowdsourcing companies that would get these super high volumes of low-skilled people. I think Scale and Surge were the primary companies that pioneered that industry. And then in this transition to higher-skilled labor, what people realized is that actually you can go a lot further with just getting higher caliber people even in smaller amounts initially, and now subsequently scaling that back up once they're able to meet the quality bar.

**Brendan Foody** (00:32:35):
And I think that there's a bunch of companies that after our success and very rapid revenue growth that started early last year have chased after that, which makes sense. And seeing that the market was changing very quickly, we were taking off, and trying to pursue a similar thesis on the market.

**Lenny Rachitsky** (00:32:56):
It's interesting. There's always been these companies, AlphaSights and GLG, that did this before AI or is paid to connect to an expert and ask them questions about stuff. And essentially, okay, it turns out this is really useful for models. We don't need the person in the middle.

**Brendan Foody** (00:33:11):
Exactly, yeah. Well, but one core difference is that AlphaSights would generally be a one-off call versus a lot of our work is really hiring people for projects of how do they work on something for a longer period of time. And so that's, I think, one of the reasons that some of the traditional expert networks have struggled to get into this. And also how do you retain those people and think about all the incentives where it actually looks more similar in some ways to one of the traditional labor marketplaces of an Uber or DoorDash, just with much higher-skilled talent that's treated exceptionally well?

**Lenny Rachitsky** (00:33:50):
It's such a good opportunity for me to learn so much about this, so I'm going to ask questions.

**Brendan Foody** (00:33:55):
Yeah.

**Lenny Rachitsky** (00:33:55):
It's so interesting to me. How much of the experts are focused on specific concrete knowledge versus personality and softer skills? How much of it's like, "Here's how you do an exam. Here's how you do an x-ray"?

**Brendan Foody** (00:34:09):
It depends on the lab. It's a lot of both. I think that previously it might've been more softer skills, but now a lot of the labs are focused on their business models of what are the economically valuable capabilities that drive revenue and leaning a lot into these professional domains. But I think the creative side is also still really important to everyone. And so we're seeing a meaningful amount of both. We hired all the people from the Harvard Lampoon a couple of months ago, their comedy club, to help with making models funnier. And so do all sorts of stuff like that, hiring Emmy award-winning screenwriters and everything across the board on creative capabilities that you'd look for.

**Lenny Rachitsky** (00:34:51):
That is amazing. What a cool story. I'm excited for this to kick in. How fast do these things turn around? Say you hired this team, how fast are we going to see the impact potential? Is like months? Is it years?

**Brendan Foody** (00:35:03):
Well, so it depends, because some models or some labs will release iteratively where they'll just improve the model behind the scenes.

**Lenny Rachitsky** (00:35:11):
Without announcing a new model?

**Brendan Foody** (00:35:12):
Exactly. Every couple of weeks versus others do these big releases. And so it depends a lot. We're behind all of them, but we move really fast. It would be a customer gives us a request of we need these award-winning screenwriters, and within 24 hours we'll turn around the experts. And then there's also this really interesting dynamic where in a set of 100 people that we hire, oftentimes the top 10% of people will drive majority of the model improvement. It's like a company. If you have 100-person company, oftentimes the top 10% of the company will drive majority of the impact. And what that means is that when we're able to build proprietary advantages in identifying who are those top 10% of people, both in so far as how do we have them on our platform but also identify and match them effectively, it creates so much value for customers that it's difficult to compete against.

**Brendan Foody** (00:36:08):
And so it really does tie back to the founding thesis of the company, which is how do we find these extraordinary people and identify them so that we can reliably deliver these top 10% or top 10X experiences for our customers.

**Lenny Rachitsky** (00:36:25):
So on that, so is the idea, you hire Jane. She's incredible at coding and she now works for Anthropic and that's her full-time job doing this? Or is this a part-time thing? Is this a project thing mostly?

**Brendan Foody** (00:36:38):
It would sometimes be part-time. Sometimes it would be full-time. I would say most often it's part-time where it's like someone might work at a thing company where they're underemployed, maybe one of the ones that's moving slower where they have an extra 20 hours a week and then they're able to do this on the side or whatever the equivalent is across a bunch of different industries. But we also do a lot of 40 hour a week roles as well.

**Lenny Rachitsky** (00:37:08):
And how much are they making? Is it meaningful enough for a AI engineer to spend time on this?

**Brendan Foody** (00:37:13):
Yeah, very meaningful. So our median pay rate in the marketplace is $95 an hour, but it can flex up, well up into $500 an hour based on the depth of someone's expertise. And one thing that highlights this difference relative to a lot of the crowdsourcing companies is if you look at the economics of the crowdsourcing companies, oftentimes they would pay $30 an hour to town as the average. And so think about the people that you can hire, the undergrads for $30 now versus the Goldman bankers, the McKinsey analysts, the Fang software engineers. And ultimately it comes down to what are the capabilities that labs want their models to have? And it much more falls in the latter bucket than the former one.

**Lenny Rachitsky** (00:38:02):
I know there's only so much you can talk about with this stuff, but so Anthropic, Claude has been so good at coding so much better historically than other models. I also use it for writing, giving feedback on writing. What is it that allowed them to get so good at this and continue to be so good at this?

**Brendan Foody** (00:38:19):
Well, I can't go too much into detail about customer work, but I think that it's this trend of reinforcement learning and being very thoughtful about defining the right rewards that we're releasing across the board. And how we could mitigate reward hacking, set up the right rewards, that's super impactful.

**Lenny Rachitsky** (00:38:42):
Evals. Again, evals is all you need.

**Brendan Foody** (00:38:44):
Back to evals.

**Lenny Rachitsky** (00:38:45):
Yeah.

**Brendan Foody** (00:38:45):
One of my favorite quotes from customers is that, "Models are only as good as their evals," which has always held true.

**Lenny Rachitsky** (00:38:53):
I think Greg Brockman tweeted this once. "Evals are all you need."

**Brendan Foody** (00:38:56):
Yeah, truly.

**Lenny Rachitsky** (00:38:58):
Let's talk about Mercor a little bit more. One of the maybe, not even maybe, I believe the data tells us it's the fastest growing company in history.

**Brendan Foody** (00:39:07):
Yeah.

**Lenny Rachitsky** (00:39:09):
I want to understand what you did to make this happen. So let me just ask, what do you think are some of the core tenets of how you built Mercor that most contributed to being this successful?

**Brendan Foody** (00:39:20):
I think the most important thing is looking at the leading indicators in fast-moving markets. I remember when I used to think... Everyone in venture talks about the why now, and I used to think about the why now of how from a product standpoint, less from a market standpoint of now we can automate the way that we review resumes or the way that we conduct interviews, et cetera. But ultimately there is this legacy market that's has all these incumbents and it's relatively stagnant. But what matters a ton is actually figuring out what are the new markets, the new pockets of demand that are changing very quickly where the wealthiest customers in the world are willing to pay whatever it takes to improve model capabilities, and how do we focus on the leading indicators of those markets to make sure that we have the best solution for the flagship customers in the market and optimize everything around that.

**Brendan Foody** (00:40:18):
And that's what I found has been most impactful in building the business. I think maybe that's one thing is leading indicators in markets. If I had to choose another, it's customer obsession. We have had for the last... We're starting to have a couple of product managers help out with go-to-market, but for the last year and a half of the business, we've had no one in sales and marketing. And so we're immature from a sales and marketing standpoint because we focused 100% of company resources on how do we build great products and experiences for our customers. Just getting word of mouth, the people that have worked with us at other businesses want to keep working with us and leaning into creating those great experiences. And so that's where I spend all my time. And I think that some founders can get caught up in how do they get really good at marketing before they've figured out the thing that really drives a lot of customer love and creates the six-star experiences that you're used to building.

**Lenny Rachitsky** (00:41:19):
I'm going to go back to that first point, which is like, okay, you found this pocket, maybe the biggest business opportunity in history. How did you first find... What was that moment of, "Wait, this could be really big"?

**Brendan Foody** (00:41:31):
So there's some crazy stories here. I remember we started the company as I mentioned in January 2023. And then in August 2023 when I was still in college, one of our customers introduced us to the co-founders of xAI over a Zoom call saying how we had these really smart Indian software engineers that were great at math and coding. So we met them and we explained how the software engineers we had were really good at math and coding because they weren't distracted by all the humanities. They didn't have to study history and English and all these other things, and they loved it. So they had us in two days later to the Tesla office and we met the entire xAI co-founding team except for Elon, while I was still a college student. And xAI was just getting started at that point and they were super excited about our focus on the quality of the experts.

**Brendan Foody** (00:42:22):
And so while they were still doing pre-training, they weren't ready for human data at the time and we didn't start working with them at that point. We just knew from that point forward before we even dropped out that the market was about to change radically and we needed to be at the frontier of that. And so then fast-forward a few months, one of the crowdsourcing players came to us and actually used our platform to hire over 1,000 people where this is very interesting experience because we started getting flooded with support tickets about how those people weren't getting paid. And we obviously felt horrible because we had referred them to this opportunity. It was this reputable company. And we realized that a lot of the incumbents were resting on their laurels with respect to what was needed in the experiences they were creating for talent in their marketplaces to help improve models. And there was this opportunity to work directly with the labs in a way that kept the dignity of the experts in the marketplace, paid them extremely well, and cut out the middlemen.

**Brendan Foody** (00:43:31):
And so we started doing that in May of last year, and then the rest is history.

**Lenny Rachitsky** (00:43:37):
Wow, okay. Hundreds of millions of dollars in revenue since. So what I'm hearing here is you were very open to looking for poll. You saw some poll, you explored it. And then once you saw that there was something really meaningful there, you just went deep on making that an incredible experience as amazing as possible.

**Brendan Foody** (00:43:57):
Exactly. I think if I had to distill it into advice for founders, one thing I've realized is that I spent a lot of time trying forced product-market fit. And in some ways you should be persistent. You should have these theses that you have conviction about how the world will change. But sometimes you just need to sheer it from the market and know that it's there, the poll, to know the right places to focus. Because if it's difficult to sell, if it's extremely difficult to sell the marginal customer, you're not going to be able to grow a huge business. What you actually need to find is the customer that's surprisingly easy to sell into where you're going to be able to grow with them. You know that it's a large pain point. And so it's some combination of being stubborn with respect to your thesis around how the world will change, but also very open-minded with respect to exactly what form that takes and how the market's developing and how your company will fit into it.

**Lenny Rachitsky** (00:44:55):
That's an amazing insight. In the moments you described, felt like it was a combination of this xAI meeting feeling like, "Oh wow, they really, really want this thing that we have. We're now doing an amazing job," and then it's 1,000 people hiring in the platform. Was that those two moments that are like, "Wow"?

**Brendan Foody** (00:45:09):
Exactly. And those happened, keep in mind, while we were a seed company, right? Well, so the first one was before we even raised any seed funding, we were totally bootstrapped because we bootstrapped the company to a million dollar revenue run rate and have always remained super capital-efficient. We've never burned money. We were lifetime profitable. And then we raised our seed round in September from General Catalyst, and it was the other experience after we raised our seed round where we really knew that there was an enormous amount of demand in this market where we saw the volume and we saw that the incumbents were sleeping with respect to how the market was changing and the kinds of people that were needed to make that change happen.

**Lenny Rachitsky** (00:45:50):
It's one thing to see this opportunity and start to execute on it. It's another to actually succeed at this scale and consistently win. You guys have very specific values within the business. Talk about those. It feels like that's a big part of your success too.

**Brendan Foody** (00:46:04):
It totally is. So I'll give the three and maybe a brief story associated with each of them.

**Brendan Foody** (00:46:10):
So the first one is having a can-do attitude, which everyone gives me a little bit of a hard time for because it's a funny saying, but we've always set these ridiculously ambitious goals, and then somehow the trajectory of the company forms around those goals. Where I remember when we were talking to Benchmark before they led our Series A, we were at 1.5 million in run rate. And I said we'd be at 50 million in run rate by the end of the year. And they said we were absolutely insane, right, as anyone would. And plus or minus two weeks, we hit it. And then we've now well blown past the tracking to 500 million in run rate, which was initially our goal for this year. So setting these incredibly ambitious goals with respect to the revenue scale of the business, the caliber of experiences for talent, all those dimensions is super important to first have a can-do attitude.

**Brendan Foody** (00:47:04):
The second thing is really high standards, which is who we hire and what we expect of them. We have an incredibly high hiring bar where we hire tons of former founders, people that have incredible experiences. We just hired or partnered with Sundeep Jain who joined us as president. He was previously the chief product officer and chief technology officer at Uber and joined our relatively small in the grand scheme of things company to help scale up all the processes where Uber is of course the largest labor marketplace in the world. So super high standards is of paramount importance.

**Brendan Foody** (00:47:41):
And then the third one that we really lean on significantly is intensity. And that if you look at the early cultures of the legendary companies, thinking of Meta or Google, they have these incredible, intense early-stage cultures of people just moving heaven and earth and doing whatever it takes to push the frontier of model capabilities. And so still very much output-oriented of what do people achieve rather than input-oriented of the specific hours they work, but recognizing that it takes a lot to build a legendary business, and that's ultimately what we're optimizing for.

**Lenny Rachitsky** (00:48:18):
I could see why this works. Can-do attitude plus high standards plus intensity, I could see how that leads to success. There's a lot of talk these days about this 6-9-9 culture, working six days a week, 9:00 AM to 9:00 PM. A lot of people are like, "Why? That's terrible. Why would you make people do that?" But at the same time, I'm just constantly hearing this from the most successful AI companies. This is just the way it is to be successful. Things are moving so fast. This is an opportunity you'll never see again. Just talk about your thoughts on that.

**Brendan Foody** (00:48:50):
Yeah. Well, to clarify, we've never mandated hours. It's more been a byproduct of people that care a lot where we care a lot about the trajectory of the business. And so a lot of people come into the office and stay late. But if they need to leave early and get dinner with their kids or travel on the weekend, of course that's totally fine. And for us, it's much more about finding people who have a lot of ownership and are really bought in, less so about the specific hours in the office, even though we found that oftentimes it's the people that are most bought in, not always, but oftentimes it's the people that are most bought in and that burn the midnight oil with us.

**Lenny Rachitsky** (00:49:30):
When you say high standards, is there something you could share that gives us an example of what you mean there? Because a lot of people think they have high standards and they don't.

**Brendan Foody** (00:49:37):
If you are very patient, there's always some trade-off between speed and quality when hiring. And I remember especially for our first 10 people, we were just so patient and disciplined about finding some of the best people in the world. Half of them are... Our second employee, Sid, as an example, our second employee in the US, Sid was previously the head of growth at Scale who joined us when we were a seed stage company. Daniel who joined us was previously scaled to consumer apps to over 100,000 users and all sorts of just extraordinary backgrounds of our first 10 hires. And I think that that initial talent density shaped so much of what the rest of the org looks like as you scale it up.

**Lenny Rachitsky** (00:50:28):
I know you also have this perspective that people talk about waiting to hire, to hire really slowly, but it's actually not necessarily the right advice. Talk about that.

**Brendan Foody** (00:50:39):
It's painful because it's a double-edged sword. On one hand, I'm thrilled that our first 10 people are so phenomenal and I think that that has paid dividends for the business. But on the other hand, I think that companies do get to the point where you just need to hire really fast. And there's some things where you need a lot of people to do them and you need to recognize that there's going to be some variants associated with hiring, but moving quickly is the priority.

**Brendan Foody** (00:51:07):
And I think that in some ways, we move too slowly with how we scaled out the team. And so the benefit is that everyone is extraordinary. We have this super high bar and we want to maintain that over time. But I think the downside is that while the company has grown incredibly quickly, we likely could have grown even faster if we had moved a little bit more quickly with especially ramping from call, like 10 to 100 people.

**Lenny Rachitsky** (00:51:37):
Okay, I was going to ask. So it sounds like the first 10, be very careful, take your time, 10 to 100, maybe speed up a bit.

**Brendan Foody** (00:51:44):
Yes, though I wouldn't say it's necessarily 10. It's determined by the point where you know it's really working. And I know that's still not a bright line, but it's like once you know that there's so much more demand than you can handle, that's when you want to step on the gas and optimize for speed in a lot of ways. But I think especially until then, it's important to be patient, be disciplined. Get the best people is always important, but speed becomes more important once you find the market opportunity, the market vacuum.

**Lenny Rachitsky** (00:52:20):
I know you've started a couple companies in the past, much smaller scale. In this new role as CEO of this massive hyper growth company, what surprised you most about where you spend the time most or just what the role involves? Because a lot of people want to start companies dream about being in your shoes. What are they maybe not understanding about where a lot of your time goes?

**Brendan Foody** (00:52:42):
Yeah, it's actually not too surprising. The top two buckets are always working on hiring and time with customers of how do I really deeply understand what customers need and how we can support them? And then how do I build the team and a lot of the processes around that? Of course, there's all of the ad hoc things I didn't expect of dealing with the people questions of how do we set up our levels and our comp bands and all of that, which you learn as you scale a business. But I think that the core places that I spend my time are in line with what I expected as well as what I love doing, which is very fortunate.

**Lenny Rachitsky** (00:53:27):
So these two companies you've started in the past, maybe share what they work because they're fun, and then how do they help you be successful in this? What's something that they taught you that helped you in your current role?

**Brendan Foody** (00:53:38):
Yeah, so there's been like a dozen, but I'll choose my favorite two. So when I was in eighth grade, I started Donut Dynasty where I saw that Safeway Donuts were selling for $5 a dozen, and I was amazed because I felt like as an eighth grader, this was such an incredible deal. And I started to bike down to Safeway, buy Safeway Donuts for $5 a dozen, and then go back to my middle school and then sell them for $2 each, running really good margins of course. It sold out super quickly. And so then I need to scale up. So I would pay my mom $20 to drive me in her minivan down to Safeway, buy 10 dozen donuts, go to my middle school, sell them all out.

**Brendan Foody** (00:54:19):
And then the school tried to shut me down because I was selling food on school campus, which they didn't like. So they had me in the principal's office asking me to not do that. And then I moved my donut stand over 50 feet, so it was off school campus, saying that they could no longer police me. I remember we had competitors pop up where the competitors were charging. They bought these Chuck's Donuts, which if anyone in the Bay Area knows, are higher end donuts than Safeway Donuts, but they have a higher cost basis. They cost a dollar per. And so I dropped my prices to $1 for two weeks to run them out of business before I knew what anti-competitive practices were. And I'd hire all my friends, paying my friends in donuts because they perceived the donuts as $2 each where they could sell them throughout the school and I could have a lower cost basis on them.

**Brendan Foody** (00:55:12):
So I had all of these fun experiences in selling donuts, and then I could talk more about my high school business as well, which was a more significant scale. But I think the takeaway from that was just like you can just do things. So many people have ideas, but the barrier to more companies being built, I think, is just initiative and taking the steps to build the product or experience that customers want and investing the time and the ambition to scale that up. And so I think it was really getting reps of that that enabled me to realize that I should do it later on at a much larger scale.

**Lenny Rachitsky** (00:55:51):
Amazing story. I love how wholesome that is versus drugs, selling donuts.

**Brendan Foody** (00:55:56):
Then my mom was very worried. She was like, "Oh, is there any pot of these donuts?" I was like, "No, mom, I assure you these are pure donuts."

**Lenny Rachitsky** (00:56:05):
I love that you paid your mom $20 to drive.

**Brendan Foody** (00:56:07):
Yeah. She was adamant it couldn't be a handout that she was taking her time to drive me, so she needed to make a little bit of money off of it. We haggled over her title where eventually she wanted to be head of global operations, which we found very entertaining.

**Lenny Rachitsky** (00:56:22):
I hope that's on her LinkedIn.

**Brendan Foody** (00:56:24):
Not yet. Maybe she'll have to add it.

**Lenny Rachitsky** (00:56:27):
So you said that you've started a dozen companies?

**Brendan Foody** (00:56:29):
Yeah.

**Lenny Rachitsky** (00:56:29):
Wow. Okay.

**Brendan Foody** (00:56:30):
Well, a dozen projects, but I think it was that, and then my AWS company were the two that I scaled up.

**Lenny Rachitsky** (00:56:39):
What's the story behind Mercor as the name?

**Brendan Foody** (00:56:42):
Mercor means marketplace in Latin or to buy, sell, trade. And we want to build the largest marketplace in the world, the marketplace for how everyone finds jobs, and that was really the draw to it.

**Lenny Rachitsky** (00:56:55):
Okay, maybe a last question. This is going back to earlier in discussion because it's something I've been thinking about as we're talking. There's been this shift from data as the fuel for models, and now it's experts. Do you think there's a next step, or is this just will take us to AGI, superintelligence?

**Brendan Foody** (00:57:15):
I don't think it's necessarily changing from data to experts. It's more just the paradigm of realizing that labs need this close collaboration with experts to help understand what are the evals that they're building and how can they push the frontier. But I think it's very clear that evals are evergreen, that so long as we want to improve models, we'll need experts to create evals for them and to create the post-training data for them to learn those capabilities. And of course there might be changes in the exact way that people do training with RL or otherwise, but they will always need an eval to measure what does success look like across every domain that they want to build.

**Lenny Rachitsky** (00:58:01):
Okay. So then building on that, a question that comes up a lot these days is, and I know we're talking about fun stuff but I'm getting to serious stuff again, scaling laws and just progression of model intelligence. A lot of people are feeling like, "I don't know, it's slowing down. We're not going to really get to superintelligence at this rate." What is your sense?

**Brendan Foody** (00:58:21):
I totally agree with that. I know there's been some executives to big labs that say we'll have superintelligence in three years, but I think the truth is that it's a longer road. And that's not to diminish from how extraordinary the models are. I think we'll be able to automate a majority of knowledge work tasks in the next 10 years for sure, but that long road is paved with all of the evals that help to make those capabilities possible. And it's not going to be 10X more pre-training data that gets those capabilities. It's much more going to be all of the post-training data sets that are far more data-efficient and thoughtful that help us get there.

**Lenny Rachitsky** (00:59:05):
David Sachs tweeted this interesting point that the situation we're now is almost the best case scenario where AI is not in this fast takeoff to superintelligence. There's a lot of competitors keeping each other in check. Models are already very valuable and only getting valuable, more valuable, but there's not just this winner superintelligence taking over the world situation.

**Brendan Foody** (00:59:26):
Yeah, I think that's true. I think a lot of the super intelligence fearmongering is probably overrated, but at the same time a lot of people's framing around that is even if there is a 5 to 10% chance of this P-Doom, then we should be careful, which seems logical. But I think that it's going to be an extraordinary 10 years for all of Silicon Valley and all of the world as this technology is able to create abundance and giving everyone better medical treatment, the best access to legal recommendations, and the ability to build great products more than we've ever seen before.

**Lenny Rachitsky** (01:00:06):
And education feels like is transforming.

**Brendan Foody** (01:00:08):
Absolutely, right. I even have felt bits of this over the last 10 years where I remember ever... My parents would give me a hard time for not going to classes in college and I'd be like, well, there's way better lectures on YouTube. Why not just listen there? But I can only imagine as the models get extremely good at conveying information, better than the best professor, what that'll mean and access to all sorts of information to better forward humanity and upskill everyone.

**Lenny Rachitsky** (01:00:41):
So I'll use that as a segue to a final question. I'm going to take us to AI Corner, which is a recurring segment on the podcast. What's some way that you personally use AI to do better work to help you in life?

**Brendan Foody** (01:00:52):
Well, let's see. I use it a lot to write documents, as you would expect. I also talk to get advice on problems. I find it helpful to just reason through almost as a thought partner because, yeah, I don't know. I find I think better sometimes when I'm talking something through, but I can't talk through everything with colleagues or people around me.

**Lenny Rachitsky** (01:01:15):
And so this is like ChatGPT Voice Mode mostly or something else.

**Brendan Foody** (01:01:16):
Yeah, I like ChatGPT Voice Mode a lot. There's stuff-

**Lenny Rachitsky** (01:01:16):
Me too.

**Brendan Foody** (01:01:21):
... or room for improvement, but I am very excited about the future of Voice.

**Lenny Rachitsky** (01:01:25):
Let me show you something I built, actually. I wasn't planning to talk about this, but there's this guy, Eric Antonow, who's been recommended by a lot of people to get him on this podcast. He's this creative product person that's under the radar now. He's at Facebook for a long time. He built this project called Pirate GPT, which is you basically put ChatGPT into a stuffed animal to talk to it. So built a little wise owl. I don't have it on right now.

**Brendan Foody** (01:01:25):
Wow.

**Lenny Rachitsky** (01:01:49):
But basically you sew in a little speaker right here and you put a little magnet underneath and you can put it on your shoulder and then you just talk to it.

**Brendan Foody** (01:01:57):
That's so cute. Wow. I love it. I'll have to get one of those. Because I have some of the voice assistants in my apartment, but I really want a ChatGPT voice assistant, so I'm excited for-

**Lenny Rachitsky** (01:02:07):
I was just thinking that. Yeah, just come on. Why can't we have a ChatGPT voice just sitting around listening to us all the time. And you can't on your phone because it goes to sleep and it's like, "Hello, what?"

**Brendan Foody** (01:02:17):
Exactly. Yeah.

**Lenny Rachitsky** (01:02:18):
Yeah, so it's what this is trying to be. Well, there's a kickstarter he started that we'll link to that. You could help out.

**Brendan Foody** (01:02:22):
There we go.

**Lenny Rachitsky** (01:02:23):
That's really easy.

**Lenny Rachitsky** (01:02:25):
Brendan, is there anything else that you wanted to share or touch on or maybe leave listeners with before we get to a very exciting lighting round?

**Brendan Foody** (01:02:32):
Tying to the point around initiative and that you can just do things, I encourage everyone, especially with AI and it being so much easier to build, just take the initiative to go out and build products and talk with customers and take that leap of faith because I think that that is in so many ways, the largest barrier to more innovation, the economy in any way that we can support that.

**Lenny Rachitsky** (01:02:58):
Yeah. There's so many people that just, let's not bash the podcast, but just listen to podcasts, read posts, just keep reading and listening and don't do anything with that information. And there's never been an easier time to actually build stuff and try stuff.

**Brendan Foody** (01:03:12):
Totally.

**Lenny Rachitsky** (01:03:12):
So definitely take that advice. Just you can do things. You can move your donut stand 50 feet and get out of their jurisdiction.

**Brendan Foody** (01:03:21):
Yeah.

**Lenny Rachitsky** (01:03:21):
Okay, Brendan, with that, we've reached a very exciting lightning round. I've got five questions for you. Are you ready?

**Brendan Foody** (01:03:26):
All set.

**Lenny Rachitsky** (01:03:27):
What are two or three books that you find yourself recommending most to other people?

**Brendan Foody** (01:03:31):
Let's see. I would say in order, High Output Management is a phenomenal book on running companies. Second is Zero to One, which of course is a classic. And then third is Shoe Dog, where I just find it to be a really inspirational story.

**Lenny Rachitsky** (01:03:46):
What is a recent movie or TV show you really enjoyed?

**Brendan Foody** (01:03:49):
I really liked Oppenheimer. My favorite TV show of all time is Suits, so I know not recent, but if I had to choose a recent one, probably Oppenheimer.

**Lenny Rachitsky** (01:03:58):
Very cool. Suits, first time someone's mentioned that. Favorite product you recently discovered that you really love?

**Brendan Foody** (01:04:05):
I love using Codex, like the new version. I know it's sort of new in terms of version. Yeah, I think it's incredible and just a huge, huge improvement. So yeah.

**Lenny Rachitsky** (01:04:19):
Do you have a life motto that you find yourself coming back to, sharing with folks, finding useful in work or in life?

**Brendan Foody** (01:04:25):
I think it's you can just do stuff, what we were talking about earlier. Take the leap of faith.

**Lenny Rachitsky** (01:04:31):
I thought you were going to say can do, which is in your Twitter profile.

**Brendan Foody** (01:04:34):
Can do as well, yeah.

**Lenny Rachitsky** (01:04:36):
Two great ones. Final question. So we were chatting before this about things that we could talk about and you shared this interesting thing that you haven't shared anywhere else, which is that you're dyslexic. Why don't you share that with folks? And just how do you get around that having built the fastest-growing company in history?

**Brendan Foody** (01:04:55):
I don't hide it at all. I think a lot of my colleagues know. And I think on one hand it definitely makes it difficult to go through 1,000 emails a day or read every document that I'm supposed to, but on the other hand, I feel like it helps me to think a little bit differently, to be more creative, and perhaps see that markets are changing that not everyone sees. And so it's turned out okay so far. And so I think one thing it's helped me realize from a management standpoint is that we focus much more on how we can leverage people's strengths rather than helping to improve weaknesses, because there's some things that I'm not great at and I'll never be the best in the world at, and there's others that I can hopefully refine and strive to be.

**Lenny Rachitsky** (01:05:46):
That's such a also recurring theme on this podcast of just focusing on strengths and not focusing over all your focus on weaknesses.

**Lenny Rachitsky** (01:05:53):
Brendan, this was incredible. I learned so much. I have a billion more questions, but you got shit to do. Two final questions. What should people know about what you're doing and roles you're hiring for? And then how can listeners be useful to you?

**Brendan Foody** (01:06:06):
Absolutely. We're hiring a ton across the board on our team. We're hiring strategic project leads on our operations team, software engineers in our engineering team, as well as researchers. And so please go to mercor.com and we would love to work with you, and that's the largest way that you can help us. Share it with your friends as well. Over half of people in our marketplace come from referrals because we have a platform of people that love us. And so any jobs that you want to apply to or send your friends to, we would love to have you.

**Lenny Rachitsky** (01:06:37):
Brendan, thank you so much for joining me.

**Brendan Foody** (01:06:39):
Thank you for having me.

**Lenny Rachitsky** (01:06:41):
Bye, everyone.

**Lenny Rachitsky** (01:06:43):
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.

---

## [17/18] Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course)

**Lenny Rachitsky** (00:00:00):
To build great AI products, you need to be really good at building evals. It's the highest ROI activity you can engage in.

**Hamel Husain** (00:00:05):
This process is a lot of fun. Everyone that does this immediately gets addicted to it. When you're building an AI application, you just learn a lot.

**Lenny Rachitsky** (00:00:12):
What's cool about this is you don't need to do this many, many times. For most products, you do this process once and then you build on it.

**Shreya Shankar** (00:00:18):
The goal is not to do evals perfectly, it's to actionably improve your product.

**Lenny Rachitsky** (00:00:23):
I did not realize how much controversy and drama there is around evals. There's a lot of people with very strong opinions.

**Shreya Shankar** (00:00:28):
People have been burned by evals in the past. People have done evals badly, so then they didn't trust it anymore, and then they're like, "Oh, I'm anti evals."

**Lenny Rachitsky** (00:00:36):
What are a couple of the most common misconceptions people have with evals?

**Hamel Husain** (00:00:39):
The top one is, "We live in the age of AI. Can't the AI just eval it?" But it doesn't work.

**Lenny Rachitsky** (00:00:45):
A term that you used in your posts that I love is this idea of a benevolent dictator.

**Hamel Husain** (00:00:49):
When you're doing this open coding, a lot of teams get bogged down in having a committee do this. For a lot of situations, that's wholly unnecessary. You don't want to make this process so expensive that you can't do it. You can appoint one person whose taste that you trust. It should be the person with domain expertise. Oftentimes, it is the product manager.

**Lenny Rachitsky** (00:01:09):
Today, my guests are Hamel Husain and Shreya Shankar. One of the most trending topics on this podcast over the past year has been the rise of evals. Both the chief product officers of Anthropic and OpenAI shared that evals are becoming the most important new skill for product builders. And since then, this has been a recurring theme across many of the top AI builders I've had on. Two years ago, I had never heard the term evals. Now it's coming up constantly. When was the last time that a new skill emerged that product builders had to get good at to be successful?

**Lenny Rachitsky** (00:01:41):
Hamel and Shreya have played a major role in shifting evals from being an obscure, mysterious subject to one of the most necessary skills for AI product builders. They teach the definitive online course on evals, which happens to be the number one course on Maven. They've now taught over 2,000 PMs and engineers across 500 companies, including large swaths of the OpenAI and Anthropic teams along with every other major AI lab.

**Lenny Rachitsky** (00:02:07):
In this conversation, we do a lot of show versus tell. We walk through the process of developing an effective eval, explain what the heck evals are and what they look like, address many of the major misconceptions with evals, give you the first few steps you can take to start building evals for your product, and also share just a ton of best practices that Hamel and Shreya have developed over the past few years. This episode is the deepest yet most understandable primer you'll find on the world of evals. And honestly, it got me excited to write evals, even though I have nothing to write evals for. I think you'll feel the same way as you watch this.

**Hamel Husain** (00:05:04):
Thank you for having us.

**Shreya Shankar** (00:05:05):
Yeah, super excited.

**Lenny Rachitsky** (00:05:07):
I'm even more excited. Okay, so a couple years ago, I had never heard the term evals. Now it's one of the most trending topics on my podcast, essentially, that to build great AI products, you need to be really good at building evals. Also, it turns out some of the fastest-growing companies in the world are basically building and selling and creating evals for AI labs. I just had the CEO of Mercor on the podcast. So there's something really big happening here. I want to use this conversation to basically help people understand this space deeply, but let's start with the basics. Just what the heck are evals? For folks that have no idea what we're talking about, give us just a quick understanding of what an eval is, and let's start with Hamel.

**Hamel Husain** (00:05:49):
Sure. Evals is a way to systematically measure and improve an AI application, and it really doesn't have to be scary or unapproachable at all. It really is, at its core, data analytics on your LLM application and a systematic way of looking at that data, and where necessary, creating metrics around things so you can measure what's happening, and then so you can iterate and do experiments and improve.

**Lenny Rachitsky** (00:06:22):
So that's a really good broad way of thinking about it. If you go one level deeper just to give people a very, even more concrete way of imagining and visualizing what we're talking about, even if you have a example to show would be even better, what's an even deeper way of understanding what an eval is?

**Hamel Husain** (00:06:36):
Let's say you have a real estate assistant application and it's not working the way you want. It's not writing emails to customers the way you want, or it's not calling the right tools, or any number of errors. And before evals, you would be left with guessing. You would maybe fix a prompt and hope that you're not breaking anything else with that prompt, and you might rely on vibe checks, which is totally fine.

**Hamel Husain** (00:07:11):
And vibe checks are good and you should do vibe checks initially, but it can become very unmanageable very fast because as your application grows, it's really hard to rely on vibe checks. You just feel lost. And so evals help you create metrics that you can use to measure how your application is doing and kind of give you a way to improve your application with confidence. That you have a feedback signal in which to iterate against.

**Lenny Rachitsky** (00:07:44):
So just to make very real, so imagining this real estate agent, maybe they're helping you book a listing or go see an open house. The idea here is you have this agent talking to people, it's answering questions, pointing them to things. As a builder of that agent, how do you know if it's giving them good advice, good answers? Is it telling them things that are completely wrong?

**Lenny Rachitsky** (00:08:04):
So the idea of evals, essentially, is to build a set of tests that tell you, how often is this agent doing something wrong that you don't want it to do? And there's a bunch of ways you could define wrong. It could be just making up stuff. It could be just answering in a really strange way. The way I think about evals, and tell me if this is wrong, just simply is like unit tests for code. You're smiling. You're like, "No, you idiot."

**Shreya Shankar** (00:08:29):
No, that's not what I was thinking.

**Lenny Rachitsky** (00:08:31):
Okay. Okay, okay, tell me. Tell me, how does that feel as a metaphor?

**Shreya Shankar** (00:08:35):
Okay. I like what you said first, which is we had a very broad definition. Evals is a big spectrum of ways to measure application quality. Now, unit tests are one way of doing this. Maybe there are some non-negotiable functionalities that you want your AI assistant to have, and unit tests are going to be able to check that. Now, maybe you also, because these AI assistants are doing such open-ended tasks, you kind of also want to measure how good are they at very vague or ambiguous things like responding to new types of user requests or figuring out if there's new distributions of data like new users are coming and using your real estate agent that you didn't even know would use your product. And then all of a sudden, you think, "Oh, there's a different way you want to kind of accommodate this new group of people."

**Shreya Shankar** (00:09:24):
So evals could also be a way of looking at your data regularly to find these new cohorts of people. Evals could also be like metrics that you just want to track over time, like you want to track people saying, "Yes. Thumbs up. I liked your message." You want very, very basic things that are not necessarily AI-related but can go back into this flywheel of improving your product. So I would say, overall, unit tests are a very small part of that very big puzzle.

**Lenny Rachitsky** (00:09:56):
Awesome. You guys actually brought an example of an eval just to show us exactly what the hell we're talking about. We're talking in these big ideas. So how about let's pull one up and show people, "Here's what an eval is."

**Hamel Husain** (00:10:06):
Yeah, let me just set the stage for it a little bit. So to echo what Shreya said, it's really important that we don't think of evals as just tests. There's a common trap that a lot of people fall into because they jump straight to the test like, "Let me write some tests," and usually that's not what you want to do. You should start with some kind of data analysis to ground what you should even test, and that's a little bit different than software engineering where you have a lot more expectations of how the system is going to work. With LLMs, it's a lot more surface area. It's very stochastic, so you kind of have a different flavor here.

**Hamel Husain** (00:10:47):
And so the example I'm going to show you today, it's actually a real estate example. It's a different kind of real estate example. It's from a company called Nurture Boss. I can share my screen to show you their website just to help you understand this use case a little bit, so let me share my screen. So this is a company that I worked with. It's called Nurture Boss, and it is a AI assistant for property managers who are managing apartments, and it helps with various tasks such as inbound leads, customer service, booking appointments, so on and so forth. It's like all the different sort of operations you might be doing as a property manager, it helps you with that. And so you can see kind of what they do. It's a very good example because it has a lot of the complexities of a modern AI application.

**Hamel Husain** (00:11:40):
So there's lots of different channels that you can interact through the AI with like chat, text, voice, but also, there's tool calls, lots of tool calls for booking appointments, getting information about availability, so on and so forth. There's also RAG retrieval, getting information about customers and properties and things like that. So it's pretty fully fleshed in terms of an AI application. And so they have been really generous with me in allowing me to use their data as a teaching example. And so we have anonymized it, but what I'm going to walk through today is, okay, let's do the first part of how we would start to build evals for Nurture Boss. Why would we even want to do that?

**Hamel Husain** (00:12:36):
So let's go through the very beginning stage, what we call error analysis, which is, let's look at the data of their application and first start with what's going wrong. So I'm going to jump to that next, and I'm going to open an observability tool. And you can use whatever you want here. I just happen to have this data loaded in a tool called Braintrust, but you can load it in anything. We don't have a favorite tool or anything in the blog post that we wrote with you. We had the same example but in Phoenix Arize, and I think Aman, on your blog post, used Phoenix Arize as well. And there's also LangSmith. So these are kind of like different tools that you can use.

**Hamel Husain** (00:13:29):
So what you see here on the screen, this is logs from the application, and let me just show you how it looks. So what you see here is, and let me make it full screen, this is one particular interaction that a customer had with the Nurture Boss application, and what it is is a detailed log of everything that happened. So it's called a trace, and it's just the engineering term for logs of a sequence of events. The concept of a trace has been around for a really long time, but it's especially really important when it comes to AI applications.

**Hamel Husain** (00:14:12):
And so we have all the different components and pieces and information that the AI needs to do its job, and we are logged all of it and we're looking at a view of that. And so you see here a system prompt. The system prompt says, "You are an AI assistant working as a leasing team member at Retreat at Acme Apartments." Remember, I said this is anonymized, so that's why the name is Acme Apartments. "Your primary role is to respond to text messages from both current residents and prospective residents. Your goal is to provide accurate, helpful information," yada, yada, yada. And then there's a lot of detail around guidelines of how we want this thing to behave.

**Lenny Rachitsky** (00:14:56):
Is this their actual system prompt, by the way, for this company?

**Hamel Husain** (00:14:58):
It is. Yes, it is.

**Lenny Rachitsky** (00:14:58):
Amazing. That's so cool.

**Hamel Husain** (00:14:59):
It's a real system prompt.

**Lenny Rachitsky** (00:15:01):
That's amazing because it's rare you see a actual company product's system prompt. That's like their crown jewels a lot of times, so this is actually very cool on its own.

**Hamel Husain** (00:15:08):
Yeah. Yeah, it's really cool. And you see all of these different sort of features that are different use cases, so things about tour scheduling, handling applications, guidance on how to talk to different personas, so on and so forth. And you can see the user just kind of jumps in here and asks, "Okay, do you have a one-bedroom with study available? I saw it on virtual tours." And then you can see that the LLM calls some tools. It calls this get individual's information tool, and it pulls back that person's information. And then it gets the community's availability. So it's querying a database with the availability for that apartment complex.

**Hamel Husain** (00:16:01):
And then finally, the AI responds, "Hey, we have several one-bedroom apartments available, but none specifically listed with a study. Here are a few options."

**Hamel Husain** (00:16:12):
And then it says, "Can you let me know when one with a study is available?"

**Hamel Husain** (00:16:16):
And then it says, "I currently don't have specific information on the availability of a one-bedroom apartment."

**Hamel Husain** (00:16:23):
User says, "Thank you."

**Hamel Husain** (00:16:25):
And the AI says, "You're welcome. If you have any more questions, feel free to reach out." Now, this is an example of a trace, and we're looking at one specific data point. And so one thing that's really important to do when you're doing data analysis of your LLM application is to look at data. Now, you might wonder, "There's a lot of these logs. It's kind of messy. There's a lot of things going on here. How in the hell are you supposed to look at this data? Do you want to just drown in this data? How do you even analyze this data?"

**Hamel Husain** (00:17:07):
So it turns out there is a way to do it that is completely manageable, and it's not something that we invented. It's been around in machine learning and data science for a really long time, and it's called error analysis. And what you do is, the first step in conquering data like this is just to write notes. Okay? So you got to put your product hat on, which is why we're talking to you, because product people have to be in the room and they have to be involved in sort of doing this. Usually a developer is not suited to do this, especially if it's not a coding application.

**Lenny Rachitsky** (00:17:47):
And just to mirror back, why I think you're saying that is because this is the user experience of your product. People talking to this agent is the entire product essentially, and so it makes sense for the product person to be super involved in this.

**Hamel Husain** (00:17:59):
Yeah. So let's reflect on this conversation. Okay, a user asked about availability. The AI said, "Oh, we don't really have that. Have a nice day." Now, for a product that is helping you with lead management, is that good? Do you feel like this is the way we want it to go?

**Lenny Rachitsky** (00:18:30):
Not ideal.

**Hamel Husain** (00:18:32):
Yes, not ideal, and I'm glad you said that. A lot of people would say, "Oh, it's great. The AI did the right thing. It looked, it said, 'We didn't have available,' and it's not available." But with your product hat on, you know that's not correct. And so what you would do is you would just write a quick note here. You would say, "Okay." You might pop in here, and you can write a note. So every observability application has ability to write notes, and you wouldn't try to figure out if something is wrong. In this case, it's kind of not doing the right thing, but you just write a quick note, "Should have handed off to a human."

**Lenny Rachitsky** (00:19:19):
And as we watch this happening, it's like you mention this and you'll explain more. You're doing this, this feels very manual and unscalable, but as you said, this is just one step of the process and there's a system to this. That was just the first one.

**Hamel Husain** (00:19:30):
Yeah, and you don't have to do it for all of your data. You sample your data and just take a look, and it's surprising how much you learn when you do this. Everyone that does this immediately gets addicted to it and they say, "This is the greatest thing that you can do when you're building an AI application." You just learn a lot and you're like, "Hmm, this is not how I want it to work. Okay." And so that's just an example.

**Hamel Husain** (00:19:58):
So you write this note, and then we can go on to the next trace. So this is the next trace. I just pushed a hot key on my keyboard. Let me go back to looking at it.

**Lenny Rachitsky** (00:20:09):
And these tools make it easy to go through a bunch and add these notes quickly.

**Hamel Husain** (00:20:13):
Yes. And so this is another one. Similar system prompt. We don't need to go through all of it again. We'll just jump right into the user question. "Okay, I've been texting you all day." Isn't that funny? And the user says, "Please." Okay, yeah, this one is just like an error in the application where this is a text message application, sorry, the channel through which the customer is communicating is through text message, and you're just getting really garbled. And you can see here that it kind of doesn't make sense. The words are being cut off like, "In the meantime," and then the system doesn't know how to respond, because you know how people text message, they write short phrases. They split their sentence across four or five different turns. So in this case-

**Lenny Rachitsky** (00:21:16):
Yeah, so what do you do with something like that?

**Hamel Husain** (00:21:18):
Yeah, so this is a different kind of error.

**Lenny Rachitsky** (00:21:19):
Mm.

**Hamel Husain** (00:21:19):
This is more of, "Hey, we're not handling this interaction correctly. This is more of a technical problem," rather than, "Hey, the AI is not doing exactly what we want." So we would write that down too.

**Lenny Rachitsky** (00:21:20):
Which is still really cool.

**Hamel Husain** (00:21:20):
Yeah.

**Lenny Rachitsky** (00:21:31):
It's amazing you're catching that, too, here. Otherwise, you'd have no idea this was happening.

**Hamel Husain** (00:21:35):
Yeah, you might not know this is happening, right? And so you would just say, "Okay." You would write a note like, "Oh, conversation flow is janky because of text message."

**Lenny Rachitsky** (00:21:51):
And I like that, I like that you're using the word janky. It shows you just how informal this can be at this stage.

**Hamel Husain** (00:21:56):
Yeah, it's supposed to be chill. Just don't overthink it. And there's a way to do this. So the question always comes up, how do you do this? Do you try to find all the different problems in this trace? What do you write a note about? And the answer is, just write down the first thing that you see that's wrong, the most upstream error. Don't worry about all the errors, just capture the first thing that you see that's wrong, and stop, and move on. And you can get really good at this. The first two or three can be very painful, but you can do a bunch of them really fast.

**Hamel Husain** (00:22:38):
So here's another one, and let's skip the system prompt again. And the user asks, "Hey, I'm looking for a two- to three-bedroom with either one or two baths. Do you provide virtual tours?"

**Hamel Husain** (00:22:51):
And a bunch of tools are called and it says, "Hi Sarah. Currently, we have three-bedroom, two-and-a-half-bathroom apartment available for $2,175. Unfortunately, we don't have any two-bedroom options at the moment. We do offer virtual tours. You can schedule a tour," blah, blah. It just so happens that there is no virtual tour, right?

**Lenny Rachitsky** (00:23:16):
Mm-hmm. Nice.

**Hamel Husain** (00:23:16):
So it is hallucinating something that doesn't exist. Then you kind of have to bring your context as an engineer, or even product content, and say, "Hey, this is kind of weird. We shouldn't be telling a person about virtual tour when it's not offered."

**Hamel Husain** (00:23:32):
So you would say, "Okay, offered virtual tour," and you just write the note. So you can see there's a diversity of different kinds of errors that we're seeing, and we're actually learning a lot about your application in a very short amount of time.

**Shreya Shankar** (00:23:55):
One common question that we get from people at this stage is, "Okay, I understand what's going on. Can I ask an LLM to do this process for me?"

**Lenny Rachitsky** (00:24:04):
Mm, great question.

**Shreya Shankar** (00:24:04):
And I loved Hamel's most recent example because what we usually find when we try to ask an LLM to do this error analysis is it just says the trace looks good because it doesn't have the context needed to understand whether something might be bad product smell or not. For example, the hallucination about scheduling the tour, right? I can guarantee you, I would bet money on this, if I put that into chat GPT and asked, "Is there an error?" it would say, "No, did a great job."

**Shreya Shankar** (00:24:34):
But Hamel had the context of knowing, "Oh, we don't actually have this virtual tour functionality," right? So I think, in these cases, it's so important to make sure you are manually doing this yourself. And we can talk a little bit more about when to use LLMs in the process later, but number one pitfall right here is people are like, "Let me automate this with an LLM."

**Lenny Rachitsky** (00:24:55):
Do you think we'll get to a place where an agent can do this, where it has that context?

**Shreya Shankar** (00:24:58):
Oh, no. No, no, no. Sorry. There are parts of error analysis that an LLM is suited for, which we could talk about later in this podcast. But right now, in this stage of free form, note-taking is not the place for an LLM.

**Lenny Rachitsky** (00:25:13):
Got it. And this is something you call open coding, this step?

**Shreya Shankar** (00:25:14):
Yes, absolutely.

**Lenny Rachitsky** (00:25:17):
Cool. Another term that you used in your posts that I love and that fits into this step is this idea of a benevolent dictator. Maybe just talk about what that is, and maybe, Shreya, cover that.

**Shreya Shankar** (00:25:27):
Yeah, so Hamel actually came up with this term.

**Lenny Rachitsky** (00:25:29):
Okay, maybe Hamel cover that, actually.

**Hamel Husain** (00:25:33):
No problem. And we'll actually show the LLM automation in this example, because we're going to take this example, we're going to go all the way through.

**Lenny Rachitsky** (00:25:40):
Amazing.

**Hamel Husain** (00:25:41):
And so benevolent dictator is just a catchy term for the fact that when you're doing this open coding, a lot of teams get bogged down in having a committee do this. And for a lot of situations, that's wholly unnecessary. People get really uncomfortable with, "Okay, we want everybody on board. We want everybody involved," so on and so forth. You need to cut through the noise. And a lot of organizations, if you look really deeply, especially small, medium-sized companies, you can appoint one person whose tastes that you trust. And you can do this with a small number of people and often one person, and it's really important to make this tractable. You don't want to make this process so expensive that you can't do it. You're going to lose out.

**Hamel Husain** (00:26:36):
So that's the idea behind benevolent dictator, is, "Hey, you need to simplify this across as many dimensions as you can." Another thing that we'll talk about later is when it goes to building an LLM as a judge, you need a binary score. You don't want to think about, "Is this like a 1, 2, 3, 4, 5?" Like, assign a score to it. You can't. That's going to slow it down.

**Lenny Rachitsky** (00:26:59):
Just to make sure this benevolent dictator point is really clear, basically, this is the person that-

**Lenny Rachitsky** (00:27:00):
Make sure this benevolent dictator point is really clear. Basically, this is the person that does this note-taking, and ideally they're the expert on the stuff. So if it's law stuff, maybe there's a legal person that owns this, it could be a product manager. Give us advice on who this person should be?

**Hamel Husain** (00:27:16):
Yeah. It should be the person with domain expertise. So in this case, it would be the person who understands the business of leasing, apartment leasing, and has context to understand if this makes sense. It's always a domain expert, like you said. Okay. For legal, it would be a law person. For mental health, it would be the mental health expert, whether that's a psychiatrist or someone else.

**Lenny Rachitsky** (00:27:41):
Cool.

**Hamel Husain** (00:27:42):
Though oftentimes, it is the product manager.

**Lenny Rachitsky** (00:27:44):
Cool. So the advice here is pick that person. It may not feel so super fair that they're the one in charge and they're the dictator, but they're benevolent. It's going to be okay.

**Hamel Husain** (00:27:52):
Yeah. It's going to be okay. It's not perfection. You're just trying to make progress and get signal quickly so you have an idea of what to work on because it can become infinitely expensive if you're not careful.

**Lenny Rachitsky** (00:28:07):
Yeah. Okay, cool. Let's go back to your examples.

**Hamel Husain** (00:28:09):
Yeah, no problem. So this is another example where we have someone saying, "Okay. Do you have any specials?" And the assistant or the AI responds, "Hey, we have a 5% military discount." User responds, and it switches the subject, "Can you tell me how many floors there are? Do you have any one-bedrooms available or one-bedrooms on the first floor?" And the AI responds, "Yeah, okay. We have several one-bedroom apartments available." And then the user wants to confirm, "Any of those on the first floor and how much are the one-bedrooms?" And then also, it's a current resident, so they're also asking, "I need a maintenance request."

**Hamel Husain** (00:28:56):
You could see the messiness of the real world in here, and the assistant just calls a tool that says transfer call, but it doesn't say anything. It just abruptly does transfer call, so it's pretty jank, I would say. It's just not-

**Lenny Rachitsky** (00:29:13):
Another jank.

**Hamel Husain** (00:29:14):
Another kind of jank, a different kind of jank. So when you write the open note, you don't want to say jank, because what we want to do is we want to understand, and when we look at the notes later on, we want to understand what happened.

**Hamel Husain** (00:29:24):
So you just want to say, "Did not confirm call transfer with user." And it doesn't have to be perfect. You just have to have a general idea of what's going on.

**Lenny Rachitsky** (00:29:39):
Cool.

**Hamel Husain** (00:29:39):
So, okay. So let's say we do, and Shreya and I, we recommend doing at least 100 of these. The question is always, "How many of this do you do?" And so there's not a magic number. We say 100 just because we know that as soon as you start doing this, once you do 20 of these, you will automatically find it so useful that you will continue doing it.

**Hamel Husain** (00:30:07):
So we just say 100 to mentally unblock you, so it's not intimidating. It's like, "Don't worry, you're only going to do 100." And there is a term for that, so the right answer is, "Keep looking at traces until you feel like you're not learning anything new." Maybe Shreya should talk about-

**Shreya Shankar** (00:30:30):
Yeah. So there's actually a term-

**Hamel Husain** (00:30:31):
... that.

**Shreya Shankar** (00:30:31):
... in data analysis and qualitative analysis called theoretical saturation. So what this means is when you do all of these processes of looking at your data, when do you stop? It's when you are theoretically saturating or you're not uncovering any new types of notes, new types of concepts, or nothing that will materially change the next part of your process.

**Shreya Shankar** (00:30:57):
And this kind of takes a little bit of intuition to develop, so typically, people don't really know when they've reached theoretical saturation yet. That's totally fine. When you do two or three examples or rounds of this, you will develop the intuition. A lot of people realize, "Oh, okay. I only need to do 40, I only need to do 60. Actually, I only need to do 15." I don't know. Depends on the application and depends on how savvy you are with error analysis for sure.

**Lenny Rachitsky** (00:31:25):
And your point about you're going to want to do a bunch. I imagine it's because you're just like, "Oh, I'm discovering all these problems. I got to see what else is going on here."

**Shreya Shankar** (00:31:33):
Exactly.

**Lenny Rachitsky** (00:31:34):
Is that right?

**Shreya Shankar** (00:31:34):
And promise, at some point, you're not going to discover new types of problems.

**Lenny Rachitsky** (00:31:39):
Yeah. Awesome. So let's say you did 100 of these, what's the next step?

**Hamel Husain** (00:31:42):
Yeah. Okay. So you did 100 of these. Now you have all these notes. So this is where you can start using AI to help you. So the part where you looked at this data is important, like we discussed. You don't want to automate this part too much.

**Lenny Rachitsky** (00:31:59):
Humans will still have jobs. This is the takeaway here. That's great.

**Hamel Husain** (00:32:02):
Yes.

**Lenny Rachitsky** (00:32:02):
Just reviewing traces. At least there's one job left for now. Great.

**Hamel Husain** (00:32:06):
So, yeah. Exactly. And so, okay. You have all these notes. Now, to turn this into something useful, you can do basic counting. So basic counting is the most powerful analytical technique in data science because it's so simple and it's kind of undervalued in many cases, and so it's very approachable for people.

**Hamel Husain** (00:32:33):
And so the first thing you want to do is take these notes, and you can categorize them with an LLM, and so there's a lot of different ways to do that. Right before this podcast, I took three different coding agents or AI tools in how to categorize these notes. So one is, "Okay, I uploaded into a cloud project, I uploaded a CSV of these notes, and I just exported them directly from this interface." There's a lot of different ways to do this, but I'm showing you the simple, stupid way, the most basic way of doing things.

**Hamel Husain** (00:33:13):
And so I dumped the CSV in here and I said, "Please analyze the following CSV file." And I told it there's a metadata field that has a note in it, but what I said is I used the word open codes, and I said, "Hey, I have different open codes," and that's a term of art. LLMs know what open codes are and they know what axial codes are because it is a concept that's been around for a really long time, so those words help me shortcut what I'm trying to do.

**Lenny Rachitsky** (00:33:46):
That's awesome. And the end of the prompt is telling it to create axial codes?

**Hamel Husain** (00:33:50):
Yes. Creating axial codes, so what it does is-

**Shreya Shankar** (00:33:54):
So maybe it's worth talking about what are axial codes or what's the point here? You have a mess of open codes, and you don't have 100 distinct problems. Actually, many of them are repeats, but because you phrased them differently, and that you shouldn't have tried to create your taxonomy of failures as you're open coding. You just want to get down what's wrong and then organize, "Okay, what's the most common failure mode?"

**Shreya Shankar** (00:34:19):
So the purpose, axial code basically is just a failure mode. It's the label or category. And what our goal is, is to get to this clusters of failure modes and figure out what is the most prevalent, so then you can go and run and attack that problem.

**Lenny Rachitsky** (00:34:36):
That is really helpful. Basically, just synthesizing all these-

**Shreya Shankar** (00:34:36):
Absolutely.

**Lenny Rachitsky** (00:34:39):
... into categories and themes. Super cool. And we'll include this prompt in our show notes for folks so they don't have to sit there and screenshot it and try to type it up themselves.

**Hamel Husain** (00:34:49):
Yeah. Great idea. And so Claude went ahead and analyzed the CSV file and decided how to parse it, blah, blah, blah. We don't need to worry about all that stuff, but it came up with a bunch of axial codes. Basically, axial codes are categories, like Shreya said. So one is, okay, capability limitations, misrepresentation, process and protocol violations, human handoff issues, communication, quality. It created these categories.

**Hamel Husain** (00:35:18):
Now, do I like all the categories? Not really. I like some of them. It's a good first stab at it. I would probably rename it a little bit because some of them are a bit too generic. Like what is capability limitation? That's a little bit too broad. It's not actionable. I want to get a little bit more actionable with it so that if I do decide it's a problem, I know what to do with it, but we'll discuss that in a little bit. So you can do this with anything, and this is the dumbest way to do it, but dumb sometimes is a good way to get started, so-

**Lenny Rachitsky** (00:35:49):
And this is what LLMS are really good at, taking a bunch of information and synthesizing it.

**Shreya Shankar** (00:35:53):
Absolutely. Synthesizing for us to make sense of, right? Note that it's not automatically proposing fixes or anything, that's our job, but now, we can wade through this mess of open codes a lot easier.

**Shreya Shankar** (00:36:05):
Another thing that's interesting here in this prompt to generate the axial codes is you can be very detailed if you want, right? You can say, "I want each axial code to actually be some actionable failure mode," and maybe the LLM will understand that and propose it, or, "I want you to group these open codes by what stage of the user story that it's in." So this is where you can be creative or do what's best for you as a product manager or engineer working on this, and that will help you do the improvement later.

**Lenny Rachitsky** (00:36:40):
So there's no definitive prompt of, "Here's the one way to do it"?

**Shreya Shankar** (00:36:42):
Absolutely.

**Lenny Rachitsky** (00:36:43):
You're saying you can iterate, see what works for you?

**Shreya Shankar** (00:36:46):
Absolutely.

**Lenny Rachitsky** (00:36:46):
It's interesting the tools don't do this, or do they try and they just don't do a great job?

**Shreya Shankar** (00:36:50):
No, I don't think they do it. We've been screaming from the rooftops, "Please, please-"

**Lenny Rachitsky** (00:36:54):
Oh, wow.

**Shreya Shankar** (00:36:55):
"... do this." I do think it's a little bit hard, right? Part of this whole experience with the eval scores Hamel and I are teaching are a lot of people don't actually know this, so maybe it's that people don't know this and they don't know how to build tools for it. And hopefully, we can demystify some of this magic.

**Lenny Rachitsky** (00:37:13):
And just to double-click on this point, this is not a thing everyone does or knows. This is something you two developed based on your experience doing data analysis and data science at other companies?

**Shreya Shankar** (00:37:23):
Well, I want to caveat that we didn't invent error analysis. We don't actually want to invent things. That's bad signal. If somebody is coming to you with a way to do something that's entirely new and not grounded in hundreds of years of theory and literature, then you should, I don't know, be a little bit wary of that.

**Shreya Shankar** (00:37:42):
But what we tried to do was distill, "Okay, what are the new tools and techniques that you need to make sense of the LLM error-out analysis?" And then we created a curriculum or structured way of doing this. So this is all very tailored to LLMs, but the terms open coding, axial coding, are grounded in social science.

**Lenny Rachitsky** (00:38:04):
Amazing. Okay. What's funny about you guys doing this is I just want to go do this somewhere. I don't have any AI product to do this on, but it's just like, "Oh, this would be so fun." Just sit there and find all the problems I'm running into and categorize them and then try to fix them.

**Shreya Shankar** (00:38:18):
I love that.

**Lenny Rachitsky** (00:38:19):
Hamel pulled up a video. What do you got going on here?

**Hamel Husain** (00:38:22):
Yeah. So I pulled up a video just to drive home Shreya's point. We are not inventing anything, so what you see on the screen here is Andrew Ng, one of the famous machine learning researchers in the world who have taught a lot of people, frankly, machine learning. And you can see this is an eight-year-old video, and he's talking about error analysis.

**Hamel Husain** (00:38:45):
And so this is a technique that's been used to analyze stochastic systems for ages, and it's something that it was just using the same machine learning ideas and principles, just bringing them into here, because again, these are stochastic systems.

**Lenny Rachitsky** (00:39:01):
Awesome. Well, one thing, we're working on getting Andrew on the podcast, we're chatting, so that will-

**Shreya Shankar** (00:39:01):
Nice.

**Lenny Rachitsky** (00:39:05):
... be really fun. Two, I love that my podcast episode just came out today is in your feed there, and it's standing out really well in that feed, so I'm really happy about that [inaudible 00:39:13].

**Hamel Husain** (00:39:13):
Very nice. Yeah. The recommendation algorithm is quite good.

**Lenny Rachitsky** (00:39:15):
Yes. Here we go. Hope you click on that. Don't screw my algorithm. Okay, cool. So we've done some synthesis. I know we're not going to go through the entire step. This is you have a whole course that takes many days to learn this whole process. What else do you want to share about how to go about this process?

**Hamel Husain** (00:39:31):
Okay. So you can do this through anything, and the same thing works just fine in ChatGPT, the same exact prompt. You can see it made axial codes. I really like using Julius AI. It's one of my favorite tools.

**Hamel Husain** (00:39:45):
Julius is kind of this third-party tool that uses notebooks. I personally like Jupiter notebooks a lot, and so it's more of a data science thing, but a lot of product managers that are kind of learning notebooks nowadays, and it's kind of cool. It's like a fun playground where you can write code and look at data. But we don't have to go deeply into that. Just wanted to mention, you can use a lot. AI is really good at this.

**Hamel Husain** (00:40:10):
So let's go to the fun part. Here we go. So now we have these axial codes. So the first thing I like to do, I have these open codes, and I have the axial codes, let's say, that we assigned from the cloud project or the ChatGPT. And so what I do is I collect them first and I take a look, like, "Does these axial codes make sense?" And I look at the correspondence between the different axial codes and the open codes, and I go through an exercise and I say, "Hmm. Do I like these codes? Can I make them better? Can I refine them? Can I make them more specific?" Instead of being generic, I make them very specific and actionable.

**Hamel Husain** (00:40:59):
So you see the ones that I came up with here are tour scheduling, rescheduling issues, human handoff or transfer issue, formatting error with an output, conversational flow. We saw the conversational flow issue with the text messages. Making follow-up promises not kept.

**Hamel Husain** (00:41:18):
And so basically, what I can do, what you can do now is you have these axial codes, and so I just collect them into a list, so this is an Excel formula. Just collect these codes into a list, and now we have a comma-separated list of these codes. And then what you can simply do is you could take your notes that you have, those open codes, and you can tell an AI, and this is using Gemini and AI just for simplicity, this is, again, we're trying to keep it simple, categorize the following note into one of the following categories as always.

**Lenny Rachitsky** (00:41:56):
For folks watching, I like all these different prompts and formulas you're sharing. This is the Google Sheets AI prompt.

**Shreya Shankar** (00:42:04):
Huge fan.

**Hamel Husain** (00:42:07):
And so basically, what you could do is you can categorize your traces into one of the buckets, and that's what we have here. We have categorized all those problems that we encountered into one of these things.

**Shreya Shankar** (00:42:22):
And this is automatic, which is very exciting. I mean, the AI is doing it. So this also drives home the point that your open codes have to be detailed, right? You can't just say janky because if the AI is reading janky, it's not going to be able to categorize it. Even a human wouldn't, right? It would have to go and remember why you said janky, so it's important to be somewhat detailed in your open code.

**Lenny Rachitsky** (00:42:45):
Okay. So avoid the word janky. It's a good rule of thumb.

**Shreya Shankar** (00:42:48):
Yeah. Or have it with 10 other words.

**Lenny Rachitsky** (00:42:48):
Oh, okay. What is-

**Hamel Husain** (00:42:48):
Yeah. I was being funny.

**Lenny Rachitsky** (00:42:52):
Yeah, okay. What are some of those other words that people often use that you think are not good?

**Shreya Shankar** (00:42:57):
I don't think it's specific words. I think it's just people are not detailed enough in the open code, so it's hard to do the categorization.

**Lenny Rachitsky** (00:43:04):
Great. And by the way, the reason you have to map them back is because, say, Claude or ChatGPT gave you suggestions and you change them and iterated on them, so you can't just go back and say, "Cool, whatever," in each bucket?

**Hamel Husain** (00:43:16):
Yeah, yeah.

**Lenny Rachitsky** (00:43:17):
Great.

**Hamel Husain** (00:43:17):
That's a really good question, actually. It's good to iterate and think about it a little bit like, "Do I like these open codes? Do these actually make sense to me?" Just like anything that AI does, it's really good to kind of put yourself in the middle just a little bit.

**Lenny Rachitsky** (00:43:32):
It's in the loop. Still space for us. Great.

**Shreya Shankar** (00:43:34):
One of the things that I like to do with this step if I'm trying to use AI to do this labeling, is also have a new category called none of the above. So an AI can actually say, "None of the above," in the axial code, and that informs me, "Okay, my axial codes are not complete. Let's go look at those open codes, let's figure out what some new categories are or figure out how to reword my other axial codes."

**Lenny Rachitsky** (00:44:00):
Awesome. And what's cool about this is you don't need to do this many, many times.

**Shreya Shankar** (00:44:03):
No.

**Lenny Rachitsky** (00:44:04):
For most products, you do this process once, and then you build on it, I imagine, and you just tweak it over time?

**Shreya Shankar** (00:44:09):
Absolutely. And it gets so fast. People do this once a week, and you can do all of this in 30 minutes, and suddenly your product is so much better than if you were never aware of any of these problems.

**Lenny Rachitsky** (00:44:23):
Yeah. It's absurd to feel like you wouldn't know this is happening. Watching this happening, I'm like, "How could you not do this to your product?"

**Shreya Shankar** (00:44:31):
A lot of people have no idea.

**Lenny Rachitsky** (00:44:31):
Most people. Yeah. We'll talk about that. There's a whole debate around this stuff that we want to talk about. Okay, cool. So you have the sheet. What comes next?

**Hamel Husain** (00:44:40):
Okay. So here's sort of the big unveil. This is the magic moment right now. So we have all these codes that we applied, the ones that we like on our traces. Now, you can do the ta-da, you can count them.

**Hamel Husain** (00:44:56):
So here's a pivot table, and we just can do pivot table on those, and we can count how many times those different things occurred. So what do we find? Find on these traces that we categorized? We found 17 conversational flow issues. And I really like pivot tables because you can do cool things. You can double-click on these. You can say, "Oh, okay. Let me take a look at those," but that's going into an aside about pivot tables, how cool they are.

**Hamel Husain** (00:45:25):
But now, we have just a nice, rough cut of what are our problems? And now, we have gone from chaos to some kind of thinking around, "Oh, you know what? These are my biggest problems. I need to fix conversational issues, maybe these human handoff issues." It's not necessarily the count is the most important thing. It might be something that's just really bad and you want to fix that, but okay. Now, you have some way of looking at your problem, and now you can think about whether you need evals for some of these.

**Hamel Husain** (00:46:07):
So there might be some of these things that might be just dumb engineering errors that you don't need to write an eval for because it's very obvious on how to fix them. Maybe the formatting error with output, maybe you just forgot to tell the LLM how you want it to be formatted, and you didn't even say that in the prompt. So just go ahead and fix the prompt maybe, and we can decide, "Okay, do you want to write an eval for that?" You might still want to write an eval for that because you might be able to test that with just code. You could just test the string, does it have the right formatting potentially? Without running an LLM.

**Hamel Husain** (00:46:53):
So there's a cost-benefit trade-off to evals. You don't want to get carried away with it, but you want to usually ground yourself in your actual errors. You don't want to skip this step. And so the reason I'm kind of spending so much time on this is this is where people get lost. They go straight into evals like, "Let me just write some tests," and that is where things go off the rails.

**Hamel Husain** (00:47:24):
Okay. So let's say we want to tackle one of these things. So for example, let's say we want to tackle this human handoff issue, and we're like, "Hmm, I'm not really sure how to fix this. That's a kind of subjective sort of judgment call on should we be handing off to a human? And I don't know immediately how to fix it. It's not super obvious per se. Yeah. I can change my prompt, but I'm not sure. I'm not 100% sure."

**Hamel Husain** (00:47:56):
Well, that might be sort of an interesting thing for an LLM as a judge, for example. So there's different kinds of evals. One is code-based, which you should try to do if you can because they're cheaper. LLM as a judge is something, it's like a meta eval. You have to eval that eval to make sure the LLM that's judging is doing the right thing, which we'll talk about in a second.

**Hamel Husain** (00:48:25):
So, okay. LLM as a judge, that's one thing. Okay. How do you build an LLM as a judge?

**Lenny Rachitsky** (00:48:31):
Before we get into that actually, just to make sure people know exactly what you're describing there, these two types of evals. One is you said it's code-based and one is LLM as judge. Maybe Shreya, just help us understand what code-based eval even is? It's essentially a unit test? Is that a simple way to think about it?

**Shreya Shankar** (00:48:46):
Yeah. Maybe eval is not the right term here, but think automated evaluator. So when we find these failure modes, one of the things we want is, "Okay. Can we now go check the prevalence of that failure mode in an automated way without me manually labeling and doing all the coding and the grouping, and I want to run it on thousands and thousands of traces, I want to run it every week." That is, okay. You should probably build an automated evaluator to check for that failure mode.

**Shreya Shankar** (00:49:12):
Now, when we're saying code-based versus LLM-based, we're saying, "Okay. So maybe I could write a Python function or a piece of code to check whether that failure mode is present in a trace or not." And that's possible to do for certain things like checking the output is JSON, or checking that it's markdown, or checking that it's short. These are all things you can capture in code or you could approximately capture in code.

**Shreya Shankar** (00:49:38):
When we're talking about LLM judge here, we're saying that this is a complex failure mode and we don't know how to evaluate in an automated way. So maybe we will try to use an LLM to evaluate this very, very narrow, specific failure mode of handoffs.

**Lenny Rachitsky** (00:49:56):
So just to try to mirror back what you're describing, you want to test what your, say, agent or AI product is doing. You ask it a question, it gets back with something.

**Lenny Rachitsky** (00:50:05):
One way to test if it's giving you the right answer is if it's consistently doing the same thing, that you could write a code to tell you this is true or false. For example, will it ever say there's a virtual tour? So you could ask it.

**Shreya Shankar** (00:50:18):
Yes.

**Lenny Rachitsky** (00:50:18):
"Do you provide virtual tours?" It says yes or no, and then you could write code to tell you if it's correct based on that specific answer.

**Lenny Rachitsky** (00:50:27):
But if you're asking about something more complicated and it's not binary, in one world, you need a human to tell you this is correct. The solution to avoid humans having to review all this every time automatically is LLMs replacing human judgment, and you'd call it an LLM as judge. The LLM as being the judge if this is correct or not.

**Shreya Shankar** (00:50:47):
Absolutely. You nailed it.

**Lenny Rachitsky** (00:50:48):
Great.

**Shreya Shankar** (00:50:49):
So people always think, "Oh, this is at least as hard as my problem of creating the original agent." And it's not, because you're asking the judge to do one thing, evaluate one failure mode, so the scope of the problem is very small and the output of this LLM judge is pass or fail. So it is a very, very tightly scoped thing that LLM judges are very capable of doing very reliably.

**Lenny Rachitsky** (00:51:18):
And the goal here is just to have a suite of tests that run before you ship to production that tell you things are going the way you want them to? The way your agent is interacting is correct?

**Shreya Shankar** (00:51:28):
The beautiful thing about LLM judges, you can use them in unit tests or CI, sure, but you could also use it online for monitoring, right? I can sample 1000 traces every day, run my LLM judge, real production traces, and see what the failure rate is there. This is not a unit test, but still now we get an extremely specific measure of application quality.

**Lenny Rachitsky** (00:51:53):
Cool. That's a really great point because a lot of people just see evals for being this not-real-life thing. It's a thing that you test before it's actually in the real world. And what's actually happening in the real world, you're saying you should actually do exactly that?

**Shreya Shankar** (00:52:04):
Yeah.

**Lenny Rachitsky** (00:52:04):
Test your real thing running in production? And it's a daily, hourly sort of thing you could be running?

**Shreya Shankar** (00:52:09):
Totally.

**Lenny Rachitsky** (00:52:10):
Awesome. Okay. Hamel's got an example of an actual LLM as a judge eval here, so let's take a look.

**Hamel Husain** (00:52:16):
I love how Shreya really teed it up for me, so thank you so much. So what we have is a LLM as a judge prompt for this one specific failure. Like Shreya said, you would want to do one specific failure and you want to make it binary because we want to simplify things. We don't want, "Hey, score this on a rating of one to five. How good is it?" That's just in most cases, that's a weasel way of not making a decision. Like, "No, you need to make a decision. Is this good enough or not? Yes or no?"

**Hamel Husain** (00:52:50):
It can be painful to think about what that is, but you should absolutely do it. Otherwise, this thing becomes very untractable, and then when you report these metrics, no one knows what 3.2 versus 3.7 means, so.

**Shreya Shankar** (00:53:03):
Yeah. We see this all the time also, and even with expert-curated content on the internet where it's like, "Oh, here's your LLM judge evaluator prompt. Here's a one-to-seven scale."

**Shreya Shankar** (00:53:15):
And I always text Hamel like, "Oh, no. Now, we have to fight the misinformation again because we know somebody is going to try it out and then come back to us and say, 'Oh, I have 4.2 average,'" and we're going to be like, "Okay."

**Lenny Rachitsky** (00:53:31):
It's wild how much drama there is in the evals space. We're going to get to that. Oh, man.

**Lenny Rachitsky** (00:54:00):
Meticulously designed to be an intuitive and simple experience, and Mercury brings all the ways that you use money into a single product, including credit cards, invoicing, bill pay, reimbursements for your teammates and capital. Whether you're a funded tech startup looking for ways to pay contractors and earn yield on your idle cash, or an agency that needs to invoice customers and keep them current, or an e-commerce brand that needs to stay on top of cash flow and access capital, Mercury can be tailored to help your business perform at its highest level. See what over 200,000 entrepreneurs love about Mercury. Visit mercury.com to apply online in 10 minutes. Mercury is a fintech, not a bank. Banking services provided through Mercury's FDIC insured partner banks. For more details, check out the show notes.

**Hamel Husain** (00:54:45):
Okay, so this is your judge prompt. There's no one way to do it. It's okay to use an LLM to help you create it, but again, put yourself in the loop. Don't just blindly accept what the LLM does, and in all of these cases, that's what we did. With the axial codes, we iterated on this. You can use an LLM to help you create this prompt, but make sure you read it, make sure you edit it, whatever. This is not necessarily the perfect prompt. This is just the stupid, keeping it very simple just to show you the idea. It's like, "Okay, for this handoff failure," I said, "Okay, I want you to output true or false," it's a binary judge. That's what we recommend. Then I just go through and say, "Okay, when should you be doing a handoff?" And I just list them out.

**Hamel Husain** (00:55:33):
Okay, explicit human requests ignored or looped, some policy-mandated transfer, sensitive resident issues, tool data, unavailability, same day walk-in or tour requests. You need to talk to a human for that, so on and so forth. The idea is, now that I know that this is a failure from my data, I'm interested in iterating on it, because I know this is actually happening all the time. Like Shreya said, it would be nice to have a way not only to evaluate this on the data I have, but also on production data, just to get a sense of, what scales is this happening? Let me find more traces, let me have a way to iterate on this. We can take this prompt and I'm going to use the spreadsheet again. The first step is, okay, when I'm doing this judge... I wrote the prompt.

**Hamel Husain** (00:56:28):
Now, a lot of people stop there and they say, "Okay, I have my judge prompt. We're done. Good, let's just ship it," and the prompt says... If the judge says it's wrong, it's wrong. They just accept it as the gospel, be like, "Okay, the LLM says it's wrong, it must be wrong. Don't do that, because that's the fastest way that you can have evals that don't match what's going on, and when people lose trust in your evals, they lose trust in you. It's really important that you don't do that, so before you release your LLM as a judge, you want to make sure it's aligned to the human. How do you do that? You have those axial codes and you want to measure your judge against the axial code, and say like, "Hey, does it agree with me? My own judge, does it agree with me?" Just measure it.

**Hamel Husain** (00:57:18):
What we have here is, okay, I say, "Assess this LLM trace." Again, I'm using just spreadsheets here, "Assess this LM trace according to these rules," and the rules are just the prompt that I just showed you. I ask it, "Okay, is there a handoff error, true or false?" Then this column, let me just zoom in a bit. Column H, I have, "Okay, did this error occur?" Column G is whether I thought the error occurred or not. You can see-

**Lenny Rachitsky** (00:57:53):
You're going through manually, you do that.

**Hamel Husain** (00:57:55):
Yeah, yeah, which we already did. We already went through it manually. It's not like we have to do it again, because we have that cheat code from the axial coding, we already did it. You might have to go through it again if you need more data, and there's a lot of details to this on how to do this correctly. You want to split your data and do all these things, so that you're not cheating, but I just want to show you the concept. Basically, what you can do is measure the agreement. Now, one thing you should know, as a product manager, is a lot of people go straight to this agreement. They say, "Okay, my judge agrees with the human some percentage of the time."

**Hamel Husain** (00:58:41):
Now that sounds appealing, but it's a very dangerous metric to use, because a lot of times, errors, they only happen on the long tail and they don't happen as frequently, so if you only have the error 10% of the time, then you can easily have 90% agreement by just having a judge say it passes all the time. Does that make sense? 90% agreement look good on paper, but it might be misleading.

**Lenny Rachitsky** (00:59:15):
It's rare, it's a rare error. Yeah.

**Hamel Husain** (00:59:18):
As a product manager or someone, even if you're not doing this calculation yourself, if someone ever reports to you agreement, you should immediately ask, "Okay, tell me more." You need to look into it. They give you more intuition, here is like a matrix of this specific judge in the Google sheet, and this is, again, a pivot table, just keeping it dumb and simple. "Okay, on the rows I have, what did the human think? What did I think? Did it have an error, true or false? Then did my judge have an error, true or false?"

**Shreya Shankar** (00:59:56):
The intuition here is exactly what Hamel said, where you need to look at each type of error. When the human said false, but the judge said true, or vice versa, so those non-green diagonals here, and if they're too large, then go iterate on your prompt, make it more clear to the LLM judge, so that you can reduce that misalignment. You want to get to a point where most... You're going to have some misalignment, that's okay. We talk about in our course, also how to code correct that misalignment, but in this stage, if you're a product manager and the person who's building the LLM judge eval has not done this, they're saying like, "It agrees 75% of the time, we're good." They don't have this matrix and they haven't iterated to make sure that these two types of errors have gone down to zero, then it's a bad smell. Go and ask them to go fix that.

**Lenny Rachitsky** (01:00:52):
Awesome. That's a really good tip, what to look for when someone's doing this wrong.

**Shreya Shankar** (01:00:56):
Yeah.

**Lenny Rachitsky** (01:00:56):
Actually, can you take us back to the LLM as judge prompt? I just want to highlight something really interesting here. I've had some guests on the podcast recently who've been saying, "Evals are the new PRDs," and if you look at this, this is exactly what this is. Product managers, product teams, here's what the product should be, here's all the requirements, here's the how it should work. They built a thing and then they test it. Manually, often. What's cool about this is this is exactly that same thing, and it's running constantly. It's telling you, "Here's how this agent should respond," and it's very specific ways. "If it's this, this, this, do that. If it's this, this, that, do that." It's exactly what I've been hearing again and again, you could see right here. This is the purest sense of what a product requirements document should be, is this eval judge that's telling you exactly what it should be, and it's automatic and running constantly.

**Shreya Shankar** (01:01:45):
Yeah, absolutely. It's derived from our own data, so of course, it's a product manager's expectations. What I find that a lot of people miss is they just put in what their expectations are before looking at their data, but as we look at our data, we uncover more expectations that we couldn't have dreamed up in the first place, and that ends up going into this prompt.

**Lenny Rachitsky** (01:02:05):
That is interesting. Your advice is not skip straight to evals and LLM as judge prompts before you build the product, still write traditional one-pagers PRDs to tell your team what we're doing, why we're doing it, what success looks like. But then at the end, you could probably pull from that and even improve that original PRD if you're evolving the product using this process.

**Shreya Shankar** (01:02:28):
I would go even further to say you're going to improve... It's going to change. You're never going to know what the failure modes are going to be upfront, and you're always going to uncover new vibes that you think that your product should have. You don't really know what you want until you see it with these LLMs, so you got to be flexible, have to look at your data, have to... PRDs are a great abstraction for thinking about this. It's not the end all, be all. It's going to change.

**Lenny Rachitsky** (01:02:58):
I love that, and Hamel's pulling up some cool research report. What's this about?

**Hamel Husain** (01:03:04):
This is one of the coolest research reports you can possibly read if you want to know about evals. It was authored by someone named Shreya Shankar.

**Shreya Shankar** (01:03:13):
Oh, my God.

**Hamel Husain** (01:03:15):
And her collaborators. It's called "Who Validates the Validated?"

**Lenny Rachitsky** (01:03:20):
That's the best name for a researcher.

**Shreya Shankar** (01:03:21):
Thank you, thank you.

**Hamel Husain** (01:03:24):
I should let Shreya talk about this. I think one of the most important things to pay attention in this paper are the criteria drift, and what she found.

**Shreya Shankar** (01:03:35):
We did this super fun study when we were doing user studies with people who were trying to write LLM judges or just validate their own LLM outputs. I think this was before evals was extremely popular, I feel like, on the internet. We did this project late 2023 was when we started it. But then the thing that really was burning in my mind as a researcher is like, "Why is this problem so hard? We've been having machine learning and AI for so long, it's not new, but suddenly, this time around, everything is really difficult." We just did this user study with a bunch of developers and we realized, "Okay, what's new here is that you can't figure out your rubrics upfront. People's opinions of good and bad change as they review more outputs, they think of failure modes only after seeing 10 outputs they would never have dreamed of in the first place," and these are experts. These are people who have built many LLM pipelines and now agents before, and you can't ever dream up everything in the first place. I think that's so key in today's world of AI development.

**Lenny Rachitsky** (01:04:50):
That is a really good point. That's very much reinforcing what we were just talking about and that's why I'll pull this up, is just... Okay-

**Shreya Shankar** (01:04:56):
The research behind it.

**Lenny Rachitsky** (01:04:58):
Yeah, okay, great. You still got to do product the same way, but now you have this really powerful tool that helps you make sure what you've built is correct. It's not going to replace the PRD process. Cool. How many, say, I don't know, LLM as judge prompts, do you end up with usually say... I don't know. I know, obviously, depends complexity to the product, but what's a number in your experience?

**Shreya Shankar** (01:05:19):
For me, between four and seven.

**Lenny Rachitsky** (01:05:22):
That's it.

**Shreya Shankar** (01:05:23):
It's not that many, because a lot of the failure modes, as Hamel said earlier, can be fixed by just fixing your prompt. You just didn't think to put it in your prompts, so now you put it in your... You shouldn't do an eval like this for everything, just the pesky ones that you've described your ideal behavior in your agent prompt, but it's still failing.

**Lenny Rachitsky** (01:05:43):
Got it. Say you found a problem, you fixed it. In traditional software development, you'd write a unit test to make sure it doesn't happen again. Is your insight here is, "Don't even bother writing an eval around that if it's just gone"?

**Shreya Shankar** (01:05:54):
I think you can if you want to, but the whole game here is about prioritizing. You have finite resources and finite time, you can't write an eval for everything, so prioritize the ones that are the more pesky areas.

**Lenny Rachitsky** (01:06:07):
Probably the ones that are most risky to your business if they say something like Mecha Hitler, Grok.

**Shreya Shankar** (01:06:07):
Yikes.

**Lenny Rachitsky** (01:06:15):
Cool. Okay, so that's very relieving, because this prompt was a lot of work to really think through all these details.

**Shreya Shankar** (01:06:21):
But it's a lot of one-time cost. Right now, forever, you can run this on your application.

**Hamel Husain** (01:06:30):
Okay, data analysis is super powerful, is going to drive lots of improvements very quickly to your application. We showed the most basic kind of data analysis, which is counting, which is accessible to everyone. You can get more sophisticated with the data analysis. There's lots of different ways to sample, look at data. We made it look easy in a sense, but there's a lot of skills here to do to it well. Building an intuition and a nose for how to sort through this data. For example, let's say I find conversational issues, this conversational flow issues. Maybe if I was trying to chase down this problem further, I would think about ways to find other conversational flow issues that I didn't code. I would maybe dig through the data in several ways, and there's different ways to go about this. It's very similar, if not almost exactly similar as traditional analytics techniques that you would do on any product.

**Lenny Rachitsky** (01:07:41):
Give us just a quick sense of what comes next and then let's talk about the debate around evals and a couple more things.

**Shreya Shankar** (01:07:48):
What comes next after you've built your LLM judge? Well, we find that people just try to use that everywhere they can, so they'll put the LLM judge in unit tests and they will build, "Here are some example traces where we saw that failure, because we labeled it. Now we're going to make those part of unit tests and make sure that, every time we push a change to our code, these tests are going to pass." They also use it for online monitoring. People are making dashboards on this, and I think that's incredible. I think the products that are doing this, they have a very sharp sense of how well their application is performing, and people don't talk about it, because this is their moat. People are not going to go and share all of these things, because it makes sense. If you are an email-writing assistant, and you're doing this and you're doing it well, you don't want somebody else to go and build an email-writing assistant and then get you out of business.

**Shreya Shankar** (01:08:41):
I really want to stress the point that it's try to use these artifacts that you're building wherever possible online, repeatedly use them to drive improvements to your product. Oftentimes, Hamel and I will tell people how to do this up to this very point, and it clicks for people and then they never come back again. Either they have, I don't know, quit their jobs, they're not doing AI development anymore, or they know what to do from here on out. I think it's the latter, but I think it's very powerful.

**Lenny Rachitsky** (01:09:15):
Just watching you do this really opened my eyes to what this is and how systematic the process is. I always imagine you just sit on a computer, "Okay, what are the things I need to make sure work correctly?" What you're showing us here is it's a very simple step-by-step based on real things that are happening in your product, how to catch them, identify them, prioritize them, and then catch them if they happen again and fix them.

**Shreya Shankar** (01:09:38):
Yeah, it's not magic. Anyone can do this, you're going to have to practice the skill, like any new skill, you have to practice, but you can do it. I think what's very empowering now is that product managers are doing this and can do this, and can really build very, very profitable products with this skill set.

**Lenny Rachitsky** (01:09:57):
Okay, great segue to a debate that we got pulled into that was happening on X the other day. I did not realize how much controversy and drama there is around evals. There's a lot of people with very strong opinions. How about Shreya? Give us just a sense of the two sides of the debate around the importance and value of evals, and then give us your perspective.

**Shreya Shankar** (01:10:19):
Yeah. All right, I'll be a little bit placating and I say I think everyone is on the same side. I think the misconception is that people have very rigid definitions of what evals is. For example, they might think that evals is just unit tests or they might think that evals is just the data analysis part and no online monitoring or no monitoring of product-specific metrics, like actually number of chats engaged in or whatnot. I think everyone has a different mindset of evals going in, and the other thing I will say is that people have been burned by evals in the past. I think people have done evals badly. One concrete example of this is they've tried to do an LLM judge, but it has not aligned with their expectations. They only uncovered this later on and then they didn't trust it anymore, and then they're like, "I'm anti evals."

**Shreya Shankar** (01:11:14):
I 100% empathize with that, because you should be anti Likert scale LLM judge. I absolutely agree with you, we are anti that as well. A lot of the misconception stems from two things, like people having a narrow definition of evals and then people not doing it well and then getting burned and then wanting to avoid other people making that mistake. Then, unfortunately, X or Twitter is a medium where people are misinterpreting what everybody is saying all the time, and you just get all these strong opinions of, "Don't do evals, it's bad. We tried it, it doesn't work. We're Claude Code," or whatever other famous product, "And we don't do evals." There's just so much nuance behind all of it, because a lot of these applications are standing on the shoulders of evals. Coding agents is a great example of that, Claude Code. They're standing on the shoulders of Claude base model... Not base, but the fine-tuned Claude models have been evaluated on many coding benchmarks. Can't argue against that.

**Lenny Rachitsky** (01:12:24):
Just to make clear exactly what you're talking about there, one of the heads, I think maybe the head engineer of Claude Code, went on a podcast and he's like, "We don't do evals, we just vibe. We just look at vibes," and vibes meaning they just use it and feel if it's right or wrong.

**Shreya Shankar** (01:12:37):
I think that works. There's two things to that, right? One is they're standing on the shoulders of the evals that their colleagues are doing for coding.

**Lenny Rachitsky** (01:12:45):
Of the Claude foundational model.

**Shreya Shankar** (01:12:47):
Absolutely, right? We know that they report those numbers, because we see the benchmarks, we know who's doing well on those. The other thing is they are actually probably very systematic about the error analysis to some extent. I bet you that they're monitoring who is using Claude, how many people are using Claude, how many traps are being created, how long these chats are. They're also probably monitoring in their internal team, they're dogfooding. Anytime something is off, they maybe have a cue or they send it to the person developing Claude Code, and this person is implicitly doing some form of hair error analysis that Hamel talked about. All of this is evals, right? There's no world in which they're just being like, "I made Claude Code, I'm never looking at anything," and unfortunately, when you don't think about that or talk about that, I think that the community...

**Shreya Shankar** (01:13:39):
Most of the community is beginners or people who don't know about evals and want to learn about it, and it sends the wrong message there. Now, I don't know what Claude Code is doing, obviously, but I would be willing to bet money that they're doing something in the form of evals.

**Hamel Husain** (01:13:53):
We'll also say that coding agents are fundamentally very different than other AI products, because the developer is the domain expert, so you can short circuit a lot of things, and also, the developer is using it all day long, so there's a type of dogfooding and type of domain expertise that is... You can collapse the activities, you don't need as much data, you don't need as much feedback or exploration, because you know, so your eval process should look different.

**Lenny Rachitsky** (01:14:31):
Because you're seeing the code, you see the code it's generating. You can tell, "This is great, this is terrible."

**Hamel Husain** (01:14:35):
Yeah, yeah. I think a lot of people had generalized coding agents, because coding agents are the first AI product released into the wild, and I think it's a mistake to try to generalize that at large.

**Shreya Shankar** (01:14:51):
The other thing is, yeah, engineers have a dogfooding personality. There are plenty of applications where people are trying to build AI in certain domains and they don't have dogfooding for doctors, for example, or not out there trying to get all the most incorrect advice from AI and be tolerant and receptive to that. It's very important to keep, I think these nuanced things in mind.

**Lenny Rachitsky** (01:15:16):
What I'm hearing from you, Shreya, interestingly, is that if humans on the team are doing very close data analysis, error analysis, dogfooding like crazy, and essentially, they're the human evals and you're describing that as that's within the umbrella of evals. You could do it that way if you have time and motivation to do that, or you could set these things up to be automatic.

**Shreya Shankar** (01:15:40):
Absolutely, it's also about the skills. People who work at Anthropic are very, very highly skilled. They've been trained in data analysis or software engineering or AI, and whatnot. You can get there, anyone can get there, of course, by learning the concepts, but most people don't have that skill right now.

**Hamel Husain** (01:16:02):
Dogfooding is a dangerous one, only because a lot of people will say they're dogfooding. They're like, "Yeah, we dogfooded," but are they, really? A lot of people aren't really dogfooding it at that visceral level that you would need to close that feedback loop. That's the only caveat I would add.

**Lenny Rachitsky** (01:16:24):
There's also this, feels like, straw man argument of evals versus A-B tests. Talk about your thoughts there, because that feels like a big part of this debate. People are having like, "Do you need evals if you have A-B tests that are testing production level metrics?"

**Shreya Shankar** (01:16:38):
A-B tests are, again, another form of evals ,I imagine, right? When you're doing an A-B test, you have two different experimental conditions and then you have a metric that quantifies the success of something, and you're comparing the metric. Again, an eval in our mind is systematic measurement of quality, some metric. You can't really do an A-B test without the eval to compare, so maybe we just have a different weird take on it.

**Lenny Rachitsky** (01:17:06):
Yeah, okay. What I'm hearing is you consider A-B tests as part of the suite of evals that you do. I think when people think A-B tests, it's like we're changing something in the product, we're going to see if this improves some metric we care about. Is that enough? Why do we need to test every little feature? If it's impacting a metric we care about as a business, we have a bunch of A-B tests that are just constantly running.

**Shreya Shankar** (01:17:27):
This is now a great point. I think a lot of people prematurely do A-B tests, because they've never done any error analysis in the first place. They just have hypothetically come up with their product requirements and they believe that, "We should test these things," but it turns out, when you get into the data, as Hamel showed, that the errors that you're seeing are not what you thought what the errors might be. They were these weird handoff issues or, I don't know, the text message thing was strange. I would say that, if you're going to do A-B tests and they're powered by actual error analysis as we've shown today, then that's great, go do it. But if you're just going to do them, which we find that people try to do, just want to do them based on what you hypothetically think is what is important, then I would encourage people to go and rethink that and ground your hypotheses.

**Lenny Rachitsky** (01:18:23):
Do you have thoughts on what Statsig is going to do at OpenAI? Is there anything there that's interesting? That was a big deal, a huge acquisition. A- B test company people are like, "A-B test, the future." Thoughts?

**Hamel Husain** (01:18:34):
Just to add to the previous question a little bit, why is there this debate, A-B testing versus evals? I think, fundamentally, evals is... People are trying to wrap their head around how to improve their applications and fundamentally need to do... Data science is useful in products. Looking at data, doing data analytics. There's many different suite of tools, and you don't need to invent anything new. Sure, you don't need necessarily the whole breadth of data science, and it looks slightly different, just slightly, with LLMs. Your tactics might be different, so really what it is is using analytic tools to understand your product. Now, people say the word "Evals," trying to carve out this new thing, and saying evals and then A-B testing, but if you zoom out, it's the same data science as before, and I think that's what's causing the confusion is, "Hey, we need data science thinking," and AI product is helpful to have that thinking in AI products like it is in any product is my take on that.

**Lenny Rachitsky** (01:19:50):
That's a really good take, I think just the word "Evals" triggers people now.

**Shreya Shankar** (01:19:53):
Yeah.

**Lenny Rachitsky** (01:19:53):
If you just call it, "We're just doing error analysis, doing data science to understand where our product breaks and just setting up tests to make sure we know-"

**Shreya Shankar** (01:20:00):
That's boring, sounds boring. No, no, no. We need a mysterious term, like "Evals," to really get the momentum going. Your question about Statsig, I think it's very exciting. To be honest, I don't know much about it, because I just imagine that they're this company that... There's a tool that many people use, and maybe it just so happened that OpenAI acquired them. I'm sure they've been using them in the past, I'm sure OpenAI's competitors are using Statsig as well, so maybe there is something strategic in that acquisition. I have no idea, I don't know anything there, but I think those are really the bigger questions for me than, "Is this fundamentally changing A-B testing or making evals more of a priority?" I think they've always been a priority, I think OpenAI has always been doing some form of them, and OpenAI has gone so far, historically speaking, as to go and look at all the Twitter sentiment and try to do some retrospective on that, and then tie that back to their products. Certainly, they're doing-

**Shreya Shankar** (01:21:00):
Then, tie that back to their products. Certainly, they're doing some amount of evals before they ship their new foundation models, but they're going so much beyond and being like, "Okay, let's find all the tweets that are complaining about it, all the Reddit threads that are complaining about it, and go try to figure out what's going on." It goes to show that evals are very, very important. No one has really figured it out yet. People are using all the available sources signal that they can to improve their products.

**Hamel Husain** (01:21:26):
What I'll say is I'm really hopeful that it might shift or create a focus within OpenAI, hopefully. Up until now, a lot of the big labs understandably focused on general benchmarks like MMLU score, human eval, things like that, which are very important for foundation models. Those not very related to product specific evals, like the ones we talked about today, but handoff and stuff like that, they tend not to correlate.

**Shreya Shankar** (01:22:01):
Yeah, they don't correlate with math problem-solving, sorry to say.

**Hamel Husain** (01:22:06):
Exactly. If you look at the eval products, let's say the ones up until recently that some of the big labs have, they don't have error analysis. They have a suite of generic tools, cosine similarity, hallucination score, whatever, and that doesn't work. It's a good first stab at it. It's okay. At least you're doing something, getting people, maybe it's like getting people look at data. But eventually, what we hope to see is, okay, a bit more data science thinking in this eval process. That's hopefully the tools we'll get to.

**Shreya Shankar** (01:22:44):
Yeah, Pamela and I should not be the only two people on the planet that are promoting a structured way of thinking about application specific evals. It's mind-boggling to me. Why are we the only two people doing this the whole world? What's wrong? I hope that we're not the only people and that more people catch on.

**Lenny Rachitsky** (01:23:04):
The fact that your course on Maven is the number one highest grossing course in Maven, clearly there's demand and interest, and there's more people I think on your side. Interestingly, just as an example you've been sharing on Twitter that I think is informative, everyone's been saying how cloud code doesn't care about evals. They're all about vibes, and everyone's like, and they're the best coding agent out there, so clearly, this is right. More recently, there's all this talk about Codex, OpenAI Codex being better and everyone's switching and they're so pro evals.

**Shreya Shankar** (01:23:33):
I know.

**Lenny Rachitsky** (01:23:34):
Yeah.

**Shreya Shankar** (01:23:38):
It gets me every time. The Internet's so inconsistent. My favorite thing was yesterday, I believe, a couple of lab mates and I were out getting dessert or something, and somebody said like, "Oh, do you like Codex or Claude better or whatever?" The other person said, "Oh, I like Claude." Then, someone else said, "But the new version of Codex is better." Then, the first person said, "Oh, but the last I checked was two days ago, so maybe my thoughts, maybe I'm not up-to-date." I was like, "Oh, my God."

**Lenny Rachitsky** (01:24:14):
So true, so true. This is the world we live in. Oh, my God. Okay. I want to ask about just top misconceptions people have with evals and top tips and tricks for being successful. Maybe just share one or two each of each. Let me just start with misconceptions, and maybe I'll go to the Hamel first. Just what are a couple of the most common misconceptions people have with eval still?

**Hamel Husain** (01:24:31):
The top one is, "Hey, I can just buy a tool, plug it in, and it'll do the eval for you. Why do I have to worry about this? We live in the age of AI. Can't the AI just eval it?" That's the most common misconception, and people want that so much that people do sell it, but it doesn't work. That's the first one.

**Lenny Rachitsky** (01:24:55):
Shoot, many humans are still great. I think that's great news.

**Hamel Husain** (01:25:00):
The second one that I see a lot is, "Hey, just not looking at the data." In my consulting, people come to me with problems all the time, and the first thing I'll say is, "Let's go look at your traces." You can see their eyes pop open and be like, "What do you mean?" I'm like, "Yeah, let's look at it right now." They're surprised that I am going to go look at individual traces, and it always 100% of the time learn a lot and figure out what the problem is. I think people just don't know how powerful looking at the data is like we showed on this podcast.

**Shreya Shankar** (01:25:48):
I would agree with that.

**Lenny Rachitsky** (01:25:50):
Those are the top two? Okay.

**Shreya Shankar** (01:25:51):
Yes.

**Lenny Rachitsky** (01:25:51):
Is there anything else or those are the ones solve those problems.

**Shreya Shankar** (01:25:55):
Oh, those are definitely... Then, I guess the third one I would add is, there's no one correct way to do evals. There are many incorrect ways of doing evals, but there are also many correct ways of doing it. You got to think about where you are at with your product, how much resources you have, and figure out the plan that works best for you. It'll always involve some form of error analysis as we showed today, but how you operationalize those metrics is going to change based on where you're at.

**Lenny Rachitsky** (01:26:28):
Amazing. Okay. What are a couple of just tips and tricks you want to leave people with as they start on their eval journey or just try to get better at something they're already doing?

**Shreya Shankar** (01:26:37):
Tip number one is just don't be alarmed or don't be scared of looking at your data. The process, we try to make it as structured as possible. There are inevitably questions that are going to come up. That's totally fine. You might feel like you're not doing it perfectly. That's also fine. The goal is not to do evals perfectly, it's to actionably improve your product. We guarantee you, no matter what you do, if you're doing parts of these process, you're going to find ways of actionable improvement, and then you're going to iterate on your own process from there.

**Shreya Shankar** (01:27:14):
The other tip that I would say is, we are very pro-AI. Use LLMs to help you organize any thoughts that you have throughout this entire process. This could be everything ranging from initial product requirements. Figure out how to organize them for yourself. Figure out how to improve on that product requirements doc based on the open codes that you've created. Don't be afraid to use AI in ways that present information better for you.

**Lenny Rachitsky** (01:27:44):
Sweet, so don't be scared. Use LLMs as much as you can throughout the process.

**Shreya Shankar** (01:27:48):
But not to replace yourself.

**Lenny Rachitsky** (01:27:51):
Right. Okay, great. There's still jobs. It's great. Hamel.

**Hamel Husain** (01:27:55):
Yeah. Let me actually share my screen, because I want to show something. To piggyback of what Shreya said is, if you heard any phrase in this podcast, you've probably heard look at your data more than anything else. It's so important that we teach that you should create your own tools to make it as easy as possible. I showed you some tools when we're going through the live example of how to annotate data. Most of the people I work with, they realize how important this is and they vibe code their own tools, or we shouldn't say vibe code. They make their own tools, and it's cheaper than ever before because you have AI that can help you.

**Hamel Husain** (01:28:40):
AI is really good at creating simple web applications that can show you data, that can write to a database. It's very simple. For the Nurture Boss use case, we wanted to remove all the friction of looking at data. What you see here is just some screenshots of what the application that they created looks like. It's just, "Okay, they have the different channels, voice, email, text. They have the different threads, they hid the system prompt by default." Little quality of life improvements. Then, they actually have this axial coding part here where you can see in red the count of different errors. They automated that part in a nice way and they created this within a few hours. It's really hard to have a one size fits all thing for looking at your data. You don't have to go here immediately, but something to think about is make it as easy as possible because, again, it's the most powerful activity that you can engage in. It's the highest ROI activity you can engage in. With AI, yeah, just remove all the friction.

**Lenny Rachitsky** (01:29:56):
That's amazing. Again, I think that ROI piece is so important. We haven't even touched on this enough. The goal here is to make your product better, which will make your business more successful. This isn't just a little exercise to catch bugs and things like that. This is the way to make AI products better because the experience is how users interact with your AI.

**Hamel Husain** (01:30:16):
Absolutely. If any, we teach our students, "Hey, when you're doing these evals, if you see something that's wrong, just go fix it." The whole point is not to have evals, a beautiful eval suite, where you can point at it, edit it and say, Oh, look at my evals." No, just fix your application, make it better. If it's obvious, do it. Totally agree with you.

**Lenny Rachitsky** (01:30:38):
Amazing. A question I didn't ask, but this is I think something people are thinking about. How long do you spend on this? How long does it usually take to do? The first time

**Shreya Shankar** (01:30:45):
I can answer for myself for applications that I work with. Usually, I'll spend three to four days really working with whoever to do initial rounds of error analysis. A lot of labeling, feel like we're in a good place to create the spreadsheet that Hamel had and everyone's on-board and convinced, and even a few LLM judge evaluators. But this is one-time cost. Once I figured out how to integrate that in unit tests, or I have a script that automatically runs it on samples and I'll create a Cron Job to just do this every week. I would say it's like, I don't know, I find myself probably spending more time looking at data because I'm just data hungry like that. I'm so curious.

**Shreya Shankar** (01:31:23):
I'm like, I've gained so much from this process and it's put me above and beyond in any of my collaborations with folks, so I want to keep doing it, but I don't have to. I would say maybe 30 minutes a week after that.

**Lenny Rachitsky** (01:31:41):
It's a week essentially, a week essentially upfront, and then 30 minutes to keep improving on adding to your suite?

**Shreya Shankar** (01:31:47):
Yeah, it's really not that much time. I think people just get overwhelmed by how much time they spend up front and then thinking that they have to keep doing this all the time.

**Lenny Rachitsky** (01:31:56):
Amazing. Is there anything else that you wanted to share or leave listeners with? Anything else you wanted to double down as a point before we get to a very exciting lightning round?

**Hamel Husain** (01:32:06):
I would say this process is a lot of fun, actually. It's like, okay, you're looking at data. Oh, it sounds like you're annotating things. Okay. Actually, I was just looking at a client's data yesterday, the same exact process. It's a application that sends emails, recruiting emails to try to get candidates to apply for a job. We decided to start looking at traces. We jumped right into it. "Hey, let's look at your traces." We looked at a trace, the first thing I saw was this email that is worded, "Given your background, blah, blah, blah, blah, blah." I asked the person right away, and this is where putting your product hat on and just being critical, and this is where the fun part is.

**Hamel Husain** (01:32:55):
I said, "You know what? I hate this email. Do you like the email, given your background?" When I receive a message given your background, comma, I just delete that. I'm like, "What is this, given your background with machine learning and blah blah?" I'm like, "This is a generic thing." I asked the person like, "Hey, can we do better than this? This sounds like generic recruiting." They're like, "Oh, yeah, maybe." Because they were proud of it, they're like, "The AI is doing the right thing, it's sending this email with the right information, with the right link, with the right name, everything." That's where the fun part is, is put your product hat on and get into, is this really good?

**Lenny Rachitsky** (01:33:38):
Something I want to make sure we cover before we get to a very exciting lightning round is, this is just scratching the surface of all the things you need to know to do this well. I think this is the best primer I've ever seen on how to do this well.

**Shreya Shankar** (01:33:51):
Nice.

**Lenny Rachitsky** (01:33:51):
But I think we did it. But you guys teach a course that goes much, much deeper for people that really want to get good at this and take this really seriously. Share what else you teach in the course that we didn't cover, and what else you get as a student being part of the course you teach at Maven.

**Shreya Shankar** (01:34:07):
Yeah, I can talk about the syllabus a little bit, and then Hamel can talk about all the perks. We go through a lifecycle of error analysis, then automated evaluators, then how to improve your application, how do you create that flywheel for yourself? We also have a few special topics that we find pretty much no one has ever heard of or taught before, which is exciting. One is, how do you build your own interfaces for error analysis? We go through actual interfaces that we've built and we also live code them on the spot for new data. We show how we use Claude code cursor, whatever we're feeling in the moment that day to build these interfaces.

**Shreya Shankar** (01:34:49):
We also talk about broadly cost-optimization as well. A couple of people that I've worked with, they get to a point where their evals are very good, their product is very good, but it's all very expensive because they're using state-of-the-art models. How can we replace certain uses of the most expensive GPT-5, with 5-nano, 4-mini whatnot and save a lot of money, but still maintain the same quality? We also give some tips for that. Hamel, you're on. We also have many perks.

**Lenny Rachitsky** (01:35:23):
Yeah. Talk about the perks.

**Hamel Husain** (01:35:24):
Okay, the perks. My favorite perk is there's 160 page book that's meticulously written, that we've created, that walks through the entire process in detail of how to do evals that supplement the course. You don't have to sit there and take all these notes. We've done all the hard work for you and we have documented it in detail and organize things. That is really useful. Another really interesting thing, and something that I got the idea from you, Lenny, is, okay, this is an AI course. Education shouldn't be this thing where you are only watching lectures and doing homework assignments. Students should have access to an AI that also helps them. What we have done is we've, just like there's the LennyBot that you have.

**Lenny Rachitsky** (01:36:19):
Dot com.

**Hamel Husain** (01:36:20):
Yeah, lennybot.com, we have made the same thing with the same software that you're using, and we have put everything we've ever said about evals into that. Every single lesson, every office hours, every Discord chat, any blogs, papers, anything that we've ever said publicly and within our course, we've put it in there. We've tested it with a bunch of students and they've said it's helpful. We're giving all students 10 months free unlimited access to that alongside the course.

**Lenny Rachitsky** (01:36:56):
Amazing. Then, you'll charge for that later down the road?

**Hamel Husain** (01:37:01):
I have no idea. I just take one month at a time. I don't know where we're going with that.

**Lenny Rachitsky** (01:37:04):
Eight months and then we'll have to figure it out. I was thinking this whole interview should have just been our bots talking to each other.

**Shreya Shankar** (01:37:09):
That's amazing. I would watch that, only for 10 minutes then I don't know what they're talking about.

**Lenny Rachitsky** (01:37:14):
Yeah, maybe 30 seconds. Do you guys train it on the voice mode, by the way? That's my favorite feature of Delphi's product. If not, you should do that.

**Hamel Husain** (01:37:22):
Oh, I think, I can't remember, I should look at it.

**Lenny Rachitsky** (01:37:26):
You definitely should. Now that we have this podcast episode, you could use this content to train it. It's 11Labs powered. It's so good. Okay, so how do they get to... I guess that's okay. They get to that once they become, enter your course.

**Shreya Shankar** (01:37:38):
Yeah, sign up for the course and then you'll get a bunch of emails. Everything will be clear, hopefully.

**Lenny Rachitsky** (01:37:43):
Amazing. Okay.

**Shreya Shankar** (01:37:44):
We also have a Discord of all the students who have ever taken the class. That Discord is so active. I can't go on vacation without getting notified on the plane.

**Lenny Rachitsky** (01:37:55):
Bittersweet, bittersweet. Incredible. Okay. With that, we've reached our very exciting lightning round. I've got five questions for you. Are you ready?

**Shreya Shankar** (01:38:04):
Yes. Let's go.

**Lenny Rachitsky** (01:38:05):
Let's do it. Okay. I'm going to bounce between you two. Share something if you want. You can pass if you want. First question, Shreya, what are two or three books that you find yourself recommending most to other people?

**Shreya Shankar** (01:38:17):
I like to recommend a fiction book because life is about more than evals. Recently, I read Pachinko by Min Jin Lee. A really great book. Then, I also am currently reading Apple in China, which the name of the author is slipping my mind, but this is more of an exposition, written by a journalist on how Apple did a lot of manufacturing processes in Asia over the last couple, several decades. Very eye-opening.

**Lenny Rachitsky** (01:38:49):
Amazing. Hamel.

**Hamel Husain** (01:38:52):
Yeah, I have them right here. I'm a nerd. Okay, so I'm not as cool as Shreya is. I actually have textbooks, which are my favorite. This one is a very classic one, Machine Learning by Mitchell. Now, it's theoretical, but the thing I like about it is it really drives home the fact that Occam's razor is prevalent not only in science, but also in machine learning and AI. A lot of times the simplest, and also engineering, so a lot of times the simpler approach generalizes better. That's the thing I internalize deeply from that book. I also really like this one. Another textbook. I told you I'm a nerd. This is also a very old one, and this is Norvig algorithms. I really like it because it's just human ingenuity and it's lots of clever useful things in computing.

**Shreya Shankar** (01:39:49):
They're down the street, him and Berkeley.

**Lenny Rachitsky** (01:39:54):
The people that did that research?

**Shreya Shankar** (01:39:57):
Yeah, textbook authors.

**Lenny Rachitsky** (01:39:58):
Super cool. Oh, man, nerds, I love it. Okay, next question. Favorite recent movie or TV show? I'll jump to Hamel first.

**Hamel Husain** (01:40:06):
Okay, so I'm a dad of two parents. I have two parents. Sorry, two kids. Yeah, I'm a dad of two kids, and I don't really get the time to watch any TV or movies, so I watch whatever my kids are watching. I've watched Frozen three times in the last week.

**Lenny Rachitsky** (01:40:25):
Only three? Oh, okay. In the last week. Okay.

**Hamel Husain** (01:40:30):
That's my life.

**Lenny Rachitsky** (01:40:30):
Great, Hamel. Frozen. I love it. Okay, Shreya.

**Shreya Shankar** (01:40:32):
Yeah, I don't have kids, so I can give all these amazing answers. Actually, so my husband and I have been watching The Wire recently. We never actually saw it growing up, so we started watching it and it's great.

**Lenny Rachitsky** (01:40:46):
I feel like everyone goes through that. Eventually in their life they decide, I will watch The Wire.

**Shreya Shankar** (01:40:51):
I know, so we are in that right now.

**Lenny Rachitsky** (01:40:51):
It's like a year of your life. It's great. It's such a great show. Oh, man. But it's so many episodes and everyone's an hour long.

**Shreya Shankar** (01:40:58):
I know. I know.

**Lenny Rachitsky** (01:40:58):
It's such a commitment.

**Shreya Shankar** (01:40:59):
We get through two or three a week, so we're very slow.

**Lenny Rachitsky** (01:41:03):
Worth it. Okay, next question. Do you have a favorite product you've recently discovered that you really love? We'll start with Shreya.

**Shreya Shankar** (01:41:10):
Yeah. I really like using Cursor, honestly. Now, Claude Code. I'll say why. I'm a researcher more so than anything else. I write papers, I write code, I build systems, everything, and I find that a tool... I'm so bullish on AI assisted coding because I have to wear a lot of hats all the time. Now, I can be more ambitious with the things that I build and write papers about, so I'm super excited about those. Cursor was my entry point into this, but I'm starting to find myself always trying to keep up with all these AI assisted coding tools.

**Lenny Rachitsky** (01:41:48):
Hamel?

**Hamel Husain** (01:41:49):
Yeah, I really like Claude Code and I like it because I feel like the UX is outstanding. There's a lot of love that went into that. It's just really impressive as a terminal application that is that nice.

**Lenny Rachitsky** (01:42:04):
Ironic that you two both love Claude Code when it's just built on vibes.

**Shreya Shankar** (01:42:09):
I think it's false. It's not just built on vibes.

**Lenny Rachitsky** (01:42:13):
There we go. Okay, two more questions. Hamel, do you have a favorite life motto that you find yourself using in coming back to in work or in life?

**Hamel Husain** (01:42:21):
Keep learning in. Think like a beginner.

**Lenny Rachitsky** (01:42:26):
Beautiful. Shreya?

**Shreya Shankar** (01:42:27):
I like that. For me, it's to always try to think about the other side's argument. I find myself sometimes just encountering arguments on the internet, like this race to eval debates and really think, "Okay, put myself in their shoes. There's probably a generous take, generous interpretation." I think we're all much stronger together than if we start picking fights. My vision for evals is not that Hamel and I become billionaires. It is that everyone can build AI products, and we're all on the same page

**Lenny Rachitsky** (01:42:59):
Slash everyone becomes billionaires.

**Shreya Shankar** (01:43:02):
Yes.

**Lenny Rachitsky** (01:43:04):
Amazing. Final question. When I have two guests on, I always like to ask this question and I'll start with Hamel. What's something about Shreya that you like most? What do you like most about Shreya? I'm going to ask her the same question in reverse.

**Hamel Husain** (01:43:18):
Yeah. Shreya is one of the wisest people that I know, especially for being so young relative to me. I feel like she's much wiser than I am, honestly, seriously. She's very grounded and has a very even perspective on things. I'm just really impressed by that all the time.

**Lenny Rachitsky** (01:43:18):
Shreya?

**Shreya Shankar** (01:43:43):
Yeah. My favorite thing about Hamel is his energy. I don't know anybody who consistently maintains momentum and energy like Hamel does. I often think that I would start carrying much less about evals, if not for Hamel. Everyone needs a Hamel in their life, for sure.

**Lenny Rachitsky** (01:44:06):
Well, we all have a Hamel in our life now. This was incredible. This was everything I'd hoped it'd be. I feel like this is the most interesting in-depth consumable primer on evals that I've ever seen. I'm really thankful you two made time for this. Two final questions. Where can folks find you? Where can they find the course and how can listeners be useful to you? I'll start with Shreya.

**Shreya Shankar** (01:44:29):
Yeah, you can reach me via email. It's on my website. If you Google my name, that is the easiest way to get to my website. You can find the course if you Google AI Evals for engineers and product managers, or just AI Evals course, you'll find it. We'll send some links hopefully after this, so it's easy. How to be helpful? Two things always for me. One is ask me questions when you have them. I'll try to get to the respond as soon as I can. The other one is tell us your successes. One of the things that keeps us going is somebody tells us what they implemented or what they did, a real case study. Hamel and I gets so excited from these and it really keeps us going, so please share.

**Hamel Husain** (01:45:16):
Yeah, it's pretty easy to find me. My website is Hamel.dev. I'll give you the link. You can find me on social media, LinkedIn, Twitter. The thing that's most helpful is to echo what Shreya said, we would be delighted if we are not the only people teaching evals. We would love other people teach evals. Any kind of blog posts, writing, especially that as you go through this and learn this that you want to share, we would be delighted to help re-share that or amplify that.

**Lenny Rachitsky** (01:45:54):
Amazing. Very generous. Thank you two, so much for being here. I really appreciate it, and you guys have a lot going on, so thank you.

**Shreya Shankar** (01:46:01):
Thanks, Lenny, for having us and for all the compliments.

**Lenny Rachitsky** (01:46:05):
My pleasure. Bye everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lennyspodcast.com. See you in the next episode.

---

## [18/18] The secret to better AI prototypes: Why Tinder’s CPO starts with JSON, not design | Ravi Mehta (product advisor, previously EIR at Reforge)

**Ravi Mehta** (00:00:00):
The framework I like to use with product leaders that I'm coaching is to think about a matrix. Your ideal goal is to lead in a scalable way, which means you feel really confident about the direction of your team and your team has the autonomy to move in that direction. There's another really effective way of leading, which is selective micromanagement, which if you don't feel confident in the direction that your team is moving, the right answer is not to be hands-off and to let them go in that wrong direction. The right answer is to micromanage, but do it in a very tactical and a very temporary way so that you can help them understand what is the right direction moving forward so that you can then pull back.
**Lenny** (00:00:46):
Welcome to Lenny's Podcast. I'm Lenny and my goal here is to help you get better at the craft of building and growing products. I interview world-class product leaders and growth experts to learn from their hard one experiences building and scaling today's most successful companies.
**Ravi Mehta** (00:04:00):
Yeah, thank you for having me. I'm excited to be here.
**Lenny** (00:04:02):
So I've been a huge fan of your writing for a long time. And this may sound a little weird, but I just feel like not enough people know about you and I'm just excited to learn from you and also just to share your wisdom with more people.
**Ravi Mehta** (00:04:14):
Oh, thank you. That means a lot. I've been a fan of all of your work as well. I've been following the podcast. It's been great to see how it's evolved over the years.
**Lenny** (00:04:20):
Awesome, man. I really appreciate that. It continues evolving. So just to start with a little bit of your background, can you just take a minute to share just like an overview of your career arc and touch on some of the wonderful things you've done and then just talk a little bit about what you're up to these days.
**Ravi Mehta** (00:04:36):
Yeah, I've been in the tech industry for a long time, so I will date myself. I started in the mid-90's. My dad was at American Express and he had just done a big buy of computers, one of their first big installations of computers, and he brought home an Apple TC computer, and back then there wasn't much to do on it other than to learn to code. So I started coding really young. I was nine, 10 years old and really just fell in love with technology and that's persisted with me today.
**Ravi Mehta** (00:05:03):
I started a game company in high school. I did that full time and part time in college, so I dropped out for a little bit during college. Went back and finished up my degree. And then my first role out of school was Microsoft, and so I joined Microsoft at a really interesting time when they were making a pretty significant investment in games. And so I joined as one of the first few people on the Xbox Live team. Really focused on thinking about how does a company that's building its future on the internet think about where gaming is going. And that was really different than how other companies in the space like Nintendo or Sony were thinking about gaming.
**Ravi Mehta** (00:05:36):
Spent about six years there, worked on stuff on the platform side, on the content side. It was a really great experience, but I knew I wanted to go earlier stage. So straight after Microsoft, I went to business school, dabbled a little bit in management consulting, but decided I really wanted to build things and so I went back into an early stage startup right after business school. I started as employee number one at a FinTech startup. Shortly after that I joined Brian Balfour, who's the CEO of Reforge at his first startup.
**Ravi Mehta** (00:06:03):
And my most recent few roles have been product leadership roles at Tripadvisor where I was head of the consumer product team, product leadership role at Facebook, and then I was the chief product officer at Tinder. And for the last couple of years I've gone back into the startup side of things and happy to talk about that some more.
**Lenny** (00:06:18):
Yeah, let's talk a little bit about what you're doing now and just to kind of put that out there and then we'll keep going.
**Ravi Mehta** (00:06:22):
That sounds good. So I spent about 10 years or so at bigger companies working with large product management teams and large engineering teams. I find that work incredibly fulfilling in terms of the ability to impact people at scale, but I was also really missing the idea of building something new and really thinking about where things are going and not having to solve for some of the legacy constraints that large businesses have to solve for. So I decided to leave Tinder and at that point started to explore what I wanted to do next.
**Ravi Mehta** (00:06:48):
I spent about 18 months working with Reforge as an entrepreneur in residence or an executive in residence with Reforge, helping them build and launch the product leadership program and helping them launch the product strategy program. And during the process of doing that, I had conversations with dozens of people that were at the middle point of their career and found a really interesting common challenge in that there's lots of ways to learn new skills. Now there's great podcasts and blogs. There's great cohort-based courses like Reforge. But one of the things I found incredibly helpful in my career that really helped me level up was one-on-one coaching. There was nothing that could really replace the opportunity to have a conversation with someone who had the ability to ask the right questions, had the ability to help you see around corners, do the experiences that they had had. And coaching had just not gotten any more accessible over the years.
**Ravi Mehta** (00:07:39):
And so about 18 months ago, I decided to start Outpace, which is a company focused on making elite expert-driven coaching available to everyone. And we're using a combination of really focusing on the product, using a lot of systems and content to structure the coaching process. We're also using AI to make coaches more efficient with the goal of making expertise-driven coaching a lot more accessible for folks.
**Lenny** (00:08:04):
Awesome. So a first area I wanted to spend a little time on is you talked about your career arc, you're CPO at Tinder, product director at Facebook, VP at Tripadvisor, and now you started a company and you've started companies in the past. A lot of PMs listening to this have a hope that they will start a company someday and they're probably working at a company like a big tech company somewhere or not, or they're starting a company right now and they're kind of in the process of starting a company. And I'm curious what you found to be the biggest differences between being a product leader at a bigger company versus a startup, especially your own startup, and especially what are maybe the biggest surprises you've felt from moving and making that transition?
**Ravi Mehta** (00:08:43):
There's been a couple of really interesting mind shifts I've had to go through over the last 18 months as I moved from a product leadership role to a founder role. The first one is really thinking differently about speed. I think there's this common, I would say it's a misconception that startups are faster than larger companies. And what I found initially is actually things felt slower when I started my own company because I didn't have as many engineers to work with. I didn't have a team built around things. We didn't have momentum around existing users to be able to research and target.
**Ravi Mehta** (00:09:15):
And what I realized as I've kind of been through the journey over the last 18 months is that the speed that startups have is not really about velocity. Bigger companies can always get more done, they can always spend more, they can always move with a higher degree of velocity than smaller companies. The advantage a smaller company has really is in latency. You can have an idea one day, you can test it the next day, and as a result you can have this really short cycle time between an assumption or a hypothesis and being able to validate that hypothesis. And that's just not true at larger companies where there's a lot more momentum.
**Ravi Mehta** (00:09:49):
The analogy I like to use, it's like driving a car. If a car is going really, really fast, it can't turn as quickly, the turning radius is lower. And so startups have a really tight turning radius and bigger companies have a really high rate of velocity. And so that was one of the things that for me took some adjustment in terms of thinking about how to boil down what would've been a pretty big ambitious plan at a larger company into something that has much smaller pieces and where you can iterate towards things and get data every day or every couple of weeks rather than have a bigger project that might take a quarter long to execute.
**Lenny** (00:10:23):
Just so folks understand what you mean by that, this interesting difference between speed and latency. So what exactly is the difference? Latency is basically how fast you can make decisions and change courses. Is that how you think about it?
**Ravi Mehta** (00:10:34):
I think about velocity is sort of the quantity of work and latency is how quickly you can go from an idea to actually being able to test that idea and learn whether or not that idea was the right one.
**Lenny** (00:10:46):
Cool.
**Ravi Mehta** (00:10:47):
One of the questions to test out latency that I likes to ask PMs is if there's a really simple change that you want to make to a product, like being able to change a button so you can test two different texts on a particular button, how long does it take to go from we think that this change is worth making to actually getting the results of whether or not it was the right change?
**Lenny** (00:11:07):
Got it. Cool.
**Ravi Mehta** (00:11:08):
The second thing is really thinking differently about how to make decisions. I think a lot of really effective companies today that have large audiences get to rely on an experimental way of making decisions. So you throw things out there, you run an experiment, you get to see what's statistically significant, and based on that, that provides a really nice way to learn about what users want and iterate towards an optimal product.
**Ravi Mehta** (00:11:33):
At a startup, you can't do that. You just don't have those users to test with. And I think a lot of startups make the mistake of trying to use an experimental approach too early where it just takes either way too long to get statistically significant results, which reduces that latency, or those results aren't as valid because you have to use a much smaller sample set. And so I've had to shift my mindset from an experimental-oriented approach to making decisions to much more of a conviction-oriented approach.
**Ravi Mehta** (00:12:02):
And I've often found myself asking the question of like, do we just have enough data to have informed conviction and we should move forward and stop digging, move forward in a particular direction, and then see whether or not that turns out to be the right one? Because too often in a startup you can spend a lot of time in paralysis around analyzing market research or going through all of the different things you could do strategically, thinking about all the different potential variants that you could build, all the different pricing strategies, whereas instead, a startup just makes sense to kind of get to a point where you have conviction, execute on that, and then move on to whether or not that felt like the right thing, in which case you can double down, or that was the wrong thing, in which case you can shift direction and do that pretty quickly.
**Lenny** (00:12:44):
Awesome. What else?
**Ravi Mehta** (00:12:45):
One of the things that I've found really surprising is the networks are pretty different. So I've gotten a chance to work with an incredible amount of great people over the years and when I was starting a company, I was excited to reach out to people, tell them what I was doing, and there were a number of people that I'd worked with at larger companies that I was potentially excited about working with.
**Ravi Mehta** (00:13:04):
And what I found was that the people sort of really build their lifestyles and their careers around a particular stage. And there are some people that like to move between stages, but the majority of people don't. A lot of people that are at larger companies, they like the benefits that come with that. They like the types of problems that they're working on, yet there's a whole other community of people who love to work earlier stage. It could be founders. It's also freelancers who like to help to build startups. It's investors and angels. And so that's been a really interesting part of the journey is meeting new people, getting to know those networks and starting to build out a group of people that are as passionate about that earlier stage as I am.
**Lenny** (00:13:45):
Got it. So you're finding that the network you may have had from say Tinder or Facebook aren't like the entrepreneurial type that maybe aren't... They're not necessarily as useful as hiring potential and things like that? Is that what you're finding?
**Ravi Mehta** (00:13:58):
Yeah. I think a lot of times people that are at larger companies, they're used to working in a particular way. They've mastered their craft. In terms of how they think about the next thing in their career, they really want to go deeper into that craft. And people who like the earlier stage or much more generalist, they're fine with kind of moving back in time. You're not going to find a lot of senior engineering leaders or senior product leaders that want to write codes and specs at big companies, but you will find those in those networks of people that are founders and that are interested in the earlier stage.
**Lenny** (00:14:30):
That's a really interesting insight that you think you're building this huge network from a big company you're working at and it may not be the network you need when you want to start a company. Do you have any other pieces of advice for a founder that's like, "Hey, I want to start a company in the future in the next few years, let's say at Facebook or Google"? Any other things you think they could be doing now to set themselves up for success?
**Ravi Mehta** (00:14:49):
I think it's important to plug into an early stage network as soon as possible. There's a bunch of different ways to do that today. There's communities that are focused on founder dating, there's communities focused on just being a place where founders can spend time. There's a great community of people in the indie hacker community and a few other related communities. And so I think it's important to connect with folks that are builders that are excited about entrepreneurship both on the development side and the operations side, as well as on the investment side. Connecting with angels and investors who are seeing what's happening within earlier stage companies. What are the things that are top of mind? What are the technology trends that people are really taking advantage of?
**Ravi Mehta** (00:15:31):
Another really interesting, I think, difference is the way that you market and grow for an early stage company is very different than how you might market or grow for a later stage company where you have much larger budgets. And so the people that might be great at building out marketing campaigns at a larger company are going to be very different than the people who are more sort of earlier stage. More hackers that are looking at. There's these really interesting new channels of distribution that you can take advantage of or interesting techniques on TikTok or interesting SEO techniques that you can take advantage of. So it's really two different networks as well as two different bases of knowledge. And so I think it's important for people that want to eventually found something to work on fostering that network so that you can connect into that community at the moment that you're ready to make that leap.
**Lenny** (00:16:15):
Are there any other specific communities that come to mind as places that either you found valuable or that you think are worth checking on for folks that are like, "Cool, indie hackers. I'll check that out"? Is there anything else that comes to mind as a really good place to spend some time right now?
**Ravi Mehta** (00:16:29):
Yeah, I think two of the best communities are the indie hacker community. What I really like about that is it's a lot of people who are thinking about how do I build something solo? And that's really different from being at a larger company. If you can think about a spectrum of you have a larger company, somewhere in the middle, you have VC-backed startups where you can take some of the ways of thinking about things that you learn at a bigger company and apply it because you have the ability to invest a significant amount of resources. And then at the opposite side, there's one person who's got a dream. They want to start something. They're trying to figure out how to do everything themselves. They're entirely generalists in terms of being both builders and sellers, as well as figuring out all the logistics. So I like the indie hacker community.
**Ravi Mehta** (00:17:08):
Another really good community is Everything Marketplaces. Mike, the founder of that community, has just done a fantastic job of bringing together a set of founders. He's specifically focused on marketplace businesses, which have some unique dynamics to them, especially in the very early stage. But it's a great example of even if you're not into marketplaces, I think it's worth looking at what they're creating, the events that they're running, and the people that are involved. They've just done a great job of curating that whole experience to provide a really great foundation for founders.
**Lenny** (00:17:38):
I'm also a huge fan of the community. I love Mike. We're internet friends. He'll love hearing this. I think the site is everythingmarketplaces.com to check out the community. And if it's not, you can just Google it. We'll also put in the show notes.
**Lenny** (00:17:49):
So Reforge, you brought it up a couple times, and this kind of gets to what I want to spend the meat of our conversation around. You built the Reforge product leadership program, the product strategy program. So those are two areas you spend a lot of time thinking about product leadership, product strategy.
**Lenny** (00:18:05):
So starting with product strategy. Every PM, every founder, every leader would say that they want to get better at strategy. I guarantee if I ask every PM, do you want to get better product strategy? A hundred percent would say absolutely. But it's this very mushy, vague, general idea of strategy. I'm going to get better at strategy. I'm going to be better. I'm going to be more strategic. You have this really cool kind of framework, mental model that you call the product strategy stack. And so I want to spend a little time on just talking about what is this concept and how does it help you think about strategy, mission, vision, all these things and how these things play together. So let's just start with what is the product strategy stack?
**Ravi Mehta** (00:18:43):
The goal of the product strategy stack is to help people take a set of terms that are normally conflated together, like goals, roadmap, strategy, and separate them into really clearly defined parts. And the reason I first started using this concept is I would often have PMs come to me and they wouldn't know whether to decide between doing A or B. So it might be that there's two features, they're roughly the same opportunity size, and they wouldn't know whether or not they should execute the first feature or execute the second feature.
**Ravi Mehta** (00:19:16):
And more often than not, when I talked to teams and helped to debug that issue, what it came down to was that there wasn't a deep enough understanding of what the strategy is. So what is the framework that should actually inform that prioritization? And so oftentimes I was seeing difficulty prioritizing as well as tactical issues surface in the day-to-day and be able to be tracked back to pretty fundamental gaps in terms of an individual PM's understanding of strategy. And oftentimes those gaps were not just because the person might not understand the strategy, it may also be because the strategy hasn't been completely defined.
**Ravi Mehta** (00:19:55):
And so the private strategy stack is a system that helps people understand what framework they're using in order to make decisions and what's going to drive value for the business. So the top of the stack is the company mission and the company mission is the change the company wants to bring to the world. It's really a qualitative aspirational statement of what is the company's purpose. And in some cases it might not be a company, it might be a particular team within a company or it might be a particular subsidiary depending on the environment you're in. But it's basically the overarching mission that helps to guide the process of moving forward.
**Ravi Mehta** (00:20:33):
The second thing is strategy. So whereas a mission is aspirational, strategy is rigorously logical. The strategy is the logical plan that your company's going to use to bring that mission into being. And so it's got to be very specific, it's got to be very rigorous, and it's basically the approach of the plan that the company will use to make progress on achieving its mission. And so the mission and the strategy at the company level really define what is the company trying to accomplish. And so the next level of the strategy stack is the product strategy. And the product strategy is the connective tissue between what is the company trying to accomplish and what are the day-to-day things that the product team is doing. And so underneath the product strategy, the product strategy informs a roadmap and the roadmap ultimately informs the goals.
**Ravi Mehta** (00:21:20):
And so those five pieces, the company mission, the company strategy, the product strategy, the product roadmap, and the product goals all work together as a system where if a PM is looking to define strategy, they can work top to bottom, and if they're looking to debug strategy, they can actually work bottom to top. And so if you're having trouble meeting your goals, it might be because the roadmap isn't set up so that it can help move those goals forward. If the roadmap isn't right, it might be because the product strategy hasn't been really clearly articulated. If the product strategy isn't right, it might be because the team doesn't understand deeply enough what the company's strategy is, how the product fits into it, and ultimately the company's mission that it's trying to make progress on.
**Lenny** (00:22:04):
Super cool. I have a bunch of questions. One is, interestingly, vision doesn't come up in the stack. Does it roll...
**Ravi Mehta** (00:22:11):
Yeah.
**Lenny** (00:22:11):
... into one of these? Or do you just no vision necessary?
**Ravi Mehta** (00:22:14):
I think about vision as part of mission.
**Lenny** (00:22:14):
Cool. That's what I thought.
**Ravi Mehta** (00:22:17):
I always get confused about what the difference is between vision and mission. And so when I was originally working on this, there was a version of this that had the mission and the vision together. There were versions that kept it separate. Often what I've heard of as the distinction is the vision is sort of the vision that the company sees for the future, and then the mission is the mission that the company has in light of that vision. And I think you can really bring those two together and you can both describe that world and the role that the company plays in a single statement. And that's usually enough to make progress and help to start to define the strategy.
**Lenny** (00:22:55):
Cool.
**Ravi Mehta** (00:22:55):
But I know you've written about this as well and you've put a spotlight on vision. So I'd be curious as to how you see the mission and the vision playing together.
**Lenny** (00:23:04):
Yeah. I think the most important thing is people just get stuck on these and try to define them and make them perfect. And I think the most important thing is just don't overthink it. Just put something that sounds right and people are excited about it in a [inaudible 00:23:16]. That's the most important thing. The way I think about it is mission is just like what are you trying to achieve in the world? And then the vision is what is the world look like once you've achieved it? What is the vision of the future? And the mission is what are you trying to do in this future? So that's the way I think about it. What are you trying to do? What does it look like? But I think keeping it as one thing is great. Like whatever works. There's no one way to do it.
**Lenny** (00:23:38):
I also know that you're a big believer in the vision when you think about a vision and define a vision, making it very visual versus just like a doc. Can you talk about that?
**Ravi Mehta** (00:23:47):
This framework originally started when I was at Tripadvisor and we had to develop a plan for what we wanted the strategy to be for trip planning. This was going to be a really big new feature for the company and for product. Trip planning is one of these intractable problems or been a number of startups that started as trip planning startups and nobody had really nailed it. Google at the time had a trip planning app that had some interesting elements to it, but it wasn't really clear that they were nailing it. And so we knew that there was both a really valuable problem to solve here, but also a really difficult problem. And we wanted to take an end-to-end approach to solving for this where rather than just kind of working bottoms up and getting to things experimentally where we might not actually ladder up to a clear product strategy, we had said we wanted to work top down, define what do we want to achieve, how we're going to achieve it, and what are the incremental steps we're going to use to get there.
**Ravi Mehta** (00:24:41):
And one of the things that we said with stake that we put in the ground was the strategy doc wouldn't be complete without wireframes. This was the first time that we were doing that in the context of strategy. And the thing that we were really trying to solve for is the fact that oftentimes when you talk about strategy in words alone, everyone takes away a different interpretation of that strategy, whereas when you actually can show people wireframes of what the product will look like when that strategy is implemented, it creates much more alignment.
**Ravi Mehta** (00:25:14):
And so the analogy I like to use, it's a little bit like working with an architect. You would never work with an architect that didn't provide you a blueprint of the house that they want to build for you because being able to describe a house in words alone is not enough. Everyone will come away from that with sort of a different interpretation of what is needed. But once you can see the blueprint, and the blueprint doesn't need to be high fidelity, it's a conceptual framework that shows you how things are laid out, it helps you understand how the pieces are going to come together. And most products are ultimately rendered in terms of visuals. They're pixels on a screen. And so it's important for you to understand how are those pixels going to be organized.
**Ravi Mehta** (00:25:56):
I think an interesting litmus test question for this is, and a lot of mobile apps can only have four or five things on their nav bar. What are the four or five things? If you just describe your strategy in words, people might come up with one nav bar that's completely different than another nav bar. And as a result, you then find that the moment that you're implementing your mobile app, that there's completely different perceptions of what's valuable to the company and how the functionality should be organized. And so the process of setting your strategy and then defining it really crisply in wireframes helps to get really specific and concrete about what it is that you're building, what's going to fulfill the strategy, and what are some of the trade-offs that you need to make in order to bring that into fruition because there's always going to be a limited number of pixels on the screen.
**Lenny** (00:26:43):
Imagine PMs listening to this might feel. "Okay, yes, I would love wireframes in all of my vision documents, full fidelity designs of everything I want to do. Here's what I'm doing." I imagine they often don't have a designer available, they don't have lists together for some review that's coming up. What do you suggest to these folks? Is it like as a PM, just sketch it out briefly is something better than nothing? What do you suggest for when there's like just not anyone to help them do this well?
**Ravi Mehta** (00:27:10):
I think it's great if you're able to work with a designer, but I also think it's really important for PMs to understand design, to understand UX and UI. You can always just sketch things on paper if you don't have design skills. I've also, time and time again throughout my career, I've gone back to Balsamiq, which is a really good wireframing tool. It's been around for a while. It's incredibly fast to work with, and often in an afternoon you can create a set of very high level conceptual wireframes that you can put in front of people that will give them a much clearer understanding of what it is you're trying to build than if you were just to share them with them a spec that is words alone.
**Ravi Mehta** (00:27:51):
So I would suggest learn how to sketch, learn Balsamiq. Having that ability to think at a conceptual level about how UI and UX works is I think a critical part of being a product manager. And if it's a skill that you don't have today, there's great resources to be able to work on that skill. And I think it'll make you feel a lot more empowered as a product manager as well if you don't need to feel like you've got to depend on a designer to help you visually think through your product each and every time.
**Lenny** (00:28:18):
Cool. No excuses PMs.
**Ravi Mehta** (00:28:19):
Exactly.
**Lenny** (00:28:20):
Okay. So coming back to the product strategy stack, can you share an example of a company you worked at and how that stack kind of all played out? Like an example, and just to come back to its mission strategy, product strategy, roadmap, goals. And while you're talking, I'm going to try something new. I'm going to pull up a window that shows your visual of this thing and it'll show up I think in my screen. Look at that. And so if you're on YouTube. Or you can actually watch these videos on Spotify now in case yet people that are listening have notice...
**Ravi Mehta** (00:28:47):
Oh, cool.
**Lenny** (00:28:48):
... [inaudible 00:28:48] new feature they just unlocked for my podcast or these videos are on Spotify. So cool opportunity to check it out on Spotify or YouTube. But let me come back to you with the question. Basically, is there an example you could share maybe from Tinder or Facebook or something like that of the product strategy stack in action?
**Ravi Mehta** (00:29:05):
So the article itself has an example, which I won't go through now, of Slack versus Discord. I think that's a really interesting example because the products are so similar and yet the company strategies and the missions are so different. They're serving incredibly different audiences despite the fact that many of the items on those teams roadmaps are likely the same. Threading, reactions, channels, video chat, things of that sort. I think a really interesting example from my past life is comparing Tinder versus Hinge.
**Lenny** (00:29:32):
What's that?
**Ravi Mehta** (00:29:33):
Both of them are dating apps, but they have missions that are really different. So Hinge's mission is almost created in response to Tinder. Hinge's mission is designed to be deleted. This is something that is prevalent throughout all of the marketing, which is, come to our app, we know that if our app works for you, you're going to find someone, you're going to kick off a long-term relationship and you're going to delete our app. And we consider that a success, versus Tinder's mission is really to make single life more fun. Tinder's mission is to be an app that's on people's phone whenever they're single and often throughout their 20s and into their early 30s. And so those missions are really different. One is a temporary use case, the other is a continuous use case. And so despite the fact that they're serving the same underlying use case, which is to help people meet each other, they have very different missions.
**Ravi Mehta** (00:30:24):
The company strategies are also pretty different. They have some similarity around how the apps are monetized. Both apps are freemium. You can use the product for free. And then there's particular features that are monetized. The features that are monetized share some commonality. So there's some commonality in terms of monetization model. There's a really big difference in terms of customer acquisition model. Hinge relies a lot on television ads that helps them reach the audience that is likely to use their product. Tinder relies much more on influencer marketing and event-based marketing. So there's some interesting similarities between the companies in terms of their strategies and some interesting and important differences.
**Ravi Mehta** (00:31:04):
The product strategies for Tinder and Hinge are actually really different. So Tinder was the original swipe-based dating app. It was built to be a really lightweight experience where swiping is really fast, getting into a match is really easy, chatting is really easy. And Hinge is one of the first really successful post swipe dating apps. So they deliberately did not build a product around the mechanic of swiping. Instead, they wanted people to spend more time on each other's profiles. They wanted to create more tools for those profiles. So Tinder profiles are very simple. Hinge profiles have prongs. Those prongs allow people to get to know each other. That sparks interesting conversations, that leads to deeper conversations that ultimately leads to long-term relationships. And so because of that difference in product strategy, there's some differences in product roadmap, but there's also some similarity in product roadmap. Both Tinder and Hinge made a significant investment in video chat post pandemic, knowing that people were going to spend a lot more time online before they met in person. And so as a result, they needed to enable people to talk with each other via video within the product.
[NEW_PARAGRAPH]And then the last piece is on goals. So ultimately both companies have very similar goals in terms of they measure success based on meaningful conversations. So they want people to match, they want people to chat with each other, but the specific product mechanics that enable people to get into those conversations are different. So the high level product goals are really similar. Some of the more detailed product goals are really different. And so using the strategy stack, you can get a really good feel for where strategy is informing particular decisions and when a decision should look like competitors and when a decision should be different than what one of your competitors or comparables is doing.
**Lenny** (00:32:42):
I have so many questions about Tinder. It feels it's such an interesting company and journey and product. I guess one question is you shared some examples of product features that you built because of the specific strategy. Is there any others that come to mind of just like, we built this thing and Hinge would never build it because we have such different strategies?
**Ravi Mehta** (00:33:01):
There's a counter example, which I think is really interesting, which is almost every dating app has filters and a whole set of filters. So you can filter based on occupation, income, religion, height, smoking preference. And Tinder, it's now got some ability to filter, but for the large part has resisted the urge to put those filters into place. And the reason was from a product philosophy standpoint, they wanted people to get to know each other and chat rather than to feel like Tinder's a search engine for people where you plug in a bunch of criteria, you can go into that specific filtered list, and then meet only the people that you want to meet.
**Ravi Mehta** (00:33:43):
And that really reflects in the product as well. A lot of people like using that product because they meet people that they say they never would've met otherwise. Because if they were given the ability to put their criteria in, of course they're going to put their criteria in and they're going to look at a filtered, narrower set of people. And so by keeping the product experience really lightweight, really serendipitous, they were able to create a way of meeting each other that's really different than the other dating products, which are more of those search engines for people.
**Lenny** (00:34:10):
When you think back on your time at Tinder, what's like a memory or story or wild experience that comes to mind if there's something that comes to mind?
**Ravi Mehta** (00:34:19):
So Tinder was always interesting in terms of product discovery. We did a lot of focus groups when I was there. We had people talk about their preferences around dating both one-on-one and ending groups, and those always led to really interesting conversations. One of the things that to me was the most surprising is when I was there, we noticed that there was a small set of Tinder that were spending a lot on Tinder. And so you'll often see this behavior in social games where you have users that are essentially whales, who your average ARPU might be $30 and a whale is spending $200 or $300. And so we noticed that a really significant percentage of a la carte revenue, which is microtransactions, was coming from a very small single digit percentage of users. And when we looked at how much people were spending, our hypothesis was these must be high net worth people that are looking to flaunt their wealth and they don't really care about the money.
**Lenny** (00:35:11):
What are they spending on, by the way, just to make that clear, because it's been a long time since I've tried with Tinder. What are you buying in Tinder? What are the microtransactions?
**Ravi Mehta** (00:35:18):
Yeah, so Tinder's monetization model has two pieces to it. It's got a subscription. There's a couple of different tiers to the subscription. There's a base subscription called Tinder Plus. And then there's the default subscription or the main subscription called Tinder Gold. And Tinder Gold, the advantage of Tinder Gold is it essentially allows you to break the rules of Tinder. So Tinder, normally you can't see who swiped on you and you're only going to match with someone if you swiped right on them and they've swiped right on you. Tinder Gold allows you to see all of the people who have swiped right on you, so you can go through those people and determine do you want to match with them. So really important sort of fundamental capability that people are willing to pay for.
**Ravi Mehta** (00:35:57):
On top of that, there's a set of a la carte products where you can buy... You can essentially buy them in bulk. You can use one of them. You can buy multiple of them. The two primary ones are super like. So super like allows you to send a super like to an individual person. If you send that super like, they're three times more likely to match with you. So it's a really good way, in a very targeted way, say that you want to meet and match with someone.
**Ravi Mehta** (00:36:22):
The other product is boost. And so boost works the same way that Facebook boost works or any other boost product works where your profile is going to show up a certain amount of times within the feed. If you pay to boost it, it will show up more often. And so what we noticed was that there was a set of people that were spending hundreds of dollars a month on boost and super like. Let's just identify some of these users, put together a usability study and start to talk to some of them and understand why they're using Tinder and that why and why they're willing to spend so much money.
**Ravi Mehta** (00:36:55):
And so what we found was actually it was very different than what we had assumed. It was essentially people saying, "I really want to meet someone." They have a use case. So sometimes these were folks that were in the military, so they were moving around a lot, or they were sales folks, they were often in different cities, or they were someone that was new to a particular city. And it wasn't that they were higher net worth. They weren't earning any more than the average Tinder user. They just had a much more intense use case. They wanted to meet someone. And what they were framing the cost of Tinder on was not the cost of other subscriptions. They were framing it on the cost of dating. And they were saying, "If I go on a few dates a month, that's probably a couple hundred dollars." Anyway, could be even more than that depending on whether you're in New York City or other places. And so they thought about that spend of a couple hundred dollars a month on Tinder as a small investment to make sure that they could date the people that they wanted.
**Ravi Mehta** (00:37:48):
And so it was a really interesting example of we identified something quantitatively that was really interesting that we knew was potentially a lever to grow the business. Our assumptions about why that use case was that use case were wrong. And when we ended up talking to users, we had some really surprising and fun conversations as a result, and we were also able to recalibrate and understand what those people were solving for. They're really solving for the utility of meeting people more effectively and not having to spend as much of their time to do it. And they were framing the price in very different ways than the average user.
**Lenny** (00:38:18):
I always love these examples where you see something in the data, you think it's something and then ends up being something else after you talk to customers. Can you share what you built or changed in the product because of that? Or is that a private?
**Ravi Mehta** (00:38:31):
Yeah, so there were two things that came out of those conversations. One is Tinder Platinum. So that's a third tier of the product that is a little bit more expensive and then comes with some additional features, as well as a bundle of these consumables that you can use within the product, additional super likes and boost.
**Ravi Mehta** (00:38:49):
And the other feature that came out of that is it's almost like a super swipe. It's the ability to, instead of just send a super like, you can send a super like with a note. And so it costs a lot more than a super like does, but it essentially allows you to break another rule of Tinder, which is you can't chat with anyone before you match. This allows you to send that first chat message to a person before you've matched. Basically to show that you're really interested in matching with that person further increases the likelihood that you'll match with them. And we were able to price it at a point which was much higher than we thought the pricing was going to be because we knew that people were thinking very differently about what the utility of that would be.
**Lenny** (00:39:26):
That is awesome. What a success story of a product team, product experience going through discovery research, data, designs, launch, revenue. Nice work.
**Ravi Mehta** (00:39:37):
And it was great. And the [inaudible 00:39:38] was working on it for about a week. She was running into my office a couple times every time she had a call with one of these folks to share what she learned. And so those are the high level takeaways, but it was really interesting to get to know this demographic better. And then just talk to users. I think oftentimes people don't spend enough time just picking up the phone and having a conversation one-on-one with the user of a product and getting into understanding their psychology, what value they're getting and how to really optimize for that.
**Lenny** (00:40:05):
**Ravi Mehta** (00:41:11):
Absolutely. One of the things that was unexpected when I started at Tinder was a couple times a week I would meet someone or I'd be in an Uber and the Uber driver would tell me, people would share like, "Oh, I met my boyfriend or girlfriend, or I met my wife or my husband on the platform." And it was really great to hear the stories. One of the things I didn't realize is the degree to which because of Tinder's very lightweight designs, it's been able to support the LGBTQ community much better than other dating products. And so some of my most fulfilling conversations with people who felt like they wouldn't have met their significant other without Tinder because there was just no place to do that.
**Lenny** (00:41:43):
Wow, man. Fulfilling, impactful, interesting, surprising. What a role. Actually met my wife online on a defunct dating site app called howaboutwe.com. Do you remember that one at all?
**Ravi Mehta** (00:41:56):
No. I haven't even heard that.
**Lenny** (00:41:58):
It was too good. It just matches people. It's like it reached Hinge's vision too well where they just... nobody needed to stay on. We don't just spend a lot of time on it, but basically the concept was how about we? And it's like a date concept. So instead of browsing profiles, you browse date ideas and then you say, "Hey, I want to do this date with you and let's go out and try it out." And it worked out for us.
**Ravi Mehta** (00:42:19):
That's really cool. There's so much opportunity. I think there's a lot of really good dating ideas that haven't been explored yet.
**Lenny** (00:42:25):
Mm-hmm. Interesting. All right. Good investment tip. Coming back to the product stack, getting back on track. One interesting thing about your product stack that's a little bit contrarian is you put goals after roadmap. And I'm curious why that is? Why you think goals should come after having a roadmap?
**Ravi Mehta** (00:42:45):
Yeah, it's definitely a contrarian point of view. I've had a few people yell at me about this. Typically, what happens is goals are almost the start of a strategic process rather than the end of it. A company will say, "We need to increase our revenue by X, or we need to increase our retention by Y. What's our strategy to be able to do that?" And what I've found over the years is that that goals first approach puts the entire energy of the product team on moving the goals without any sort of structure of what success looks like and why.
**Ravi Mehta** (00:43:17):
The analogy I like to use, it's a little bit like taking a road trip and starting out by saying, "Hey, we need to drive 250 miles." It's like, no, if you're going to take a road trip, you first decide where you want to drive to. If you're in LA, you might take a road trip to Vegas. And so our destination is Vegas, and we'll know whether or not we reach there if we've driven 250 miles. Because that 250-mile goal is in the context of a destination.
**Ravi Mehta** (00:43:42):
And so I think about all of the pieces of the strategy stack as being really clear about what is the end destination that you're solving for, and then you should work on goals to the extent that they help you reach that destination. And if you find that achieving your goal is actually pulling away from the destination, then there's a really important conversation to be had about do we leave that gain on the table because it's not aligned with our destination, or do we need to change our destination? And I think what happens too often when people start with goals and then create the roadmap is that the goal takes precedence and there's no context, there's no principles that are ultimately driving that. And so those decisions about the direction of the product come and go without even really being noticed because there's nothing to calibrate against.
**Lenny** (00:44:31):
So I a hundred percent agree that strategy should come ahead of goals. What's interesting is, so if your approach is strategy then figure out what you're building and then figure out your goals, how do you prioritize the roadmap? Because from my perspective, come up with your strategy how are we going to get to where we're going to get. Goals to me are how we measure progress towards that. And then the roadmap comes out of what's going to help us achieve this goal, and how do we prioritize based on what's going to most impact this goal that we have. So how do you approach prioritizing and picking what's going to be in the roadmap if you don't have your goals? Is it more like, here's the main KPI, or you have a rough sense of KPIs and metrics you're going to watch and use that to prioritize? Or how do you think about that?
**Ravi Mehta** (00:45:13):
Yeah, I think as part of the strategy, you'll typically have some quantifiable elements of that strategy. So for example, for Tripadvisor, our strategy was with trip planning, we wanted people to come directly to Tripadvisor and spend more time on Tripadvisor. And so what was happening was that most of a person's usage of Tripadvisor was interleaved with visits to Google. And so people would search for something, Boston hotels, come to Tripadvisor. They might say, "No, I want to look at New York." They Google New York hotels, and they come and look at Tripadvisor. And Tripadvisor's in a really good position to actually not have a person go back to Google because we knew about the preferences, we knew about their states, we knew who else they might be traveling with. And so more of that planning activity could happen directly within the product.
**Ravi Mehta** (00:46:01):
And so the problem was that at a company like Tripadvisor, which is very experimental, very quantitatively focused, the product teams were constantly optimizing for what's going to drive bookings in the moment. And so the thing that drives bookings from a visit to Google naturally moves a person down a transaction path and gets them to the booking and doesn't have them stop along the way to set up their trip and start to add things to their trip and create their wishlist. That actually gets in the way of the transaction itself. And so in the absence of that strategy around, we actually want to get people to come directly to Tripadvisor more often. We were doing so many things that ultimately undermined that strategy and got people to sort of leapfrog through the product instead of stay with the product.
**Ravi Mehta** (00:46:50):
And so that's a really good example of where if we know we want to generate that long-term continuous relationship with the user, there's a set of things from a roadmap standpoint that we can do to do that. We can prioritize those things, we can use numbers, we can opportunity size them, we can prioritize based on that, and then we can measure whether or not we're made progress based on that strategic and very conceptual understanding of where we want to go.
**Lenny** (00:47:14):
So the biggest takeaway I think we both fully agree on is your strategy should come ahead of having goals and coming up with your goals and aligning on goals. No question.
**Lenny** (00:47:25):
Speaking of goals, you also have some really interesting insights on just how to come up with goals and best practices for aligning and setting goals. I'd love to dig into that a little bit and then I have another topic I want to talk about.
**Ravi Mehta** (00:47:36):
Yeah, that sounds good. So I've done a little bit of writing about goals, which came out of... I've been at multiple companies that have put OKRs into practice and had a really hard time with that. And I've talked to a lot of product teams who have had a hard time. So the question I started asking is, why are companies having a hard time with OKRs? What's happening that is preventing teams from being able to set goals that they really understand how to achieve and achieving those goals?
**Ravi Mehta** (00:48:01):
And one of the things that I found, which I think was sort of a first principle that's happening at a lot of companies, is this idea of always focusing on outcomes over outputs and comes from a good place, which is ultimately, and I think this is the case, ultimately, a PM needs to measure their success based on whether or not they generate valuable outcomes for the business. But that doesn't necessarily mean that in this quarter we need to commit to a specific outcome or that we should commit to a specific outcome that we may or may not know how to move.
And so I think ultimately the goal is to drive outcomes, but oftentimes there's things that come before that that need to be addressed ahead of time so that you can really understand what the plan for meeting those outcomes is going to look like. And so I refer to that as the frontier of understanding. There's a point at which what the team knows and what the team doesn't know. There's a junction point there, which is this frontier. And it could be actually we don't know what moves retention. If you ask me to remove retention, I can brainstorm 10 experiments, but I don't actually know why people are continuing to use our product. And so then it doesn't make sense to commit to a retention goal because you're going to sort of throw a spaghetti against the wall, have a bunch of experiments, [inaudible 00:49:16] will stick, and maybe you'll be able to move the metric, but you won't have understood exactly why, or you might move the metric in a way that is not tied to the strategy that you have as a business.
**Ravi Mehta** (00:49:27):
So the first type of risk is really understanding risk. And if you don't understand how to move a particular metric, then the right goal is to set a goal to increase your understanding not to move that metric. Once you have an understanding of how to move the metric, your team may or may not be able to execute very well. It might not be able to execute those sorts of experiments. It may not have the resources that it needs to execute. And so then you might want to set an execution goal. So we want to hit 20 experiments this quarter, and if you can hit those 20 experiments, you'll know that you're executing really, really well. And even if those experiments don't work, that moves that frontier a little bit forward.
**Ravi Mehta** (00:50:04):
And then finally the ultimate frontier is strategic risk. We understand how to move retention or we think we understand how to move retention. We're going to do a set of things to do that. And then either we'll learn that our understanding is correct, in which case we can pull that lever more, or we'll learn that it's not correct, in which case we need to go back to understanding and goal ourselves based on that.
**Lenny** (00:50:24):
That is really interesting. So the term is frontier of understanding, right?
**Ravi Mehta** (00:50:28):
Yeah, exactly.
**Lenny** (00:50:29):
And there's four buckets that you just described of types of goals. Can you repeat them again?
**Ravi Mehta** (00:50:33):
Yeah. So the four buckets are, it starts with understanding risk, which is we have something that we want to do but we don't really understand what the levers are. Then the next thing is dependency risk, which is we understand what we think the levers are, but we may or may not have the tools that we need in order to make progress. Then there's execution risk, which is we have all the resources that we need, we have a really strong hypothesis, and then we may or may not be able to execute against those hypotheses. And the last thing is strategic risk, which is we have a hypothesis and it might turn out that that was not the right hypothesis.
**Lenny** (00:51:07):
Oh man. I wanted to move on to a different topic, but I want to dig into this a little bit because it's really interesting. So a lot of people work at companies where their product manager, leader is not going to be like, "Cool, let's spend a quarter understanding if we can move this metric." That seems like you have to be a really evolved leader to be okay with that, or is that even not a good idea to spend a quarter doing that? How do you think about not actually having a goal that is moving a metric that people care about and focusing on understanding and kind of pushing this frontier of understanding further versus just moving a metric that people actually want you to move?
**Ravi Mehta** (00:51:41):
It might be that for the quarter, the way that the company works, the things that it's focused on. You need to actually commit to a goal to move retention or a goal to move your follower count or something like that. There's two ways to do that. One is you can commit to that goal and then in three months kind of hope for the best and just do a lot of work that you think might actually move the lever. The other thing is to say actually that journey towards hitting that particular goal, we can break into. Initially let's spend a couple of weeks understanding. We'll talk to customers. We'll do some analysis. We'll form some really good hypotheses. And then based on those hypotheses we'll start to figure out what do we need to execute on in order to start to validate those hypotheses. And then we can execute on those things and validate those hypotheses.
**Ravi Mehta** (00:52:28):
And depending on where in the quarter things start to go off the rails, you'll have a feeling for where that frontier is. And when you miss the goal, you can then go back to the team or the leadership and say, "We missed our goal, but I think I know why. Here's the things that we did within the quarter and here's where things started to go off the rails. Here's what I'd suggest that we commit to for the next quarter so that we can be much more sure that we're going to hit our goal."
And leadership is always going to be outcome driven, but they also want to have a lot of confidence that we're going to be able to hit those outcomes. And so if you can clearly convey the learning and provide a really clear path that will get them that confidence, they're often going to be much more [inaudible 00:53:07] than you anticipate. I think the desire to always set outcome-based goals is just shorthand for we want you to move the needle and we want you to be thinking about that. That doesn't mean that you do that in the absence of really detailed understanding and really honing your execution process so that you can execute flawlessly. So approaching things in that way can help you change the conversation and make it much more specific.
**Lenny** (00:53:31):
And you also have a post about this exact topic, right?
**Ravi Mehta** (00:53:33):
I do, yeah. I've got a post on the Reforge blog. Can't remember the title. I think it's Set Better Goal with NCTs instead of OKRs.
**Lenny** (00:53:40):
Okay, cool. So if your manager is not buying what you're saying, that could be interesting to share with them and see if that'll change their mind. Your first point is worst case, you just hope for the best. You know that your frontier understanding is not that far, but still set that ambitious outcome-based goal and then hopefully works out. But in reality, it may not be realistic.
**Ravi Mehta** (00:54:00):
I think we can think about it as two by two matrix. On one axis of the matrix you have, did we hit our goals? And on the other access we have, do we know why? And ultimately you want to be in the upper right quadrant. You want to hit your goals and know why you hit your goals. Some teams are in the quadrant where they hit their goals, but they don't know why, which is good for now, but it's eventually going to catch up with you. And then an important thing to be in is if you didn't hit your goals to make sure that you're at least understanding why or at least you're making progress on understanding why. And I think too often teams get so focused on the goals, they get less focused on the learning.
**Lenny** (00:54:33):
Okay, final topic, product management competencies. So this is a post you wrote a while ago. It's the post I've shared most of your many writings online, and I'm going to pull up this image on my screen. So another plug to check it out on YouTube or Spotify. Can you talk about what this is and why it's important for PMs to think of their career in this view and in general just understand what the components of a great PM are?
**Ravi Mehta** (00:55:02):
Yeah, definitely. So we developed this at Tripadvisor. When I joined Tripadvisor, the company was newly public and as part of being a newly public company and wanted to grow different teams really quickly including the product team. And what we were finding is that hiring a product out of industry, and at the time we were based in Boston. So hiring a really good experience PM in Boston was taking between three and six months, and that just was too long to reach the sort of growth goals that we wanted to hit from a team perspective and a headcount perspective.
**Ravi Mehta** (00:55:34):
And so the head of product there came up with this program called the product rotational program, where we would hire people directly out of business school and out of undergrad into their first product role regardless of whether or not they had prior product experience. And they would go through two years of rotations. So four six-month rotations where they would be able to focus on teams that are zero to one teams or growth teams or infrastructure teams. So the goal was in about two years to get a person to be able to experience various different parts of product management and have them come out with the skills to be a senior and effective product manager.
**Ravi Mehta** (00:56:12):
And so as part of that, we really needed to define very clearly what is product management and how do we help people identify the skills that they need to be an effective product manager and give them a plan so that they can grow those skills. And so that's how this framework initially came to be. The framework consists of 12 competencies in four different areas. These competencies I think are the same for APMs as they are for CPOs, and I can talk a little bit about how they change as a person gets more senior. But these 12 areas are equally important regardless of where you are within your product management journey. The specifics might change, but the overlying structure remains the same.
**Ravi Mehta** (00:56:51):
The first thing that's really important is product execution. So PMs need to be able to work with their teams to build product, and that breaks down into three sub-competencies. The first is functional specification. So that's the ability to work with your team to define what is the PRD or the functional specification that defines what you want to build. The second thing is product delivery, which is the ability to work with engineering and design and the other teams to take that specification and turn it into working product. And the third piece, which I've changed from quality assurance is now product quality, is making sure that what you build is high quality, not just from a technical perspective, but also from a design perspective, a usability perspective and a business perspective.
**Ravi Mehta** (00:57:34):
And so ultimately that's the foundation of being a successful product manager is being able to execute. And that's as true for an APM as it is for a VP or a CPO. An APM is going to think about product execution in terms of their day-to-day individual contribution. But a CPO is going to think about product execution in terms of the systems that they create to enable teams to define really good specifications, to deliver products really effectively, to execute flawlessly and to deliver products that have a very high bar of quality.
**Ravi Mehta** (00:58:04):
The second area is customer insight. So in addition to being able to build products, you need to understand customers so you can figure out what to build. The three subcomponents here are fluency with data, which is the ability to use all of the data at your fingertips to make decisions about what customers need. The second one is voice of the customer, which is the ability to have the conversations with the customer so that the product manager can be the advocate for the customer throughout the entire company as well as the advocate within the product.
**Ravi Mehta** (00:58:31):
The third is user experience design. And so this goes back to our earlier conversation about wireframes. I think a fundamental part of being a really good product manager is the ability to think about the user experience in a very detailed way to make sure that you're not just defining functionality, but you're really clearly understanding how that functionality turns into user experience. And this is very explicitly user experience design and not user interface design because the experience of your product may vary. If you're building APIs, then your experience is actually the API spec. If you're building ML models, then your experience might be the training models or the other systems that you're using to identify the effectiveness of those training models. So this can be a skill that you can think about really broadly across a lot of different product roles.
The third piece is product strategy, and that breaks down into three things. The first one is being able to own business outcomes. So it's really important to move away from thinking about product as shipping features to driving business outcomes. And so this competency is about understanding how does your product or the features that you're working on plug into the business and drive value for the business. The next competency is product vision and road mapping. So that's the ability to take the individual pieces of work that you're doing for a product and put those together into a coherent vision and roadmap that allows you to build towards the product strategy and the company strategy over time. The third one is strategic impact. You're just like product road mapping as a sequence of features. I think about strategic impact as a sequence of business outcomes. So initially [inaudible 01:00:08] really focused on owning business outcomes and delivering business outcomes. But ultimately what's really important is does that sequence of business outcomes move the strategy forward and help you deliver impact on that strategy?
**Ravi Mehta** (01:00:21):
And then the fourth and final piece is all about leadership. So it's influencing people. The first up competency is stakeholder inclusion, so that's being able to work with all of the different people throughout your organization to rally them around the work that you're doing. The second one is team leadership. This is one that doesn't actually come into play until you have direct reports, but once you have direct reports, being able to help those direct reports become really great product managers is a critical skill. And the last one, which is always really important for PMs, is being able to manage up so that you can win the support of the leadership within your organization.
**Lenny** (01:00:53):
Amazing. What a crazy-ass job this product management job is.
**Ravi Mehta** (01:00:57):
It's crazy, isn't it?
**Lenny** (01:00:58):
Look at this thing.
**Ravi Mehta** (01:00:58):
You just got to do that and then you're good.
**Lenny** (01:01:02):
Oh man. This is an incredible framework. I've never found a simpler, more beautiful, very clear, easy to consume and share version of what the PM role is. So if people are looking for some inspiration for figuring out how to define the PM role with their company, set up their career ladders, I always point them to this and we'll definitely link to this in the show notes. And thank you for doing such a great job walking through it. There's a lot there.
**Ravi Mehta** (01:01:27):
Yeah, definitely. And then on my website I've got a downloadable kit that's got tools to evaluate yourself. It goes into each of the competencies in more detail. It talks about some of the different archetypes. So you'll find certain styles of PMs have certain clusters of competencies. If you're a growth PM, you might have a certain focus that might include a lot of focus on data and outcome ownership. If you're more of a product discovery or product innovation PM, you may have a different set of skills. So being able to map yourself out. We'll help you understand where you want to grow and what types of roles are a really good fit for you.
**Lenny** (01:02:01):
Plug the site while you're at it. Where do they find this exactly?
**Ravi Mehta** (01:02:03):
They can find it at ravi-mehta.com. So M-E-H-T-A.
**Lenny** (01:02:07):
Sweet. You have this kind of concept of exponential feedback that kind of relates to this and just partly touches on why this is so important for PMs to think about. Can you talk a bit about that?
**Ravi Mehta** (01:02:16):
Yeah, definitely. So this is something we talked about in the product leadership program. One of the most challenging things I think for both a PM as well as a product leader is to figure out how to grow yourself and grow your team. And a key way to do that is through feedback. It's really important to provide people with good feedback to help them understand how to grow. But the problem is, and this is true in a casual one-on-one, as much as it's true in an annual performance review, it's oftentimes the feedback that people provide is very surface level. It may focus on particular symptoms but not root causes.
**Ravi Mehta** (01:02:53):
And so one of the ways that this framework can help is, I'll often encourage people when they're first starting to use the framework, just to go through each competency and rate themselves needs focus, on track or outperforming on each of the competencies to quickly get a read on where you feel like you're landing. You can ask your manager to do that same thing. You can do that in five or 10 minutes. And then the areas where your manager and you see eye to eye and the areas where you guys see differently is stimulus for a really deep conversation.
**Ravi Mehta** (01:03:25):
And so I think that is like the entry point to providing exponential feedback. And I think about exponential feedback as feedback that has compounding returns. So if you give someone feedback on a particular symptom or you give them feedback on something that's tactical and they fix that in a moment, the feedback, the conclusion of that feedback, it just happens and then it's gone. But if instead you help a person understand the underlying behaviors that led to that particular situation, then they can focus on growing themselves. They can also focus on helping to diagnose their own performance more effectively, and that leads to compounding returns where they just keep getting better and better over time.
[NEW_PARAGRAPH]And so the ability to kind of apply the competencies as a lens helps you move out of that abstract, kind of surface level feedback into very specific categorizations of things that a person might need to work on, which I think gets to the root cause of areas a person can grow in and that ultimately leads to more effective feedback that has those compounding returns.
**Lenny** (01:04:25):
On that thread, just maybe a last question here. If your manager isn't good at this and isn't giving you this sort of feedback, do you have any advice for how to get feedback from people like this, mentors, anything like that? If your manager just kind of isn't doing that, isn't filling that role for you.
**Ravi Mehta** (01:04:40):
One of the things you can do if you are in a product role is ask them to do this exercise and evaluate you. Your manager will almost certainly have some impression of your performance that they haven't necessarily... If they're not doing it proactively, they probably have it intuitively. And helping them get it down on paper and getting it more specific can be a really good way to start that conversation. So that's one thing that you can do.
**Ravi Mehta** (01:05:06):
A second thing is I think oftentimes people refrain from giving feedback when they feel like that feedback is going to be intrusive. So just inviting your manager to say, "Look, I'm really looking to level up. Please give me feedback whenever you see something. You can give it to me in real time. Don't worry about wordsmithing it. I just want to make sure that I'm getting better." That agreement with your manager and giving them permission to give you that feedback will make sure that the stream of feedback has a much higher volume and starting with the quantity of feedback as a way to get eventually to quality of feedback as well.
**Lenny** (01:05:40):
As you're talking, I'm thinking of the advice Jules Walter shared on this podcast a couple episodes ago of when you get feedback, no matter how it makes you feel, whether you're melting inside or not, just be very enthusiastically. Thank you so much for that. That was really helpful.
**Ravi Mehta** (01:05:52):
It's so key because then you've rewarded the person for giving you feedback, even if it hurts inside, and then they'll want to do it in the future.
**Lenny** (01:05:59):
Yeah. Anything else that you want to touch on or share before we get to our very exciting lightning round?
**Ravi Mehta** (01:06:05):
One of the challenges I hear PMs that are moving into leadership roles is they often worry about micromanaging their teams. And so I kind of see two failure modes for people that are taking on their first leadership role. The first one is that they do actually micromanage, and so they don't let the person on their team have the autonomy that they need to figure out a path forward. And there's two problems in that, one that really makes that person feel like you don't trust them. The second thing is that rate limits the size of the team that you can manage because you can only do that for a finite set of people before you yourself are tapped out on bandwidth. And it's usually a couple of folks. So that's one failure mode where people sort of treat their first direct reports as an extension of themselves.
**Ravi Mehta** (01:06:51):
The second failure mode that I commonly see is just a completely hands-off mode of leadership where a person assumes that the new person on their team, they trust them, they give them a lot of autonomy, but as part of that, they don't give them the context that they need. So that person may be able to be successful but may actually lack the guard rails and the frameworks to channel their efforts. And so I think the right solution here is to say, actually micromanagement is not a bad thing. Some of the most innovative leaders in tech are famous micromanagers. Steve Jobs is a micromanager. Elon Musk is a micromanager. Mark Zuckerberg's a micromanager. Ultimately as product builders and products innovators, the details matter and sometimes you need to zoom into what does the text on a particular button say, and you might have a strong opinion on that. And so it's okay to engage at that level.
**Ravi Mehta** (01:07:46):
I often encourage product leaders to think about their process of becoming more senior, not as a matter of getting more and more high level, but of increasing their dynamic range. So a CPO, it's not that a CPO never thinks about tactical issues, it's that they spend a lot of time on strategy, but they also can zoom into specific issues. And so a framework I like to use with product leaders that I'm coaching is to think about a matrix. Your ideal goal is to lead in a scalable way, which means you feel really confident about the direction of your team and your team has the autonomy to move in that direction.
**Ravi Mehta** (01:08:22):
There's another really effective way of leading which is selective micromanagement, which if you don't feel confident in the direction that your team is moving, the right answer is not to be hands-off and to let them go in that wrong direction. The right answer is to micromanage, but do it in a very tactical, in a very temporary way so that you can help them understand what is the right direction moving forward, so that you can then pull back. And the two failure modes are if you're hands-off and you let that team go off the rails, that hands-off mode of leadership might feel really good in the short term. It might help you avoid micromanaging in the short term, but ultimately it's going to mean that that team doesn't get to where they need to go.
**Ravi Mehta** (01:09:03):
And then what we commonly think of as micromanagement, I think more of as micro mismanagement, which is you don't feel like you've got a sense of control or a sense of confidence about what the team's doing. The team doesn't feel like they have a sense of autonomy. There's not a clear end in sight, and ultimately both the leader and the team are frustrated. So I think the two really effective functional ways of leading are scalable leadership where the team has autonomy, you have confidence or selective micromanagement where for a brief period of time you might take away some of the team's autonomy to set them on the right track, but with the goal of getting back into that scalable leadership mode.
**Lenny** (01:09:40):
I really like this topic. I feel like this could be a whole other thread. Maybe one quick question along these lines. Would you call it selective micromanagement?
**Ravi Mehta** (01:09:47):
Yeah.
**Lenny** (01:09:48):
Is there a heuristic you have in mind of just like what does that mean in practice? Like one out of every 10 decisions, maybe you push them in a direction that you'd need them to go. How do you figure out what's selective enough or is there in your experience?
**Ravi Mehta** (01:10:02):
I think it often comes down to being overly detailed at the moment that you see a problem. So helping the team get back on track by any means necessary, including potentially you're getting really detailed about the decisions that the team is making. But as you do that, think about the frameworks that you're using to help the team make decisions and help the team understand that framework. And so over time, the goal is to replace you actively kind of going in and guiding the team's decisions with them having a framework that they really understand so that they can make the decisions that are aligned with where you think the right direction is to go. And the ultimate success is that you give enough of a framework and the team has enough autonomy that they get to answers that are even better than you could come up with. And so that gives the team an incredible feeling of power and that gives you as a leader an incredible feeling of confidence in the team's ability.
**Lenny** (01:10:59):
Got it. Yeah. Well, this makes me think about it as a product leader. Most of the time you need to push your team to do the thing that you believe is right, and maybe once in a while let them make a mistake and have them learn from it. But it's not the other way. It's not like, "Cool, let them make all the mistakes and once in a while correct." It's the opposite. Your ass is on the line if they waste time and resources and fail. So yeah, your job is to make sure they're heading in the right direction.
**Ravi Mehta** (01:11:22):
There's another framework that we talk about in product leadership which goes into this topic, which is as someone who's working with a manager, there's kind of two things that you're constantly solving for. One is the degree to which you're aligned with your manager, and the second is the degree to which your manager has confidence in you. And so if there's a high degree of alignment and a high degree of confidence, you have full support, but there might be cases where there's actually not a high degree of alignment. You want to go in a different direction than your manager wants to go in, but if you have their confidence, you'll get their permission, you'll get their support to go in that direction.
**Ravi Mehta** (01:11:56):
And so keeping an idea of where you are on that radar is really helpful for understanding the currency that you have to be able to push things in the direction that you think is the right one. And if you don't have your leader's confidence and you're not aligned with them, that's not a recipe for success. One of those things needs to change. Either you need to do things that they are aligned with or you need to do things to win their confidence in your ability to pick a different path forward.
**Lenny** (01:12:23):
I like that. One final tangential totally out of nowhere question. I had on my notes that you've been doing some stuff with AI in your coaching work. And so I wanted to ask you, how do you think AI will work with PMs and just coaches and us as, I don't know, professionals in the workplace? What have you found so far in your experience there?
**Ravi Mehta** (01:12:47):
So when we started Outpace, we knew that AI was going to continue to advance and that eventually we would want to think about AI as a way to amplify coaches and to help make them more efficient and more effective. We thought that that was going to be a multi-year journey and that we would get to it at some point in the future. But this year's been incredibly exciting with the advances that we've seen from OpenAI and Stable Diffusion and Midjourney and all of these different models.
[NEW_PARAGRAPH]And so we've actually accelerated a lot of our roadmap around that. We have an interesting opportunity to use AI in the product where one of the things that makes Outpace different from other coaching platforms is we provide both content as well as the coach. And so each week a person will go through a 20 or 30-minute session. That session includes a brief audio lesson and then includes interactive exercises that go into how would you use the things that you just learned in that lesson.
**Ravi Mehta** (01:13:42):
And so one of the things that we have is we've got text content from all of the participants in Outpace where they're providing very specific answers to very specific questions. We're using that content to prompt, in this case, OpenAI, to give suggestions to the coach. And so the coach can go in and say, "Help me with a suggestion of what I should say as feedback to this particular response." And then the coach can go in and tailor that based on what they know about that person. And one of the most amazing things is we've been able to simulate different styles of leadership by using different types of prompts. So we can have suggestions that are really action oriented that provide lists of next steps. We can have suggestions that are more sympathetic that focus on the person's feelings. We've got suggestions that are more inquisitive, which ask follow-up questions. We've got suggestions which are informative, which provide frameworks and advice.
**Ravi Mehta** (01:14:32):
So it's really pretty remarkable how far the technology has come. I know we're at an interesting time right now. It's going to be interesting to see how things play out. I think one of the most interesting things about it is not AI as a replacement for people, but AI as a way to amplify people and make them more effective. And I think we'll see a lot of that in terms of both image generation and text generation where it's less about AI doing all the work and more about AI providing a really good starting point.
**Lenny** (01:14:58):
I love the idea that people have these visions of where their product's going to go in five, 10 years, and the vision's happening so soon. And that's got to feel nice, but then you got to rethink, "Oh my God, what's our new vision of the future at this pace?"
**Ravi Mehta** (01:15:12):
It's been really exciting. I haven't been this excited about tech in a long time. And I think it was, we knew we would need to pivot in order to embrace this more, but it completely makes sense and it fits really nicely into something that we're already doing.
**Lenny** (01:15:24):
Well with that, we've reached our very exciting lightning round. I've got six questions for you. I'm going to power through them and whatever comes to mind, just share it away. Sound good?
**Ravi Mehta** (01:15:35):
That sounds good.
**Lenny** (01:15:36):
Cool. What are two or three books that you recommend most to other people?
**Ravi Mehta** (01:15:42):
I really like Hooked. That's a book that I know came out a few years ago, but I find that that model is just such an effective model for thinking about how to create products that are engaging. I also really like Working Backwards. I think Amazon has such a unique way of going about building product, and they've been so opinionated about what matters within that process. It's great to get a really detailed window into that. I was always curious about how it worked, and Working Backwards was a great way too to understand that a lot better.
**Lenny** (01:16:09):
For folks that are interested in that, we had Ian McAllister on the podcast talking a lot about that stuff. So if you're interested in Working Backwards and don't want to read the book, there's a podcast episode for you. Speaking of podcast, what's a favorite other podcast of yours other than the one you're currently on?
**Ravi Mehta** (01:16:22):
Yeah. One of my favorites is The Ezra Klein Show. I love the fact that he talks about a bunch of different topics. He's often got contrarian points of view. He just had an episode recently about a skeptical take on AI that I disagreed with a lot, but it was really interesting to think about it from a different perspective.
**Lenny** (01:16:36):
One of the best compliments I got about this podcast is someone telling me that they listened to these two podcasts as the only two podcasts they listen to, and they always have to pick one or the other when they're going in their morning walk. [inaudible 01:16:45].
**Ravi Mehta** (01:16:47):
You and I were talking a few months ago, and that's sort of the boat that I'm in. I've been listening to your podcast. I've been listening to Ezra Klein, and then there's a couple of others that are in the mix, but the ones that I keep going back to when I'm walking the dog are those two podcasts.
**Lenny** (01:16:58):
What a dream.
**Ravi Mehta** (01:16:59):
Thanks for having me on.
**Lenny** (01:17:00):
Oh man. It's not over yet. Next question. Favorite recent movie or TV show that you've really enjoyed?
**Ravi Mehta** (01:17:05):
I love Andor. I just finished watching it about a week ago. I think it's not just a great Star Wars piece of content, it's just a really great piece of science fiction. I think a lot of science fiction has gotten very samey and very dystopian recently. This was such an interesting reflection of what's happening today, really deep thinking about what the future could look like, really good expansion of the universe. So it's just great on a lot of levels.
**Lenny** (01:17:27):
Frigging love Andor. Huge plus one on that. Favorite interview question that you like to ask.
**Ravi Mehta** (01:17:34):
My favorite interview question is, tell me about a product that you love. And I can have that question last five minutes. I can have that question last 60 minutes. And so that's the first question that I'll typically ask people during a screening interview. I use the word love very deliberately. I want to see what products in their lives they really gravitate to and they engage with and that they can use that word with. That helps me understand a lot about what they value. And then I'll ask a whole series of questions, which is, why do you love it? Why do you think other people love it? What would you like to see about it in the future? Pick a feature that you'd like to build for that product. Why do you think that's a good feature? How would you measure the success of that feature?
**Ravi Mehta** (01:18:11):
So I've used this for years. It's just such a good way to help understand the product sense that a person has, help get to know a person a little bit better. It's always interesting when people pick products that are more physical products to see what they're into in terms of hobbies and things of that sort.
**Lenny** (01:18:26):
What are five SaaS products that you use at your company or on your team?
**Ravi Mehta** (01:18:32):
Airtable has been amazing. It's such a powerful tool. We just rebuilt our accounting system in Airtable. Webflow, we're using constantly. It's really changed how we think about building products. We now ask, do we need to build code or can we do something in Webflow? We're using Superhuman. I spend most of my day in Superhuman. It's an incredibly fast email client, so I love having it on my team. A lot of the team today is using Descript or Descript to edit videos. They found that to be something that works so much better than prior audio and video editing solutions. And then I've always loved Balsamiq. I've been a Balsamiq user for probably 10 years now, and whenever I get stuck on a user experience issue, I go in, I create some wireframes, and it always helps.
**Lenny** (01:19:13):
We use Descript/Descript, I also don't know how to pronounce it, on this podcast. So a huge recommend of that. Final question. You are building a company that is helping people find coaches. Do you have any tips for someone that is talking to a potential coach and what they should maybe ask them when they're trying to decide if it would be a good fit?
**Ravi Mehta** (01:19:34):
Yeah. I think one of the questions that's really helpful is tell me about the client that you're most proud of helping. What was the challenge if they were facing? How did you help them meet that challenge? The person doesn't need to go into anything confidential, of course, but I think what's really nice about that question is it get you really deep insight into what they value. You get to see where their pride comes from. It gives you insight into how they engage with the people that they're helping. And then you can understand, does that sort of map with what you're looking for in terms of a coach?
**Lenny** (01:20:03):
Ravi, this was everything I hoped it would be. I learned a lot. I had a lot of fun. Two final questions. Where can folks find you online if they want to learn more, and how can listeners be useful to you?
**Ravi Mehta** (01:20:13):
My startup is Outpace. You can find it at outpace.co. We published a lot of free resources. We just published a resource that you helped us with, Lenny, called Unlock Your Product Manager Potential. We also have a Q and A service where you can ask questions of coaches. Our goal with Outpace is to get more and more people to experience coaching, whether that's in an active coaching relationship or just a really quick conversation with a coach. So come to outpace.co. That'll be really helpful for us and hopefully really helpful for you as well. And then if you want to follow me, I'm on LinkedIn. You can also read my writing at ravi-mehta.com.
**Lenny** (01:20:49):
Amazing. Ravi, again, thank you for being here, and we'll share all these in the show notes and all these links you mentioned. Thanks again.
**Ravi Mehta** (01:20:57):
Yeah, thanks so much for having me. This has been great.
**Lenny** (01:21:01):
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcast, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at lennyspodcast.com. See you in the next episode.

---


</div>