Optimizing Pinterest’s Experimentation Strategy with Jeremy King, Senior Vice President & Head of Engineering at Pinterest

(Intro Narrator at 00:00:03) Hello, my friends. Today, we're talking to Jeremy, Senior Vice President and Head of Engineering at Pinterest. And we discuss why Pinterest is the Internet's way to figure out what people want, how Jeremy optimized the company's experimentation strategy, and how people may be missing out on personal growth in fully remote environments. All of this right here, right now on the Modern CTO Podcast. Here we go.

(Joel Beasley at 00:00:35) This is the Modern CTO Podcast. I was curious to know, when we were talking last, you were at Walmart. And then you went from Walmart over to Pinterest. And I saw that, and I got all excited. And I was like, oh, we should have him on again. And then I realized he just moved to Pinterest, so he needs to go do some stuff for a little bit so we have something to talk about.

(Jeremy at 00:01:03) Yeah. Great.

(Joel Beasley at 00:01:03) Tell me about that move.

(Jeremy at 00:01:05) You know, it was kind of an interesting move. I was at Walmart for eight years, and it was a wonderful ride. I mean, the people there were great. It was one of the best experiences in my life. I mean, trying to take, you know, Fortune One, the biggest company in the world, and transform them to be in the driver's seat, transforming them to a digital powerhouse. It was couldn't have asked for anything better. So I had been talking to Doug about how long I was going to stay at Walmart, and I'd been there eight years. And if you look at my background, I usually stay—I always tell people it's six months or five years. Right? It was sort of my, you know, it's either going to work and I'm going to stick around, or I'm out of there. Right? When I got to eight, I pretty much—not that you can never accomplish anything in any job, but, you know, I kind of done most of the things I wanted to do there. And so I started thinking about taking a little bit of time off. And just like you were mentioning your own kids, my kids were going to be home for one more summer. Little did I know that wasn't going to be true. But one more summer before they went off to college, and so I was going to take the summer off and do some consulting and maybe some advising. So I called a few friends of mine and, you know, got an introduction to Ben Silverman, the CEO of Pinterest, and he was, you know, asking for some advice about how to structure the team they were about to grow and, you know, and could I be an advisor and that sort of thing. And I said, yeah, that sounds great. I fell in love with the company and the product. As things come, you know, their CTO left, and then he'd asked me to join. And I said, you know, that sounds great. I love the company. I love what you do. I'll join after the summer, you know. And he said, you should start soon. Right? You know? And I'm like, okay. I got you. So I got into the company a couple weeks before the IPO, which was awesome. But it was, you know, how those companies that go from, you know, such a great idea and such people that love the product so much, and then you just have to scale it. I mean, you just run into this massive scaling problem. And the good news is that from a talent density standpoint, you know, the number of A players as a percentage of the company is incredible at Pinterest. It's amazing how smart the people are here. And that was another thing I was looking for in my next gig. And not that Walmart wasn't doing a lot of leading edge stuff, but, you know, this is really bleeding edge stuff. We have the best computer vision people in the world. We have the best graph database people in the world. We spend a ton of time on our machine learning engines, and it keeps me, you know, very on my toes, if you know what I mean. That's what a CTO wants. Right?

(Joel Beasley at 00:03:38) Yeah. It's got to stay interesting. If you're not solving hard problems, then you just turn to mush.

(Jeremy at 00:03:43) Yeah. Exactly.

(Joel Beasley at 00:03:45) What's the business model over there? Is it just ads? Because to me, Pinterest is my wife's on Pinterest a lot. Sometimes she'll send me a link to a collection of things. That's pretty cool. I was looking for—I moved to Tennessee from Florida. And the person who had built this house was a contractor, and he was also a prepper. So he, end of world type stuff. So he built this bunker. Right? And I was like, oh, okay. I guess I'll put my security system in there and just kind of make it that type of room. And I was looking for inspiration ideas for how to design it. And my wife sent me this Pinterest thing, and it was just unbelievable, man. It was just this curated list of the coolest safe rooms ever, and it had so many ideas. And that was one of my first experiences with Pinterest.

(Jeremy at 00:04:29) It's funny you say that because people oftentimes will say, hey, Pinterest is for women. But if you go talk to any contractor, any designer, any architect, male or female, they all have Pinterest accounts. Right? I mean, some of the, you know, the burliest guys down the street, the plumbers or whatever, you know, they use Pinterest for exactly that kind of inspiration. It's like, I don't know what color to paint this bathroom. You know, what is the latest color to paint bathrooms? And they'll have a Pinterest board for that. I just finished the—or I'm halfway through—you know, how projects get to 80%? I'm 80% of the way through building a doghouse. I just got a new puppy in the—I, of course, had to make a Taj Mahal kind of doghouse for it. You know? So I've been using Pinterest for, you know, inspirational doghouses. So which is super fun to work on.

(Joel Beasley at 00:05:17) I love it. I have this image in my head of you doing this Venetian painting type stuff.

(Jeremy at 00:05:23) It's more modern sort of architecture, and I want to do 100% recycled. So I tore apart some pallets and have, you know, been doing more slats and that sort of thing. But it's going to be good. I need another weekend to work on it before it's going to be done.

(Joel Beasley at 00:05:36) You can have one more weekend. Exactly. You've got to make it a smart doghouse too. Put an Alexa in there or something.

(Jeremy at 00:05:44) I considered putting a solar panel on it and doing heating and cooling, but it just got a little bit crazy. You know? Like, hey. Does my dog really need a fan in his doghouse? It's kind of a—

(Joel Beasley at 00:05:56) We'll see. You'll learn.

(Jeremy at 00:05:57) Right? Exactly.

(Joel Beasley at 00:05:58) Your dog will tell you.

(Jeremy at 00:06:00) Exactly.

(Joel Beasley at 00:06:00) Dude, this is cool. What type of hard problems are you solving over at Pinterest?

(Jeremy at 00:06:05) Well, another reason I love it is because it is a scale problem. I mean, again, you know, in previous lives, I've done it. When I was at eBay, we scaled early Walmart. Of course, you know, everybody in the world shops at Walmart. You know, Pinterest is a place where 400 million monthly active people come. You know, they come to do everything from, you know, finding doghouses to safe rooms to redecorating their kitchen to what are they going to wear for New Year's, how to redesign their look, you know, from a hairstyle standpoint, how to find a good pair of shoes, that kind of thing. So it's such a variety of things that people are getting inspired by. We have a ton of people that are gardeners, and they want to know how to do succulents and, you know, design a garden space. So it's a big part of the variety. And so that means ingesting billions of pins coming through into Pinterest and then using both computer vision and machine learning to do what is content understanding. What is this pin? Right? Is it safe? You know, this is obviously one of the most important things about Pinterest is making it the safest place to go on the Internet. You know? You don't have any politics. You don't have any, you know, suspect content. You don't have any of the crazy stuff you see in the social platforms. So number one, making it safe, but then making sure that we really identify, you know, what it is. Some people come here for inspiration, but they're also wanting to learn. Right? They want to—you know, we call it sometimes we call it edutainment. You know, people are, you know, building websites that are teaching how to cook or, you know, how to garden or that sort of thing. So, and then, of course, all the big hard problems when somebody does a search for, you know, New Year's Eve ideas or, you know, Saint Patty's Day party ideas. You know, I've got to collect a billion pin images and pins, and how do I reply back in a good way? So all the hard machine learning problems and the computer vision problems are great. But really, the reason I joined is because 90% of the people that come to Pinterest, they say, you know, I wish it was easier to find the thing. Once I get inspired, I want to buy it. And I don't know if you saw this, whether it was shelving or, you know, a safe for your room or something like that. Where can I buy that safe, right, or that particular item? And it's really about merging all the work that I've done for the last twenty years, you know, of Internet retail getting connected with the inspiration platforms. And so that's really what I've been working on.

(Joel Beasley at 00:08:34) Isn't it always so clear in hindsight?

(Jeremy at 00:08:38) It is funny that you say that. Bill Ready, who's my new boss, just became CEO of Pinterest. Obviously, he came from Venmo, PayPal, and Google. He's been the last couple years on Google Shopping and a few other things. So we have a lot in common now. I've been spending a lot of time with him on our shopping efforts and what to do next.

(Joel Beasley at 00:08:57) I would say that was very frustrating for me because the whatever they were mounting to the wall that then they could put all the hooks on and everything, I could see it, but I couldn't find it. And there were a couple different types, and you couldn't tell if it was plastic or metal or—and I've seen, you know, some place on some of the pins, because we did a bathroom remodel too when we moved to Tennessee, and it was 80% done for a long time. Right? But they did have a couple of, the best way for me to describe it was a hot spot type deal where it's this little pulsating deal, and you could click on it, and you're like, oh, that's the one. And so it was very sporadic in the sense that, it—I guess it just picks up on items. It's very clearly new. Right? Can creators do it?

(Jeremy at 00:09:43) Yeah. That's a new tech that we've been doing. So over—I've been here three years. The first year was, you know, all obviously, getting your feet wet and growing the team and just putting in some—you know, the team—you know, we went IPO, but we still hadn't been a mature company yet. So we spent a lot of time on just basic sort of development processes and architectural standards and that sort of thing. The second and third year have really been about both creators and shopping. Right? It just exploded. When, especially when the pandemic came, we saw such an influx of engagement that we took the opportunity to really invest in the creators' side and the platform side. And that really means, in particular, on the creator side, we needed a content ingestion platform through all the APIs, but we also needed a place where you could actually create, you know, a video pin and say, hey. Here's what I want to do. And that came, of course, with creator monetization. So, you know, creators don't just do this for the goodness of their heart. Some do. But, you know, they want to be paid, and there's lots of ways to be paid whether we do pays based off of engagement rates or whether they're using affiliate links. So a lot of people have Amazon affiliate links, and they're going to tag items. So, and that's exactly what we're thinking. Of course, we have some of the great computer vision people in the world, but identifying every single item and every single pin is a pretty hard computer vision problem. Right? And it was much better to get somebody, including our merchants, will tag the items. Many of the merchants have these beautiful hero images, they call it, or lifestyle images. You'll see a scene of a room, and it's got 10 products in it, that sort of thing. And guess what? The merchant knows that here are the ten items that you have. And so creators and merchants both will submit both images and videos with products that are tagged, and that includes affiliate links, and they can get paid for that effort.

(Joel Beasley at 00:11:32) Oh, that is excellent. Yeah. I think you're going to get really interesting lists from a certain segment of the market that—for me, example, I have a couple interests. One specifically, guns. I could put together a really sweet list, and it's so cool how you've basically made it really easy for everybody to do that, and then you get really great lists because that's the one thing that makes me want to interact with Pinterest is the list I get will not be going on Amazon and typing in, hey. I need something over in this area. It'll be this—it's almost like a guaranteed artistic result.

(Jeremy at 00:12:10) Yeah. Absolutely. We talk about Pinterest this way. It's like, there's a million places to go on the Internet if you already know what you want. There's really only one place to go if you don't know what you want, and that's Pinterest. Right? Because you get there and you say, listen. I'm thinking I want a modern barn kitchen. Right? You know? What the hell does that even mean to anyone? Right? You know? You can't do that search really accurately anywhere else. But when you do that on Pinterest, millions of people have pinned billions of items to a board called modern barn style kitchens, you know. And so we use the Internet's population to help curate and sort of tag, you know, machine learn tag the billions of images around the world and now videos. And so, that's what Pinterest is all about. And it's a hard technical problem, but I tell people when I—one of the main reasons I joined Pinterest, and a lot of people that work here that talk to me is, if you ask, you know, 10 people, you know, do you have a Pinterest account? What do you think of Pinterest? They go, oh my God. I love Pinterest. Right? You know, they show you their boards, and here's what I'm working on and that sort of thing. So it's fun to work at an inspiration company. Right?

(Joel Beasley at 00:13:16) And then what's your day to day like? Is it much different than Walmart, or is it still coaching tech leaders? And—

(Jeremy at 00:13:23) Yeah. The one thing that was killing me at Walmart was the time I was on the road. You know, I was on the road 80% of the time. You know, I had teams all over the world. When I joined Pinterest, 90% of the team was in San Francisco. So, one of the first things I had to do, of course, is diversify that because that's just an insane technical strategy these days. Right? So we opened offices in Seattle, Vancouver, Mexico, Poland, Dublin over the three and a half years that I've been here, a couple through acquisitions. But, as you can imagine, I've also sort of standardized on sort of the North American time zones. If you heard those numbers, I didn't say India or China, largely due to the fact that we're just not big enough to deal with the tax that it takes on the time zone. Everyone calls it the time zone tax. Right? You know, 11:00 at night is 11:00 in the morning. Right? And it's a pretty painful process. So that was number one, growing the team just—we need to double or triple the size of the team. We've tripled it essentially since I've been here.

(Joel Beasley at 00:14:21) Why did you need to do that? Why did you need to triple the team?

(Jeremy at 00:14:24) Well, number one, we grew our revenues. When I joined, it was under $1 billion, and now we're well over $2 billion. So we're tripling the size of the revenue. So everything from the advertising stack, the work that we had to do, but plus, in order to build a whole creator's platform and a shopping platform, we needed new team members. And, you know, my team is relatively small.

(Jeremy at 00:14:49) You take a look at my competitors. I'm very commonly, as I've done in my career, working with a small army against a much, much larger army. So part of the reason that I mentioned the talent density is so high is that we need a super high performing team because we're building products not only for the whole world, but we're competing with the biggest internet companies in the world. So we definitely have to have some scale in order to make that happen.

(Joel Beasley at 00:15:17) Like Alibaba? Like, who would be a competitor?

(Jeremy at 00:15:19) Yeah. Everybody who basically buys ads on the internet. So all the social platforms, all the big guys now that we talk about, everyone from Snap to TikTok to Instagram to even nowadays the ad platforms. You see Etsy building their own ad platform. Wayfair built their own ad platform. Walmart, of course.

(Jeremy at 00:15:41) I built the ad platform at Walmart. So it's all about making it easy for companies to understand when they're buying an ad. In our case, most of the time, it's an inspirational type of ad where they're trying to get people to understand what they sell and get lots of new suppliers coming in for us. But making sure that they understand what the results are. You know, they want reporting back.

(Jeremy at 00:16:05) They want, and it's getting harder and harder, given the IDFA work to directly understand a correlation between somebody saw an ad on Pinterest and whether they actually converted, given all the cookie changes and everything that's happened in the last year.

(Joel Beasley at 00:16:21) Did those Apple changes affect you guys at all?

(Jeremy at 00:16:24) Absolutely. Yeah. Absolutely. We didn't do a lot of retargeting. Most people around the internet, they hated retargeting the most. Even though it's one of the most useful parts of internet advertising, it's the one that everyone said is the creepiest. You know, I'm looking at something on an internet retailer, and then I jump over to some other website, and then it says, "Hey, do you want to buy this item?" And you're like, "How did this company know that I was looking at something on some other company?" So we were less impacted by those changes.

(Jeremy at 00:16:58) We also, the way Pinterest works is we use what you're pinning and what you're engaging with as the recommendation engine versus lots of companies target you based off your age or your profile and that sort of thing. Pinterest doesn't really do that at all. We know your age and a couple of other things, but we're not really targeting advertising or pins based off that. We're using the inspiration engine based off what you're engaging with. So people will come in for everything from inspirational quotes to tattoos to gun safes or whatever.

(Jeremy at 00:17:35) And guess what? You're going to get more gun safes and you're going to get more related items to those. And most of it, it's visually appealing versus it being something more targeted. And so as a result, we've been less impacted by it. But it's still, it can be hard, especially because do you know about last click and the internet advertising platform?

(Jeremy at 00:17:55) So, you know, Amazon, Google, and many of the other platforms really optimize companies for last click. If you're looking for a couch, it's very unlikely that you're going to go to a website and click and buy that couch in the same day. Right? You come home. You say, "I like this couch."

(Jeremy at 00:18:15) You show it to your wife. You say, "Hey, here's what we're going to do. Should we go look and see if they have it over at Living Spaces?" Blah blah blah. And then 30 days later, you might buy a couch. Right? Oftentimes, Pinterest is a place where we introduce those brands and those types of couches and the styles to consumers, maybe weeks before they buy. And smart advertisers like Home Depot and Wayfair and IKEA and some of the other high consideration companies understand a multi-touch attribution.

(Jeremy at 00:18:47) Like, say, "Hey, they saw it on Pinterest first, then they went to the store and checked out IKEA for a while, and then they end up buying it after they saw an ad on some other platform." Right? And they'll give us, quote unquote, each a slice of that advertising revenue because they know that that's a really good place to start your journey. The more nascent advertisers are like, "Oh, no. They clicked on Amazon and they bought it. So Amazon should get all the advertising credit." Unless they didn't find it on Amazon. They found it on Pinterest first. And so part of our shopping journey is how to connect that together.

(Jeremy at 00:19:23) Like, people don't want to leave Pinterest to go click off and go look at some other retailer. They want to be able to know, like, "Hey, this exact item is this exact item, and I'll click over to the retailer and buy it." Right? And so that way, it's really easy for people to understand that it came from Pinterest. Right? So we've got a lot of work to do there, and it makes it hard. Right? And that's just the way of the world now. Right?

(Joel Beasley at 00:19:42) Well, I experienced that firsthand, understanding attribution and awareness versus direct conversion.

(Jeremy at 00:19:48) Right.

(Joel Beasley at 00:19:49) The podcast, we sell sponsorships where people come on. Right? And we've been doing that for about three years now, and we had to figure out, like, how do we prove it to the customer? Because when you use a podcast, when you hear something, like, we're talking about Pinterest, no one's going to go into the description of this podcast and click a Pinterest link to go to Pinterest. Everyone's going to open their phone up and Google or go directly to the site.

(Joel Beasley at 00:20:13) And so we saw these customers repeat buying from us a lot. And so I set meetings with them, and I just chained like 30 meetings back to back with all of these customers. And the thing that made them successful versus our other ones was that they track it. They have a "How do you hear about us?" and there was like a Modern CTO podcast either on their page or in their CRM when their salespeople would ask in their first meeting. And so I tell that to the other people, and they'll just be like, "Yeah, it didn't work." I was like, "Well, you didn't track it."

(Jeremy at 00:20:41) 100%. And you described the problem exactly with Pinterest. The other thing is, of course, the last click people are saying, "Oh, no, no, no. We get all that credit. We're the attribute. You don't need to worry about any of that stuff." And that came from here. Right? So you're fighting against the, like, "No, those platforms don't work. They're, you know, blah blah blah." You know? It is what it is. So part of the, you just have to get smarter about doing things like tracking and customer education and all that sort of thing. The other good news is that, frankly, advertisers want to diversify their ad spend.

(Jeremy at 00:21:16) I mean, the social platforms have become pretty scary in some cases to advertise on. You know, they get a lot of bad press when crazy stuff comes out. And so there's a lot of brands that are just absolutely nervous, and they want a safe place to advertise. And we spend a lot of time talking to people about why we're the safest place on the internet and why it's a part of our DNA. We spend a lot of technology cycles making sure that's true.

(Joel Beasley at 00:21:45) That's interesting because I didn't know that. I'm learning about that now. Also, if you were to ask me when we started the conversation, like, in the first 10 seconds, if I use Pinterest, I'd be like, "No." Yes! All the examples you've given about the bathroom, the gun room, I've looked at tattoos on there. There's so many. I looked the other day, I was looking for a panda cartoon character, and a list of these panda cartoon characters came up. And it's funny that my brain didn't register that I use it, but I actually, I'm on Pinterest, you know, a couple times a year at least.

(Jeremy at 00:22:16) Yeah. Yeah. We get that a lot. Yeah. It's funny.

(Jeremy at 00:22:18) We also are really good at SEO work as well. And, frankly, it's natural because to your point, people are looking for lists of things, and Pinterest has billions of lists of curated items. And as a result, we have the best results in many cases. And so the search engines know that, and they end up seeding us a lot of data. Right?

(Joel Beasley at 00:22:42) That's almost always how I get to you, by the way.

(Jeremy at 00:22:44) Yeah. Yeah. We want to get less and less dependent on that. I mean, Google changes their algo constantly, and they definitely demote us. And we would rather have you open the app, if you know what I mean.

(Joel Beasley at 00:22:54) Well, I think I got to the point where I had to make an account, whether it was sharing something with my wife or some system that said I looked at too much stuff. I don't know how you guys do it exactly, and I'm sure you run experiments constantly.

(Jeremy at 00:23:07) Yes, we do.

(Joel Beasley at 00:23:09) So at some point, I got an account.

(Jeremy at 00:23:10) Yeah. We have about a thousand experiments running at any one time. So the other thing about Pinterest that was pretty different from Walmart is born in the cloud, born with an experimentation platform from the very beginning, where I was shoehorning in both the cloud and experimentation platforms into my other tech stack. So everybody uses, I mean, we use the experimentation platform to launch features, every single feature, every. Versus, you know, at Walmart, I had to ingest it into our search engine and all kinds of stuff that was built before there was experimentation. And so that part of it is amazing. We're one of, Amazon won't tell us if we're number one, but we're pretty close on AWS as number one customer. In the top five for sure, and probably the government agencies maybe are spending a little bit more than we are, but they won't tell us exactly. But yeah, we're spending hundreds of millions of dollars on AWS.

(Jeremy at 00:24:04) We just passed an exabyte, which I didn't want to. We were trying to prevent ourselves from growing our data stack that big. But, you know, it's a massive platform as well that we're having to manage. Right? So we do a lot of experiments that, you know, even we call it NUX, new user experience, whether you should have to have an account or not. But to your point, a lot of people share their boards with someone, their spouse or their architect or their designer or a friend. And it's hard to do any of that without one.

(Joel Beasley at 00:24:39) I got to talk a few months ago to the CTO of GoDaddy.

(Jeremy at 00:24:43) Oh, okay.

(Joel Beasley at 00:24:43) And one of the things I was super surprised about is the culture they have for experimentation. It's deeply embedded into their culture. They built their own systems to sort of, you know, how they can propose experiments, and then you run them and you watch them, and it's like one central system that will allow experiments to happen across their entire platform, all these different products that they build and own. So that's baked into Pinterest core culture too. Right?

(Jeremy at 00:25:07) Absolutely. Yeah. From the beginning. As a matter of fact, when I came here, one of the problems I had with it was, and this will sound very demeaning, but it was like our product strategy was often like, "We'll run these 100 experiments and see which one works." And I'm like, "That's not a product strategy. That's an experimentation strategy." So it's great to have it, but you need to be going somewhere. Like, "Hey, we're going this direction, and then here's the 100 experiments that are going to get us there." But yeah, it's very common to be running. I mean, we're running, we just upgraded our machine learning systems as well, and we can run like a thousand models now instead of like a hundred just to test which ones are, you know, which signals are better for relevance. And so we're just constantly improving that. So both on the machine learning side and on the client side, we're running experiments like crazy. We don't even think about it. Our, the other stat that I use a lot, which I was blown away by when I joined Pinterest, was we have an analytics system that consumes both our big data platform and some of our SQL data.

(Jeremy at 00:26:15) 60% of the company uses that on a monthly basis, like, six zero. Like, crazy. It's amazing how data oriented people are in the company about decision making. In my first few meetings when I joined Pinterest, there's this guy, Dan Lory. He's one of our best product managers. Somebody asked a question in a meeting. Literally, he opened his laptop, did a query, and gave the answer back. And, you know, that kind of thing, having real time analytics and having a company that answers questions with data, it's so, it's awesome to work at that, you know, versus the HIPPO, "This is the way we're going. I feel like we should go this way." It's literally a data oriented company.

(Jeremy at 00:26:57) And it's hard to understand how different it was from, for a company who just doesn't have access to the analytics they need in order to get answers to data in real time. It's amazing to work for a company like that.

(Joel Beasley at 00:27:08) I think this is a good time to plug the careers page. You guys have a careers page at Pinterest?

(Jeremy at 00:27:12) We do. Careers at Pinterest. Yeah. [email protected]. Yeah.

(Jeremy at 00:27:16) We're still hiring. We had, like many companies, we've been cautiously moving forward with hiring. So we cut back quite a few. We've been watching, even during the pandemic, our CFO, Todd, and I sat down and said, "Listen, we should open reqs like the beginning of every quarter. Let's see how the business is going and then open reqs." And, actually, it's one of the smartest things we could do because we were able to pick up some incredibly good talent from Airbnb and some of the people that got harder hit by the pandemic. And as a result, just, right, not only able to grow the team fast, but I got some incredibly strong talent. And while this one, it looks a little scarier from an overall economic standpoint, we're still taking advantage of trying to grab the best people we can.

(Joel Beasley at 00:28:03) Yeah. It's definitely a super interesting climate because I'm 34, so I started getting into business early. And so I lived through, in the business world, I had experienced going through the real estate crash. Right? And so when you were going through that, it was pretty clear. There was one thing happening, and that's it. Now there's like 18 things happening. Nobody agrees. They're debating whether a recession's possible, we're in one, or it's already happened.

(Joel Beasley at 00:28:34) And the way I responded as a CEO is I said, "What can we, like, let's look at our efficiency. Let's trace back. We're heavily calendar driven here." So you don't have to put exactly what you're doing, but just like the block of time. Like, "I'm spending time in this area here." That way, we can understand where time spent. And we just make sure that that connects, how far is that from revenue. Like, if we're doing something really far off, that's just kind of a fun thing that we're doing, you know, that might be something to cut. And we're just figuring out how to get lean so that regardless of what happens, the company, you know, you're always thinking about survival. Like, if you're a CEO and you're not thinking about survival, you're lying.

(Jeremy at 00:29:14) Yeah, absolutely. I think that's really smart. I think we talked about this even when I was at Walmart. I've been obsessed with engineering productivity for twenty years, and it's an incredibly hard thing to measure.

(Jeremy at 00:29:27) And I've had various forms of success of looking at things like NPS in combination with things like build times and number of pull requests, KLOC, and number of lines of code you're able to commit. But to your point, it does a whole bunch of things when you focus on this. Number one, you're increasing the productivity of your team. Awesome for the company. Awesome for everyone.

(Jeremy at 00:29:50) Number two, it makes the engineers incredibly happy. Right? Most engineers I know want to write code. They don't want to deal with some approval systems or security review or some compliance thing or a build that takes more than thirty minutes to run.

(Jeremy at 00:30:06) And so I've always seen much higher return on investment on productivity efforts for the engineers in particular that have, in these times, really good returns on investment. And even my product team here is 100% aligned with that. The amount of gains we've had both on engagement and revenue over the last three years that came from just ML, machine learning productivity efforts, is a lot higher than what we would have expected, let me put it that way. And so we just continue to invest, like, how to make the machine learning people more effective. Right? It returns every time.

(Joel Beasley at 00:30:47) And then when you see, you know, speaking of algorithms and machine learning, I was thinking when we were preparing for this call about TikTok. Because before we talked, TikTok was a thing. Kids used it. And in the past two years, past Google as one of the most visited sites. It's hugely addictive, and they've got their algorithm. But what I've noticed after using that, and then using Instagram Stories and then Shorts on YouTube, this is a trend that's going across all of these media companies. Have you guys done anything related to that?

(Jeremy at 00:31:21) Yeah. Your experience is what we call doom scrolling, and I think the internet is commonly using that. And, you know, we want people when they come to Pinterest, you know, and they use it for ten minutes or an hour or whatever to walk away more inspired and happy. And oftentimes, that's the opposite of what happens when you're doing it on a TikTok or one of these others. You walk away and you're like, oh my god, I just wasted an hour of my life.

(Jeremy at 00:31:45) So, yes, is the short answer to your question. Short form video is something we invested largely with creators and the ability to ingest not only videos to Pinterest like a pin, but the ability to do it in a way, sometimes we call it edutainment. So it's entertaining, but you're learning how to plant a garden. And, you know, most of the creators know that they can't just be fact people. They have to have a style and that's where you see all these cooking shows where the people are really skilled with their craft, but they're also really creative when they're talking about how to do substitutes and that sort of thing. So we look for creators that are in this sort of edutainment space that like the short form video. So we've been experimenting with both putting videos into your feed, which if you scroll through on mobile, you'll see it pretty often. But we also have another one we call the watch tab that have just video threads, and that's been going okay.

(Jeremy at 00:32:51) We see, not surprisingly, a lot of the early Pinterest users don't like the video format, and a lot of the new Pinterest users love the video format. So it's a little bit of a thing that we have to deal with on experimentation, which is always fun. But, yeah, and I don't think it's going away because I think the video, it's so much more engaging. And, like, you know, even when I was looking for, like, how do I repair the brakes on this particular car. You know, a series of images isn't going to work. You need some videos. Right? You know, you need some instructions on how to do that. And, you know, so oftentimes, when I think about our competitors, I actually am more nervous about YouTube than on some of these other platforms because that's a place where you go for this sort of medium term long format. You know, how do I build a cabinet or remodel my bathroom or replace the O-ring in my sink or something. Right? You know, these sort of things is what really engages really well on Pinterest.

(Joel Beasley at 00:33:51) Yeah. I discovered YouTube Shorts two months ago, and I spent twenty minutes watching it.

(Jeremy at 00:34:00) Yeah.

(Joel Beasley at 00:34:01) And it was just one of those rare times. You know, I've got a four year old, a three year old, and then one that'll be here this week.

(Jeremy at 00:34:06) So you got no time.

(Joel Beasley at 00:34:07) Yeah.

(Jeremy at 00:34:09) So—

(Joel Beasley at 00:34:09) When I say I spent fifteen, twenty minutes on something, that's all my free time. Right?

(Jeremy at 00:34:13) Yeah.

(Joel Beasley at 00:34:14) And I was like, I found it. I was like, this is so addictive because Google's known me for twenty years.

(Jeremy at 00:34:19) Yeah.

(Joel Beasley at 00:34:20) So it was so relevant to who I am, and it was so perfect. But when I found that format a couple months ago, I went to my production team, and I was like, this is addictive. This is fantastic. We need to do clips like this. When we condense that down and cut out to your most important sixty seconds and put some graphics in there that don't have to be crazy complicated graphics, just some words or whatnot, man, does that perform.

(Jeremy at 00:34:49) That's what we call idea pins. I know everybody calls it something different, but it's the same type of thing. You take a hero image, a short video, a list of ingredients, a technique on how to stir or sauté or whatever. And to your point, it's really engaging and really helpful. I mean, it's really helpful, right, in a very short period of time. And so I think that I think the format is here to stay. The, of course, to your point, there's stuff that gets lots of engagement that is irrelevant. And so we want to be the place where that doesn't happen. You know, we want to be super relevant and that sort of thing. So that's what I think about the challenge.

(Jeremy at 00:35:30) And it's a lot of the same computer vision and the content understanding problems that we have with images. You just have a bunch more images. Right? You take a segment shot of a video and you kind of understand what's going on with computer vision. And it's great. It's a rapidly adopted tech that we have to use. Right? And it's going really well. We've even experimented a little bit with live shopping as well on Pinterest.

(Joel Beasley at 00:36:01) What's that?

(Jeremy at 00:36:01) So very typical, almost like, you know, home shopping network style. Hey. I'm going to sell this item for twenty minutes, and I'll show you how it is and that sort of thing. It's not, you know, as you read about it around the internet, it doesn't really work in the United States. It's very popular in Asia, and it's a pretty interesting format, but it just hasn't taken off. And even some of the social networks have cut their live shopping stuff. So we've been playing around with it a little bit, but it's good for some products, but it just doesn't work for everything. So we'll see how that format. We're constantly experimenting with the new formats and capabilities to see what pinners love. Right?

(Joel Beasley at 00:36:38) Oh, yeah. That format's a no go for me. I saw Amazon did it, and they happened to do a live shopping one of something I was buying. So I clicked on it just to be like, what is this? Because you have to explore. Right?

(Jeremy at 00:36:49) Yeah.

(Joel Beasley at 00:36:49) And there was so much filler because they needed to reach this audience within this range that they weren't, it wasn't good. I don't know how to tell you it was bad. I just saw it, and I was like, this is not good. I can just go read the bullet points and the descriptions and read the reviews, and I can get the information I need to make this purchase without watching you try to engage an audience.

(Jeremy at 00:37:11) Some of them, you guys saw a bad one. Some of them are really good. The question is about reach and, you know, to your point, audience. Like, you know, we're making this video, and it's being seen by a few hundred or maybe even a couple of thousand people. But from a scale standpoint, that's not enough. Right? I mean, we're talking about reaching, you know, a video, our videos reach hundreds of millions people. Right? You know? So if you do one coffee show and you sold a thousand items. Yes. And we've seen live where people do it on Pinterest. You know? They increase their sales. That's great. You know? But is it really enough to warrant a whole format and everything? So we're playing around with it. We'll see how it goes. That format's good.

(Jeremy at 00:37:48) The other one that we're playing around with is AR, of course. So I talked a little bit about our tech there. We've done all the early easy stuff, which is like, hey, I want to see a chair or a couch in this room. I want to put a new piece of art on my wall, and I'm going to use my phone to see it. But the innovative stuff we've done there is we've done what we call AR try-ons. So you've seen this now. Some other companies have copied us. But you can use makeup and do hairstyles and all that kind of work. And we have L'Oréal and all the big beauty brands ingesting all their content so that you can try out and buy a lipstick.

(Jeremy at 00:38:24) And the great part about this is it's literally, if you use it, you're five times more likely to buy it. Right? Which is crazy. I mean, that kind of lift just doesn't exist on the internet. Right? And so these kind of technologies, when you're using and engaging with it, you know, you really know that you have a committed buyer to your point earlier. And that's one of the great parts about Pinterest. So we've done that with makeup lipstick, and we've now done, we call it inclusive search again, a place to be the safest place in the internet, but also the most inclusive place in the internet. You do a search for hairstyles. You can choose your skin tone. You can choose your hair pattern type. So you can, like, curly hair or straight hair or wavy hair and that sort of thing. And it just makes it so much easier for people to find people that look like them. You know? I want styles that look like this. Right?

(Jeremy at 00:39:10) And, you know, those are really hard engagement problems. Like, you know, if you don't have information about a person, I told you earlier, we're not spending a lot of time on personalization data or capturing this data. But if you specify and say, hey, I've got a lighter skin tone range or a darker skin tone range or I've got this wavy hair, then your pins just are magic. Right? They come out just perfect for you. Right? And it's a wonderful experience. And, again, all powered by the same computer vision team and technology. That is what's made Pinterest special for so many years.

(Joel Beasley at 00:39:42) Pinterest is how I found my beard.

(Jeremy at 00:39:45) There's tons of that. I, you want to put back to my point. I mean, everyone says, you know, hey, men don't use Pinterest, but they use it for fashion, they use it for DIY ideas, they use it for paint colors, they use it to find their beard and mustache styles. That's another thing about Pinterest is it's a really safe place to explore a new you, if you know what I mean. Like, people, I mean, you're not worried about being judged, and we hear that from creators too. Creators come to Pinterest and they create a piece of content on Pinterest, and they go to Instagram, they create another piece of content. It's like, people that use Pinterest don't hate on them. You know, they go publish this, like, your content sucks. It's terrible. Blah blah blah. You know? They come to Pinterest. They're like, hey. I like this. Well, you could think about that. You know? Like, lots of helpful people. And they just say, it's so much easier and nicer to build on Pinterest that they just love it.

(Jeremy at 00:40:42) Many have abandoned all the other platforms just to build on Pinterest because of the audience. And, again, that's not just because the people that use it. It's part of our technology stack to eliminate sort of troubles and all that stuff that we have to deal with. It's our problem. Hard tech problem.

(Joel Beasley at 00:40:50) Yeah. Well, you guys have always had, I think, one of the, I always notice brands that have really good design or at least software that has really good design because I've been building software my whole life. Right? And when I see, I was like, oh, they invest in design. And that was one of my early memories of Pinterest was—

(Jeremy at 00:41:09) Yeah.

(Joel Beasley at 00:41:09) It was really minimal, and it looked really nice. They did really great things with typography, which I love when people understand how to do that. But, yeah, that's one of my earliest memories of Pinterest. Because you come across stuff and you're like, that looks like it's going to be huge.

(Jeremy at 00:41:20) Yeah. It came from Evan, our cofounder, who is really a design expert. As a matter of fact, now that he's retired, he's been working with Jony Ive on his design studio. So, I mean, this is the kind of caliber of designers we have here. I talk about my team a lot, you know, the computer vision team and the machine learning team, but we have some of the world's best designers as well. We built, you know, it's been a design first company for a long time. So it's just a wonderful place to work. I mean, our product team has been a design centric machine learn, it's hard to find people that play both sides. Like, you know, we really know that, you know, oftentimes when I describe Pinterest to people, it's like, it's this app, this beautiful website that's showing you billions of beautiful images powered by the most unbelievable machine learning and graph database system that you ever would have thought of. Right?

(Jeremy at 00:42:09) And that's what Pinterest is. It's literally both sides. You know, I grew up, my father was an engineer and my mother is an artist, so it kind of makes sense that I ended up a place like this. But it is sort of the best of both worlds, like this beautiful place in the world powered by this incredible engine that makes it go. Right? Makes it scale globally. Right? I mean, 400 million people around the world. Right? So it's many, many languages, many different cultures. You know? I always love seeing Diwali ideas and things, celebrations I've never even heard of. There was a huge thread on, I think it was on Reddit talking about home designers in Nigeria, and Ben and I were just riffing on. Who would have known that, you know, home designers in Nigeria are going deep for Pinterest. Right? You know, it's just a wonderful thing.

(Joel Beasley at 00:42:57) That's cool. Yeah. It sounds super rewarding. I want to make sure that we watch your time here because I don't want you to be late to your next meeting.

(Jeremy at 00:43:04) Got it.

(Joel Beasley at 00:43:04) But as we wrap up, last question. What are you learning right now as a leader?

(Jeremy at 00:43:09) Oh, I'm learning how to run a large team when we never see each other. Right? It's, you know, from my time at Walmart, one of the things I did, I spent a ton of time with the team all over the world. And the pandemic's been tough. I mean, trying to run a large organization, you really have to triple down on your communications.

(Jeremy at 00:43:31) You have to really understand documentation and build a culture of investing in docs. You know, I was just talking to a woman who was a lawyer, and she was talking about young lawyers and how do they learn law. And, you know, in a law practice, when you're young, you're usually working fifteen hours, and you're working with clients, and you're working with twenty different clients, and you're randomly going to meetings to hear things. And this is how engineering is exactly. And so I've been very nervous about how do we bring on new engineers and ramp them at the same pace as we did pre-pandemic.

(Jeremy at 00:44:10) And I know the freedom that the work-from-home lifestyle has given people, especially for folks who have young kids and that sort of thing. It's been freeing and amazing. But I look back at my time when I had young kids your age. I was spending—I slept in the office sometimes. And I look back on that part of my career, not only the best time I had in my career from a people and accomplishment standpoint, but I learned more than I ever had. Right?

(Jeremy at 00:44:37) I'm just worried that people have forgotten that this is an important part of culture. Right? And how do you build that in an environment where you're completely remote? So that's the hardest thing to do in leadership right now, is to figure out how to balance the fully remote environment. Right?

(Jeremy at 00:44:54) We're not quite fully remote. We're coming in once every other week or once a month. Most teams come in about once a month. So people get to know each other, but it's still pretty different. Right?

(Jeremy at 00:45:03) You know, if you can't lean over to your buddy and say, what's that thing again that rebuilds the system when it crashes? You know, I'd say everything that can't be solved in Slack that isn't a half-hour meeting is like this no man's land. Like, the death of the five-minute meeting. I just need somebody for five minutes. You have to schedule it next Thursday for thirty minutes. Right? You know? It's like, ah. So productivity, I've been worried about productivity. I've been worried about onboarding.

(Jeremy at 00:45:30) And then just worried about, you know, how fast people grow in their careers in this situation.

(Joel Beasley at 00:45:36) Excellent. Well, thank you so much for sharing, man. It's always good to see you.

(Jeremy at 00:45:41) Yeah. It's great to see you. So much. Yeah.

(Joel Beasley at 00:45:43) Thank you so much for listening. And if you found this episode useful, please share it with a friend or colleague who you think would get value from it. And if you have topics that you'd like to hear discussed on the podcast, either add me on LinkedIn or send me an email, [email protected]. Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.