Lei Yang - VP of Engineering at Quora

(Joel Beasley at 00:00:00) Hello, my friends. Today we are talking to Lei, the VP of Engineering at Quora, and we discuss ways to evaluate and track developer velocity, why there's no substitute for experience when it comes to learning, and the benefits of quarterly hack weeks to drive collaboration and innovation. All of this right here, right now on the Modern CTO podcast. Here we go. This is the Modern CTO podcast.

(Joel Beasley at 00:00:36) Where are you at right now?

(Lei at 00:00:39) I am in this conference room at Quora headquarters in Mountain View.

(Joel Beasley at 00:00:47) Very cool. Is it nice out today?

(Lei at 00:00:50) It is. The weather is really nice. It's quite hot. But usually when it's hot outside, the office is freezing. So the room where I'm at is very comfortable and just the right temperature, so I feel good.

(Joel Beasley at 00:01:05) Nice. Yeah. Here in Florida, it rains like three times a day, and it feels like a tropical island in the summer.

(Lei at 00:01:14) Oh, wow.

(Joel Beasley at 00:01:15) Yeah. I just came back from lunch and we ran in and it's raining. It just randomly, it'll be sun shining and raining at the same time.

(Lei at 00:01:23) It's typical tropical weather, you know. I'm kind of jealous of that. It never rains in the summer in California.

(Joel Beasley at 00:01:31) Oh, does it?

(Lei at 00:01:32) No. California is like a desert. It really only rains in the winter.

(Joel Beasley at 00:01:38) So that's when the trees get fed in the winter?

(Lei at 00:01:41) Yes. Yeah. So in the summer, we usually have a drought condition. We've had a drought condition for like three years in a row or even longer. Yeah.

(Joel Beasley at 00:01:49) Were you born there?

(Lei at 00:01:51) No. I was brought up in China. So I went to college in China, and then I went to grad school in the U.S., and then immigrated over after I got my PhD degree.

(Joel Beasley at 00:02:07) Oh, where did you live in China?

(Lei at 00:02:09) My hometown is in the Midwest of China. I went to college in Beijing. But if I tell you where in China I grew up, you probably wouldn't know it.

(Joel Beasley at 00:02:21) My sister lives in Shenzhen.

(Lei at 00:02:24) Oh, she did. Yeah. Yeah. Shenzhen is a big city. It's in the south. Yeah.

(Joel Beasley at 00:02:29) Yeah.

(Lei at 00:02:29) Nice.

(Joel Beasley at 00:02:30) She's been there four years, got some jobs, taught English mostly, came back, went back, and now she kind of bounces around everywhere. And she really likes it. She's lived in a couple different spots and always live streams and shows me like the outside on Skype and stuff. It's pretty cool.

(Lei at 00:02:49) Yeah. Yeah. Shenzhen is a cool city.

(Joel Beasley at 00:02:52) So then you grew up in China, then you come over, you go to college, and then you just land at Quora or did Quora not exist then?

(Lei at 00:03:02) Quora didn't exist then. So I came to the U.S. I went to grad school at Northwestern University. I got my PhD there. And then when I graduated, that was 2008. So my first job was with Google. I spent a couple of years there. That's my first job. And then at Google, I started off as a software engineer, and then later I got into management. I managed and grew a few teams with different products. So I worked at Ads. I worked at Google+, the social network product by Google. And then I also worked at Search for a short period of time. And then, you know, eventually, by that time, I'd been at Google for seven years. I really felt like I reached a ceiling, and my growth kind of became stagnant. It just feels like I'm underutilized, and I really needed to learn more and grow more. So I decided to look for new opportunities, and I joined Quora back then. So that was about four years ago.

(Joel Beasley at 00:04:12) And that was the right move?

(Lei at 00:04:14) Definitely. Absolutely. I feel like that's probably the best career move I've had in my career.

(Joel Beasley at 00:04:20) You're growing a lot. You're learning a lot.

(Lei at 00:04:22) Yes. Yes. So yeah. Do you want me to expand on that?

(Joel Beasley at 00:04:27) Sure. Yeah.

(Lei at 00:04:28) Yeah. So, you know, at Google, even though I worked at different teams, I feel like I had the most fun and I grew the most when I worked on Google+. I know that, you know, from a product point of view, that was never a really successful product. It really didn't pick up the user traction. However, the team at Google was really strong. They really put together the best people to work on this. It was a huge bet for them. And so it was like a fast-moving startup within a huge company. So it's a really fresh feeling. Like, you know, we have a lot of agility, very experiment and data-driven, great culture, great people. I really enjoyed that. And then as the product started to go on the downfall, you know, there were organizational changes. My team got moved into Search, and that was a totally different culture and totally different feel. I don't like that. So when I decided to look for a new opportunity, I knew for sure that I wanted to go to a smaller company, fast-growing, and I wanted to work on a product that I feel proud of. So I chose Quora. So basically, I'm really attracted by the mission, and the company is still small. And, you know, in all my time here, I really feel like every couple of months, I get to wear a different hat and play a different role. So the growth is constant. I never felt like I'm comfortable and I'm good at my job. It's always like I'm doing something new and I'm learning, and that feels great.

(Joel Beasley at 00:06:03) Yeah. It's like you're competing against yourself to become better.

(Lei at 00:06:06) Yes. Yeah.

(Joel Beasley at 00:06:08) So right now at the organization, like, what are you learning right now? Like, what's the current lesson that's on the top of your mind?

(Lei at 00:06:17) The current lesson. I feel like, you know, in a role like this, there are just constant struggles, a lot of things, a lot of challenges, a lot of things that I should be learning every day. I think probably right now, the biggest lesson that I'm learning is the importance of building a strong leadership bench, being able to grow and hire great leaders for the company. I feel like that is probably the most important thing that I need to do in my role, and it's definitely not easy. So in both ways—hiring great leaders, experienced leaders, and integrating them successfully in your culture is definitely a challenge. You know, developing leaders and growing them is also challenging. So these are the things that I'm constantly learning and trying to do better at.

(Joel Beasley at 00:07:15) Have you—we obsess about this, by the way. I don't know if you know this about me. Do you know this about me or no?

(Lei at 00:07:23) No. Tell me about it.

(Joel Beasley at 00:07:24) Okay. So we have a leadership company for technologists. Like, we just completely obsess about how to create better technology leaders. And so we work with companies who do that all the time. But I'm curious though, this might help you. Have you worked with the learning and development function inside of Quora about this?

(Lei at 00:07:45) Oh, yes. Definitely. Yeah. I mean, you know what? I feel like when you're working in a place where there's some amount of structure and support—it's not like there is no training, no leadership development available at Quora—but I feel like in the end of the day, the thing that really moves the needle, the thing that really gets the leader better is not by going to some classes. That's my personal opinion.

(Joel Beasley at 00:08:14) That's our marketing.

(Lei at 00:08:15) You know, reading books or going to classes or going to seminars, going to meetups, listening to podcasts like this, they all help. And these things build up over time. It's not like one night is a complete flip over. At the same time, I also feel like, personally, what really gets the leader stronger is—I like this analogy of throwing them into the water and just forcing them to learn to swim. I feel like just learning from doing it and giving them big challenges, giving them opportunities to do new things and learn new things, it tends to be the more effective way rather than teaching them from some manual. You know? I tend not to believe in that 100%.

(Joel Beasley at 00:09:02) Yes. And here's exactly why. So you sound like our marketing material, by the way.

(Lei at 00:09:08) Really? Good.

(Joel Beasley at 00:09:09) For action-based leadership development. And you made the exact point. If you read a book, there's 80 great things that you could do in that book. All sorts of insight and things you could apply. But what happens is it's not in a format where you do it. So you just read the book. You're like, okay. That's great. You forget everything. You have this false sense that you improved because you just spent a couple hours reading a book. You go back into your team. You've actually taken no action in the real human world, and you haven't improved. So we started obsessing with, like, how do we get people to take an action? Like, don't go to a workshop and learn about giving credit or learn about individual ownership. Actually, go deploy the concept in your team, get dirty, get experience with it, and then now you'll remember that experience forever.

(Lei at 00:09:53) Mhmm.

(Joel Beasley at 00:09:54) Yeah.

(Lei at 00:09:54) Yeah. And you know what? Usually, I think what works for me is I read a book and I learn something about it. I was like, oh, yeah. This sounds very reasonable, and I will forget about it. And later on, when I get to experience a situation, that will start to resonate. I was like, oh, by the way, that thing that I learned before from that book actually makes sense. So I think you have to connect theory with practice. Right? So the action-based thing, you just, you know, totally makes sense.

(Joel Beasley at 00:10:23) Yeah. We figured out the formula for experience is knowledge plus action. So you've got to learn only enough to go take some action, and then you gain experience.

(Lei at 00:10:32) Okay.

(Joel Beasley at 00:10:33) You would like, um, so one of the principles that I built this on that I connect this with is my brother and my stepmom are both physicians, doctors. Right? And when they learn surgeries, they have this very simple step where they watch—they learn about it for a short period of time. They watch someone do it, and then they do it. And that's like the process. You will learn a surgery in like an afternoon. It's like a very simple process. And now these are high availability, like, there's lives in your hands. They're very complicated things, but the way you're learning it is super important because lives are at stake. Right?

(Lei at 00:11:13) Yes. Totally.

(Joel Beasley at 00:11:14) So I like your style. I really do. You're pretty cool.

(Lei at 00:11:18) Cool. Thank you.

(Joel Beasley at 00:11:22) So let's take it back to when you were a kid. I'm curious. What was your first interaction with technology? Like, how did you fall in love with technology?

(Lei at 00:11:33) I think it's going to be really hard for me to tell you when I started to fall in love with technology. I think as a kid, I've always been fascinated with science and technology. My childhood dream is to be an astronaut. Obviously, that did not come true.

(Joel Beasley at 00:11:48) Not yet. Not yet.

(Lei at 00:11:50) Not yet. I mean, I'm still young. I still have a chance. But I remember I like problem solving. Like, you know, I really like working on puzzles. I like disassembling and fixing small electronic devices, appliances at home. At school, I was always more interested, and I did better with science and technology classes. And then when I went to college, I chose to work on electrical engineering, which is more on the hardware side. After graduation, I went to grad school, and then I started working on and researching into computer engineering and then transitioning to software. So in my opinion, these are connected, and I've always just been really, really fascinated about the technology, this whole domain overall.

(Joel Beasley at 00:12:42) So you were way more attracted to the software than the hardware at first?

(Lei at 00:12:46) Oh, no. At first, I was—I mean, at first, when I went to college, to be completely honest, I didn't really know exactly what's the difference, you know, software, hardware. That was a long time ago. I just knew that I wanted to work on engineering and, you know, electrical engineering back then was very cool. I worked on signal processing, optical devices even. And then, you know, I got exposed to what was so-called embedded systems. So this is, I think, in like 2002 or something. Like, that's before all the mobile, like, smartphones. I remember, you know, working on design of a—they called it personal digital assistant, PDA. Yeah. It's like the early version of the iPad. We would have this little circuit board where we would be installing an operating system on it, and we'd be developing some applications and then try to make the whole system, software and hardware, work as a whole. And it was really cool work. That's 20 years ago. Back then, I don't think smartphone, mobile devices—exposure into building a device like that that could have a potential of changing the way people live their lives. So, you know, that really got me into computer engineering. That's why I decided to do research into it.

(Joel Beasley at 00:14:12) Oh, that's very cool. Yeah. You know what? They actually kind of stuck around a little bit in the medical field. Like, they went away in the business world really fast when smartphones came out. But I remember the early Palm Pilots and the Windows ones and all those. Those were so cool.

(Lei at 00:14:26) Yeah. Yeah. Yeah. Yeah. Definitely.

(Joel Beasley at 00:14:28) That's cool that you were working on that type of stuff.

(Lei at 00:14:31) Yes. Yes. Definitely. Yeah. I really enjoyed doing that when I was in college. It was fun.

(Joel Beasley at 00:14:36) So your engineering teams over there, how do you structure them? Do you structure them based on like product or, like, what's your methodology for structuring engineering teams?

(Lei at 00:14:47) Oh, sure. The engineering team at Quora. So we are right now around 120 people or so. The way we structure the engineering team is by function, if I could call it, or by skill sets. So essentially, you would group people together by the kind of work they do. And then we have four organizations at the high level. So there is, at the foundational level, the infrastructure team that handles, you know, cloud computing, network security, database storage, caching, all that stuff. And then on top of infrastructure, we have a team called Platform, and really the mission of the team is building developer tools to make developers' jobs easier. So they will be building tools, abstractions, systems that kind of just improve developer velocity. And then at the very top application layer, we have two teams. One of them focusing on machine learning and the other one focusing on product engineering. So they would be building the core product, and then they'll be building various features to improve user experience and make the product better. That's directly user-facing.

(Joel Beasley at 00:16:03) I think that's the most articulate response to me asking that question I've ever gotten. So you win that award.

(Lei at 00:16:09) Really? That's good to know. Yeah. Yeah. I've got to talk a lot about that when I interview candidates. They will ask me, like, hey, how are the teams structured? And then I would dive down into some of the teams they're more interested in. But thank you.

(Joel Beasley at 00:16:26) Yeah, I like how you described the team that helps build internal products that help with developer velocity and developer performance. Do you track developer performance? Have you seen those new tools out? I've seen one called Pinpoint. Have you seen those types of tools?

(Leigh at 00:16:43) Yes. I mean, there are definitely a lot of open source and third party tools that we use at Quora as well. We do build a lot of in-house systems and tools. You have two questions. One is, how do I feel about those tools out there? And then the second question is, how do we evaluate developer velocity. Right? Did I get it right?

(Joel Beasley at 00:17:05) No, you got it better than I said it. Yeah.

(Leigh at 00:17:08) Thank you. Well, I mean, yes. So we do use a combination of open source third party tools. We also build a lot of in-house systems. For example, as a company, Quora is heavily invested in machine learning technology. And in order for us to be able to iterate very fast using new machine learning models, ship new features, we also have to have very good tooling and platform support in order to plug and play. Right? So if there is some new research that people in a university or in other companies are finding great results for, then we would want to quickly experiment with them. But you want to have the system or the platform that enable you to do that very easily, that hooks things up with production. So our team would build that glue logic that enables you to use a model that Google develops and then apply that to our problem set and run experiments very quickly. So those are the tooling that are more specific to Quora that may not necessarily be supported by third party or open source.

(Joel Beasley at 00:18:15) That's interesting. I'm going to start asking other people, other tech leaders, if they're doing that or how much time they're putting into it, because that sounds good and it sounds like it should be happening everywhere.

(Leigh at 00:18:26) It should. You know, honestly, I think this is a really good advantage, a big advantage for us, because the fact that you can iterate quickly on technology would really influence everything. That has a huge impact. Now coming down to your next, the second question, do you still want me to answer that? How do we track developer velocity? You know, of course, Quora is in general a very data driven, metrics driven company. So we have a lot of metrics on the product, on people management as well. We have metrics for engineering—reliability, availability, speed, performance, all of that. I would say that, to be completely honest, the developer velocity is one of the things that is the hardest to evaluate using a metric. Right? So a lot of that is how people are feeling. How easy is it for them to do things. Now there are some basic metrics that we tend to look at just to get a sense of it. They're proxies. They're not perfect. But for instance, how often and how long do people spend on reacting to pager duties. Right? How often are they paged during on-call, and how easy is it to recover from issues? How fast is it to recover from a site downtime? Right? And then there are things also like the onboarding time. If you have the right amount of tooling and support, then it will be easier for someone to pick up things and then learn and be productive when they join the company. So we have some rough metrics that we could use to measure, but I wouldn't say that any of them is really the magic. Right? None of them is truly the magic. It's not as easy as uptime or reliability.

(Joel Beasley at 00:20:18) The coolest one I've seen—I was talking with Archana, who's the CIO over at Atlassian, and they're actually out in your area too. I'm sure you're familiar with them. And we were talking about Pinpoint and some other ones, and I actually did a demo with this company because when a new whole category emerges they start making all these claims about engineering performance management. I was like, I should check it out. I should explore it just to see what I'll learn. And they're actually pretty fascinating. I won't bore you with all the details about how they do it, but I think that it would at least be something you should take a look at. Because I think two things would come of it. One, you would learn something to improve your internal tools, or two, you would learn something and then probably help them in that call by sharing what your experience is with that whole pager duty thing. And I just say that because I like to help the whole industry move forward. Right? And when I see that new category come out, I'm like, oh, it was growing really fast. And I was just like, it takes a lot to impress me. I've been writing code for seventeen years. It takes quite a bit to get me excited about something. And when I saw that and watched what they had done and how they connect all these different dots to ultimately put a really solid metric on performance management, I was super impressed.

(Leigh at 00:21:44) Mhmm.

(Joel Beasley at 00:21:45) So I like to pass things along that I find that are good.

(Leigh at 00:21:48) That's very cool. You have to send me some reference links after this. I'd like to look into it.

(Joel Beasley at 00:21:52) Because then you'll like me. If you pass good information along, then people are like, oh, I like that person. Right?

(Leigh at 00:21:59) Yeah. Yeah. You're right. I mean, there's definitely a lot of value to kind of push the whole community forward. Right? Yeah.

(Joel Beasley at 00:22:07) So you mentioned the interviews. People ask you a lot about that in interviews. What are you looking for when you're bringing on people to your team?

(Leigh at 00:22:15) Oh, that's a good question. Well, in any role, when we conduct interviews, it's usually several rounds. Right? So you would be evaluating the candidate from different dimensions. Of course, within engineering, some of the most important dimensions are technical skills, right, coding, problem solving. When you're working with the code base, just how skilled are you navigating through and solving problems and debugging. So definitely a lot of technical interviews that cover those dimensions. The interviews that I usually conduct and the interviews that I usually value the most is the culture signal interview. So I'll be spending a lot of time just trying to ask questions to the candidates and have a good sense of how well they will fit in the Quora values. I mean, there's not really good and bad. It's really fit or not fit. Right? So we do, at Quora, we do have a really strong value system that is clearly articulated. There are certain traits that we are exclusively looking for from candidates. For instance, one of those things is mission first. So I think this really distinguishes all people. Right? So a lot of people that I've worked with at Google, like a huge company, I wouldn't say that they are mission driven. I think a lot of them are career driven. Rightfully so. Everybody cares about their career. But it means differently when you apply that into a small startup company that is trying to succeed. So we really need to look for people who are here to make the company successful. And that means that you have to make trade-offs, maybe sacrifice your personal interest, and that you have to be a really strong team player. Right? So things like this, just, you know, just circling back to your question, I think the things that I usually spend the most time on and evaluate the most from candidates is their culture fit. And in general, kind of, in addition to how they fit into our culture, also some norms, like, is this a reliable person? So a lot of things on the soft skill side.

(Joel Beasley at 00:24:40) And are you local, remote? How is your engineering spread out?

(Leigh at 00:24:46) Oh, yes. So all of Quora is really in two buildings that's right across the street in downtown Mountain View in California. So we are completely local. We don't have a remote office yet. Although I think as the company scales and grows, I can see there's a tipping point where we would want to have a remote office. And, you know, in the end of the day, you have to go where the talents are. Yeah. And just that right now, with our size and stage, and the product development is so tightly integrated, and there's a lot of benefit for us to be all in the same place, and then you can talk to each other face to face all the time. It just makes things so much easier.

(Joel Beasley at 00:25:27) It does. It really does. There's nothing can replace the in-person communication.

(Leigh at 00:25:32) Yep.

(Joel Beasley at 00:25:32) And so what is the mission at Quora?

(Leigh at 00:25:36) The mission is to share and grow the world's knowledge.

(Joel Beasley at 00:25:39) Yeah.

(Leigh at 00:25:39) And you are a Quora user?

(Joel Beasley at 00:25:42) Yes, I am.

(Leigh at 00:25:43) Yes. Great. Great. So, you know, what I'm proud of that mission is Quora was founded ten years ago, and we've never changed our mission. We've always been marching through on that mission, and we're making a lot of good progress on that. Today, I think Quora, in my opinion, is the number one question and answer platform in the US. We're expanding internationally, and we are already in a lot of international countries and languages. So a lot of users around the world are using the product to share knowledge.

(Joel Beasley at 00:26:22) I love it. Yeah. I don't think you can exist and not know what Quora is, because you come up for all the questions. Half the questions I ask Google is a Quora result. So you have to be a user. You're like, I've got to know the answer to this. It looks so good. And then you read a couple answers, and then you just become a fan. You're like, okay. A couple times, I actually just bypassed Google and just went to Quora. And I'm like, let's just ask the question there. Let's just search it from there.

(Leigh at 00:26:51) Yeah. Yeah. That's the goal too. I think eventually you want people to kind of just think of Quora as the destination to get their questions answered. A lot of people already use Quora like that. And it's, you know, sometimes I also feel like it's not just as simple as I have a question that I want to get an answer for. Myself and a lot of other users use Quora in a way that, casually, I'm just opening up the Quora app on my iPhone, and then I will be reading through lots of fascinating stuff that people are talking about. Those are the questions that I wouldn't think of asking, but those are the questions that are really, really interesting. A lot of people are sharing really interesting thoughts and views. It's not just about the—I think the human knowledge is different from an encyclopedia. It's not just facts. It's also about how people react and feel and think about different situations and then they share their experiences. So that to me is something that I don't think any of the product out there can replace the value of Quora.

(Joel Beasley at 00:27:58) I fully agree. I've used it before for market research, for lack of a better term. I want to understand this problem. Is this problem existing? What are people saying about this? How are they currently solving it? Is there a gap there? And it actually helps because we'll have ideas, and we can explore them through asking the right questions, which is a super important thing.

(Leigh at 00:28:20) Mhmm. Yep.

(Joel Beasley at 00:28:22) So as we wrap up here, I want to know what are you most excited about today at Quora?

(Leigh at 00:28:28) Oh, today?

(Joel Beasley at 00:28:29) Yeah.

(Leigh at 00:28:31) I have a great answer to that. Okay. So this week happens to be a hack week for Quora. So internally, we call that the Off Road. The idea is you could spend a week working on quick ideas that have a potential to make the product better. And then you would work on that in this week, build some prototype, you know, get some early ideas. And then eventually, after the end of this week, if you can prove that it is impactful, you can actually upload it and then make this official project that will be launching to users. So, you know, as I just walk around the conference room, I can see a lot of people kind of working on their hack week project. And the nice thing about Off Road is, because you're not working on your normal day to day job, you get to work with other people. So I see a lot of people forming teams and groups, cross-functionally. Engineers working with designers and data scientists, you know, and even maybe writer relations. Everybody forming new ideas. So this is really where I think a lot of creativity is happening. And this week is definitely one of the most exciting times for Quora. And so that's the most exciting thing for me today.

(Joel Beasley at 00:29:50) That is so cool. I love that you do that. Is this something that you do annually?

(Leigh at 00:29:54) We do that three times in a year.

(Joel Beasley at 00:29:56) You do it three times a year?

(Leigh at 00:29:57) Yes. Almost quarterly, except that usually in the last quarter, in the wintertime because of the holiday, the quarter is already very short. So we tend not to do that that quarter. But otherwise, we do that every three months, basically.

(Joel Beasley at 00:30:14) This is actually really cool. It sounds like it gets them engaged, something to look forward to, something unique and interesting.

(Leigh at 00:30:22) Yes. And I really think it is also helpful kind of to train the muscle of people to be creative and innovative, because it's so easy to be lost in a day to day execution. And sometimes you really have to think outside of the box and to exercise your creativity. We definitely hire the best people out there. And, you know, when you give them the right motivation and empower them, they can do amazing things. And I think Off Road is one of the opportunities where we can show how much potential they have.

(Joel Beasley at 00:30:57) I love it. I love your passion too and how you light up when you're talking about it. You are definitely—what you love in life is 100% aligned with what you're doing there and the work that you're doing.

(Leigh at 00:31:09) I believe so. Yeah. And that makes me happy.

(Joel Beasley at 00:31:12) Me too. It makes me happy too. Just trying to do podcasts, it just makes me happy now too. Thank you so much. We did it. We made a podcast. We did the first podcast for you.

(Leigh at 00:31:23) Yes. Yes. That's awesome.

(Joel Beasley at 00:31:26) Right? It was fun. Right?

(Leigh at 00:31:28) Yes. It is very fun, and you made it very easy for me too.

(Joel Beasley at 00:31:31) Oh, thank you so much for the compliment.

(Leigh at 00:31:33) Thank you, Joel.

(Joel Beasley at 00:31:34) See you.

(Leigh at 00:31:35) Thank you. Bye.