Get Good at Delivering Software with Dave Mangot, Principal at Mangoteque

(Intro Narrator at 00:00:03) Hello, my friends. Today Joel is talking to Dave, principal at Mangoteque. And they discuss how to get good at delivering software, how CTOs can step back to let the CEO decide what problems need solving, and why you might want to reevaluate what KPIs you use to measure engineering productivity. All of this right here, right now, on the Modern CTO podcast.

(Joel Beasley at 00:00:32) Here we go. This is the Modern CTO podcast.

(Dave at 00:00:44) So I

(Joel Beasley at 00:00:45) was hoping to get started to sort of introduce you to the audience. Could you give me a quick 30,000-foot backstory about your career and what you're interested in?

(Dave at 00:00:56) Yeah. I moved to California in the sort of mid-to-late nineties and had no idea that there was this dot-com boom thing happening. All I knew is I wanted to move to San Francisco, and somebody said, "Hey, wow, you know some things about computers?" And I was like, "Yeah, I've been writing code and getting paid for it for a while." And they're like, "Here's a job." And that was 1996. And so I've sort of been involved in Silicon Valley and startups and big companies and medium companies and all that other stuff for more than 25 years at this point.

Last couple of jobs: I was an architect in infrastructure engineering at Salesforce, designed a lot of the way that Salesforce runs, also led the DevOps transformation there, which was sort of right around the time where DevOps was actually becoming a thing. So pretty soon after AllSpaw and then the Flickr speech at Velocity, and pretty soon after Patrick and Andrew had their Agile Infrastructure thing. And then went on from there to run the global SRE group for the SolarWinds cloud companies—companies like Pingdom and Papertrail and stuff like that. And then about three years ago or so, I started my own business, working now primarily with private equity portfolio companies to help them maximize their growth using their technology organization during what they call the holding period, which is the time that the private equity firms actually own the company.

And it's for me, it's just been awesome because I get to do all the transformation kind of work, making engineers' lives better, which is kind of really why I got into this whole thing. But I get to do that as a job. I don't do that just like, "Hey, I work for Joel's company and I have this job and this." I get to do that with all kinds of different companies. So it's actually really fun.

(Joel Beasley at 00:03:12) Yeah. Sounds super interesting. For a while, I was working with some venture capital companies, and when they would make investments into some of their portfolio companies, sometimes they had just scotch tape and bubble gum—and they knew it. And so they would connect with me and say, "Hey, can you help them build an engineering org?" And so I got some experience doing that for a while. But then I learned about private equity, and a lot of people will use them interchangeably—venture capital and private equity—but they're fairly different.

(Dave at 00:03:41) Oh, yeah.

(Joel Beasley at 00:03:42) And when you say holding period, that's probably the time that the private equity firm has purchased the company from the entrepreneur, typically the founder or another PE firm. And until—what do they do with it after, though? What comes after the holding period? I don't know that.

(Dave at 00:03:58) Yeah. So the holding period would be broken into sort of three sections, you would call it. So the first one you would think of is planting. So that's when they're going to invest all the money. We want to improve the engineering organization's output, which is where I come in. We want to fix up the financials. We want to do all that stuff that happens in the first—and this is not for every PE firm, don't get me wrong. There's a certain style of PE firm that I like to work with. So that happens maybe in the first year or two.

And then really the middle part of that period is just growth. That's what they want—they want growth. And it's funny, we're on a CTO podcast, right? And I talk to the CTOs and they're like, "Well, how do we save money?" And I'm like, "No. We're not—it's not what we're doing here." They're like, "What? No. But my whole career, everyone's told me I have to save money on everything." And I'm like, "Yeah. No. The saving money is good and it's important and all that other stuff. But we're going for growth. You have a product that people care about. That's why they're investing in you. It's not like venture capital. No one's looking for product-market fit. Product-market fit is established. You have a valuable product and somebody wants you to grow the crap out of it."

So that's sort of where—that's why I'm in sort of the planting phase, to make sure that the growth stuff happens. And then there's the harvesting, which is basically exactly like you said. You sell to another PE firm, you go public, you do whatever the exits are where the firm gets the benefits of what they invested. And so, yeah, does that mean the things I typically work with tend to be really messy? Yes. Is that what makes it fun? Yep. That's the best part. I do work with startups in VC a little bit, not very often, but you've got a 60-person team and two people are having trouble figuring out how to get something done. You're like, "Hey, you should try talking to them about it." And then things get resolved. But when you have a thousand engineers, it gets a little bit more complicated. So, again, I think it's so fun. I love getting involved with that stuff.

(Joel Beasley at 00:06:28) Yeah. I've just been learning about it primarily the past five years. I kept seeing everybody do really well with money, and so I said, "I need to surround myself with some really smart money people." I had built a financial software app, and then I met a couple investors from Wharton Business School. And they were older and they were looking to—they needed my help with some technology. And so I said, "Hey, can I hang out with you guys and spend time with you and learn about how this stuff works so I could develop some sort of mental picture?"

And what I got from the whole experience was there's various levels of capital in the world. And so there's a whole group of people who are just making seed investments. There's a whole group of people who are just making growth investments, wanting the PE exit or the public exit. Then there's the lower-level PE that's trying to buy things that are $5 million, maybe doing M&A and then packaging it up for 25 and then selling it up to this other group, different capital level up there. And they only deal on $50 million-plus stuff. And I was just like, "Whoa." It was so mind-blowing, because you don't know that. No one teaches you that in school. And unless—you don't need to know that as a CTO, but to understand how that field is laid out is crazy.

(Dave at 00:07:42) Yeah. I kind of went on the exact same journey as you, so I can definitely relate to it. Sometimes I'm just like, "Hey, you just learned what capitalism is." That's really fascinating because, you know, we're engineers, right? Like you said, nobody ever taught us that. I think the important thing or the good thing or whatever is that the CTOs who understand that are the ones who are really going to do well.

And so, you know, I'm in some discussion groups with engineers and stuff like that. And they're like, "I can't"—you probably had this already on your show, but there's a whole discussion about what technical debt is. And then people get confused because when you go to a business person and you say there's all this debt, they're like, "Great, let's package it up and sell it." And you're like, "No." And it's like, "No, no, no, but no." And they think about things in a different way. And so they're like—one of the things we talk about in engineering sometimes is if you want to get something accomplished, bring a finance person with you, bring someone from the CFO's office, because they're talking that language that the business people understand. And then you go and say "technical debt," their mind goes immediately somewhere that's not even close to what you intended.

And so I think it's really important. One of the things I love the most about the DevOps movement is that it really helps to tie the engineering that we do to business outcomes. And I think that's sort of exactly what a CTO is in a lot of respects, right? They're representing what engineering is to the business because it's about business outcomes. And so I always tell my clients, you know, they're like, "Well, what should we be working on next?" I'm like, "Work on what is the most important thing for the business." And they're like, "What?" And I'm like, "I don't know anyone who's ever gotten fired for spending all their time working on the things that are most important to the business. So this is perfect job security. Just keep working on the things that are most important to the business."

And when I was young in my career, I was like, "Oh, bigger hardware." Because, you know, I was a Sun Solaris administrator and I got super excited about, "Oh, bigger E450s, E10Ks." They had even bigger numbers, but it's not why we're at work. We're not at work to play with bigger hardware. We're at work to achieve business outcomes. And I think that the DevOps movement and the State of DevOps reports and the Accelerate book and all that stuff really is starting to reinforce that with people—that that's why we're here. We're here to solve customer problems.

(Joel Beasley at 00:10:27) Absolutely. Yeah. I've seen people get that advice of, "Work on the hardest problem." And then what they do is they'll go try to figure out what the hardest problem is by their own reasoning. And I'm always like, "Nope, that's what your CEO is for." Go to your CEO and ask them, because the most important problem isn't the subjective thing you're trying to seek out and understand. It is whatever the leadership of the company has determined is the most important item. So just ask, you know?

(Dave at 00:10:58) Yeah. I often tell the story of this guy back in 2008 who was a network administrator for the city of San Francisco. And they told him to give the Cisco enable password—for people who are of that generation, I guess, which is the administrator password for the network—to these consultants. And he was like, "No." And they were like, "What?" And he's like, "No, I won't give it to them." And you're like, "What?" And they eventually wound up throwing this guy in jail, which is crazy, right, to begin with.

But so Gavin Newsom, who's now the governor of California, was the mayor of San Francisco at the time, had to go and negotiate with this guy to give up the enable password so he could get out of jail. And to me, it's like the most DevOps story ever. It's like this guy thought that the reason that he goes to work is to work on technical challenges, not understanding that the reason that you go to work is to serve the people of the city of San Francisco in the best way possible. And to your point, it's the CEO or the mayor or whoever who's figuring that out. You have a window into the technical part of it, which is super cool—don't get me wrong—but that's not how we figure out what we're trying to get done overall. So, yeah, it's pretty crazy.

(Joel Beasley at 00:12:23) Yes. Now, are you, when you're going in and working with these companies, are you helping them with all things? Are you helping them make this transition of understanding—for what you said earlier about saving money, most people, "Here's my budget, I've got to cut the budget, got to shave the budget." They just have done that their whole lives. And their personal finances, they've done that. And they do that at work to some degree. So it's just this drilled-in muscle memory thing.

The big change for me happened when I started listening to people like Robert Kiyosaki or Grant Cardone or just some of the billionaires like Warren Buffett or Elon Musk. And they gave me this perspective that just cleared everything out. And it was just this concept of capital allocation. It's like the people that have the biggest companies, they're the best at capital allocation. And so when I started to run my business from that perspective, everything changed. I wasn't worried about, "Okay, this one subscription, that's $50 a month," I'm not spending, you know, whatever, trying to figure that out. I'm just constantly focused on where does the capital that we have need to go. And often it's to growing sales and then making the product better. I mean, that, for me, that's what it is. It's like, "Let's get more sales and let's make the product better." And whenever I'm confused, I just go to one of those things.

(Dave at 00:13:41) Yeah. So I really only work with engineering parts of the organization. I do work with the CIOs or the CEOs, certainly. But for me, you know, I wrote an article for CIO.com called "Get Good at Delivering Software." And really it's got a lot of those elements from the Lean Startup and things like that, which is basically, hey, especially for the kind of companies I work with, you deploy four times a year, right? That means you can run four experiments a year as to what your customers want, stuff like that. Joel's company deploys every day. That means at a minimum, probably 20 experiments a month. You're deploying four times a year. Joel's deploying 20 times a month. He can run 20 experiments. You can run four. Who's going to win that one?

And really, you know, it's not fair. It's asymmetric warfare. Joel's company is going to completely destroy them because you can do all kinds of fancy things. So what I work with these companies on is getting good at delivering software. And the great thing about things like Accelerate and the 2019 State of DevOps report was they determined that companies that are high performers are twice as likely to meet or exceed their organization's performance goals. So those are things like customer satisfaction or revenue or whatever. And so what I'm working with these companies on is a lot of the stuff from those reports. But basically, this idea of if you can deliver with speed and quality consistently, that's where you're going to get your growth. And that's what the PE firms want—they want growth. And you're going to get that growth because you're literally just going to outcompete everybody else in the marketplace.

(Joel Beasley at 00:15:39) So how do you do that? How do you deliver software better?

(Dave at 00:15:43) So, you know, lots of different things come into it. Sort of the microcosm of all that is we do an assessment, which we don't do for everybody, but we do it for a lot of both portfolio companies and just non-portfolio companies as well. But it's based on the State of DevOps report, which I know you know about. But there were the four KPIs in the State of DevOps report, which are deployment frequency, lead time for changes, change failure rate, and time to recover—and they call it "mean time to recover," and if you and I want to go off on a fun tangent, we can keep talking about that.

But basically, there's a lot of companies out there that'll measure those things or whatever, but when I created the assessment during lockdown—because, you know, needed some fun to do—what I wanted to set out to do was really create something that talked about people's capabilities to deliver things. And I didn't really get all that worried about the actual numbers, because you can deploy 500 times a day. Is that way better than deploying 300 times a day? It's just kind of meaningless at that point.

(Dave at 00:16:58) Right? And so there's this law called Goodhart's Law, which is when a measure becomes a target, it ceases to become a good measure. I didn't really like measuring deployment frequency or any of that stuff. What I want is people to get good at those things. And so, you know, I've checked out a whole bunch of the Modern CTO podcasts and you've had people come on and talk about resilience engineering.

(Dave at 00:17:23) You've had people come on and talk about feature flagging, like all these kinds of things. And those are things that are actually really important to being a high performer. So we ask people, you know, questions about what happens from when they check in code until it's in production. We ask about same things for infrastructure. We ask about incident response.

(Dave at 00:17:44) And then we talk to people about like, hey, you know what? If you, let's use the example of feature flags. You know, if you were to implement feature flagging, that would really help with your four DORA metrics, whatever. Not that I really care what that number is, but we already know that those metrics or those measures are what's indicative of a high-performing engineering culture. And because, like, let's say I was to focus on just lead time for changes and I knocked it out of the park and everything else was garbage. That's not a good engineering culture. So those things are just a proxy for what's a good culture, and we try to help people get there. And yeah.

(Dave at 00:18:30) And feature flags, you know, I wind up doing... We were doing Evolve last year.

(Joel Beasley at 00:18:35) Was that the LaunchDarkly episode?

(Dave at 00:18:37) Yeah, I had that. That was really interesting. But, like, we did them all last year, so I did some analysis at the end of the year to see what we learned. And it turns out that in the highest performers and in the lowest performers, we would tell people about feature flags because they are that important.

(Dave at 00:18:56) Right? So if I'm using feature flags, I can deploy more frequently because I can turn it off when I deploy. You know? And then I can dial it up to 1% where my lead time for changes is gonna go down because it's just less risky to put things out there. And, you know, if I have a failure, I can recover faster because, you probably know, like, Etsy used to deploy everything under a feature flag so that if an SRE got woken up in the middle of the night, they just turn the thing off.

(Dave at 00:19:26) Boom. Problem goes away. You know? And so it turns out that that's like a huge thing, and that factors into a lot of these different measures. Whereas, you know, other things that we might recommend, like, you know, having really good documentation about your incident response procedure, it's not gonna affect your deployment frequency or your lead time quite as much as it will certainly your change failure rate or your time to recover or things like that.

(Joel Beasley at 00:19:59) Like at the CTO Summit or something? Where did you give that talk at?

(Dave at 00:20:03) Yeah, it was a private equity firm CTO Summit. It turns out that a lot of these firms have these types of things where they wanna get people in their portfolio together, both to talk to each other and learn from each other, but also to bring in other people that can help them, you know, take a step forward, whether that's cybersecurity or whatever, all kinds of different topics.

(Joel Beasley at 00:20:27) So when you give that talk, there's so many things to focus on. Everybody has their own unique context and issues that they're bringing in. So they're all seen as slightly different. How do you structure that talk? What's like the three or four takeaways I would get from listening to that talk?

(Dave at 00:20:42) You know, for me, for that particular talk, I was basically saying, you know, focus on creating the environment that enables the good scores on those four KPIs like we talked about: deployment frequency, lead time for changes, change failure rate, and time to recover. And don't obsess about the actual number itself. Like, that's not really the important thing because, you know, people will go off and... I think Deming had a quote something to the effect of like, give a man a target and he will achieve that target even if it drives the company out of business. And so you don't wanna focus so much on the actual number, but you wanna focus on the capabilities. And for them, I went over what the things that people did best were, you know, over the last year and what are the things that people struggle with the most.

(Dave at 00:21:40) And, you know, and why? Like, not why did they struggle so much, but why are these things important? And so, like, for example, the thing that everybody did the best on was storing their software in a revision control system. And that sounds, you know, pretty basic, right?

(Dave at 00:21:59) And you would hope that everybody would do well on that. And they did, which is good, you know, because a few years ago, people had stuff on their desktop and they were, like, tarring it up and SCPing it somewhere. And, you know, we don't wanna deploy software like that. And the second thing that people did best on was building code into artifacts automatically, like when you commit to code. And it sounds funny to say it out loud, but, like, you're talking about a piece of software that started out in 2007 and is this big gigantic monolith.

(Dave at 00:22:32) You know, some of those things aren't actually all that easy to do, like to build all that stuff. And that's what we call modern software development. But the reason that those things are good, right, is because we can deliver software quickly, and we can deliver software with high quality. But those things are enabling that kind of thing. Right?

(Dave at 00:22:53) If I can't check into revision control, then I can't get Joel to do a code review for me. So that's gonna automatically reduce the quality of my code. And I can't build the artifacts. That means I can't test things. And I mean, there's all kinds of knock-on effects.

(Dave at 00:23:09) And so for this talk, I talked about what the top hits were, what the top misses were. And then we went through a couple of examples of companies that we assessed over the last year, sort of what the scores were that they got and, you know, what are the things that they did well and what are the things they didn't do well. And I was joking with the CTOs. I'm like, hey, you're all consultants now. Because I was giving them, like, sort of like, hey, this was an add-on that we were doing onto a bigger company.

(Dave at 00:23:39) They were a small startup. They were very agile. They were in, you know, this space. They had these characteristics. So, you know, it gave people, to your point, like everyone's bringing their own sort of their own biases to it, I guess, or their own background.

(Dave at 00:23:53) It gave them a little background, a little context on what they were looking at. And then they could start thinking like, yeah, we kinda have that problem too, you know? And just really trying to get them to understand because, you know, it's not a fault of anybody. It's just CTOs generally have a really great background in software development, not as strong a background in software delivery, especially if, you know, they're the right age. Like they grew up at a time where everyone threw the software over what Andrew Shafer calls the wall of confusion. Like, I write it, you run it. And that's sort of... They never got to sit in those environments and really see how it behaves and all the other stuff. So I'm trying to sort of help them understand like, hey, this is kind of what it looks like when we're delivering quickly.

(Dave at 00:24:45) And, yeah, I mean, a lot of startups just get this off the bat because they're born in the cloud. Hey, I'm born in the cloud. I got all this stuff, blah, blah, blah. But not every company is like that.

(Joel Beasley at 00:24:57) I didn't understand artifacts. I haven't heard that before.

(Dave at 00:25:01) Just like, you know, any kind of thing that's shippable. So like a Docker container would be an artifact.

(Joel Beasley at 00:25:07) Oh, anything that's shippable?

(Dave at 00:25:09) You know, any package. Like, you know, we used to talk about like when we deployed JAR files with Java, we're like, hey, should we put that in a Docker container? And we're like, well, it's already in a container. And so putting it in Docker is like, yeah, there's a lot of advantages in terms of the ecosystem, but there's also a pretty good ecosystem already around Java. Like, it's not a brand new whiz-bang, whatever. So, you know, over the years when you've been doing this as long as I have, like there's Debian files and RPMs and Chocolatey things for Windows, and now there's containers and then there's JAR files.

(Dave at 00:25:51) And like, I don't really care what the actual thing is that you're shipping, but, you know, you wanna ship an artifact in whatever form that artifact happens to take, because that's something that you can transport around and use, and just makes things a heck of a lot faster. I was telling people I used to write a lot of Puppet back in the day. And one of the things I love about Puppet is it made you do things the right way, which is sort of what I'm trying to encourage people to kind of do now. But it wouldn't like... You could do the thing that we all used to do back then, which was dot slash configure, make, make install, right?

(Dave at 00:26:33) That's how you deploy software. You compile it, you install it. And Puppet was like, yeah, if you wanna do this, go ahead, but it's gonna be hard. And they were like, instead, you know, create an RPM repos, a YUM repository for RPMs, or create a Debian APT repository, whatever, and just deploy a package out of there.

(Dave at 00:26:53) And it was just so easy when you followed the correct patterns. And I loved that it was opinionated in a way that made you do really good engineering. So, like, you know, an artifact for me is just any kind of thing that I can ship that's not, you know, here's the source code. I'm gonna compile it like this. You know, those days are long gone.

(Dave at 00:27:17) Hopefully. Hopefully. There's people who still do that and, you know, they should call me, I guess.

(Joel Beasley at 00:27:23) I've talked to companies that all they do is maintain massive facilities of these, like, old mainframe type things.

(Dave at 00:27:32) Yeah. There's some really interesting stuff out there about, like, how to do DevOps on mainframes, which I have to admit is not something I would like to jump into. But, yeah, it's really fascinating stuff. I mean, I think, you know, the classic one for the DevOps movement is Gary Gruver's HP Jetdirect cards where they were actually doing DevOps-type, you know, continuous delivery kinds of things on these hardware cards. And people are just like, what?

(Dave at 00:28:08) But, yeah, mainframe's probably a whole other level of interesting that I may not want to get involved in.

(Joel Beasley at 00:28:16) What other struggles did you see? Like, what were the top two or three struggles that you saw from your most recent talk?

(Dave at 00:28:24) I mean, some of the stuff is things that you would expect, right? Deployment, like, being deployed on demand by developers without a ton of intermediate steps or whatever. People who are having trouble, like, they have this whole thing where it goes to this... And then the QA department gets it. And then they run off and do a bunch of manual testing, and then they bless it or whatever.

(Dave at 00:28:49) And then it goes to a change advisory board, and you have to write all this documentation for the change advisory board for them to allow you to release it in production. You know? And it's like, look. That's finished. Like, we're not doing that.

(Dave at 00:29:04) And, you know, we can dig into a lot of that, but—

(Joel Beasley at 00:29:08) What's the—

(Dave at 00:29:08) One of the things I learned the most at Salesforce was the difference between QA and QE and how quality engineering is actually a real discipline. That's not any of the stuff that we've all thought of as QA over the years, which is, you know, their big focus is on automating this stuff. And they cover that in, you know, Humble and Farley's Continuous Delivery book about automating your tests and all that stuff. And then the change advisory boards, you know, one of my favorite things of the past few years was Dr. Nicole Forsgren, who was the lead author on the Accelerate book, stood up at a DevOps Enterprise Summit and said, change advisory boards are worthless.

(Dave at 00:29:50) And everybody in the audience was like, you know... And, but you know, and this is something that when we do assessments and we see people that have change advisory boards, you're like, you gotta get rid of that because, you know, I've been that guy. I've gone to the change advisory board. And this is, you know, I'm gonna pick something that's super old so that nobody feels targeted. You know, I was like, we wanna deploy this thing on the FTP servers in six countries around the world. And the change advisory board's like, well, you know, we're gonna review this.

(Dave at 00:30:25) And I'm like, so who here knows how the FTP protocol works? Silence. So you don't even know what I'm talking about, yet you're going to approve or deny what I'm doing, and that's gonna make it safer. Like, this doesn't make any sense. Like, it's—

(Joel Beasley at 00:30:44) It kinda makes sense. That's how a lot of things are with humans. Like, sometimes you just walk into these processes and you're like, scratch your head. Like, what's going on? Why?

(Dave at 00:30:53) Yeah. So the change advisory boards, turns out they don't add any safety or value whatsoever. All they do is slow things down because all the things that you wanna test, you don't wanna rely on some human anyways. Like, you want code, you want all kinds of other stuff because humans are really fallible, which is, you know, that's what the computers are there for is to help us along and make us really awesome, not to get rid of us or any of the other things. So, you know, the easy deployment stuff is a thing that people struggle with.

(Dave at 00:31:28) People understanding even the deployment process is something that people struggle with.

(Joel Beasley at 00:31:33) Yeah. Can I—

(Dave at 00:31:34) Can I stop you there? Moving parts.

(Joel Beasley at 00:31:37) I've got a question for you. I don't wanna get too far away from it because I think there's a lot of people that are curious about this. So you had described this workflow of going to the QA department, having them do their things, then going over to this change advisory board and like this. And you're like, ah, that's the archaic way of doing it. But what is the modern way to do it?

(Dave at 00:32:00) You wanna have automated tests throughout. That's how you make things safe, you know. And so this isn't even... So depending what you wanna call modern, you know, this was in Continuous Delivery by Jez and Dave, which is... So I have this classic example I give. I went to an executive vice president at Salesforce, and I was like, hey, man. We wanna build this, you know, continuous testing environment... Testing pipeline, really better way of saying it, for being able to deploy this stuff after production. And he was like, great.

(Dave at 00:32:33) How much does it cost? It's a reasonable question, right? And I said, well, how confident do you wanna be in what we're releasing? And that's the thing that Jez and Dave talk about is the more testing we do, the more confidence we can have in what we release.

(Dave at 00:32:49) And so if I run like three unit tests and then I'm done and it goes out, I'm probably gonna have a pretty low degree of confidence in the code that I wrote. It's not gonna break something. But if I have really pretty good unit test coverage... You know, people talk like 80%, things like that. I have my integration tests.

(Dave at 00:33:07) You know, I'm running my SonarQube stuff for security. I'm doing all this other stuff. When that thing gets out to production, I have... And I have regression tests, blah, blah, blah. I have a pretty good feeling of confidence of the thing that I deployed. And so, you know, it's a balance.

(Dave at 00:33:25) Like, that's the big joke about engineering, right? It's not just all ones and zeros. Like, it's an art. How much testing do we do? When do we feel the most confident?

(Dave at 00:33:35) If we do too much testing, we slow the business down. If we don't do enough testing, then our customers get super angry, whatever. So it has to be a balance. But you really, you know, in this environment, certainly what you described as modern, like speed kills. Speed is the name of the game. Being able to run all those experiments, outcompete your competitors, that's a big deal.

(Dave at 00:33:59) And so you want the code, you want the computers to make you kind of a rock star in that regard. You want the computers to run the test, to do all those things. And it doesn't mean there's not a place for exploratory testing, for QA or whatever. It's just that can't hold up the process. That's an important thing that can get done, but it doesn't block the critical path.

(Dave at 00:34:27) Does that make sense?

(Joel Beasley at 00:34:29) Oh, yeah. Yeah. And, you know, I get to talk to people off the show and I would say just very little data, just completely subjective. My feeling is that at least half the people, at least half the companies out there, they'll have some concept of testing, but it won't be in the culture, you know. Maybe some of their better developers test and then some don't really test as much. And then I always, my ears always perk up when I hear things like, oh yeah, but we can't really test that because throughout my entire growth of learning how to test, I was like, you can't test it.

(Joel Beasley at 00:35:05) And I figured out how to test it, and then I figured out how to test it, and I just kept going all the way up and then learning new tools and new ways of how do you create more firm, you know, tests versus more brittle tests. And how do you attach it? And all of these different things about, like you said, it's stylistic, what type of test to use, where and when.

(Joel Beasley at 00:35:23) And then I realized that if you couple that with feature flags, QA gets real small, real fast, like human QA.

(Dave at 00:35:32) Oh, yeah. Yeah. I mean, the quality engineers that I worked with at Salesforce just blew my mind sometimes. I was like, so this is what I'm trying to do. And they're like, yeah. What about if we did this instead? I'm like, oh my god. That totally solves the problem. And they're like, yeah. This is kind of what I do for a living. I'm like, oh, yeah.

(Dave at 00:35:53) That makes a lot of sense. So yes, I totally agree with you. And, you know, it does. It shrinks down that human side of it, but it doesn't make it any less valuable. It's still tremendously valuable, especially for software delivery, you know, for shipping things out quickly.

(Dave at 00:36:11) Some of the other things that people struggle with is stuff that you've also talked about on your show before, which is resilience engineering. You know, what happens when something fails? Do we even know? What you don't want is the, hey. It's 3AM. The only person who knows about this is Joel. Wake Joel. You know? That's not a great place to be. Well, it turns out that Joel's in Fiji relaxing on a beach.

(Dave at 00:36:39) Well, now we're in trouble. You know, that's not something that's desirable. So, you know, we go as far in the assessment as asking things about chaos engineering and stuff like that because, you know, turns out that thing's huge. And what I really try to do is help people understand why that's huge. Right?

(Dave at 00:36:58) And it's the reason that Netflix, you know, did the chaos monkey and all the fun things that we love to hear about. It's not because they were like, hey, look, we're so awesome at delivering software. We're just going to turn things off. It was that they understood, and which most people probably are going to understand in about thirty seconds, is when I deploy software in January, you know, some microservice that I have and I deploy that same thing in August, it's not really the same software anymore. Right?

(Dave at 00:37:29) I've been deploying for months, and that's been changing the characteristics of that software. It changes how it behaves. It changes the things that it can do. Things are constantly changing. And the only way to really stay on top of that is the chaos monkey kind of stuff. Whereas we want to have high confidence that we actually understand how this software works. And if it loses connection to the database, then it's going to do the same thing that it was always supposed to do, and it doesn't do something different. And you could stop the release every single time and do all kinds of tests, and I'm not saying you shouldn't do some of those things. But there's a lot of interactions between all this stuff in these complex distributed systems. I've worked on a lot of really big complex distributed systems that you just can't reason about. You can't just sit back and look at the entire system and be like, oh, I understand everything.

(Dave at 00:38:27) This just doesn't work that way. And so that's why they're doing these chaos experiments, is because, you know, you want to break the software when everybody's awake and understands what's going on. You don't want to break software at 3:00 in the morning and then everyone's reaching for their Jolt colas to try to figure out what the heck is going on.

(Joel Beasley at 00:38:46) That's back in the nineties. Does Jolt still exist?

(Dave at 00:38:48) Does Jolt still exist?

(Joel Beasley at 00:38:49) No. Surge. Surge and Jolt. Yeah.

(Dave at 00:38:53) Monster energy.

(Joel Beasley at 00:38:55) Some of my favorite reliability episodes that you just started talking, they started, I started remembering. Yeah. We have one with Gremlin. It was this guy who had left Amazon. I think he was on the reliability engineering team for Amazon's website. And then he said that somebody would run around and just randomly yank out cords, and they called it a gremlin or whatnot.

(Dave at 00:39:16) Right. And just to see if it would still work.

(Joel Beasley at 00:39:17) Yeah. They'd run around the server room and just yank stuff out and be like, oh, does it work? Does it still work? And so he took all his learnings, and then they built this gremlin thing. And I believe this was a year or two ago when I talked to him. But the thing that caught my interest about it was when you have this issue, right, it does, I think, a post mortem on it, and you can write code against it. And then it goes into this library of, so Gremlin will pick up on, you know, the incident and then you can track that. It wasn't full incident tracking, I don't think, but I don't know. It was worth looking into.

(Joel Beasley at 00:39:53) And then I had a guy, it was a cool guy. So it was a good episode. And then Adam Burat, I think I don't know how to say his last name, but he's got this Apex Ridge reliability company. He does hardware, industrial stuff, but he also does software. And he's crazy, and he turned, he put, he turned his Porsche into something that could pull a sailboat. It was crazy. So, uh, yeah, those are two of the reliability episodes that I really like.

(Dave at 00:40:23) Yeah. And, you know, it turns out that's a big part of being good at delivering software is caring about that stuff. You know, one of the things I talked about during that talk was incidents are socio technical constructs. You can define an incident as whatever you want. And I tell CTOs, if you want to get better nines, just change your definition of an incident. You can go from 97% reliability to 99.5 nines, no problem. Just change your definition of an incident. But, you know, the kind of the jokey part is that really what we've learned is airlines that have more incidents are actually safer than airlines that have less, which makes no sense when you say it out loud.

(Dave at 00:41:09) But it's because the airlines that have more incidents have a lower definition of what is an incident because they want to learn from those incidents, and they want to get better, and they want to be safer and stuff like that. The airlines that only care about what the public will see and the number, and it's all for vanity and whatever, they don't take security seriously. And so they're going to have more incidents. They're going to have a harder time recovering or whatever. And I don't think it's any different in the software industry.

(Dave at 00:41:41) It's like if people aren't interested in that and it's all about features, features, features. Yeah. Well, you're going to have longer incidents. You're going to have things where, you know, what I always tell people is the golden rule of SaaS. Don't lose customer data. You're going to lose customer data when you don't take this stuff seriously. So there's a lot of things that go into getting good at delivering software, which is why I wrote that article for InfoQ.com. And it's not just about, you know, let's argue about this JavaScript framework. That's not all there is to delivering good software.

(Joel Beasley at 00:42:18) At the beginning of your career, when you're new, it is. Because that's all you really see. You get so excited about it, and your whole life is trying to just figure out how this thing is working.

(Dave at 00:42:27) That is true. And also, you know, that's why people sometimes get wrapped up in these silly things like lines of code. How come you didn't write enough lines of code this week? Like, what is that a proxy for? That has nothing to do with value. It has nothing to do with quality. It has nothing. And Camille Fournier, who wrote A Manager's Path, this great book, whatever, wrote a great blog post last summer where she, it's something to the effect of, you know, a list of things that you need to be able to do as a senior engineer that's not writing code. And it's like how to convince other people that your idea is a good one, how to mentor a junior engineer, how to write good technical documentation. I mean, you even said it yourself a couple minutes ago, what's a good test?

(Dave at 00:43:15) Where is it appropriate to put that test? All of that, none of that is writing code. Yet you and I both know because we've been doing this for a while, those are way the most important parts of delivering software. It's not how I can write code. Yeah. Okay. Fine. But that's literally the easiest part. But to your point, early in your career, you're like, look how good I am. I'm writing code. And it's like, yeah. After four or five years, the fact that you can write code and write pretty good code, we all kind of consider that something that should, if you haven't, can't do that after four or five years, there's a big problem because you're never going to be super effective in your career. And, yeah. That's a great skill.

(Dave at 00:44:02) But the hard part is all of the things that are hard to measure and all that other fun kind of stuff.

(Joel Beasley at 00:44:10) Yeah. It was sobering for me to realize one day I just had this epiphany that, oh, wow. Writing the code is turning the wrench. Because my brother-in-law has a motorcycle shop and I was there and I was like, oh, it's like your entry. It's your entry. It's like, here's the low level job that you can do to get your foot in the door and, you know, on that side of the organization. But there are some people who like turn it into an art and just take it way farther. You know, there are those few. So I have a lot of respect for those people and, um, I don't know. I just I tend to have the most respect for people that just really care and try and push things forward regardless of the topic. You know?

(Dave at 00:44:54) Yeah. And I, you know, and I have the most respect for the people like that who can make everybody else around them even better.

(Joel Beasley at 00:45:00) Oh, yeah.

(Dave at 00:45:01) So there was a guy on one of my teams who whenever I wrote code, I was like, I don't want you to do my code review. Because I knew every time I was going to learn something really, really valuable because he could do what you're talking about. He could just sit there and write some amazing stuff, but he also made everybody else around him better all the time. And, you know, how do you outcompete a team where that's happening? That's a real 10 x engineer.

(Joel Beasley at 00:45:33) You mentioned a couple times this book Accelerate. I have not read this book. Can you describe what it is?

(Dave at 00:45:39) Accelerate is the book that's the science behind the 2019 state of DevOps report. So DORA research was back in the day, Jez Humble, Nicole Forsgren, and Gene Kim and some other assorted folks. And so, you know, when Nicole came in to get involved with that survey, she really brought her PhD statistics, experimental design, all that kind of background to the actual study, we'll call it. And so when they were able to publish these findings, like we said, which is why I based my assessment on it, high performers are twice as likely to meet or exceed their organization's performance goals.

(Dave at 00:46:28) She, there was real math behind it. And so the Accelerate book talks about a lot of the learnings of the report, but the back half of the book is like, here's the experimental design. Here's the math. Here's the everything. So that, you know, a lot of engineers and I certainly was guilty of this perhaps early in my career are like, oh, that's all human touchy feely garbage. None of that matters. And it's like, yeah. Except here's the math that proves that it matters. And, like, it's kind of hard to argue with the math.

(Dave at 00:47:08) And so they call it the science behind or whatever. But that's what the Accelerate book is, and most people know it from the four key metrics that we were talking about.

(Joel Beasley at 00:47:15) The first time I heard, Cody, he was the CIO at T Mobile. I think he's retired now. But he had told me, he goes, I was asking what secrets are or things like that. He was in an early interview. I think he's in the first 50 interviews. But he said, Joel, it's all about the people. And then from then on, I had heard that from so many because, you know, there's a gradient. There's different people of different levels of success, you know, personally and professionally that I get to talk to.

(Dave at 00:47:46) And

(Joel Beasley at 00:47:46) I always try to figure out, all right, the top 20% of everyone I get to talk to, what are the trends there and how can I be spending, you know, more time there? And I just kept hearing it over and over and over. And then as my business grew as an entrepreneur, I started focusing on that because I heard, you know, the whole gardener concept. You mentioned it earlier, you can't make the plant grow, but you can create an environment where the plant can grow.

(Dave at 00:48:11) Right.

(Joel Beasley at 00:48:11) And then you can help shape the plant and whatnot. And so when I started to take that perspective, you know, everything changed for me, and things started working a lot better. And I believe that perspective can be applied as an individual contributor all the way up to manager, all the way up to owner. You can always find ways to apply that.

(Dave at 00:48:31) Yes. I agree 100%. I also think that's why, you know, we have this horrible thing that we do in our industry, which we take a really great engineer

(Joel Beasley at 00:48:41) and

(Dave at 00:48:41) we're like, congratulations. Here's your promotion. You're now an engineering manager. And they're like, what? Oh, I guess this is the way that you move up in engineering. And it's like, first of all, that whole pattern is broken. You know, and the good organizations will have a track for people to move up technically and then a track for people to move up in management. And those aren't the same tracks. And you can go up all the way to distinguished engineer and CTO and all those other things in a technical track. But, uh, but yeah, I think that's why people struggle is that their whole career, they've been like, I'm an engineer, I'm writing all this code. And now I'm basically in a people job because an engineering manager is not in a writing code job.

(Dave at 00:49:27) And now I'm in a people job and you're like, "Hey, just get the best you can out of this engineering team." And like, you know, your boss is basically like, "Well, that just means you're gonna yell at people about how they should write code." And it's like, that's not what being an engineering manager is at all. And so it's a completely different job. And we throw people into these jobs without the training that they need and the understanding that they need in order to get it done.

(Dave at 00:49:55) And it's funny for me because in the work that I do consulting, like not in the assessment, those lower level managers are the ones that I wind up working with the most. Not just because not that they're bad at their jobs or anything, but like they're the closest to the information because that's where all the engineering is, where the information is. And then they also have to sort of mediate between what management wants and what the engineers are doing. So that's a very, very high leverage place to make change. But a lot of those people are just like, I'm like, "So how did you get into this role?" Well, I was the best engineer on my team, so they made me a manager. I'm like, "So you don't get to do the best engineering anymore?" And they're like, "No." I'm like, "Okay."

(Joel Beasley at 00:50:42) Well, I think what you were talking about, like with that, you know, it's very prevalent. Right? This concept, it's talked about a lot. But to tie it back to our earlier conversation, like if your goal as a leader, a leader of people, is, you know, to put them first and make sure that what they're after in life, them going after that and what you need done in life, that those things connect. Right?

(Joel Beasley at 00:51:11) And by promoting someone without knowing that it's what they want, I think that's like where the mistake was. And, you know, I learned this and I think it's worth saying that it's not just engineering. So I was doing this across the board. I thought everybody wanted to run a team and wanted a big company. Some people were absolutely frightened and like wanting to quit. And I'm sitting there thinking I'm pumping them up.

(Joel Beasley at 00:51:34) Like, "We're gonna get big. You're gonna become the leader. You're gonna have teams of teams." And they're sitting there like, "How do I get out of this job?" And I'm sitting there...

(Joel Beasley at 00:51:41) Yeah. And so what I learned was, you know, when I start talking to people, I just say that. I just tell that story and I say, you know, you need to tell me what you want. And I go, and you also need to remember that it will change. It's dynamic.

(Joel Beasley at 00:51:58) Like when I'm having, like we're having our third kid right now. My wife is pregnant again. Yeah. And things definitely changed the first couple months I have a new kid. Like my focus definitely is different than it is, you know, after they're out of that very, very early stage. And so, you know, I might not want a promotion at that point in time. So knowing your people and being a great leader, I think that's super important. I think that would solve a lot of the people's question of, "Should I promote this lead engineer, or should I convince them of the talk track?" You shouldn't do anything other than talk to them.

(Joel Beasley at 00:52:34) Like you should talk to them and try to figure out, you know, where they're at in their journey and how you can help them based on what, I mean, your biggest thing helping them might be letting them stay exactly where they're at. They might be going through some stuff right now.

(Dave at 00:52:47) Yeah.

(Joel Beasley at 00:52:47) And they might need to just have the comfort of a stable work environment. You know? You don't know that if you don't have a relationship with them.

(Dave at 00:52:55) Yeah. That's the "have you tried talking to them" as a service. Yeah. Yeah. I gave advice to someone this morning. They're like, "So I saw this engineer and I thought he was really struggling. So I gave him another resource and now he's super angry. It doesn't make any sense." And I was like, "Did you try asking him about it?" And they're like, "Well, no, I saw him struggling, so I knew what the answer to his problem was." And I was like, "Yeah, I think you probably had a much better answer to his problem if you just asked him what's going on, you know?"

(Joel Beasley at 00:53:29) And it's honestly a lot easier on you as the leader too, because then you don't have this big burden of trying to like solve everything for everyone else. Right?

(Dave at 00:53:39) You don't have to read everyone's mind. You just ask them what's on their mind.

(Joel Beasley at 00:53:43) Yeah. That person, that engineer was probably stressed out, had a bunch of work, or had something going on in their personal life. And then when giving them that resource, thinking it was the solution, just gave them somebody to train, which gave them more work. And now they're like even extra stressed.

(Dave at 00:53:57) Yeah. No good deed goes unpunished. Amen. But yeah. No. I think it's huge. And I think that's why I love like the Accelerate report and those all the all those other things, is it's about creating that culture, and it's not about the actual numbers or or whatever. And and I think that's, you know, as someone who's made that transition from engineer to engineering leader and and all those other fun, exciting transitions that you have the scars to prove that you did them. You know, those are definitely some of the things that I learned on my journey. And I'm definitely out on a mission to help other people to understand those things so that they don't get quite as many scars on the journey, even though some of them are, you know, that's just the nature of the business.

(Dave at 00:54:47) But, you know, I want to help people have better lives. I want them to be able to produce more stuff. I want them to be happier at work. Because, ultimately, that's where all this growth comes from in these companies is like having really happy employees that are stoked, if they use the California word, you know, to come to work every day.

(Joel Beasley at 00:55:08) So if people wanna get good at delivering software, they like you, they've been hearing you, they're like, "I wanna talk to this guy." You do this. Right? It's not just for PE firms exclusively. Like they can reach out to you?

(Dave at 00:55:18) Absolutely. Yeah. It's not just for PE firms. I just, PE firms tend to have a lot of the most interesting problems to me because they're really hard. But it's, I, you know, I've worked with tons of non-portfolio companies as well. I love working with them too. It's, you know, again, like I just want people to come to work every day really happy because everybody, like there's so few opportunities we have in life for win-win. Like this is win-win. The engineers are happy. The business is happy.

(Dave at 00:55:53) Like let's take advantage of that as much as possible.

(Joel Beasley at 00:55:58) Yeah. You're getting good at delivering software. That's like your current, your current banner that you're waving. Right?

(Dave at 00:56:04) Yeah. It is definitely my current banner that I'm waving.

(Joel Beasley at 00:56:08) How can people reach out to you? LinkedIn, website?

(Dave at 00:56:11) Yeah. They can go to Mangoteque. Uh, it's m-a-n-g-o-t-e-e-q-u-e. So like the, a play on the French idea of my last name and then tech. Or they can certainly find me on on LinkedIn. And then if they go there, they can also sign up. I do a monthly, sort of email about DevOps and private equity. So for people who wanna hear about those kinds of problems, it's not just about private equity. It's about the kinds of problems that I see in these private equity kinds of companies. So it's also another great way to keep in touch and or get in touch and, you know, certainly can reach out.

(Joel Beasley at 00:56:56) Thank you so much for listening. And if you found this episode useful, please share it with a friend or colleague who you think would get value from it. And if you have topics that you'd like to hear discussed on the podcast, either add me on LinkedIn, or send me an email, [email protected]. Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.