How GoDaddy Experiments, with Charles Beadnall, CTO of GoDaddy

(Intro Narrator at 00:00:03) Hello, my friends. Today, Joel is talking to Charles, CTO of GoDaddy, and they discuss how Charles uses experimentation in his teams to further agile development, challenges of running simultaneous experiments across many development teams, and how aligning your motivation with company needs is a key to success. All of this right here, right now, on the Modern CTO Podcast.

(Charles at 00:00:31) Here we go.

(Joel Beasley at 00:00:32) This is the Modern CTO Podcast. I want to talk to you about engineering. When I was talking with the team, we always get together in our group production meetings and try to figure out what's the unique angle for every episode. And when it came up for yours to talk about experimentation within a company, I thought that was really interesting because we've done 500-plus episodes and we've never had an episode about experimentation. And so I was like, wow, boy, do I have a lot of questions. So if it's cool, I want to talk a little bit about that.

(Charles at 00:01:08) Yeah, that's great. And a colleague of mine said that experimentation is like crack for engineers. And I think a super simple way of thinking about it for me, when we're talking about agile software development, I mean, you have the product owner sitting with you every week or two weeks or whatever your sprint cadence is, telling you what they want and giving you feedback on what you've delivered. And that works great if you have a single customer. When you've got 20 million customers out there, kind of different regions of the world, different countries, different cultures using your products, it's really hard to be able to do that. And it's also, in some ways, almost unfair to expect a product manager will have interpreted the entire user population in a perfectly accurate way. And so, really, we've been using experimentation to what I would call furthering agile development. It gives us a way upfront stating a hypothesis. Hey, if we deliver this feature, we make this faster. This is the behavior that we're going to get in the end. We go off, we do the work, we run the experiment, and we prove, hey, it worked or it didn't work. And I think most engineers, that's kind of what you want to know is, did my work have value and meaning to the customer? And so I think that's really the way I think of experimentation at a very high level.

(Joel Beasley at 00:02:35) Can you give me a more specific example of an experiment?

(Charles at 00:02:38) Yeah. So at this point, we're running hundreds of experiments somewhere in the range of a classified experiment that runs a control and a treatment, well over a hundred a month, and many more than that in various different versions where we maybe have less rigor around them. And it could be something as simple as changing text or putting or not putting an ad on a particular page, or it could be something much more complex and more involved. And right now, we're actually looking at how do we take even bigger swings, multivariate tests across lots of different products and areas. But I think it's helpful to think about it in a way that you're not trying to change too many things all at once, at least to get started. I think you can get more advanced as you go. But I think just the rigor of thinking about it as an experiment in terms of what is your hypothesis, what is the metric that you're going to use to determine whether your hypothesis was correct or not? That's a reasonable amount of work just to come up with those basic building blocks and then obviously going off and doing the work. And I'd say the overall philosophy that we've had is most similar if you've heard of the British Cycling team and their story. The coach basically said, we're just going to be 1% better at everything. So it wasn't that they had the best bikes. It wasn't that they had the best training program. It wasn't they had the best logistics or whatever, but they just focused on making lots of changes. And those accumulated into a world-class winning cycling team. And I think it's helpful to think of it that way. It doesn't mean you don't take big swings as well, but you've really got to get the rigor of running and experimenting and the experimental mindset. And so it's yes, there's a technical aspect here, but it's also just cultural in terms of how are we going to run experiments across the product and engineering teams and really making sure that that layers up into the business and the P&Ls and things like that. So that's really, I think, the big change that's required to deliver experiments. And again, I mean, there are lots of different examples, and we're trying to run it as a program such that we share those experiments. And we go through now the top 10 experiments that we run each month that we want to share with the rest of the company. And so I think another big and important part of this is just sharing the learnings. So if one team runs an experiment where they remove advertisement from a particular type of page, that other folks see that and understand why it was or wasn't successful. And, of course, half the learnings here are not the successes. I mean, we'd like for all of them to be successes, but half the learnings are in the failures or learning opportunities. Because ultimately, when you make a hypothesis and you're wrong, there's a lot of good learning that usually comes with that because obviously, the customer reacted in a different way than you expected. And it's important to understand why that is.

(Joel Beasley at 00:05:55) How do you teach new engineers this? Is this something that you actually teach them in onboarding or training, or do you pair them with somebody? How do they learn this skill?

(Charles at 00:06:04) Yeah, that's a really good question. And on our journey, I think we started with the right path, but I think there have been some significant learnings along the way. We actually started this path with experimentation by trying to figure out how to deliver better machine learned models in a number of different areas. And we found that most teams didn't have a good way to collect the metrics to determine whether that model was improving or not in the areas that we were deploying them. And so it really, in some ways, set us back in terms of, we've got to figure out a way to measure these as experiments such that we can prove whether this investment, whether this new ML model is actually being successful or not. And so we went down the path of looking at where we were running experiments, how we were collecting data, and really just set out on a process to build a tool that we would be able to collect the data and assess whether we were successful or not. And so, ultimately, that was a big kind of first step for us. And over time, that's evolved. We've built a common tool that allows us to run experiments across user experiences, which I'd say are generally speaking a little bit easier. There are commercial tools out there you can kind of plug in. But also API and other types of back-end tools, especially that's where we're potentially running an ML model that makes a different decision based on a bunch of different inputs. And so that's an area that we've now consolidated onto. It's actually technically, it's two tools, but really a homegrown one with some additional components from other providers. And so having that tooling in place certainly helps the engineers to instrument. We've got feature flags so that you can turn on a particular feature or test for a time duration and then collect the data at the end. And so that was really on the engineering side. I think the interesting thing about experimentation is it really takes a partnership with the folks developing products. And I'd say that's where we ended up focusing a lot of our time was on that cultural change because in some ways, some folks initially, I think, saw experimentation as threatening. Like, you're telling me what to do, as opposed to you have the freedom to define experiments, define the metrics, define the outcome that you're looking for. And so it's a big shift for us as a company from that product perspective. And so in some ways, I'd say that the engineering side of this equation, and it was partially that's where we came from, was easier because, okay, you're just building tools. You're snapping all the bits together such that you can operate an experiment. But ultimately, it's bringing the product and business along for the ride in terms of their understanding and willingness to format their work in that fashion as opposed to, here's a product roadmap with features A through Z, and we'll do them in this order. And we may shift them a little bit based on learnings over time, but it really kind of upends that whole method of developing products.

(Joel Beasley at 00:09:27) How do you propose an experiment? Do I just knock on your door and say, hey, Charles. Got this idea. Want to run an experiment? Or is it can any engineer do it? Is it a specific team that's just the experimentation team? How does that work?

(Charles at 00:09:40) Yeah. So GoDaddy has well over a hundred development teams that are all working on different aspects of GoDaddy. Everything from marketing pages or campaigns through domain products, website building products, hosting products, commerce, and payment products. And so it really has to be something that is completely federated such that any team can basically work with their products manager to come up with an experiment. And that is something that I think is also beneficial here is that I think using this format allows tighter collaboration between the product and engineering teams to come up with a proposal for their area. And so I'd say a lot of the initiative comes from the product teams. Some of it also comes from the engineers, but it's basically federated out such that if you're working on a particular team, you can say, I have a different idea of how we can help our customer. Maybe it's making more money, reduce latency, which will have X effect. They can go off and run that experiment. And then, basically, we bubble it up, and we share the learnings from those experiments across the company such that if one team finds that, hey, reducing latency from four seconds to two seconds or whatever it was had a really positive benefit on customers, that's learnings that we can share and hopefully get other teams to implement those experiments in their own areas as well.

(Joel Beasley at 00:11:12) You mean people like things that work fast?

(Charles at 00:11:15) Yeah. That's definitely one of those areas where I like to call it the ilities, like scalability, dependability, all those things where I think engineers have a unique capability to really drive improvements in those areas. And, yeah, I think security is a feature, stability, reliability, performance, those are all features. And I think in our research with customers, they tend to run really high up there. And, yeah, sometimes you're looking for features as here's a capability to make something easier or whatnot. But, yeah, being reliable, being secure, being performant, those are all major contributors to customer satisfaction.

(Joel Beasley at 00:12:00) And then is there one team that manages the tool itself?

(Charles at 00:12:03) Yeah. So we have a team that has built our internal tool. We call it Hivemind because essentially, it really is the hivemind of all of us building experiments, collecting that data, learning from the collective intelligence. And we've also built the mechanisms to collect the data from across all the different systems such that we can then funnel those metrics into that system.

(Joel Beasley at 00:12:29) Nice. That sounds pretty cool. And so then you have this interface where you can literally look across your entire org and see every single experiment that's happening.

(Charles at 00:12:38) Yeah. Yeah. No. And I think that's really the benefit. And it took us some time, and I'd say in some ways, it's a journey because there's always some pressure to do something different. Maybe a particular team wants to move faster, but obviously, there's statistical significance. And so, the faster you move, the less statistical significance you're probably going to have, the fewer data samples you're going to have because you want to move faster. And so I think we're working through, I think, those kind of in real time right now in terms of what's the right trade-off between getting learnings kind of at any cost, and you may have less precision and having high precision and learning from those and having a definitive answer. And so we've kind of erred, I'd say, close to accuracy, but backing off a little bit such that we also make sure that we're continuing to move forward with lots of new experiments. But it is an interesting trade-off because we are talking about probabilities and statistical significance and not absolutes. And one thing that we had seen early on with some teams is that they would say, hey, I've got this winning experiment that will generate $2 million, and I got this other experiment that will generate $2 million, and I have this other experiment that will generate $2 million. But the total of those experiments didn't add up to $6 million. And so I think you also have to be kind of careful about the rigor with which you run those experiments such that you have reasonable assurance that they're adding up in the right trajectory even if the precise accuracy may not be 100%.

(Joel Beasley at 00:14:22) Have you had to teach people about things like statistically valid sample sizes and how to do that, or is it kind of baked into the tool?

(Charles at 00:14:33) It is baked into the tool, but there's definitely a good learning process here in terms of what is best practice. And so we've created a community, especially with our products team members, to help coach along that path. And we've created kind of an internal certification that basically gamifies the process such that the best run experiments get a higher status. And, again, I think that that's then in turn turned into our ability to share those more broadly in these corporate-wide events where we talk about the experiments that we're running and the learnings, as well as the underlying the probabilities of the different experiments.

(Joel Beasley at 00:15:20) Nice. So this internal certification seems like you guys are a teaching company if you have systems set up like that. I think this would be a good time. Are you hiring currently?

(Charles at 00:15:30) We are hiring. Yeah. And I think in a number of areas that are specifically related to data, both the collection, as well as in the products where we're expanding use of experimentation right now. And that's something, yeah, I think it's an interesting environment right now. I think none of us know exactly which path the overall economy will traverse. But I think something that we have seen repeatedly is that people are starting small companies almost irregardless of the macroeconomic constraints. And I think what's interesting about GoDaddy is that there are a lot of companies that address the traditional small-medium business market, which I think most people classify as having 20 employees or 200 employees or something kind of of that size and scale. And obviously, if you've got 200 employees, you can probably write reasonably large checks to a systems integrator or an enterprise software company or whatnot. Whereas, I think most of our customers from a numerical perspective, number of customers. Obviously, we have some pretty big customers as well.

(Charles at 00:16:39) But many of our customers are small, almost what I would call proto-businesses in terms of they have one employee or a half an employee or something of that nature, such that our average is actually significantly under 20 employees. And obviously there are many that have 20, there are many that have 200 and above, but we have so many customers that have one employee. It's a solo entrepreneur. I think it's an interesting space for us to be in that I think the competitive offerings in that market are basically commercial consumer tech, such that, I mean, you can obviously use Gmail or, you know, maybe Facebook for certain things. But ultimately, we service that market, and we become essentially like the IT department for those very small businesses.

(Charles at 00:17:30) And that's really our focus is to pull the different pieces together such that there's a more integrated offering for the business, the seller. And I think what's interesting for us too is that we're not necessarily just looking at how do you sell on Amazon or how do I sell on a particular marketplace, but rather how do I sell across different marketplaces. And I think most very small companies, they want to do a couple things. I mean, one, they want to get identified and found. So it's kind of, here I am.

(Charles at 00:18:03) I'm hanging my shingle up. You can find me here. And then the other is they want to transact. And so, really, our focus is helping really small businesses do both of those things. And so it's a little bit, you know, a Robinhood analogy isn't a perfect one, but it's really trying to provide tech for very, very small companies.

(Charles at 00:18:22) And something that I think we've seen in past economic downturns has been that more people tend to kind of step out on their own even when larger companies are not growing. And so that's an area that I think we have a kind of a moral mission to really help those very small businesses get up and off the ground and find customers and get business.

(Joel Beasley at 00:18:48) Yeah. We've talked about mission a lot on the show. Last time we were talking about it was with this company called Release. They're like an environment as a service type company. And the guy I was talking to, his name is Tommy.

(Joel Beasley at 00:18:58) Pretty cool dude. But they were very, very mission driven. Their whole thing was like enabling developers to add more value to the world. And when you have the GoDaddy mission, what's like the sentence that you use around the office?

(Charles at 00:19:13) I don't know that I have a sentence, but I guess I have a story. I was helping out my local county, the economic development office during the first months of COVID. And, you know, what I saw was a lot of companies that were in a period of upheaval in terms of they were used to having people coming into their restaurants or shops, and they had their point of sale terminal, and they were able to take orders. But all of a sudden, people weren't coming in, and they had to scramble like crazy.

(Charles at 00:19:44) And most of these folks are not software developers. They're not necessarily even like the most technically savvy people in the world. This is what they need to do to run their business, but, I mean, their business is making things or cooking things or doing things and not fussing around with the technology. And I think it was really at that point where it became super clear to me what the value of GoDaddy was. And it was to pull those pieces and parts together in a way that allows them to focus on their business and not everything else.

(Charles at 00:20:19) And we have subsequently gotten into the point of sale terminal business that allows you to purchase something in a store. It allows you to sell that same product through your own online store or sell it through a number of different external marketplaces. And I think that's really the benefit of our mission is helping these customers out with it because I don't see anybody else really stepping in and trying to help them. If you've got 20 employees, 200 employees, yeah, sure. There are lots of companies out there that'll help you.

(Charles at 00:20:50) But if you're very small and just getting started or you're in a smaller environment, that's something I think that we can help with.

(Joel Beasley at 00:20:59) Well, I'm a fan. I mean, I've been a customer of GoDaddy for over 15 years. I buy a lot of domains from GoDaddy. But, yeah, I remember signing up with you guys.

(Joel Beasley at 00:21:09) I must have been 12 years old, and I'm 34 now. So more than 15 years. And the one thing when I was hearing you talk about experimenting and running different experiments, I was like, that makes so much sense. Because every other time I log in to GoDaddy, the interface is different.

(Charles at 00:21:25) Hopefully, mostly better.

(Joel Beasley at 00:21:28) Yeah. And it makes sense too, honestly, now that I was thinking about the different changes. And I always ask myself, you know, from an engineering perspective, oh, I wonder why they did this, and I wonder why they did that. And the moment I heard that, like, the average customer that you have of your 20 million customers has 1.4 employees, I go, that makes so much sense. That makes so much sense.

(Joel Beasley at 00:21:47) They're probably a lot of these changes I saw, some of them were probably to reduce support issues. Like, one thing that I noticed was the DNS settings kept getting moved farther and farther back. Because all I do in GoDaddy, just so you know, like, all I do is I buy a domain, direct my DNS to Cloudflare, and then go do what I want to do. I just like GoDaddy. I know I can buy domains other places, but I just, I don't know, have it. Right?

(Joel Beasley at 00:22:07) And slowly but surely, that button jumped farther and farther back, and then it required additional things. And after it went to you having to check a box to double confirm or whatever that you're changing this, I said to myself, I bet you people are changing their DNS settings, and they don't know what they're doing because they're Googling stuff. And I bet you that's creating a ton of support for them, and that's probably why they did this.

(Charles at 00:22:30) Yeah. I think that's fair. I think the other thing is, again, if you think about somebody running a pizza store or a barber salon or something like that, why should they ever know what an A record is? It's like this most arcane bit of information that, of course, we all know what it is. We know how it helps, but it's really just an impediment to kind of getting the job done.

(Charles at 00:22:49) And so we actually wrote an IETF standard around a thing we called Domain Connect, which basically provides a mechanism to create templates. So if you're hosting your email with Gmail, there's a Gmail template. And that's something that works at GoDaddy. It actually works with a number of other DNS providers on the Internet. I think at this point, almost all major service providers are supporting Domain Connect, and it provides an easy way for you to just click a button and say, yeah, my email is here, my site is there, whatever. Click a button.

(Charles at 00:23:15) If you haven't logged in to GoDaddy or whatever your DNS provider is, you have to log in, but you just click apply, and it'll do the error checking for you to make sure that if you're going to stomp on some other template, it warns you. Are you sure you want to do this? It'll also, if you don't own the domain, you can buy it.

(Charles at 00:23:46) It simplifies a whole bunch of things that customers otherwise can easily get lost. And so, I mean, yeah, maybe there's in some ways a cost of resolving that. But I think more importantly is from a customer experience perspective, that's a really crappy experience if you're going through and trying to figure out what A records are and Googling all this stuff. It's like that's needless waste of time of the customer. And so if we can simplify that, it's in our interest to do it, but I think more importantly, it's a better experience for our customers.

(Joel Beasley at 00:24:22) Absolutely. Yeah. Like I said, I'm a huge fan. I love the product. More fun questions for you.

(Joel Beasley at 00:24:28) Do you know where the name GoDaddy comes from?

(Charles at 00:24:31) You know, I know that it was the founder and I think his head of marketing at the time. It was one of those kind of just funny names that they came up with that sounded catchy, and it kind of took off from there. I think like you had mentioned, when you start a company, essentially the first thing you do these days, I think, is try to find the domain name because ultimately you're not going to file paperwork to incorporate with a name that's already been taken or you're going to get some really long, elaborate permutation of it.

(Charles at 00:25:23) And so I think, yeah, naming companies is its own exciting phase and naming a product as well in a larger company. I think looking for the domain name and being able to find alternates quickly, that's an area that we spend quite a bit of our time focusing on because 70% of the time, the domain name that you want to find isn't available. And so that means you're going to iterate a few times to come up with a different name that is available. And so, yeah, obviously, at the time GoDaddy was founded, it was more likely to be available than it would be even today, but I think it was a similar type of process.

(Joel Beasley at 00:25:40) Yeah. I noticed your tools did get more advanced for domain picking. When you type the domain in that you want, it'll provide all these different alternatives now. And they're actually, like, at first, everybody kind of had it. Right?

(Joel Beasley at 00:25:51) When it was emerging and it was whatever. It was okay. But then they started to get really good. Like, actually coming up with some really creative things and understanding the name better and the context better. And so I was pretty impressed about the improvements there in the past two or three years.

(Charles at 00:26:06) Yeah. And it's an interesting area because most search engines are really an inverted index of available documents. You're just basically sorting and ordering those in an efficient way, whereas finding a domain name is a very different problem because essentially you're looking for the blank space. It's like you want to find the thing that isn't there. Obviously, we look at what other people, you know, their permutations of they don't find it the first time, what do they look for the second time, and what do they end up buying to basically help short circuit for everybody else, building models that incorporate those different methods.

(Charles at 00:26:48) And obviously, doing that in different languages, different regions. I mean, there's quite a lot of complexity that goes into that.

(Joel Beasley at 00:26:57) It's exciting. What's the next thing that's coming that you can talk about?

(Charles at 00:27:01) That's a good question. In some ways, you know, the way I think about technology is it's in service of the business, and the business is in service of its customers. And so it kind of flows in that order. And so a lot of what we're focusing on right now is figuring out how to order the technology in a way that better services those customer segments. And in some ways, it's like a giant software engineering project in that if you're building a particular system, you think about how you componentize certain things, divide that work across multiple teams.

(Charles at 00:27:39) And when you're doing it at a company of our size and scale, you've got to take a different tack to approaching that. So I think looking at how we do domain-driven design and define those domains and what the capabilities are and kind of order things in a way that we get out of each other's way. I think that's a really important thing for us to be able to do to continue to grow and evolve as a company.

(Charles at 00:28:32) And then, ultimately, I think that there's huge opportunities for us just leveraging the data in smart and intelligent ways. And whether that be running experiments or helping customers directly, I think we are continuing to evolve our machine-learned models to be able to help customers make their way through our systems more effectively, to help the folks that we have answer phones, answer them in more effective ways. Obviously, we have marketing campaigns and running those more effectively. There's lots that we can do to help using the data that we already have available to us. And so it's a matter of making sure that it's in the right formats that we can build models on effectively.

(Joel Beasley at 00:28:48) Can we talk about leadership for a minute?

(Charles at 00:28:50) Sure.

(Joel Beasley at 00:28:51) Just want to, I'm watching time, and I want to make sure I pick your brain on this real quick. You're the CTO of one of the most known brands on planet Earth, arguably in the universe. How did you get to where you are today?

(Joel Beasley at 00:29:03) Like, if you have to look back, obviously, it's probably more than one thing. But when you're talking with the next generation of technology leaders and they're like, you know, Charles, how did you do that? How did you become, you know, CTO? How did you get to that point in your career? When you look back, what do you think got you there?

(Charles at 00:29:20) You know, I had a mentor a long time ago tell me something kind of pithy. And it was something to the effect of good things happen to people who continue to be useful. And I think it's easy to say, hey, I want to do this or I want to do that. But I think when you're talking about a company, you kind of have to look at what does the company need. And I think it's ideal when your motivation and your skills line up with what that company wants and delivers.

(Charles at 00:29:51) But I think kind of counterexamples where folks are really focused on building a thing that they really want to build, but it may not line up with what the company really needs. And so I think you really have to ask yourself a question. You know, first is making sure that you understand what the company needs and not just in this particular moment in time, but even looking out six, 12, 24 months into the future. And I think lining yourself up with things that will help drive the company in a better direction, those are the things that I think you really want to focus on.

(Charles at 00:30:31) And sure, I wanted to be a CTO in my career for probably a long time. I don't remember exactly when that original nascent thought occurred, but I always focused on making sure that what we were doing was lined up with a benefit to the company and our customers. And so I think that that's maybe the, I guess, advice that I would give folks is to take that look at what the company needs. And obviously if you're in a company where you can't find that, maybe you need to find someplace where you can. But otherwise, I think that that's the most productive path that I can see to evolving.

(Joel Beasley at 00:31:08) Nice. Nice. I like that. Yeah. It's very important to understand how the actions you take and the work you do brings value to the customers because that's why they give you their money, because you're solving some problem for them, and that's how you get your paycheck. So it's like the circle of life. Right?

(Joel Beasley at 00:31:20) Well, Charles, we did it, man. We made a podcast. How do you feel?

(Charles at 00:31:27) Yeah. It went by fast.

(Joel Beasley at 00:31:28) I know. Right? Thank you so much for listening. And if you found this episode useful, please share it with a friend or colleague who you think would get value from it. And if you have topics that you'd like to hear discussed on the podcast, either add me on LinkedIn, or send me an email, [email protected].

(Joel Beasley at 00:31:47) Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.