Strengthening The Relationship Between Product and CTO with Chris Boyd, Head of Product at Trunk Tools

(Intro Narrator at 00:00:00) Today, we're talking to Chris Boyd, Head of Product at Trunk Tools and an old friend of Joel's, about strengthening the relationship between product and CTO. You're listening to Joel Beasley, Modern CTO.

(Joel Beasley at 00:00:18) Okay, I want to talk about that company where you had six different CTOs in five years. Tell me about that one.

(Chris Boyd at 00:00:24) Yeah, it was six different CTOs over about seven years. And it was interesting because I got to see a lot of different flavors, if you will, of CTOs. I got to work with a guy who's really, really sharp from a data science perspective. I got to work with another guy who was one of those examples of probably more of a senior engineer that got the CTO title. And then the category of CTOs that I described is they like to shake hands and kiss babies, but aren't necessarily in the code or thinking strategically about where the tech is going. And so then there were a few other flavors in there as well. But yeah, it was a lot of turnover for a lot of different reasons, but it was interesting to see how each of them operated.

(Joel Beasley at 00:01:06) So you were running product, interacting with these different people. The shake hands and kiss babies type CTO. I want to know more about that one. That's the politician.

(Chris Boyd at 00:01:15) Yeah, exactly. It was the one who, first of all, when you're in there for such a short tenure, you're trying to get your feet under you and understand which way is up. And so I'm not even sure it's fair to say that they had enough context for the business if they're only there for three or four months and then things start getting rocky or otherwise. But yeah, it seems like that category of CTO in multiple cases was just trying to understand where is the business trying to go and how do I help it get there, but not always being very reflective on the current state of things. So there was a frequent disconnect between the reality of the tech stack, the reality of just where things were at in the product itself in contrast with the vision of where we were trying to go. And so if you're in an executive meeting trying to encourage everybody, like, yeah, we're totally going to get where we want to go, but you're not taking inventory of where you're at. That's kind of a hallmark, I would say, of that type of CTO.

(Joel Beasley at 00:02:03) So did the attrition stop at some point? Did they find the right person, or did you move on from the project?

(Chris Boyd at 00:02:11) So the most recent CTO who's there now has remained there. So I think he's coming up on having the longest tenure. We won't hold our breath. But in general, a lot of the attrition was for extremely different reasons. Like, in some cases, it was somebody has the CTO title, but actually wants to be the COO, and so it's not a fit. And then you've got another situation where somebody is constantly trying to promise the moon from a technical perspective of what can be done and then missing those targets. And so that has consequences. And then there are others, I think, who just got absolutely burnt out by this. We want to be here, but we're currently here. And getting from A to B requires making some sacrifices and forecasts for sales and not promising things, et cetera. And if you fight that battle every single day for months and months on end, it's just exhausting. So you have people who get burnt out. So it was just a lot of different reasons for why people ended up walking.

(Joel Beasley at 00:03:08) But the company did well?

(Chris Boyd at 00:03:09) Yeah. Yeah, they did really well.

(Joel Beasley at 00:03:11) What was it like when you joined the company size-wise and then when you left?

(Chris Boyd at 00:03:15) Yeah. So I joined that company as employee number 17. They didn't really have anybody from a product perspective at that point. So I was coming in as kind of an individual contributor slash scrum master helping get organizational processes in place. And at that point, it was just so early. It was about trying to understand product market fit and there was one product at that point. And so we were able to grow the company at a pretty steady clip, you know, accomplish the whole triple, triple, double, double, double, double, which was really, really exciting. But there were just some decisions early on from a technical perspective that kind of plagued us. For example, specific aspects of the tech stack, like a lot of it was done in PHP in the back end and that's because people were just really, really loyal to that. But then we had all this trouble hiring people. And so anyway, there was one challenge after another, but kind of in spite of ourselves, we still were able to build a product that gave us a powerful wedge in the market and then launch numerous other products, acquire other companies, and then ultimately spin up other business units inside of the company. So when I left, it was at the beginning of last year, and we were up to just under 500 employees. We'd hit a $1.5 billion valuation and recently closed the Series D. And yeah, it was a really fun ride. Learned a ton in the process. It was just a lot of CTO turnover in the midst of it.

(Joel Beasley at 00:04:36) Did you have the same CEO the entire time?

(Chris Boyd at 00:04:38) Yeah. Yeah, absolutely. So he's one of the co-founders, and so worked with him consistently. Still stay in touch.

(Joel Beasley at 00:04:44) Nice. Nice. And then, you don't have to share this on the show if we don't want to, but I'm just trying to think of the listener's perspective. So during this turmoil of this, let's call it the CTO turnover turmoil, the executive members around that position are having these conversations. What type of conversations are they having around that CTO coming in? Were other positions like CMOs coming in and out, or was it just that CTO position turning over?

(Chris Boyd at 00:05:15) Yeah. I mean, there was definitely turnover in aggregate, if you will, at the executive level. I think that the challenge is that a lot of people are not necessarily qualified to interview a CTO, right? Like, that's a unique role. You have to have kind of been there, done that, and seen some things, and experienced the good and the bad, and then also have a sense of understanding as to what are the business needs, what do you need from a CTO? They're not all created equal. Each of them have different strengths, bring different attributes to the table. And so I saw a lot of conversations happening where there's like the CTO you think you need and then the CTO you actually need. And evidence of that, which you see a lot in businesses, of this disconnect between the executive leadership, which is and should be thinking multiple years into the future, sometimes being disconnected from the current realities and that influenced the hiring of CTOs. So it's like, can the CTO catch the vision? Do they get excited about where we're going? Check all of those boxes. That doesn't necessarily mean they're really well qualified to help us get from where we're at to where we want to be. And that's where I saw those disconnects emerging.

(Chris Boyd at 00:06:09) And so, where are you at now? So now I'm at Trunk Tools, which is also in the construction tech space. I'm personally just really passionate about construction. So the premise for Trunk Tools, I joined there in April last year to lead product, is that essentially these construction workers are kind of underserved from a technology perspective in general. You know, a lot of the tech built for construction is built for the back office or built for the architects or project managers, things like that. So Trunk Tools takes kind of a field-first approach and so we launched our first product a couple years ago that was focused on financial incentives for the workers who are actually swinging hammers, boots on the ground, so that if everybody knows what the goals are on the project and those goals are set with a financial compensation for achieving those goals by a specific target, then they receive that money instantly, in our case on debit cards. But then on top of that, we recently launched last year a generative AI product leveraging RAG, retrieval augmented generation for construction to accelerate all the user's ability to get to the information they need. The pain point in construction, one of the many pain points is that on a large commercial construction project, one of them, for example, that we have up in Manhattan right now, it's about a 400-foot tower and it's two towers, and it's about a $750 million project, and it has 3.5 million pages of documentation associated with it. And so the ability for any one person to ever comprehend all of that or recall that is just impossible. And so we have this running joke where if you took every page of documentation associated with that project and you printed it out and you stacked it, it would actually be taller than the towers that are being built. And so our whole premise for that product is allow people to ask questions about the project data and get information back immediately, including out in the field where they're having to address a lot of these questions in real time. So those are the two products we've launched. The whole premise for Trunk Tools is how do we leverage technology in a field-first approach to give people a suite of tools on the construction site that they can pull out of the trunk and leverage at the right time?

(Joel Beasley at 00:08:15) How did building your custom house help you with this? Well, were you at Trunk Tools before you built?

(Chris Boyd at 00:08:20) Yes. So the timing was that the house finished right as I was leaving my last company and I was actually using the software, one of the pieces of software, actually two of them that I helped develop at my last company while building the house. One of those for interacting with the construction lender and then the other for actually paying the GC and the subs and managing the invoicing process. But then, yeah, going into Trunk Tools on the heels of that experience, which in total was almost three years from acquiring the raw land through clearing it and creating a finished lot and actually building the house, learned a ton about the perverse incentives in construction, where in many cases the plumber comes on-site, does the work, and then doesn't do something really well, or it's incomplete and is excited about, oh, we got to come back and finish it. If he's getting paid hourly, then that's just more money in his pocket. In the same way, a lot of the trades don't necessarily communicate or have incentive to communicate with one another. And so if the plumber says, well, the HVAC guy is cutting off my route here, so I got to go around him. And then the HVAC guy gets upset because he's intersecting with something, like all of these clashes, if you will, occur on a regular basis, and you end up footing the bill. So as I was encountering a lot of that and answering the same questions over and over and over again, I saw a lot of the appeal of what we're doing now at Trunk Tools.

(Joel Beasley at 00:09:35) Nice. And so you've been at Trunk Tools for two years now?

(Chris Boyd at 00:09:38) No. It'll be a year actually this month.

(Joel Beasley at 00:09:43) Very cool. Why are you passionate about construction?

(Chris Boyd at 00:09:48) There was a study that came out. It got a lot of press in circulation. It was almost ten years ago now. I think it was McKinsey that put it out talking about technology adoption across different verticals. And the very top of that list was things like banking and finance, stuff like that. The very bottom of that list was agriculture, and then the next one from the bottom was construction. And that always intrigued me. Like, if construction is lagging in its technology adoption, then that means there's tons of opportunity. But the other thing that's really fascinating is that construction is one of the only industries that's becoming progressively less and less efficient. And so if you go back and you read about, for example, the construction of the Empire State Building in New York, like from the moment they broke ground to it actually having legal occupancy, it was like 14 months, which you imagine today a skyscraper getting built anywhere would take years, and they were able to do it so quickly. Now, obviously, perfect storm, there are a lot of things that contributed to that, but it's indicative of the fact that we are moving away from a lot of the efficiencies we've had historically in construction. And so that just attracts me because there's so many opportunities to make that better. Some folks are attacking it from the prefabrication route. Others are doing it from a strictly technology-oriented, what software do I use at what times. And BIM models have been all the rage for a while, but they have a lot of shortcomings, and they're expensive to maintain. And then others are looking at the robotics side of things and 3D printing. So there's lots of different things that are kind of converging, and there's still just so much opportunity to make it better. So that's why I'm excited about it.

(Joel Beasley at 00:11:16) Yeah. Early on in the podcast, I made a friend with Ben from BIMobject, and they were out of Sweden. And great guy, ended up going over there and hanging out with him and his team. Very big cultural differences from the U.S. office versus the Swedish office. But I asked him why he was interested in doing what he was doing, and he had been like an architect turned technologist, and he said to stop the waste. The amount of... And then from there, I got to meet, couple people over at Trimble and go out and speak at their company, and that was the thing I heard there as well, is there's apparently massive amounts of construction waste. Then I bought this place and started building stuff and renovating and all of that and doing the projects. And because this room that we're in now, this was just a metal building with a concrete floor. So we heated, cooled it, put drywall on it, insulated it, did all of that. And when I started working with the contractors, like, oh, yeah, you just buy like 30% more than we should need. And I was like, what do we do with the rest? Return it? They're like, nah. You just throw it in the dumpster. And I'm like, are you... Yeah. I get it. You don't want to have not enough, right? Because then you have to run back to the store in 30 minutes or anything, right? So I get it. But the fact the whole industry is designed that way is phenomenal. It's crazy.

(Chris Boyd at 00:12:33) Yeah. And on top of that, it's not that way by accident, right? So there are aspects where it's not just, oh, I have to go get more materials. It's that, oh, the sub is now booked solid for the next three weeks and can't come back. And so if the work that they're doing is part of the critical path and everything gets delayed, so you kind of overcompensate to try and prevent some of those delays. And so, yeah, you get a tremendous amount of waste. And I think waste is this overarching term in construction that's not just about material waste. There's waste in efficiencies where there was a study that was done that said basically 15 minutes out of every hour for a construction worker is actually productive work. And so that's a massive amount of headroom from an efficiency perspective just in time. But then on top of that, you have waste in the form of rework of I implemented this thing the wrong way, and now I got to go back and do it. The specification is what I followed, but there was a request for information that went out where the owner made a decision to change something and now it's got to be redone. And some of those are simple, right? It's like, oh, you painted it the wrong color. Others are ludicrously expensive where they poured concrete in the wrong place and now they got a wrecking ball to come in, jackhammers, so they can go back and redo it. So, on average in the U.S., construction projects assume between seven and nine percent of the total project costs will be attributed to waste. And so that's, you know, if you're talking about a $500 million project, that's a lot of money.

(Joel Beasley at 00:13:51) Yeah. We just need robots to build it, right? Have you seen any 3D-printed houses?

(Chris Boyd at 00:13:56) What do you think about that? I think that the application in housing makes a ton of sense because you can do a lot of things over and over again. The more you get into commercial construction, the more bespoke the construction process becomes to the project. And so I think that's where there's a lot of challenges. And, you know, again, I still think there's application. A building is comprised of tens of millions of parts.

(Chris Boyd at 00:14:18) And so some of those parts can be fabricated really easily on-site. We're also seeing more and more of that happening. I saw this custom home where there were so many custom window sizes. It was more cost effective to actually create the windows on-site. They had a truck there.

(Chris Boyd at 00:14:33) That's all it was doing was outputting these windows and to do it off-site and ship it. So, certainly a lot of opportunities there, but the cookie cutter aspect of residential construction lends itself more to that approach than commercial.

(Joel Beasley at 00:14:45) And then I noticed in your profile, you had we've talked a lot about efficiency and software and all of that, but and this might be outdated, but solving America's labor shortage.

(Chris Boyd at 00:14:56) Mhmm.

(Joel Beasley at 00:14:56) Is that like a different product or tell me about that?

(Chris Boyd at 00:15:00) Yeah. So the labor shortage problem is massive. First of all, I think more so than a lot of people realize. There was a negative impact on it with COVID and all of these jobs that were lost. But on top of that, you have an older generation that's been in construction that's getting ready to retire.

(Chris Boyd at 00:15:18) And I don't remember the exact numbers off the top of my head, but it's approximately 50% of the US construction workforce is of retirement age in the next five to seven years. And so, name another industry where that's the case. That's just ludicrous. And so, there's just not as much attraction to new individuals entering the job market to enter into construction. And so, solving the labor shortage is a multifaceted aspect. Obviously, you want to try and make it as attractive as possible.

(Chris Boyd at 00:15:45) So, one of the ways that we want to try and do that is make it more financially attractive and that's where the financial incentives come in. The other aspects we want to try and remove some of the, I won't call it grunt work, but the work that's just tedious. That's not about being great at construction. It's about just finding the right information. And so it's interesting to watch the younger demographics entering the construction space and being so surprised by the absence of technology.

(Chris Boyd at 00:16:09) Right? They're met by green screens. We're back in the nineties and a lot of construction has basically been treating that as it's normal and the expectation is much higher, obviously, for the younger folks that are coming in. And so we see opportunities there with both of the products that we've created and other products that we have planned to make it more attractive financially, to make it much easier for them to do their jobs, to empower them to focus on the things that are actually important, that require their thought process, not just tedious, let me find the right Excel file that has the right row in it that I need.

(Joel Beasley at 00:16:40) The green screens like at Guitar Center.

(Chris Boyd at 00:16:42) Yeah. Yeah. Yeah. Yeah. Like what's powering most of our ATM machines still to this day.

(Chris Boyd at 00:16:47) Right.

(Joel Beasley at 00:16:48) I heard Mike Rowe talking a lot about this. Mhmm. And so when we've had plumbers and HVAC people and all of that, I've asked them, and it's absolutely the case.

(Chris Boyd at 00:16:58) Mhmm.

(Joel Beasley at 00:16:58) I mean, to get service has been an interesting thing. So pre-pandemic in Florida, I never had a problem with anything. I needed it called, it's there. Yep. Out here, it's like there's a couple plumbers, Mhmm.

(Joel Beasley at 00:17:13) And they're all the same thing. You call them, voicemail, I'm retired now, but if it's really a problem,

(Chris Boyd at 00:17:20) you can

(Joel Beasley at 00:17:20) leave a message and I'll come on out. And I'm like, oh my goodness. Mhmm. And there are very few 20 or 30 year olds that we interact with at all. And when they do exist, we had a friend at church that was that way.

(Joel Beasley at 00:17:34) Mhmm. And he was only there long enough to learn the business and he started his own. Yeah. Because they just instantly realized, I can just buy a truck, wrap it, and there's so much demand here that I don't need to work there and I can make more money.

(Chris Boyd at 00:17:46) Yeah. Yeah. Yeah. And the demand is great as far as it making the job market attractive in construction, because obviously there's money to be made, but it also means that I think your individual scrutiny of who you're hiring often gets lower. Like, they're available.

(Chris Boyd at 00:18:02) I'm just going to take them. And that's where I've seen personally so many situations where it's like that plumber or whoever was not actually of the right caliber for what we were needing. And so, you end up again, a lot of rework, a lot of like, I gotta hire plumber number two to fix what plumber number one did and things like that. And so, you know, in general, in construction, there's not enough scrutiny and there can't be in many ways by a homeowner, for example, of like, you're not qualified to evaluate how effectively

(Joel Beasley at 00:18:31) it goes out.

(Chris Boyd at 00:18:31) Yeah. You call it in and Google

(Joel Beasley at 00:18:33) shows you who can come now. Yeah. And you just cross your fingers and look for the most stars.

(Chris Boyd at 00:18:38) Mhmm. Yeah. And then the swing between the bids can be outrageous. Yeah. I mean, I remember situations, even the building of our home, where I was trying to get bids on the electrical portion for just wiring it up and rough-ins, etcetera.

(Chris Boyd at 00:18:51) And it wasn't like, oh, that one's 10% higher. That one's 5% lower. It was like that one is 80% higher, and they don't claim to be doing anything any differently. But you don't wanna go to the lowest common denominator. And so, but you know that price is not always a dictator of quality.

(Chris Boyd at 00:19:05) And so it's just this weird, like, what do I do? And sometimes you just don't know until it's done and you gotta fix something or you find out the hard way.

(Joel Beasley at 00:19:12) Yes. Yeah. That's why the relationships are everything. Mhmm. Even in our building of product.

(Joel Beasley at 00:19:18) Yeah. You know, to know I used to look at, when I was 18, 19, 20, and I had software experience. I used to look at the older Mhmm. guys that were in the field that were 20 or 30, and I was like, I'm as good as they are at engineering and writing code. I can see their GitHub or whatever it may be.

(Joel Beasley at 00:19:36) And then I realized as I've gotten older that the value in the people who are more experienced, Mhmm. is partly in them from having seen a bunch of things

(Chris Boyd at 00:19:45) work

(Joel Beasley at 00:19:45) out and not work out. Yep. But also a huge value in their network. Mhmm. The fact that I can pick up that phone and I know there's five different guys I can call, Mhmm.

(Joel Beasley at 00:19:53) that could take a multimillion dollar engineering project and get it done right the first time. Yep. Those were expensive relationships to figure out how to build. Right.

(Chris Boyd at 00:20:01) Yep. And it's true, you know, just to go back to construction, construction in general has a knowledge transfer problem where, you know, the most experienced individuals in construction bring around them fewer mistakes, fewer situations of rework, less waste, etcetera, just because they've been there, they've seen things. But, you know, if you hire a new individual straight out of college to be a PM on a construction job site, there are going to be more issues. There's going to be more rework. There's going to be more waste.

(Chris Boyd at 00:20:29) And so, most of that, again, is knowledge transfer. We had an interesting example the other day where we had a project manager who was building a very kind of unique theater, old school kind of opera style theater. And he works for one of the largest general contractors in the US. And he sent out a message to everyone in the business and just asked like, has anybody ever done this before? And there were two people in the whole company and one of them had just retired. And he was like, can I just set up an hour with you?

(Chris Boyd at 00:20:55) And the story was so funny because he talks to the guy. He's like, what do I need to know? And of course, this guy's just rattling off all this stuff. My favorite example that came out of it, he goes, make sure you have every single person on the job site take a bathroom break at the same time after lunch. Put it on the clock, tell everybody to do it at that time.

(Chris Boyd at 00:21:10) He said, because you need to simulate what's gonna happen when intermission hits and everybody goes to the bathroom at the same time. And sure enough, the first day they did it, everything blocked up. It wouldn't work. And of course he wouldn't have known until prime time when it would have been hell to pay. So, just a prime example of like, without this knowledge transfer that we need and not just construction, but in software as well, you lose a lot and it gets really expensive, really fast.

(Joel Beasley at 00:21:34) And so, I wanna take just a minute to talk about product in general. Mhmm. So I've built products, but I always kinda did it as like the CTO under the CTO title, and I Yeah. I follow a couple different people. Marty Kagan's one of my favorite guys.

(Joel Beasley at 00:21:51) You know him? Oh, yeah. Yeah. Inspired. And I got to interview him.

(Joel Beasley at 00:21:54) Yeah. Nice. Like way he was so cool. Yeah. He was like episode 20 or something.

(Chris Boyd at 00:21:59) It was

(Joel Beasley at 00:21:59) super early. Mhmm. I sent an email to him. He's like, yeah. Because it was on my, one of my best friends, his name's Derek, and he had it on his bookshelf.

(Joel Beasley at 00:22:09) Mhmm. The Inspired book. And I was over at Derek's house, and I saw it, and he's like, that's

(Chris Boyd at 00:22:12) the best book I've ever read on product. Oh, it's great.

(Joel Beasley at 00:22:14) So I read it, and then I emailed Marty, and we set it up. But you tell me a little bit about because you have this working experience as the role of product specifically, Mhmm. Yeah. at a company that has scaled, another company that's in the process of scaling. Mhmm.

(Joel Beasley at 00:22:31) What are what are the one or two let's say the one piece of advice that you would give a young product person, someone getting into product right now?

(Chris Boyd at 00:22:38) Oh, man. I gotta whittle it down to one. Yeah. I mean, what I would say in general is that some of the best product folks I've ever worked with have an inherent sense of humility that they are not responsible for coming up with the best solution. They're responsible for doing the discovery with the customers to understand the pain points and then working collaboratively with the engineering teams and product designers and otherwise to propose solutions and then validate or invalidate those solutions with the customers.

(Chris Boyd at 00:23:06) And so I just think that that combination of humility and empathy is really, really important. This notion that products and product managers in particular, where you see folks who used to be BAs or business analysts and they're coming in and they're like, oh, I just write requirements. That's not a product manager. Not a good one anyway. And so I just think it's really important to have that balance.

(Chris Boyd at 00:23:27) You've got to have the humility and empathy and the business savvy to know that, like, you're not building this for fun. You're building this because you're trying to solve a problem, which should result in revenue and business stability. So, if you can dedicate time and get excited about understanding a customer's pain points and trying to represent those effectively to the rest of the business. I think there's huge potential there. And a lot of the talk about product focuses on these, you know, hard skills, if you will, of what you need to learn, how to interview and how to do discovery and how to translate that into tickets and all this stuff.

(Chris Boyd at 00:23:58) I think that the soft skills are just super important because you can build a subpar first iteration of a product. And if you've built good relationships and trust with those customers, they'll stick with you for years as you keep building out the better solution over time. They just need to feel heard, and feeling heard doesn't mean you always build what they want. And so, I just think that those soft skills are really important.

(Joel Beasley at 00:24:20) So, that

(Chris Boyd at 00:24:20) was more than one, but that's what I would really hone in on.

(Joel Beasley at 00:24:23) And then do you is that something people practice or?

(Chris Boyd at 00:24:29) Yeah. I mean, I would say that you practice it inadvertently, right, in the sense that you're doing these things on a regular basis in a product role, whether you realize it or not. But I know that for myself, having hired a lot of product managers and product associates and training them, and I've had product designers and product marketers and all of those folks rolling up through me, I think frequently expect that I'm gonna really drill in on these specific hard skills they need to improve in. And, we certainly spend time on that. But in many cases, it's just the soft skills because product managers have so much influence about the trajectory of a business. I think it was Andreessen Horowitz had described it as, you know, they're being entrusted to spend millions of dollars on the company's behalf, and they should know what it costs to build feature X and feature Y and why that thing is being created.

(Chris Boyd at 00:25:19) And if they can't answer those questions, they're probably not in the right role. And so, that again just ties back to this kind of overall sense of the influence that you have, the importance of the decisions you're making. And, you know, you don't want to over index on having data for everything because you'll get paralyzed. There is some intuition associated with it as well, but that intuition doesn't get developed unless you spend a lot of time with customers.

(Joel Beasley at 00:25:40) And what does your product team look like currently? Is it just you right now? Yeah.

(Chris Boyd at 00:25:45) So right now, I'm operating as the head of product and an individual contributor for the product management role, but then I have a product designer on my team and then actually implementations for now rolls up through me. That's my preference early stage in businesses because every time you implement a customer, you learn so much early on about what they need and friction associated with that process and ways that we can better manage their expectations. So that's the composition for now. We expect later this year, if we continue the growth, that I'll hire a PM or two that would be a part of that org.

(Joel Beasley at 00:26:16) And what was it like at BILT?

(Chris Boyd at 00:26:19) Much, much bigger org over time. I think at the peak, there were about 34 people in the product org. And like I said, that included product marketers, product designers.

(Joel Beasley at 00:26:29) What do product marketers do specifically?

(Chris Boyd at 00:26:32) Yeah. It's an interesting role. I actually really value it because traditional marketers think a lot about just how do we message this externally, and they're kind of looking to product to give them the ammo they need to know how to describe these things and the value it provides. What I really like about product marketers is they're also responsible for that external communication, but they're a little bit closer to the product team itself. And even early stage, like when a feature is in early ideation, if the expectation is that this feature is going to unlock a new market or unlock new revenue streams or otherwise, they're helping think through and shape how do we describe not only the value it provides, but what are the specific stakeholder groups that we expect to benefit from this.

(Chris Boyd at 00:27:15) They're spending time interviewing those people. How do I make this appeal to you? What are the pain points it's solving for you? Things like that. So, in the same way that product management is doing customer discovery to understand what to build, product marketing is doing customer discovery to figure out what to use to describe what's being built and to make sure that it resonates so that you're not just kind of throwing stuff up against the wall to see what sticks.

(Joel Beasley at 00:27:36) That's pretty cool. And we've gotten to work with a couple, like, sponsors that are product marketers. It's always somebody different at the company. Yeah. But I figured since you're mister product, which is what I'm gonna call you from now on, mister product.

(Joel Beasley at 00:27:51) Yeah. You could answer that for me. No. That's, uh, I'm glad you like Mr. Kagan.

(Chris Boyd at 00:27:57) Yeah, because, no, he's great. He keeps rewriting his book.

(Chris Boyd at 00:28:01) I know why not? New versions all the time.

(Joel Beasley at 00:28:04) Every year or two, it's like a new colored cover and it's a refreshed version.

(Chris Boyd at 00:28:08) Mhmm.

(Joel Beasley at 00:28:08) I'm like, that's not a bad way to do it, you know?

(Joel Beasley at 00:28:12) How did you guys come up with this idea? What was the exact moment that you came up with this idea to do the RAG AI for this 3,500,000 page document?

(Chris Boyd at 00:28:23) Yeah, so it was kind of funny. You know, as often happens, you're working on your core product, which in our case is our incentives product, and you're identifying opportunities where it could improve and stuff that you would like to make available to customers to make it more sticky or appealing. And one of the challenges we face with the incentives product is that, surprisingly maybe, a lot of people have a hard time translating work to be done into goals for a team to execute against. And so as we were encountering that friction, we saw customers having this kind of deer in the headlights look when they would go into the software and it's time to create your first goal and they're like, "I don't know," you know. And so we were having to do some education, almost consultative implementations of helping them understand how to do that.

(Chris Boyd at 00:29:09) And so in the back of our heads, we were thinking like, how do we help them? How do we take the documents and have the context for their project and maybe even recommend goals that would be really valuable for them? And so we started doing that internally using various large language model solutions and things like that. And we kind of had this like, wait a minute, what if we could do this for all the construction documents? And we started playing with it. And what was really interesting is our first prototype of the software solely manifested over text message because we're like, well, these guys are out in the field. They may not even have enough service to have a data connection. And so it's literally just you go into your phone, you have a phone number for your project, your phone number has been whitelisted, you text a question to that phone number, and then it responds within a few seconds. It gives you an immediate, like, almost joke of sorts of like, "I'm Bob the Builder and I'm gonna go excavate this question for you," you know, whatever.

(Chris Boyd at 00:29:57) And then it would respond and give the source documents where we found the information.

(Joel Beasley at 00:30:01) It says that?

(Chris Boyd at 00:30:02) Yeah, I mean, it's awesome. It had multiple different responses, but it was just that like, hold please, let me get this for you, you know, because we're dealing with text message, so we don't want them to feel like we're just ignoring them. And so we were amazed at the response. We were like, oh my gosh, you're answering questions. And in some cases, it was taking me thirty minutes to go back to the trailer, go through my documents folders, try and find the document where I think it is, Command-F in the document, all that stuff. So that was the point of origin. It was like, how do we help recommend goals for the incentives product? And then we built that prototype, and the feedback was really positive.

(Joel Beasley at 00:30:32) That's pretty crazy. Do you know how many of those queries you guys are getting a day? Are you talking tens or hundreds or millions?

(Chris Boyd at 00:30:41) Well, so there's a few different ways that we slice it. First of all, we have a little over $2,000,000,000 worth of construction volume on the software now, and that's growing significantly. We've actually waitlisted new customers because there's so much appetite for it right now, just because we try and manage expectations. You know, you have some folks who they hear about AI and assume that every day...

(Joel Beasley at 00:31:01) Yeah, exactly. It's not gonna do your laundry.

(Chris Boyd at 00:31:05) You know, we gotta do some expectation... No, exactly. It's not gonna replace you. It's also not gonna take over your life. And so we have been queuing that up such that we can manage expectations, we make sure we can facilitate integrations, because we did some initial, hey, give us a one-time data dump for the information on your project. And immediately people were soured on it three days later when there was new documentation that we didn't have and they're asking questions and the answers are from old data. And so we just said hard stop. We're not even gonna do this unless we're integrated. So built a bunch of integrations and that allows us to bring on new customers when their integrations are live and when we feel like we can facilitate it effectively. And so, yeah, I mentioned that one project that has three and a half million pages, but we're bringing on numerous projects every single day. And then we're seeing, we had this one case study we just did recently where we took four superintendents on a $500,000,000 project up in Wisconsin. And over the course of about forty-five days, they were asking a little over 300 questions during that time. And it was at the very end of the project, which is when you kind of expect things to ramp down. And on average, if they were out in the field, they said they were saving between twenty and forty minutes in the answers that they were getting back.

(Joel Beasley at 00:32:17) Nice.

(Chris Boyd at 00:32:17) And so, yeah, we expect...

(Joel Beasley at 00:32:19) It's a huge win.

(Chris Boyd at 00:32:19) It's a huge win. Yeah. And so in that case, it's a very small number of people using it on that job site, but it was just to get them excited about it and confident, and then we're rolling it out to other folks on the same site. And so that's where we're gonna measure project volume, number of projects on the system, number of users, and the number of questions that are being asked.

(Joel Beasley at 00:32:36) I'm a huge fan of starting small. Anytime I've ever seen a project or have been a part of a project where they tried to build the end thing with billions of users in mind to start, it has never worked. And every time I've been on a project where it's like you can essentially land the contract before the software exists because there's such a demand for it, and all you have to do is deliver these one or two features and they're happy, it's always grown into millions of dollars.

(Chris Boyd at 00:33:02) Yeah. Yeah, yeah. As long as you don't stay too sales-led for too long where it drives your roadmap, then yeah, I agree. I think that if you can listen to the customers, understand where the demand lies, make sure that the demand is coming from the people who actually have influence from an economic purchasing power perspective, then yeah, absolutely. So that's one of the things that we have to be really cognizant of, is that a lot of our users are project managers, superintendents, et cetera. They don't necessarily have purchasing power for their whole business. And so we have to make sure we're exposing the value we provide to those groups to the executive decision makers as well. And then as they... you know, then you get this kind of dichotomy of the features that are being asked for. What do the executives want versus what do these folks who are out on the ground want? And we're kind of trying to walk that tight rope as we go. But yeah, man, there's just so much opportunity, and we could spend hours talking about AI and RAG.

(Joel Beasley at 00:33:52) You get to experience it in your personal life. I'm a part of a couple newsletters that I get that are every day, there's a new data dump of 50 crazy tools. What's the most interesting, weirdest, craziest thing you've seen with the AI?

(Chris Boyd at 00:34:09) Oh, man. So there's a company that... it's kind of, I won't say it's weird, it's just good to see that it's kind of coming together. You know, Microsoft's had their Copilot thing for GitHub to help engineers. And now there's a group that, of course I'm gonna blank on the name, but they have a product. They're calling it Devin, which is supposed to be the first AI developer.

(Joel Beasley at 00:34:30) Oh, I just heard... yeah, yeah. I forget. It was a Chinese guy was on the video, and he was explaining the Devin, and then he did pull requests and participated in GitHub projects.

(Chris Boyd at 00:34:42) Yeah.

(Joel Beasley at 00:34:42) Yeah, we asked for an interview with that person.

(Chris Boyd at 00:34:45) Yeah. So actually, one of the guys who's been a member of our team at Trunk Tools had was going back to school and that sort of thing, and so he's actually gonna start working with them. So I think what they're doing is really unique and powerful for sure. That one jumps out. Otherwise, I mean, worked with some groups, all the stuff that is happening within AI around its overall progression and how it's enabling things like the video generation. Before it was just Midjourney generating images. Now it's entire videos that are being created.

(Joel Beasley at 00:35:13) You've seen Sora?

(Chris Boyd at 00:35:14) Yeah, Sora. It's wild. Yep. And I actually have some friends... Cognition. I did it for you. Cognition.

(Joel Beasley at 00:35:21) Yeah, that's right. Yeah, this is the guy. Here he is. Scott Wu. He reached out to Scott and we're like, dude, we gotta interview about this one.

(Chris Boyd at 00:35:26) Yeah. Well, I might have an in for you, so...

(Joel Beasley at 00:35:29) That'd be great. Yeah.

(Chris Boyd at 00:35:30) Yeah. But I actually have a guy that I spent some time talking to for a while. He is a head of product at a company called VEED, V-E-E-D.

(Joel Beasley at 00:35:39) Yeah, yeah, yeah.

(Chris Boyd at 00:35:39) And they do these little kind of micro videos that are all AI-generated, and then you can kind of choose the language that you want, the voiceover, and do you want it to have transcripts? Just stuff that would have taken forever in video editing software. I think all that stuff's awesome. But in general, what I'm excited about is that I think that historically in AI, there was this notion that you have to build a lot of this stuff. And now it's more about stitching together a lot of things that already exist to power a lot of different workflows. And we have kind of a philosophy at Trunk Tools, which is if AI is a wave, we don't want to be so far ahead of the wave that when it catches up to us, because we're investing all this money in building something, that when it catches up to us, what we've built is kind of... you're drowning. Yeah, it's not even relevant anymore. But we also don't wanna be so far behind it that we're not benefiting from all of these new advances. So we're trying to sit right on the wave and the sweet spot there and ensure that when a new model comes out, we can use it. And so we have no model allegiance today. We jump between Claude and between, you know, OpenAI and all these different solutions, the different vision models, et cetera, so that we can bring the best technology to bear in each use case. And that just gives us so much more freedom over somebody who's like, oh, I've got a Microsoft relationship, so I have to use what they're providing.

(Joel Beasley at 00:36:52) Yeah. I didn't ever understand that, and I walked into that later in my career where there's a tribal aspect in the Fortune 500 B2B world where it's like, whoa, we're a Microsoft shop or we're an Amazon shop. And I'm like, well, I'm a human, and I use the best tool that's available. And if it's just a cloud industry, you can just go on there and use it. It's like, you can just type it in and use it.

(Chris Boyd at 00:37:14) It's like being teenagers again, when we were like Xbox versus PlayStation fanboys. It's like...

(Joel Beasley at 00:37:19) Yes, Josh's house. Yeah. For people who don't know, Christopher's an old childhood friend who has recognized that I've grown a lot.

(Chris Boyd at 00:37:27) Yeah. Oh, boy.

(Joel Beasley at 00:37:31) Oh, man. Oh, man.

(Chris Boyd at 00:37:32) I like Josh's idea of a podcast after hours version where we can go deep into that.

(Joel Beasley at 00:37:41) Oh, man. So I had on... who did... Josh, you're gonna help me with this name recalling. Who did we just have on, but it hasn't aired yet? The guy that we talked to that does... you see Dune?

(Chris Boyd at 00:37:52) I haven't seen the new one yet, which is killing me. I'm trying to avoid the internet until I see it.

(Joel Beasley at 00:37:56) Me too. Yeah. I got kids, so I need babysitters to go see it.

(Chris Boyd at 00:37:59) Yep. Paul Salvini from DNEG.

(Joel Beasley at 00:38:02) From DNEG. So they do every movie you can imagine, they're doing the VFX for. They were the primary lead on Dune and Dune 2 and a bunch of other ones. And so I had him on, and I was, hey, what's going on with Sora? And when are you guys gonna be out of business?

(Chris Boyd at 00:38:18) Yeah, they've got 5,000 people.

(Joel Beasley at 00:38:21) They're not going out of business anytime soon. But he gave me some really good insight onto how they're strategically attacking it. So what they do is they've booted up teams within their org. We're gonna have to fact check this by listening to the actual episode. But they put them in a couple areas, and one of the areas was essentially a team to look at the other teams to see what low-hanging tedious work can be... they can use technology for. Because my mind was like, are you making the prompt to reality technology? And the answer, I believe, was no. What they're doing is they're using all of that AI technology to make their current workforce more efficient on the things they do.

(Chris Boyd at 00:39:01) It makes sense. I mean, as AI gets all this buzz, it's like all other tech advancements. You've gotta take it back to what's the actual business value it provides. And so I think a lot of discovery and kind of introspection on where are the areas that we know LLMs are really good or other AI models are really good that we could actually replace some of the stuff that we're doing today. And yeah, it's interesting because it kind of comes back to, you know, in data science, machine learning in general have all been the rage for a while. And it's like every time people start applying those things, they'd realize how crappy their data was. Like, oh my gosh, I can't actually get anything out of this unless I organize all this. It's kind of true in some cases with large language models as well, right? It's garbage in, garbage out. And so in many cases, these enterprise rollouts of AI solutions, they go like, oh, I'm getting bad answers back. Well, because all the data I'm giving it is terrible. It's unstructured. It doesn't know what is a good answer versus a bad answer, et cetera. So it's kind of an interesting problem to solve. I think people will oversimplify the complexity, get in there, realize shortcomings, and then start to figure out how deep the rabbit hole goes.

(Joel Beasley at 00:40:03) Yeah, that's exactly my experience with it. So, you know, you see the models and you're like, oh, I can train it. And then you learn there's different types of retrieval. There's different types of models and how they learn. And so I did what I was describing to you before, is I just went and got some really smart people. And I was like, hey, come tell me, I'm trying to do this, which pieces do I need to pull together? But basically, every CEO saw AI and they called their product or CTO or whatever and said, we need AI in the product. And they all got a back of the napkin education on it and then tried to put something together. Meanwhile, there are people who've been doing this since the eighties out there that are real deep, and they're doing it before it was cool, and there were all these different use cases.

(Chris Boyd at 00:40:48) They're charging a lot of money now.

(Joel Beasley at 00:40:50) They're charging a lot of money now to do... yeah, yeah.

(Chris Boyd at 00:40:52) That's true. And I think that's kind of unfortunate to a certain extent as far as the CEO pressure to just launch something with AI because you get a lot of half-baked, you know, "Oh, this must be AI because they called it AI." And so much of it is just kind of slapped on. You know, we see the support chatbots that everybody has on their site, which are hilarious, by the way, to play games with—the ones that don't scope their answers back to the data that's specific to the business.

(Chris Boyd at 00:41:17) I think there's an example online the other day of a dealership that put one of those chatbots on, and somebody asked it about, you know, "What do I do in this divorce situation?" And it gave a detailed answer. It's like, you've got to manage some of this stuff. But I do think that the buzz around it is positive. We're seeing so many advancements so quickly, so much capital being invested.

(Chris Boyd at 00:41:38) The report that came out a few months ago said that in the venture capital space, the valuations for AI-based companies was 30% higher than any of the other companies, you know, all that type of stuff. But there's going to be, I think, a bit of a reset or a reckoning of sorts about, you know, what actually survives through all of this because AI products in general have a bit of a churn problem. Because they get all this buzz and people use it and they're like, "It helps sometimes." And then it kind of tapers off. So seeing how these different product businesses manage that and ensure the value keeps people coming back, I think it's going to be really interesting to watch.

(Joel Beasley at 00:42:11) Well, that's where you go from people that aren't experienced with product, essentially, building something that's really cool, that's flash in the pan versus figuring out how do I build something that is consistently useful to people, not just consistently useful, but in a way that they're willing to pay me money every month. Yeah. Because that's, you know, my experience went like this. It's like, alright, my first projects, I accidentally had success.

(Chris Boyd at 00:42:36) Mhmm.

(Joel Beasley at 00:42:39) Yep. And then the next one was like, oh, since I'm successful, I'll just build this part because I can see it in the market. And then I realized what made my first project successful was the fact that I was sitting there in the office with the real estate agents.

(Chris Boyd at 00:42:47) Mhmm.

(Joel Beasley at 00:42:47) And I was listening to them and I was building solutions versus me being in my own office thinking about what they might need. And so that was one of the first maturities—learn how to build it. The second maturity was asking people if it would be useful and doing my discovery. This is all before I found Marty Cagan.

(Chris Boyd at 00:43:06) Yeah.

(Joel Beasley at 00:43:07) It was asking people if they'd use it, and they'd be like, "Oh, that's so cool." Well, first of all, everyone's going to tell you that, especially your friends. But there are times when it is really cool, and they might play with it, but it's not meaningful enough for them to continuously pay for it. And so that's when I came up with my second bar: you have to pay me money. Not only will you use it, but will you pay me money for it? And that's usually when everyone's like, "Well, you know, I, yeah, you know?" And so if you get to the point where they want it, they say it's useful, and they say, "Yeah, I'll pay you money," my third one is: will you pay me money before it exists? And that'll tell you how much of a problem it is. Because the big companies that are experiencing problems, they're looking at, "I'm going to have to go boot up a team and solve this problem," or "Use some consultant group to solve this problem," or "I've got this person right in front of me who's like, I can just be their first customer."

(Chris Boyd at 00:43:52) Yep. And one of the things I like to do in that specific discovery and analysis is that I will put a product in their hands, even if it's an early prototype, because I'm very cognizant of the fact that if I describe a product in detail of what it should be able to do, the individual is going to make assumptions about it that are probably inaccurate. So I need to give them something that they can actually touch and feel and put their hands on. So once I do that and they use it for a little while, I will then say, you know, "Give me a sense of your emotional response if I were to take this away from you and never give it back." And so it's almost an equivalent to, you know, "Would you pay money for this?" Because what I'm looking for is if they say, "I would be furious," or "I would be less productive," or "I would have to go back and do these stupid things that take up all this time," or whatever, then I can get them to pay money for it. And that becomes a secondary issue. And so sometimes we talk about the difference between a vitamin and a pain pill. I don't love that differentiation, but it's like, you know, if you stop taking the vitamin, you don't necessarily feel a big difference. But if you stop taking the pain pill, something's going to hurt, you know? And so if it's a pain pill and they feel its absence in a big way, right, then you're onto something. And so that's been one of the ways that we kind of stress-tested these things early on to get a sense of feedback. And even if it's not really positive initially, but you have demand from them for the things they want to see, if you see it moving in a positive trajectory when you ask them the same question a month later, that can also be a really, really strong signal.

(Joel Beasley at 00:45:17) Yeah. Was there anything that we wanted to get out there? Any advice, insights, Chris wisdom that we wanted to share out there into the world that we didn't get to yet?

(Chris Boyd at 00:45:29) Oh, man. Well, you know, a couple of veins. One, I think that on the product side, one of the things that gets talked about a lot are the different roles and responsibilities and what is the relationship between product and CTOs and tech. And, yeah, first of all, I'm not anti the structure, but I'm not as big a fan of the structure that you sometimes see in tech where product actually reports up through the CTO. And the reason why is because I think there should be a healthy tension between product and technology. And that healthy tension exists because product should always be pushing for building software as fast as possible to meet the business needs, to address user demand, to create new opportunities, et cetera. And the CTO and the technology org should be working really closely with product to figure out the how and what's going to take to get there and the resourcing and all that stuff. And the reality is, if you have a technology org that wants to build tech for the sake of tech, then that's going to work against the business goals. If you have a product guy who doesn't care at all about the tech and then runs the whole org, you can run it into the ground and you've got tech coming out of your ears. So I think there needs to be a healthy tension there. And I've had that before, and I value it in such a big way. I love it when a CTO tells me, "I understand you want to build that. I want to build that too, but we have to do these other things first. Otherwise, this thing's going to fall on its face. And here's what that's going to look like. And I've got a delivery plan in place. Here's when it's going to be done so that we can start building this other thing that we need to for the customers." And that trust that builds just allows so much more effective execution than the contrast I've seen, which is either the CTO grabs it by the horns and goes, "I'm going to dictate what this roadmap is going to look like and why because I'm the tech person, blah, blah, blah." Or the technology individual doesn't have the sway, the product person does, and the product guy just runs it into the ground because he builds all the great features and products, but they can't stand the scale or whatever else. So I think that tension is really, really important. And I think it's something that is frequently missed. I know in my experience, it was missed frequently in the interviewing process with CTOs. Like, what relationship have you had historically with product? Don't talk to me about pie in the sky. We're not talking about singing Kumbaya and going to a spa day together. Tell me about what you've done in the past and how you've worked with product. I think that is one of the most important things to suss out because if your CTO and your product guy are not on the same page or working in or pulling in the same direction, there's no more expensive conflict in the business. Engineers are the most expensive resource you can bring to bear. And if you guys are pulling in different directions, it's just going to go the wrong way.

(Joel Beasley at 00:48:00) You can tell the more experienced CTOs come in knowing a lot of this, and then they figure out what the organization's needs are. That's when I was looking at different people doing interviews, and I've been a part of different projects doing interviews, I found out that the best people, the people who we hired and then worked out the best, the entire interview, they were essentially asking us questions to help them understand the organization. And not in a lame way, right? In a way where they had tons of experience, and they knew their strengths. And they were trying to figure out the lay of this org and if what they had—because they know what they had—would fit in well. And if you don't have that, you just have people getting the job just there for a job. Yeah. Or they're still getting their experience.

(Chris Boyd at 00:48:51) Yeah. Red flag for me in the interview process is if I ask them if they have any questions for me and their answer is, "No, I don't think I have any." I'm like, what are we doing?

(Joel Beasley at 00:48:59) Yeah. How is that possible?

(Chris Boyd at 00:49:02) This is a major decision you're about to make, and you know your own skills and your own experience far better than I do, even though I've asked you a bunch of questions. So, yeah, I totally agree. I think that finding that connectivity there, I think, is really important. And the questions they ask are usually pretty indicative of what's important to them that they need to understand. And that can just tell you so much about what that synergy or lack thereof is going to look like once they come on board.

(Joel Beasley at 00:49:25) Yeah. Definitely. Definitely some red flags.

(Chris Boyd at 00:49:30) Switching topics for a minute. Another thing I think to touch on in this AI space is kind of fascinating. Jumping back a few topics. But in general, in the AI world, you have this comparison of whatever's come out to, you know, "Oh, what if I just use ChatGPT?" or "What if I just use Claude 3 or whatever?" And what's fascinating is that tied into this notion that these large language models are only as good as the data that you put into them, the huge dependency there is what data you're putting in in the first place. And so, you know, you can go into GPT-4 right now and you can upload a single document, no problem, ask questions about it, rock and roll. But a lot of these RAG solutions will say, "We'll hook up your Google Drive account or your SharePoint account. We'll pull all the documents out of it." Also great. A lot of value there. But what we're doing that I think is interesting, and I think there's a lot of opportunity for this in other verticals, is that there are document types, information types, et cetera, that are very bespoke to different verticals. And so in construction, we've got blueprints and all these drawings and detail views and reflected ceiling plans and electrical conduit makeup and all this stuff. And then you've got these really detailed construction schedules, these massive Gantt charts and that sort of thing. And if you can't extract all of that information and make it available to these large language models that you can ask questions and interrogate it and leverage the data, then, you know, the reality is that all RAG is not created equal. And so as a lot of these off-the-shelf solutions come out, Microsoft has their Copilot solution, which Copilot, by the way, is the most overused word in all of AI right now.

(Joel Beasley at 00:51:04) They're stressing that. Microsoft did it to themselves. They have their one tool called Copilot on GitHub, and then they're like, "You know what? Everything. Paperclip. It's going to be the new Clippy. We're going to put it on every product."

(Chris Boyd at 00:51:15) It kills me. It kills me. Yeah, you can literally take inventory of every single company that's launched something with AI. It's like, "Our Copilot." Anyway, so all that being said, there is huge value in creating RAG solutions that are bespoke to the industry that you're trying to approach. And that's what we're experiencing personally in the construction space. And I think that's just an untapped opportunity. And a lot of it's not necessarily sexy. Like, how do you get information out of documents? That's been a semi-solved problem for a while, but how you structure it, organize it, augment it, because you can do a lot with those models to improve the data that you're getting in. There's just so much opportunity. You get into agents, a slew of other things, but I would say we're still just scratching the surface of what's possible on a lot of that. But again, the key dependency is what data do you get in there in the first place?

(Joel Beasley at 00:52:00) I have a prediction that'll most likely be wrong, but because neural networks are designed based off of humans and how we think, I believe we'll end up with a bunch of different specialized narrow AIs and then AIs that specifically understand how to route and retrieve. And so because I noticed as these models get bigger and bigger and bigger, they can be good at conversation and they can be good at very specific things. But as you—I'm sure you've had it where you have a model, and then you realize you almost overtrain it. It's like over-explaining it to someone, right? And then you ask them and then they kind of jumble things because they're new at learning this, and then they give it back to you. And it's like, well, that's kind of right. So I learned in my personal experimentation of it, there's a certain right amount. I need a model, and I'm going to train it on all my past episodes in the context of there's multiple speakers, there's a recurring speaker, and I can tell it the stuff, and then I just leave that model there. And so when I need to go access a model that has that skill, I just go do that. And then one over here where it's like, you know, "Here's how I like to—here's my style of commenting," and I use that for my LinkedIn commenting or whatever it may be. And so I think that if you scale that up to the agents-type concept, that that's what we'll ultimately end up with. And then AIs are good at routing. Like the GPT store. Have you seen all the apps that have come in? So I got really fascinated with how it understands which app—if you install 80 apps, how does it understand which one to use in the context?

(Chris Boyd at 00:53:39) Yeah. That's part of the reason I have a love-hate relationship with the term copilot because it kind of works. It's somebody who's junior to you that's trained just like you are and could in theory take over if something happened to you. You know, there's a lot of relevance there, I suppose. But yeah, in general, to your point about routing, we're already doing a lot of this, which is really fascinating where you can use large language models in so many different ways. And one of the ways we're doing it is when a user asks a question, we use the model to classify the type of question they're asking. That allows us to leverage routers to determine which prompts make the most sense based off of the question type. And potentially, the entire retrieval process is very different. You know, one of the examples I like to use is sometimes it's best to use a large language model to actually retrieve information in the big corpus of data.

(Chris Boyd at 00:54:22) Sometimes it's better to use a large language model to write a SQL query to pull the information out of a structured database because then it's gonna be deterministic. You can rely on it, things like that. And so, again, I think those routers are really, really valuable. What I get excited about is, and we're starting to play with this, this notion that if the model can effectively understand the user's intent based off of their role and a slew of other things, the question they're asking, the context for the project, otherwise, then arguably, it's going to be best suited to dynamically write a prompt on the fly to represent what they're trying to accomplish.

(Chris Boyd at 00:54:54) And so that's where, like, I don't think we'll ever see a world where, in my space, a construction worker writes these detailed prompts. Like, it's just not gonna happen. But a world where the construction worker confidently can ask questions and the system creates prompts on the fly to reflect what they are needing, that I think is really powerful.

(Joel Beasley at 00:55:11) Yeah. That design pattern I was talking about with a guy, not on the podcast, just in my own personal life, but he was telling me about how they're using the prompts to then actually write good prompts and, like, rephrasing it. There was some term that they had for it. But that's exactly that. I had that conversation that week, and the following week is when I think Gemini got busted for their diversity.

(Chris Boyd at 00:55:34) Yeah.

(Joel Beasley at 00:55:34) Sorry, about the generating of past historical images.

(Chris Boyd at 00:55:38) World War II soldiers.

(Joel Beasley at 00:55:40) Yeah. And it's because they were doing that. They would, and what you want, it's a hard thing to describe what you want because I want you to guess correctly. But I don't want your nonsense added on top.

(Chris Boyd at 00:55:51) Yeah. Right.

(Joel Beasley at 00:55:52) So it's like, I wanna type into Google, like, two or three words, and it be unstructured, and I wanna get, I want it to know exactly what I want.

(Chris Boyd at 00:55:58) Right. Yep.

(Joel Beasley at 00:55:59) But I don't want your belief system injected into it. You just be the search engine.

(Chris Boyd at 00:56:05) Yeah.

(Joel Beasley at 00:56:05) Just be the thing. And that's where I believe the line gets crossed. Line gets crossed when they'll take their personal beliefs and then modify your search term.

(Chris Boyd at 00:56:15) Yeah.

(Joel Beasley at 00:56:15) For the sake, uh, who was it? I was on the call with a designer or somebody, and he was looking for pictures of something about, like, something men, like software men wear. Half the results were, like, Asian and black women. And I'm like, what? I'm like, your search term is literally, like, man at computer. You know? Like, or male at, and I was like, this is obviously altered search results because you don't have to try to get that type of result because it's already tagged, man. It's already, you would have to specifically hijack the query to return equitable results or whatever.

(Joel Beasley at 00:56:55) And it's like, that's not the point of this technology. The point of this technology is to serve us for what we're trying to accomplish.

(Chris Boyd at 00:57:04) Right. Yeah.

(Joel Beasley at 00:57:04) Not Big Tech's view of the world.

(Chris Boyd at 00:57:06) Yeah. Absolutely. And that's where it can do a lot of that stuff really well. But again, if you're manipulating what the user is putting in in such a way that goes against what they're trying to accomplish, then again, that's why I think the agent thing is so interesting because an agent should, in theory, be the best representation possible of the user's profile or their role or otherwise, and then they can better represent them in a lot of these things. But even, you know, I can't think of a single scenario where it would radically change the user's intent.

(Chris Boyd at 00:57:35) If anything, it's gonna do things like, oh, you said X and you meant Y in construction terminology, which would not necessarily make sense to most people walking around. So, I think that these models kind of layering on top of one another where the sequence gets really important is just so powerful because you can, and you can jump between models. Like, we played with, what if the user asks the question, the question gets classified by GPT-4, and then the classification and the question gets sent to Claude 3 Opus or Anthropic because it's really good at writing prompts. It writes a prompt, which gets passed back, was already optimized knowing it's gonna be used in GPT-4. It's actually sent against GPT-4 to return the result. Like, you can do stuff like that. And there's no reason that you shouldn't be able to flex these different models in these different ways. And then your users benefit. And that's what I love is that in many cases, you don't ship new features, you just use new models.

(Chris Boyd at 00:58:28) And then suddenly the product improves significantly. And that's just so exciting.

(Joel Beasley at 00:58:32) So I went down a rabbit hole because my background in engineering for hard software was Ruby.

(Chris Boyd at 00:58:38) Yeah.

(Joel Beasley at 00:58:39) Most recently. Right? And so I'm very familiar with, like, test driven development, Ruby and Java and those types of things. And so I was trying to wrap my mind around this of how do you test driven develop these models and deal with them. And one of the guys helped me, and he's like, look. The most common thing that's happening right now is you explain what the result should be, and you have a secondary model tell you if that result is correct. Yep. Exactly what you described. Yeah. Similar to what you described, but, like, the fact that you're using models to check other models is a fascinating concept, because I've got the TDD methodology, like, drilled into my head.

(Chris Boyd at 00:59:21) And so,

(Joel Beasley at 00:59:21) I'm wanting, like, specific code. And, uh, have you seen that happening at all?

(Chris Boyd at 00:59:26) I mean, even OpenAI, when they had their conference at the end of last year and they were releasing some of the new models, they were talking about the RAGAS, R-A-G-A-S, scoring and other things that you can use to try and objectively score the retrieval and accuracy of what's being found and that sort of thing. So, yeah, I think there's a lot of opportunities to score these things and to use models on top of models, et cetera. The reality though, again, kind of bespoke to some of these different industries, is that you can use these generic off-the-shelf solutions like that, but in many cases, you have to make it more specific. Otherwise, you're just like, well, it's better than nothing, but it's not great. So, yeah, I think that there's a lot of headroom there.

(Chris Boyd at 01:00:07) And I know that, for example, one of the things that we do is we create a dataset of questions with correct answers and linked documentation that we know about a project, for example, and then we run all of those through the system. And if we're not getting very similar answers and the same source docs and things like that, then that's an early indicator that something's off. And so, again, that's very bespoke to what we're doing. And so, I think that's one of the opportunities, but yeah, you gotta be careful because it's so easy to make a change in the sequence, the prompt, the model, or otherwise, that positively impacts a whole category of questions and negatively impacts another category. And so from a TDD perspective, how do you make sure that you're not absolutely wreaking havoc on a lot of things in order to prove something else over there?

(Joel Beasley at 01:00:50) Well, it's gonna be fun.

(Chris Boyd at 01:00:52) Yeah. It is. It is.

(Joel Beasley at 01:00:54) Uh, last thing. Have you seen the, there's a GitHub project called Autonomous LLM, and it's essentially just a bunch of links to the most interesting autonomous large language model type projects.

(Chris Boyd at 01:01:04) Yeah.

(Joel Beasley at 01:01:04) So I highly recommend checking it out.

(Chris Boyd at 01:01:08) Okay.

(Joel Beasley at 01:01:08) I think if you Google Autonomous LLM, GitHub, it'll just be the first result. I saw one the other day with these agents, these autonomous agents. That go, you give it an objective, and it uses its library of agents to think, reason, solve, task, all, it's basically a very small version of what we were talking about. But they have to be built in that ecosystem. Right? But it was, have you seen that project specifically?

(Chris Boyd at 01:01:36) I haven't seen that one specifically, but I'm familiar with that notion, because, again, like you said earlier, there are things that are purpose built for certain things, and you should leverage those and not create a generic one-size-fits-all solution. So, yeah, I think to your point, I see a lot of this going that direction as well, which just in general, I think there's so much opportunity in exploring the different workflows that you can influence with this stuff and then figure out, you know, what's the best approach to bring to bear because the approaches can be radically different to get to a specific solution. And so, at this point, like, a lot of what we're doing is just this experimentation. Like, you just gotta dedicate time to play with different ways to do this stuff. Um, and, yeah, it's improving so rapidly that it's hard not to get excited about it.

(Chris Boyd at 01:02:23) Like, I get that there's a buzz and I totally believe there's gonna be a bit of a reckoning, but I haven't been this excited about tech in a long time, because it's just moving forward so quickly.

(Joel Beasley at 01:02:32) Yeah. Well, this is great. We did it, man. We did it. We made a podcast.

(Chris Boyd at 01:02:36) We covered a lot of ground. We did. Yeah.

(Joel Beasley at 01:02:39) Thank you so much for listening. And if you found this episode useful, please share it with a friend or colleague who you think would get value from it. And if you have topics that you'd like to hear discussed on the podcast, either add me on LinkedIn or send me an email, [email protected]. Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.