Why Data Quality is Crucial in the Age of AI with Adam Dille and Zeba Hasan

(Intro Narrator at 00:00:00) Today, we're talking to Adam Dille from Quantum Metric and Zeba Hasan from Google about how you can optimize your data and make the most of it with tools like Felix AI from Quantum Metric. You're listening to Joel Beasley, Modern CTO.

(Joel Beasley at 00:00:19) Zeba, you're from Google. Adam, you're from Quantum Metric. What brought your companies together to do a partnership?

(Adam Dille at 00:00:26) Yeah, the history is almost ten years ago now. So 2016, about, we were in kind of a traditional database migration kind of exercise. Old school traditional database not serving our needs, and we were out there looking for new technology. And we were trying a bunch of different things, but one of them was Google BigQuery. And we were also a little bit unique in the way we were using it. It was kind of like more of a research tool back then, people pushing huge datasets into it, okay with waiting a little bit of time to get these massive answers back from it. And we jumped in and said, hey, we want to use this like a relational database, basically. And so they were kind of like, oh, these people are interesting slash crazy. And from there, we just kind of jumped into the menu of Google Cloud products, and we've engaged with a lot more than BigQuery over the years. And they've been amazing partners, hooking us up with PMs and just giving us all the information we needed at every turn. So just great partner to have building the company.

(Joel Beasley at 00:01:29) So, Zeba, are you running BigQuery over there?

(Zeba Hasan at 00:01:32) We're running everything, but I will say, Quantum Metric has been fantastic and just helping our product teams learn as well what the capabilities are and understanding more use cases. And Quantum's really been leading the way for our teams as well.

(Joel Beasley at 00:01:44) Nice. What are you both most excited about right now?

(Adam Dille at 00:01:48) Everyone's excited about AI, so gotta throw that one in there. But we definitely are, we've been a little bit averse to kind of like the shiny object over the years at Quantum. So really finding that applied AI use case over the last year was exciting for us, something that we can easily say this saves customers a bunch of time that they were spending in the past, and excited about serving our customer base. We work with the most amazing brands in the world. You just kind of go down the list of biggest bank, biggest retailer, biggest airlines. And so seeing those brands day to day, working with digital teams there has just been kind of the experience of a lifetime for me.

(Zeba Hasan at 00:02:29) We're learning so much every day. And, Adam, you touched on this, but I feel like AI erupted so quickly and we're moving so fast. There's always something new to learn and just understanding what the limits are, whether they exist or not. So I think that's been super fun to see.

(Joel Beasley at 00:02:44) Yep. Yeah. Well, Google's known for testing boundaries. I mean, they've pushed the envelope. I've been around for thirty-seven years now, and it's been crazy watching Google grow from startup to where it is today. Well, I want to talk about, because we have two very intelligent people here in regards to data, I was hoping you guys could tell me a little bit about how the buzzword data quality has evolved. That's been in the industry a while now. It's changed its meaning over the past couple years. Tell me about that.

(Adam Dille at 00:03:14) Yeah. I think about ten, fifteen years ago, talking about data quality. And back then, I think it was more about, do we have clean data? Are the signals coming in consistent? Or are they dirty every now and then? And how do we either kind of code around that or make sure that we clean up what we've got? And that's always going to be a problem. Like, even today, you gotta ensure clean data or have an idea of how to handle it. But nowadays, everyone's got a flood of data too. So back then, it wasn't so much like, oh, yeah, we bring in terabytes and terabytes of data per day. And now it is the case that everyone's bringing in their huge dataset. And I think data quality these days is more about what pieces out of that massive dataset indicates something that's really, really valuable to me or my business.

(Zeba Hasan at 00:04:03) Yeah. I'd like to add on to that. I just think that previously, data quality was really just seen as a cost center concern versus a strategic one. Right? So it was just letting the IT people handle it and making sure database entries were input properly. But now it's seen as a way that directly impacts a business's outcome. So I think customer experience, going to some of the things Quantum Metric does a great job and kind of tying the ropes with. But seeing how data quality really ties directly to customer experience, I think, comes full circle here. So, you know, poor quality data isn't just a technical problem anymore. It means having bad data could involve customers getting wrong deliveries or receiving incorrect communications or just making decisions based off of bad information. Right? So I think the stakes of that data are so much higher now than they were in the past.

(Adam Dille at 00:04:52) Yeah. I also like to think of data as kind of two tiers. Because we've got so much of it, I usually try to break the data that I'm working with into sort of tier one and tier two. And I want tier one, you know, we work a lot in analytics at Quantum, and I think of tier one as, like, what things are going to tell me whether or not everything is okay. So they're like my most important metrics, my most important data points. And then if something's not okay, my tier two is there for me to dig into why not. You know? So breaking those two apart helps you think about most important signals and then kind of your indicators of why that particular problem might be happening.

(Joel Beasley at 00:05:31) Now is this what Felix does?

(Adam Dille at 00:05:33) Yeah. So part of Felix is, you know, we deal obviously with Gen AI, we deal with context windows limited in the amount of data that we can pass to an LLM, and Quantum brings in a ton of it. So a single session, if you're on a retail site for thirty minutes, tons and tons of signals coming in, and we can't pass it all to an LLM because we're limited by that context window. So there's an exercise of what pieces of that data are likely to tell me some of the questions we want to answer about that user, like, what were they there to accomplish? Did they solve that or not? And if not, what's sitting in their way? So that is definitely an exercise with Felix of figuring out what we want to include and what we want to strip away.

(Joel Beasley at 00:06:15) Yeah. It's amazing to see how the advancements are happening in the large language models too.

(Adam Dille at 00:06:20) Mhmm.

(Joel Beasley at 00:06:21) So let's talk a little bit about AI initiatives. Right? Have you seen organizations succeed or fail with AI because of the data?

(Zeba Hasan at 00:06:31) So I always like to use this metaphor, but you can't build a solid house with a messy blueprint. Right? You would have the best blueprints and the best architects in the world, but if the foundation itself is weak, the whole house ends up being compromised here. So I think that same principle applies perfectly to AI. So your data essentially acts as the blueprint for your models. And if that blueprint is unclear, inconsistent, or incomplete, no amount of sophisticated, the newest and greatest technology can make up for those fundamental flaws here. And I've seen that play out in a lot of organizations and how they approach AI initiatives. And I feel like the successful ones invest heavily in their data infrastructure and their governance before diving deep into AI. So they're going in and making sure their data is properly labeled and that they understand the data lineage. And they usually end up starting with smaller focused AI projects where they know their data is actually solid and then go on from there, and they'll expand after that. On the flip side, I've seen companies rush into AI projects without this foundation because, honestly, I mean, like we've said, AI is moving so quickly. They feel that pressure to get involved, to jump into AI. So they usually end up with models that might perform well in test environments, but when they move to production, because the real-world data is a lot messier than what they tested on, it just ends up leading to unstable results. So I've seen both ends of it, and it's clear to see that, you know, just making sure you have that solid foundation is the way to lead an organization to success there.

(Adam Dille at 00:08:01) Yeah. I plus one on that, the strategy. I think about another topic in engineering that's not necessarily AI, but something we've been dealing with for a long time. I've seen over and over again, logging around engineering projects. You know? It turned into kind of like we started by logging something because it would be really important to know when this error or this particular condition occurs. And then we just sort of piled on and piled on and piled on over time. And we went from something that contained really useful messages to something that had such a mess in it that we don't even listen to any of it anymore, including the useful messages in there. And so when we would have a logging strategy, then we would end up with something that could be maintained in a way that was going to be useful over time. So it's like, you know, the same old problems we've been dealing with will rear their heads when you approach AI without that kind of strategy.

(Joel Beasley at 00:08:55) Yeah. I mean, my background is software engineering, and I remember getting the bill for our logging data. And I was like, whoa. Alright. We gotta be more intelligent about what we're logging here. Yep. So we're talking about LLMs and stuff. What happens if you don't have good data? Do you just get bad results?

(Zeba Hasan at 00:09:15) I think there's a lot more of a risk factor involved there. I think when you're giving a model bad data or improper context, it can lead to a super misleading picture. So I think one of the biggest risk factors we see here is hallucination, which is just a way of saying that the AI model may have made things up to fill in its gaps in knowledge. And the tricky part, I think, with that is that the model will do it in a way that sounds completely convincing. So if you give your model partial information about your company's return policy, it might tell your customers, hey, yeah, you can return this after sixty days. That's completely fine. Because to the model, it sounds reasonable enough even though your actual company policy might say, no, you only have up to thirty days. So it's creating reasonable-sounding outputs, but it's incorrect because it's filling in the gaps of what it knows. So I think there's a bigger risk at play when you're feeding in improper or not properly labeled data at the end of the day.

(Joel Beasley at 00:10:12) Yeah. How do you know, Adam, how much data is enough data?

(Adam Dille at 00:10:16) That's a good question. I mean, there's obviously a round and round process in something like prompt engineering, working with an LLM. And so I think you do get to a point where you realize, I'm taking out the pieces that I don't need and getting the result that I do want as a result of that. And that's the point where, you know, you've struck a good balance. But there's also not just bad data, but bad prompting to go with it. And if you, for instance, take good data and pass it to an LLM with minimal prompting, you're kind of at the whim of what that LLM has been trained on publicly, you know, massive datasets that have gone into training. And you're probably going to get some results you don't expect because you didn't coach the LLM on this is what these data points mean. This is how it's structured. When this thing points to that thing, it means this relationship or whatever is specific to your data. So your prompt engineering really has to get to the point where it describes the data uniquely or, you know, in detail. And then your prompting asks for what it needs based on that data and that description of the structure.

(Joel Beasley at 00:11:28) What is feature training? I haven't heard this before.

(Adam Dille at 00:11:31) Zeba? You want...

(Zeba Hasan at 00:11:32) You could go ahead with that. I know you've done a lot of that.

(Adam Dille at 00:11:35) Yeah. You mentioned this. You're actually the one that taught me the term. But it's, I'm using an LLM, and I have my dataset. And then I begin prompting, and I've got an idea of some kind of problem I want to solve. So in the world of Quantum, one of the things we did was, we have session replays. You know, every one of these brands I talked about, every user who goes there to that airline or that retailer, there's a representation of what happened in their session. And our users, one of the typical patterns with our users, they go back and watch a lot of these replays a lot of times to figure out what was the user there to accomplish and did they run into a problem along the way. It might come from a VOC survey or an executive escalation. So they're putting all that time into watching that replay and finding those things that are pretty consistent questions that they want to answer. And our feature was we want someone to be able to pop into that replay and click a button and in two seconds figure out what was that user there to do, did they have a problem on the way to that goal of theirs, and did they bail, and what problem actually stood in their way. So that's our feature. And to train the LLM on what that feature is means we gotta describe the data that we're giving to it about that session, what kind of summary we want out of it, what kind of questions we want answered. So the, you know, those few things I keep mentioning, we want that in a short summary format. And we describe what does this session data mean. This is a user online, potentially shopping or, you know, looking to open a checking account or buy a flight, that sort of thing. And that helps it to understand what's our goal. And so we're training that LLM for that particular feature.

(Joel Beasley at 00:13:19) Alright. So it's similar to prompting. Like...

(Adam Dille at 00:13:22) Yeah. Yeah.

(Joel Beasley at 00:13:23) Yeah. Okay. Alright. Cool. So I had new term. Whenever I see new terms, I'm like, I gotta understand what this is. Yep. Do you do any training across public data?

(Adam Dille at 00:13:32) We don't do training across public data. In fact, we don't even do training from, for one customer from another customer's dataset. It's all kind of siloed per customer that we've trained the model to understand for our particular feature, what are we trying to accomplish for our customer. And then we do a little bit of, you know, training one industry versus another so that the LLM understands airline things versus insurance things versus banking versus retail, et cetera. But no public training other than what that LLM that we're using, Gemini Pro, in our case, has been trained on publicly.

(Zeba Hasan at 00:14:12) And I know in BigQuery, we have a ton of public datasets as well. So things like weather, you know, just public insights as well that a lot of customers use to try augmenting any data that they already have.

(Joel Beasley at 00:14:23) Where do people go wrong building AI?

(Zeba Hasan at 00:14:26) I think one of the biggest mistakes I see when, you know, building out AI or just new products in general is a lot of people assume the more data, the better. So I often use this analogy. Adam, we talked about this before as well, but adding more hay to the haystack doesn't necessarily make it easier to find the needle. Right? So it actually makes it harder. So I think one of the mistakes people make is not being intentional about the data that they're actually collecting and how they're using it. So we talked about how BigQuery has a bunch of public datasets. Which one of those datasets is actually useful here? Which one is actually going to make an impact on, you know, your end results over here? I think, I mean, some examples that I've seen in the past are, you know, for a health care company, for instance.

(Zeba Hasan at 00:15:11) Maybe they're building an AI system to predict patient recurrent visits or no-shows, and they're going full throttle on collecting every single piece of data they can. So things like patient demographics, medical histories, interactions with the hospital systems, and then throwing in things like weather forecast or air quality. This takes a lot of effort, and they might go through this entire exercise in putting all these resources forward to getting that data, only to realize a few months later that they found that over 90% of their accurate predictions came from just a few of those data points. So it's a ton of time and resources that they spend only to realize, hey, we didn't actually need all of this data.

(Zeba Hasan at 00:15:49) So I think this goes back to the whole haystack example. Adding more hay to the haystack isn't going to help getting accurate quality results at the beginning.

(Joel Beasley at 00:15:57) We want to throw all the hay in the baler, Zeba. Yeah.

(Zeba Hasan at 00:16:02) I think that's just human nature. The more, the better, right? But not necessarily in this case.

(Adam Dille at 00:16:06) It's also an easy first step of just, like, I don't have to think about how to get to the right data. Let me just pass it all the data, pass it too much, and then that's your downfall there. And I think another general problem with product development that applies to AI as well is just not understanding the core problem from the very beginning that you're trying to solve. So in our case, we scoped all the way down to that. I want to save one of our customers, on average, twelve minutes that they're spending watching this session replay by telling them the answer to these few questions.

(Adam Dille at 00:16:41) But if we hadn't thought about that from the first place and just kind of gone broad at replay summarization, then we're not scoped in on the actual problem we're trying to solve. We're just kind of trying to broadly summarize a bunch of things, and we're more likely to get things wrong in that case than we are if we're really scoped in on those three questions that we're trying to answer. And then, of course, from there, we had a first solution, and we went more broad because we were able to target more personas and more problems we wanted to solve from there.

(Zeba Hasan at 00:17:12) I think that also leads to how important it is to understand domain expertise too, Adam. Like, I feel like a lot of leaders will go in and just decide, hey, we need AI experts to lead this project and then sideline people who actually understand what the business problems are. So you might be trying to understand specific aspects on why a customer might be purchasing something. You want to predict that a bit more, but you bring in an AI expert who doesn't really have interactions with the sales teams. They don't actually interact with the customer to know why a customer might actually be upgrading. But speaking to sales teams, customer success managers, and knowing the business, they might be the ones to help lead you into understanding which data points to actually hone in on versus looking at the entire population of data that might exist out there.

(Adam Dille at 00:17:55) That's a topic I love talking about. I'd get on a rabbit trail, but I love talking about having the field teams in the product development process. Not just talking to them every now and then, but having them as part of the team working day to day.

(Zeba Hasan at 00:18:07) For sure.

(Adam Dille at 00:18:07) I love it.

(Joel Beasley at 00:18:08) Well, I mean, this is one of the oldest problems in product development, right? That's where I spent a lot of my career. Right? And people just building stuff because it was cool, or they pick the technology and then try to find the problem. Right? And the way that you have success is by really clearly defining the problem, and then that will dictate what technology and what resources and what people need to be involved. And then to Zeba's point, spending time with the customer or whoever is experiencing the problem. All right. So tell me a little bit more about Felix, Adam. What can Felix do?

(Adam Dille at 00:18:40) So some of the basic things I talked about are tell you what a particular visitor to a web application or native application was trying to do and how that experience went down the line. Did they make it to their goal or not? But one of the cool things we've figured out we can do with that same exact data is if you engage support for, say, changing your flight—you went to the website or the native application, you're trying to change your flight, something went wrong, and you engage support—you know, they have massive call centers. And today, you're going to call in and someone's going to say, hi, tell me what you're calling about today. You know? Felix has all the context of why I was on that airline's website or native application just now, right before I called in. And we figured out that we can put a summary of that session in front of the call center agent or the chat assistant agent and say, Adam was just trying to check into his flight to Phoenix tomorrow. He was trying to change his seat along the way, and he had an error at this point, and he bailed, and then he called, of course. And so that agent can pick up the phone and say, hey, Adam, I saw you were trying to check into your flight and change your seat on the way to Phoenix tomorrow. Can I help you get that done? I'm going to be way happier hearing that when I call in instead of, hey, why'd you call today? Oh, let me transfer you to the next person. Then they say, why'd you call today? So it's the same exact solution, just put in front of a different audience and with slightly different prompting to give the right information to that agent. And then it creates a magical experience for that customer who's engaging with the support team. So that's one of the other things it's able to do.

(Joel Beasley at 00:20:26) I mean, the only thing better than that is if it just worked.

(Adam Dille at 00:20:29) That's right. And yeah, we have a whole path to that too, which is, you know, quantify those errors that are leading to calls and then sending them back to your product team so that you can knock down the calls in the first place. So it's like, reduce the calls that are coming in, but when they do come in, handle them better for the customer.

(Joel Beasley at 00:20:47) Yeah. If you can prioritize based off of number of calls, frequency...

(Adam Dille at 00:20:52) Yeah. Time spent on the line. All that.

(Joel Beasley at 00:20:54) That's good. That'll save a lot of time in meetings and human interactions, right? Because usually, the way you get that is you're spending time with the head of that department, and you're talking with them, and you're engaging and interacting with them, and you hear what the problems that they're experiencing. But now it just cuts right through all that.

(Adam Dille at 00:21:13) Yep.

(Joel Beasley at 00:21:13) Which is great. Which actually is my next couple questions here about the balance between human and AI, right? So everyone's talking about it, at least on X. But the role of humans and AI—what do you think the future is that we as humans are going to play in this AI game?

(Zeba Hasan at 00:21:36) I feel like a lot of people think AI is taking over, machines are taking over. And the reality is AI still ends up being just a tool, right? And like any other tool, you need human guidance. So I feel like people shouldn't be worried as much about being replaced by AI. It's more about collaboration there, right? So I like to think about it like the introduction of Excel for spreadsheets. Right? Excel didn't end up eliminating the role of an accountant. It eliminated a lot of the manual calculations that were taking place in the back end and freed accountants to focus on more higher-level analysis and strategy. The accountants who ended up thriving weren't the ones who were best at doing manual calculations right in their head. They were the ones who could actually use those tools to provide better insights and make better strategic decisions at the end of the day. So while AI is super, super powerful and we can see that with all these use cases taking over the world, it still struggles with things that we do need humans for, like making nuanced judgment calls and letting you know whether a decision actually aligns with your company's values or long-term strategy. So I think we're moving more towards a future of augmented intelligence versus artificial intelligence replacing humans. And the organizations that are actually successful, they understand that. They're not asking the question of how can AI replace my workers. They're asking how can AI help make our experts better at making decisions faster. So I think it's more just a mindset change there.

(Joel Beasley at 00:23:02) Yeah. It turns out that's not a popular conversation.

(Zeba Hasan at 00:23:05) Right. Yeah.

(Joel Beasley at 00:23:07) It's important, though, right? Because we're all trying to figure this out together, to be honest with you. Like, we're all in this weird space, and it's like, all right, how are things going to change? And then we still—Adam, I'm sure you see this—we still live in this world where there's mainframe credit card processors that are running code from the eighties and nineties. So it's like even when the technology does come out, there's just that natural human resistance of the time it takes for us to experiment with it and adopt it. So on the long term, I'm optimistic. I do think there will be pockets of chaos in that. As humanity, as humans, we're going to have to bind together and figure out how do we help that? You know? Like, if graphic designers go out overnight, it's like, okay, we've got to figure something out to help them bridge that gap between however they're going to spend their time in the future or whatever it may be.

(Adam Dille at 00:23:58) Yeah.

(Zeba Hasan at 00:23:58) I think we're all learning as we go, right? Things are just moving so fast. We don't really know what the wider implications are. But I mean, the only one to judge whether something is actually reasonable or something that the company can align with are humans at the end of the day. AI will do some work for you, but at the end of the day, someone needs to give a little green light to actually move forward.

(Adam Dille at 00:24:18) And I think there's so many use cases that are ripe for just improving our lives before you even talk about changing headcount. You know, I just talked about two that save our customers time and save our customers' customers time that would have been spent either watching session replays or sitting on a phone waiting for an airline support agent for twenty minutes instead of two because the agents are getting through calls quicker because they're getting straight to what that customer needed. And those don't change headcount numbers. It just means a better experience for the end customer. And I think there's also a connection to risk. So when I'm thinking about human in the loop versus, you know, straight fully AI-controlled solution, I'm going to think about risk of whatever I'm building. You know? So I might be building something where I'm just trying to summarize some data for a user, and the risk of me getting that wrong is low. And I can put a message there that says the AI is not always right, you know, check the results. Versus there's AI in Tesla's self-driving car solution. You know?

(Joel Beasley at 00:25:26) Have you used that, by the way?

(Adam Dille at 00:25:27) I have used it. And yeah, most of the time, it's magical. But super high risk. You know? So I'm going to think about how quickly I want to roll a solution like that out to customers, how often I want to put a human in the loop to check results before they go to a customer. And, you know, the more human in the loop we have, the slower it is to that thing that everyone worries about, which is just robots are running everything all the time.

(Joel Beasley at 00:25:53) Have you tried 13, Full Self-Driving 13?

(Adam Dille at 00:25:57) I have. Latest one? Oh, yeah. Pull out of the parking—I love it. You never even touch the brake or gas. Just push the button and it takes you to the spot.

(Joel Beasley at 00:26:03) Zeba, have you had this experience yet?

(Zeba Hasan at 00:26:05) I drove a Tesla once, and I freaked out because I didn't realize once you put your foot off the accelerator, it's—

(Joel Beasley at 00:26:12) The regenerative braking. Yeah.

(Zeba Hasan at 00:26:14) Yeah. That made me super uncomfortable. So never again.

(Joel Beasley at 00:26:17) No worries.

(Adam Dille at 00:26:17) You get used to it, though. Like, you can't go back.

(Joel Beasley at 00:26:19) You can't go back to the gas to brake. Yeah.

(Zeba Hasan at 00:26:22) I haven't gotten on that bandwagon yet.

(Joel Beasley at 00:26:25) My wife doesn't love it, to be honest with you. The Tesla's my car, so she's like, you drive that, you do what you want. But she likes her gas cars.

(Zeba Hasan at 00:26:36) I do like the little music and Christmas stuff that they add into it. It makes it fun.

(Joel Beasley at 00:26:40) Yeah. It dances. Yeah.

(Zeba Hasan at 00:26:42) Yeah. The light show is great.

(Joel Beasley at 00:26:44) I know. It's like a high school nerd got to make the car.

(Adam Dille at 00:26:48) I was going to say as a tech nerd, it just makes so many bells ring in my head of good buzz. You know?

(Joel Beasley at 00:26:56) So company buy-in. I want to talk a little bit about that. How do you get company buy-in for AI projects?

(Zeba Hasan at 00:27:03) I think the best way to start with leaders is to make sure that they understand this whole data foundation piece is not just a data concern, but actually a strategic one, like you mentioned earlier. So instead of saying, hey, we need better data quality, maybe bringing it as a way of saying we can reduce customer churn by X percent if we have these specific data capabilities. So just bringing it in a way and tying it to data initiatives that directly measure business metrics, I think that really puts it in a way that leadership will actually care about. So talking about revenue, cost reduction, customer satisfaction optimization, you tend to get much better engagement with leaders from that standpoint and get them more excited about introducing AI. And I think just going to that, I feel like a lot of leaders, again, they'll just brush into AI because it sounds exciting. But making sure that they understand that you need that basic data infrastructure and governance first so that that AI project doesn't fail, I think that's super, super important. Because otherwise, they'll go in and start this AI passion project and realize, hey, it's not what it's cut out to be, only to realize that it's because of the data. You didn't spend time on that first.

(Adam Dille at 00:28:12) And not only understanding the need for data quality and some strategy and data initiatives, but there's an opportunity for more evangelism of the technology itself and what it actually does. So building the understanding of what Gen AI is good at and what it isn't good at. And it's so interesting that it's a technology that is so kind of broadly out there right now. You know, if you asked an executive ten years ago, like, what do you want to do with SQL? He'd be like, what's SQL? Is that a cereal or something? And you ask them about Gen AI now, and all of them know. You know? Everyone is using it in their day-to-day life. They all know the need to get behind it, but they don't all know what it can do and what the core of an LLM is built to do. And so there's this line of understanding where on one side, everyone thinks it's magic. And, you know, the solutions proposed are just pass all the data to it, and it'll just tell you what you need to know, just magically. And on the other side, it's like understanding what it can truly do, and the solutions proposed from that side of the group are way more in line with reality. And I think we need to shift that line so more people understand what the technology does. And so I'm just advocating for, you know, us teaching more people what the technology is capable of and what it's not so that we're all thinking in terms of realistic solutions.

(Joel Beasley at 00:29:43) Yeah. And for me, you know, my background is software engineering, right? So I build lots of software companies and projects. And when I first started playing with the LLMs, I was like, this sucks. Like, these things aren't that good. But then you kind of figure out how to use them correctly. You figure out what their limits are. Like, at first, have you guys heard this term "one-shotting an app"? Have you heard this term? It's basically where you make one prompt and the output is a fully functional application. And so there's all these people on X, Twitter, whatever. They do these one-shot contests where they write one prompt and make like a Spotify clone and it works, you know?

(Zeba Hasan at 00:30:23) Well, that's awesome.

(Joel Beasley at 00:30:24) Oh, I know. I know. Yeah. So they've been doing this for like two years. They call them "one-shotting apps." And at first, that's how everyone's mind kind of thinks it would be, right? Because you go in there and you describe it all. But it turns out that while that's a fun game and a fun thing to watch, the real benefit of the LLMs in programming is figuring out at what level of resolution can you use this thing? And once you start figuring that out, it becomes a very powerful tool that gives you massive efficiencies. But you have to play with it, and you have to resist that urge to dismiss it early on.

(Adam Dille at 00:31:01) Yeah. It's probably in that area of coding assistance. It's probably going to be really good at helping you reduce time doing boilerplate things. It might be decent at coming up with a greenfield solution. Probably not great at taking an existing code base and adding new features or providing maintenance on top of it.

(Joel Beasley at 00:31:23) Oh, it is now.

(Adam Dille at 00:31:24) It is now. It's getting better.

(Joel Beasley at 00:31:25) Yeah. It's getting really wild out there. I mean, it's my full-time job to try to keep up with this stuff. And every week, I'm texting a buddy or two about, like, look at this advancement or that advancement. I won't go down the rabbit hole there.

(Zeba Hasan at 00:31:40) I mean, I think it helps really seeing the art of the possible, right? And I'm just thinking about it too, especially for these new grads or people coming into the field now. There's so many new tools out there for them to learn and, I guess, help onboard. It's amazing to see.

(Joel Beasley at 00:31:56) Yeah. Well, that's what I think—I think one big Fortune 100 company, they put out some report about where the largest improvement is. And from their data that I read a year ago, it's getting new developers to mid-level developers. You can do that so much faster. There's such a huge benefit of upskilling them and giving them more resources. And then just getting stuff done faster. Like, my buddy Derek—he's my best friend. We've been programming and doing stuff together for almost 20 years. And he called me the other day. He's like, "Hey, I've gotten really into poker." And he's like, "I like to play this version called five-card poker. It's something that we play in real life, but I went to go download an app store version of it, and they don't have that game mode digitally." And he's like, "So I spent like three days, and I built a fully functional application where now I've been playing with my friends, and I've got analytics, and I can track how they play, and we can just—" And he's like, "I just did it in three days."

(Adam Dille at 00:32:58) And he did it all with an LLM?

(Joel Beasley at 00:32:59) Yeah. He just did it all with the LLM. That's crazy talk.

(Zeba Hasan at 00:33:03) Yeah. That is awesome.

(Joel Beasley at 00:33:04) Granted, it's only got 10, 15 friends on there, right? We don't know about scale and stuff yet, but just the fact that you could do that in three days is crazy.

(Adam Dille at 00:33:14) Yeah. On that scale point, I think the thing that I worry about is a less experienced engineer having a fully LLM-driven coding solution. The thing I worry about is not enough experience yet to know what a bad and a good solution looks like. So that example of scale—if I'm a junior engineer, I might think, well, it works, so it's going to keep on working. But if I've got more years of experience, I probably know where to spot the ways that it's going to fall down later on. And so I think that's just something we just have to keep in mind, like, how are we both upskilling an engineer using an LLM, but also still training them on traditional things that would cause something to fail, and they need to be able to spot it on their own.

(Zeba Hasan at 00:34:01) And I think that's where that human-in-the-loop piece comes back in, I think even for training. Yeah. They have all these awesome resources out there to help them upskill, but they still need to be trained, right? They need to understand the business pieces behind it because every company runs things differently. They need to understand those nuances.

(Adam Dille at 00:34:17) Yeah. And back to the topic of risk. That five-card poker game or whatever he's building—pretty low risk if it doesn't scale. But if you're building something that has connection to higher risk, then more experience, please.

(Joel Beasley at 00:34:29) Yeah. You're exactly right too. I mean, you guys live in Fortune 100 world, okay? So you're talking about massive billions of dollars of revenue, mission-critical systems, right? That's a different context and a different strategy of applying this stuff carefully than it is for a Saturday afternoon startup situation.

(Adam Dille at 00:34:51) Yeah. For sure.

(Zeba Hasan at 00:34:52) Yeah.

(Joel Beasley at 00:34:53) Yeah. And one thing that humans are going to be around for a lot, to your point, is the wisdom to know what to implement and how to implement and who to give the keys to, and then the management of people to help the ideas become reality. I mean, you have to work together with other humans, and so that's leadership skills, communication skills, which is something I wanted to talk with both of you about since you're experiencing this every day. What type of leadership lessons have you learned recently, Adam?

(Adam Dille at 00:35:25) One for me is, I think, early in my career, I figured out early on that an engineer needs to be able to communicate to those outside of engineering in terms that they understand. You know, it's a common thing that we teach engineers. Make sure you understand your audience and translate for them. But I think there have been a lot of times where I took that too far and kind of dumbed down things to the point where you lose the excitement of some of the engineering accomplishments you're making. And back to that topic of making sure people understand what GenAI and LLMs do, I might have kind of stripped away all that detail and just left it at a very high-level description, in which case people are left at the point where they're like, "Oh, it's magic." So I'll give you some magic solutions we could build with it. So I've learned to kind of let pieces of the engineering accomplishments and the work we're doing and the details of the solution kind of slip through and educate. So we do, and people really appreciate it. They're like, "We geeked out or we went into the weeds that time, and I loved it." You know? So it's about finding the right time to do that and the right time to speak in terms of the business and just keep it under the covers.

(Joel Beasley at 00:36:44) What are you learning, Zeba?

(Zeba Hasan at 00:36:46) Yeah. I think just to go on with that, I mean, Adam, you said it's not magic. It's a tool, right? So I think a huge piece of lesson is just setting realistic expectations. I've seen a lot of leaders promise their boards that AI is going to transform their business overnight, which is an impossible expectation. There's so much work that goes behind it. So I think a key lesson learned is just understanding and communicating the AI is powerful, but it's not perfect at the end of the day. So focusing on the small use cases, things that are more achievable, and building from there versus promising these big promises of transformation to the business.

(Joel Beasley at 00:37:20) I just want to flip the AI light switch and have money come out. That's how I want it.

(Adam Dille at 00:37:24) Don't we all?

(Joel Beasley at 00:37:27) Awesome. So where can people learn more about Quantum Metric, Adam?

(Adam Dille at 00:37:32) Certainly, our website has a wealth of information, even on Felix, like, the some of the things I talked about with it. And also the other solutions that are the rest of our platform at large provides. If you're into some of the product circles, we show up at ProductCon, and we show up at different industry events. Like, we'll be at Google Next with Zeba here in April and Adobe Summit. So depending on the industry you're in, we were in NRF a couple weeks ago. So we show up there often with the opportunity to take a look at the product, see it in action, see demonstrations, ask us all the questions you want.

(Joel Beasley at 00:38:12) Do you usually go to those yourself?

(Adam Dille at 00:38:14) I do. Yeah. I'll be at Adobe Summit. I'll be at Google Next this year. I've done a few ProductCons in a row. Got to speak at a couple of them, actually.

(Joel Beasley at 00:38:23) Oh, nice. Very cool. Yeah. Zeba, anything to add before we wrap up?

(Zeba Hasan at 00:38:28) I'd say just take an assessment of where you are in your data maturity journey. I think it's important to be honest about that. From a GCP perspective, we have some awesome AI capabilities, and it's really only as good as the data that you feed it. So I'd say look at your data quality, data governance structures, and we have tools like BigQuery to start organizing and analyzing that data. And then you can jump into some of our more AI-advanced capabilities like Gemini or our different LLM models. So take a look at what you have today and start from there.

(Joel Beasley at 00:38:59) Thank you so much for listening. And if you found this episode useful, please share it with a friend or colleague who you think would get value from it. And if you have topics that you'd like to hear discussed on the podcast, either add me on LinkedIn, or send me an email [email protected]. Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.