Why You Can’t Go Off Vibes in Software Development with Stephen Poletto, Field CTO at Span

(Intro Narrator at 00:00:00) Today, we're talking to Stephen Poletto, Field CTO at Span, about the modern landscape of software development and how devs are screwing up by going off of vibes. You're listening to Joel Beasley, Modern CTO.

(Joel Beasley at 00:00:20) Why are we here, Stephen? Are we the caterpillar turning into the digital butterfly? Is that what's going on here?

(Stephen Poletto at 00:00:31) I don't know. I mean, it depends what your p(doom) is, right? You're either an optimist that this stuff is going to alleviate human labor and suffering, and we're just going to live in a techno-optimistic future, or we're all not going to have jobs, and the AI overlords will treat us hopefully like pets and treat us benevolently, but they could also treat us poorly. And who knows which version of that reality we're living, Joel? The universe, the timeline that we've been living has been very strange. It's fun, though. I don't know which path we're on.

(Joel Beasley at 00:01:05) It's fun. It's interesting. It's not boring, is it?

(Stephen Poletto at 00:01:09) No. Definitely not boring.

(Joel Beasley at 00:01:11) And so what are you doing? You're over at Span. What do you do over there?

(Stephen Poletto at 00:01:15) Instrument development processes. Like, what's the workflow of your engineering team? How are they bringing products to life? And we're at the epicenter of this because AI is changing software development practices very quickly, and we're trying to help engineering leaders make sense of it. So I have a front row seat to this whole transformation, which is—yeah, it's not boring. It's very interesting.

(Joel Beasley at 00:01:38) And you're out there and you're giving talks. You have a popular talk called "Don't Go Off Vibes." What's that all about?

(Stephen Poletto at 00:01:45) Yeah. So it's hard to consume anything in the mainstream media about AI and AI's impact on software engineering because you either get the hyper-optimistic, "It's making us 10x more productive, 100x more productive," or you get the doomerism that's like, "It's a toy. It's causing all these quality issues, AI slop." And I realized there was no grounded, neutral voice in the market that was just like, "Let's use data to inform what's actually happening and what's working and what's not working." And so I gave that talk, and I'm hoping to continue to study this space, but just be a reasonable, non-polarized voice. I don't have anything to gain.

(Joel Beasley at 00:02:35) Get out of here, Stephen. No, no, no, no, no, no. That's not the game you're playing.

(Stephen Poletto at 00:02:37) I know.

(Joel Beasley at 00:02:39) It's not going to do anything for the virality of your podcast.

(Stephen Poletto at 00:02:40) I'm sorry. I'm just going to be a neutral, objective observer as much as I can.

(Joel Beasley at 00:02:47) We don't want rational, logical people. No. But it is kind of interesting, though, because I'm incentivized when we go to write that title to do the craziest, hardest title we could possibly think of that's like, would still deliver—like, clickbait title, but it still delivers. And I realized the other day, like, if everyone's incentivized for that, everyone from me to the billion dollar media company, we're all incentivized to be as hype as we could possibly be. What does that do long term to humanity?

(Stephen Poletto at 00:03:18) Yeah. The erosion of—it's good.

(Joel Beasley at 00:03:20) I don't think it's good.

(Stephen Poletto at 00:03:21) I mean, you're talking to someone like—I pay for my journalism. I get a physical copy of The Economist once a week, and I try to read the news. I think they do a pretty good job of outlining the facts of what's happening in the world because of that polluted social media environment and just how difficult it is to find rational voices in the world. So I agree. I agree. I don't know that it's good, but it's what the algorithms reward, and we're all having our attention monopolized by these systems.

(Joel Beasley at 00:03:56) You know, I found—I did an interview with the guy that did a lot of projects with Google X, you know, like the moonshot type projects and all of that. And if you ever go back through and read the history of each one of those projects, it is so cool because it takes these very edgy pieces of technology, and then it boils it down. It says, "We tried it, and here's the results." And it was cost effective or it wasn't, and it's going to move forward or it's not. And when I was reading that, preparing for that interview, I thought, "Wow. It's been so long since I've had just, like, unbiased, brilliant—this is just what's happening—content versus trying to get me to think a certain way." Felt good.

(Stephen Poletto at 00:04:36) Yep. Well, I'm trying to be that on the subject of AI transformation and using data and not going off hype and not letting our emotions and polarized headlines drive important decisions about how you staff engineering projects and lead the team through this transformation, but instead instrument, use hard data, take best practices from organizations who are doing it well, and just try to make it more grounded.

(Joel Beasley at 00:05:04) Who is doing it well?

(Stephen Poletto at 00:05:08) Ramp is doing it well. They're a very innovative engineering org, and they've talked publicly about the agents that they're building and some of the innovative technical work that they're doing. So, yeah, Ramp's doing it well for sure.

(Joel Beasley at 00:05:25) Ramp's doing it well. You know, is this good timing for this conversation too? Because, you know, you mentioned one side is 10x to 100x productivity improvement. The other side is doomerism. I always like looking in the market, like, what's happening in the market. I was getting early reports that companies were laying—this is like a year ago, two years ago—that they were laying off engineering people prematurely. But then you had the guy from Block, I think, Dorsey, right?

(Stephen Poletto at 00:05:54) Mm-hmm.

(Joel Beasley at 00:05:54) He laid off half of his company.

(Stephen Poletto at 00:05:56) Correct.

(Joel Beasley at 00:05:58) Yeah. And he seems—if you've heard him in interviews, he seems pretty level-headed.

(Stephen Poletto at 00:06:04) Mm-hmm.

(Joel Beasley at 00:06:04) So for him to do that was actually a surprise to me. He must really have strong feelings about it.

(Stephen Poletto at 00:06:12) Yeah. I mean, there's a lot of speculation about that Block layoff and how much it's a reaction to overhiring that happened during the growth era and a correction for that overhiring and overstaffing. But the messaging that he provided publicly and the public markets responded well to was all about AI efficiency and how they feel like they can do a similar or greater workload as an organization with fewer people. So it is on the extreme end of what we've seen. We've seen smaller layoffs, 20% layoffs from other organizations claiming the benefits of AI productivity. But this was a really extreme one, and it's kind of reminiscent of, you know, what Elon did at Twitter when he bought Twitter. And that kind of set off a cascade of subsequent layoffs and market changes and people following the path that he laid out. And I could totally see business leaders looking at what Jack has done and saying, "Hey, could we do the same thing in our organization? Like, how do we derive this level of productivity gain out of these tools and get the same benefit to our gross margins or whatever?" The thing that's really interesting about Block is, like, their business is doing well. You know, it's not like they're stressed and doing this layoff in response to business distress. They're doing it in this more offensive maneuver, which greatly polarizing and, you know, there's a lot of discussion and speculation out there about it. But, yeah, huge news that came last week or two weeks ago.

(Joel Beasley at 00:07:47) I can see how it could actually be costing them money by keeping them, and I don't mean just, like, the direct salary. You and I have been building software long enough to know that there's different types of people that you need involved in the process at different points. And so I've never worked at a massive organization. You have, but I have not. I've always built an idea prototype, turned it into a small team, grew it to about 30 people, and sold it off. That's my trajectory. So I've never been in a giant Fortune company. But what I could see happening when these teams get really big and what I saw happening early days with my team is, like, there's one type of person who I can give an outcome to. I'd be like, "I need you to go achieve this outcome." And then depending on the complexity of that outcome that spins up several different teams underneath them and all. But it's almost like it's compressing. It's like, now I—rather than—I can get more done if I have fewer people in this loop of this outcome we're trying to achieve, and then there needs to be fewer people between them because there's more efficiency in actually getting this hard work done. I could see how it could be a benefit to be smaller and leaner to do something more complicated.

(Stephen Poletto at 00:08:59) Yeah. I mean, there was an interview with Brett Taylor earlier this week that, you know, he was describing the atomic unit of productivity as a process. Because almost any business process or if you think about product development and software development and you think about the stages involved in bringing a product to life, it's like you do market research, you do customer discovery with the insights and the pain hypotheses that you're developing based on those conversations. You turn that into problem statements and problem identification. And then with those problems identified, you go ideate solutions and you test those solutions maybe in early design stage or prototype stage with the market. And then as you validate the solution, you turn that into a technical specification, you turn that into engineering implementation, and then you deploy rollout and scale. That's a process by which software comes to life, and what AI has the potential to do is compress the cycle time. Right? It has the potential to say this process used to have this many handoffs, and it used to take this many hours or this many days or weeks or months because it has all of these handoffs involved, and there's this coordination cost associated with passing context from one stage to the next as humans have to get up to speed and understand what happened in the stage prior. And AI has the potential to compress that process. Right? And I think that's a good way to view the world because, like, yes, large teams—the challenge of large teams is coordination overhead, getting everybody to row in the same direction, getting everybody to have shared context on the goals. And, like, the potential of AI is that you're able to achieve and optimize those processes with fewer people. Now I think there's kind of two schools of thought at the moment. One being what we saw Jack do with Block, which is, "Well, hey. We can achieve the same amount with way fewer people. Let's improve our efficiency, our gross margin. Let's just, like, optimize and reduce coordination cost and slim the team." And then there's other people who are thinking, "Well, hey. If we can do more and we can compress these processes, what more can we do? We've always had a backlog that we've never been able to service because we've always been resource constrained. We've always had these ideas for things that we've wanted to do that we haven't been able to staff. How do we basically promote top line revenue and do more and pursue more of those things that previously might not have cleared the hurdle threshold?" Right? Maybe before it was too expensive to build something relative to the benefit to the business, so you'd say, "Oh, we're just going to discard that or it's going to sit in the backlog. It's not important enough to do." And now if the cost is reducing, maybe the ROI equation changes, and it's actually better to go do that thing and create more net new value. So I think you're kind of seeing multiple schools of thought emerge. And, you know, what is the future shape of teams? Do engineers feel more empowered to make product decisions? Do product managers feel empowered to write code? The two pizza team that we have basically treated as the industry standard for the past twenty years, is that the new standard when, you know, people can do more and minimize these handoffs and maybe team sizes can compress? So there's a lot of speculation right now. I have not yet seen kind of, like, a golden path emerge where it's like, "Oh, this is the clear answer. People have experimented with multiple different staffing models. Here's the pros and cons, and here's the winner." It's very much like experimentation mode right now, and people are—you see people taking different approaches in the market.

(Joel Beasley at 00:12:30) There will likely be different approaches by industry and company maturity too, right?

(Stephen Poletto at 00:12:35) Totally. I mean, it's not too dissimilar from, like, the COVID disruption. Right? You saw people embrace remote. You saw people say, "No. We're in office three days a week, whatever." There's this huge debate about the right way to do it, and people chose different strategies for their business. And some of them worked, some of them had new trade-offs, and, like, I think the same thing is going to happen here. People are going to reimagine how should a software development team be staffed in the era of AI. They're going to try different things. And, you know, that's the beauty of free market capitalism is, like, we'll see a competition of ideas emerge, and we'll see which ones work, and we'll see which ones don't.

(Joel Beasley at 00:13:11) I think that's a mistake a lot of people make where they'll just look externally. They'll listen to some great interviews on, like, the Modern CTO podcast or something like that. And then they'll say, "Hey, I want—they do that. You know, Stephen does that or so and so does that. That's what we're going to do." I'd say, you know, I tend to be very introspective. I always look like, what are the needs of this team and this organization at its current point? Because so many companies are so different, and you can get one piece of advice, but it's from somebody who's 10 levels ahead of you, and it means something entirely different to you now than it does to them where they're at. And so you really have to focus on, like, what's my team need right now? What's the best move for my company? And then I just look outside to see if there's anybody that's in my situation that's also experiencing this that has come to the same con—I'm trying to validate my conclusion. Like, I came to this conclusion for my team. Is there validation here or am I way off? And usually through the act of trying to validate my decision that I'm making internally, I then learn some stuff, and it shades the decision a little bit. That's how I do it.

(Stephen Poletto at 00:14:15) That's why—as every business has different pressures on it, different regulatory hurdles, different product complexities, you know, different markets. And so as a byproduct, you need different skills to navigate those complexities. Some businesses, you know, if you're building a greenfield pure software play product, you might be able to execute with engineering doing a lot of the quote, unquote product work. But if you're in, like, a highly regulated environment with a lot of compliance and things like that, like, you know, maybe it's better to have a product manager who is an expert in navigating those things or what have you. So, yeah, I agree entirely.

(Joel Beasley at 00:14:59) Yeah. Like, when you're talking about the two paths, right—like, Jack at Block, he took one path. You know, I would say from my experience from the types of businesses that I've run, which are more aligned with Jack's, I would tend to reduce down and become profitable and then intentionally start new experiments and then boot up teams. But you might be in a situation where you're a healthcare company building physical devices, and there's very few people that even know how to do this type of very specific work. And at that point, you might want a strategy where you say, "Hey—"

(Joel Beasley at 00:15:37) We're gonna reallocate this chunk of people to these new areas, and we're gonna consider this an investment. Like, over here, we're gonna optimize our core product and get really lean over here, but we're gonna take these people. Instead of letting them go, we're just gonna reallocate them for an investment of an area that we know is gonna be important because that talent is so hard to acquire and bring up to speed on this. So there is—totally, you could have different strategies.

(Stephen Poletto at 00:16:02) Totally. Yeah.

(Joel Beasley at 00:16:04) But there's not just one answer for everyone, Stephen. That would be so easy, wouldn't it?

(Stephen Poletto at 00:16:09) It would be. I mean, I think what we're seeing with AI transformation in general is there is a set of emergent practices that are unproven, but people are talking openly and publishing their learnings, and it's kind of a marketplace of ideas right now. You know? And you see this even with things like harness engineering or what I call constraint programming. You know?

(Stephen Poletto at 00:16:36) Last year, it was vibe coding. It was like, I'm just prompting the generation of these big swaths of code and features and things like that. And now people are reckoning with the strain that that has introduced in their organization, whether it's like, we're producing too much code to code review and keep up. Like, human code review can't pace well enough to hold pace with the volume of code generation that we're doing, or it's actual quality issues in production. Like, earlier this week, we saw reports that AWS had an engineering all hands where they were talking about incidents in production that were due to AI-generated code and quality issues escaping their quality guardrails and making their way into prod, including like a six-hour outage.

(Stephen Poletto at 00:17:22) And so people are dealing with the hangover of just, like, kind of vibes-based programming and realizing, oh, we need to put constraints in place. We need to invest in the environment that these agents are operating in to make sure that they can't—you know, we don't fuck ourselves and run into these issues. And but that's an emerging practice. Like, what is harness engineering? What is constraint-based programming? It's like that's basically a new field that has developed in the last sixty days. You know?

(Joel Beasley at 00:17:51) You're gonna have to explain harness engineering to me because I haven't heard this before.

(Stephen Poletto at 00:17:56) Well, yeah. So harness engineering is the idea that you are delegating workloads to agents. So you are describing product specifications that you'd like Claude or code to go build, or you're providing a bug report as an input into an agent and having it go try to fix it on its own. And without the right oversight mechanisms to constrain the solution space, who knows what the agent might do? They might go look at your code base, see that there's some pattern of solving the issue somewhere else in the code base and replicate it. But that might be a tech debt pattern that you actually don't want to be replicated. So instead, you want to define constraints in the environment that make it so that when the agent tries stuff, they're more likely to succeed in accordance with your standards. Right? And so this can range from—I mean, agents.markdown files were kind of the first attempt at this, where it was like, well, let's just tell the agent these are our architectural standards. These are our design principles. Like, please code in this way. But, unfortunately, I don't know, you've probably used LLMs a bunch. They don't always follow your instructions. Right? Like, they're kind of like children or junior engineers or something, where they sometimes follow your instructions and sometimes don't.

(Joel Beasley at 00:19:15) They do what they wanna do.

(Stephen Poletto at 00:19:16) Yeah. They do what they wanna do. Right?

(Joel Beasley at 00:19:18) They're people too, Stephen. Don't be racist.

(Stephen Poletto at 00:19:22) I mean, I do think a reasonable mental model of these agents is, like, they're junior engineers. Right? So you need to provide oversight and constraints so that they're going to be successful. And so, you know, if it started with kind of agents.markdown files, it's progressed into, you know, how do you do custom linters with remediation messages? How do you do pre-commit hooks? How do you have browser automation tests per session so that the agent can validate the correctness of what it's building more real time? So before the agent ever opens a pull request, you have a priori constrained the solution space such that it can't pass until it does certain things, until it satisfies certain criteria. Right? And so I think there's a spectrum of constraints ranging from things that you express in English language and kinda hope for, to things that you make impossible by design. Like, you don't give the agent access to raw SQL. It can only go through an interface layer where you control, you know, the way that it can mutate the database, for example. And so all of these kind of practices of how do you create an environment of constraints, and then under those constraints, the agents can thrive—that whole practice is kind of being referred to as harness engineering.

(Joel Beasley at 00:20:38) Interesting. Have you come across promise theory yet? It was new for me.

(Stephen Poletto at 00:20:45) No. No. No. I've heard that one. Yeah.

(Joel Beasley at 00:20:46) Yeah. I would butcher it. I had the scientist PhD people on that came up with it and then a company that had used the theory to implement it in swarms of agents. But the TLDR is they've experimented with different ways to get the agents to do what you want them to do without derailing. And the way that they did it was fascinating because they tried a bunch of ways that didn't work, like command and control and then like all of these different things, like demanding it do certain things, but then they found a way to get it to do what they wanted it to do. And we did a whole episode on it.

(Stephen Poletto at 00:21:27) Very cool. I'll have to go listen to that episode. I don't remember what I—because there's emerging practices. Right? It's like spec-driven development, agents.markdown, and how to get the most reward for that, this, you know, constraint-based programming or harness engineering. I'm seeing new frameworks like BMAD, which is like how you apply AI to every stage of the product development process. So you can actually have agents go do stakeholder interviews and user interviews to help frame and shape the product requirements that you're defining. So there's—yeah. It's a marketplace of all these different techniques and approaches that people are trying, and it's a fun time to—you know, very intellectually stimulating. So, yeah, I haven't heard of this promise theory, but I'm gonna go check that out.

(Joel Beasley at 00:22:16) I think the endpoint is fairly obvious. I think how we get there is gonna be exciting, and we're not gonna be entirely certain until it actually happens. But I do think the endpoint is, basically, we just have these interfaces where we tell it what we want, and it gives us what we want. I wanted to do this, and then it just comes back perfectly and does it right the first time. I think that's where we're headed.

(Stephen Poletto at 00:22:38) Yeah.

(Joel Beasley at 00:22:39) I don't know if it's gonna be five years, ten years, a hundred years.

(Stephen Poletto at 00:22:41) Yeah. I think—I mean, I think we will get closer and closer to that. Something that I believe is that taste and judgment has not yet been expressed by these systems. It's entirely possible that it will be, and then we really are just expressing high-level intent and offloading everything. And that's a very different world than the world we live in today. But at least today, I feel like there's still a level of judgment and taste about, like, which problems are worth solving and what does a good solution look like and how do you shape that into a cohesive product experience for your customer? The CEO of Linear was tweeting about that earlier this week. He was saying, you know, the cost of shipping features is now so low that they're shipping too many features, and the features aren't cohesive, and the product experience is degrading. And it's like there's this taste and judgment layer that sits on top of everything that curates into something good. And I don't know. It's entirely possible that we'll get to the point that agents can do that on their own. But I think for the time being—

(Joel Beasley at 00:23:53) You think we will? 100%. Do you? Because it already happens in life. So imagine you're a king of a very wealthy nation. You just express high-level intent to one of your head officers, or maybe like, I want this to occur. Now how many thousands of people are part of that pyramid to make the outcome occur? Who knows? But—

(Stephen Poletto at 00:24:13) If you don't like the outcome, you fire the person. Right? Or—

(Joel Beasley at 00:24:16) Right. Depending on the era, we had them. Or—

(Stephen Poletto at 00:24:17) Or what have you.

(Joel Beasley at 00:24:18) Or they come back with an outcome, and then you provide more input to get a better outcome. And so that already happens. The thing is, historically, information itself and that ability to direct large amounts of intelligence, whether it's human or silicon-based, was for the most wealthy people. Right? Even way back in history, just access to books itself or access to experts. We can turn on podcasts now, and we couldn't do that a hundred years ago. You'd have to know these people and bring them in and talk with them. And so what we're doing is we're giving that to the average person, and I think it's a brilliant thing. I'm not like 1,000% optimistic about every little thing that'll happen from it. There's a ton—it's like a knife. It can kill you. It can feed you. Like, it can do a lot of different things.

(Stephen Poletto at 00:25:11) Yeah. But in that world order, what do you think we spend our time doing as humans?

(Joel Beasley at 00:25:18) Good question. So humans—I like to think about money a lot because money is a way that humans exchange value with each other. Like, we don't give dollar bills to platypuses or elephants or something like that. They don't have any use for our money. It's just paper to them. But like money is how we exchange value with each other, and how what we value over time changes. And so I'm certain that whatever we value today, we probably won't value a ton of it identically in a hundred years, but there will still be the transfer of value among humans. So that's what I—when I think about the economy, I don't think it's gonna go to zero. I don't think it's crashing to zero and like there will be no economy and no way to trade value. I think there's going to be incredible amounts of peaks and valleys within different industries at different points in time that we, as a civilization, are going to have to try to figure out.

(Stephen Poletto at 00:26:14) Mm-hmm.

(Joel Beasley at 00:26:15) Like, it's gonna be hard. Like, imagine, Stephen, you wake up tomorrow and there's this new tool that comes out that completely automates everything that you and your specific niche job does. And then now you're gonna have to find a new value thing in the marketplace to do.

(Stephen Poletto at 00:26:32) Yeah. And if you think about, like, the animal kingdom, right, and the, you know, superiority of Homo sapiens, the reason that humans have become the top of the pyramid is due to our intellect. Right? The ability for us to coordinate, communicate well, and our intellect to orchestrate, you know, behaviors at scale. Like, that is what, you know, made us kind of masters of planet Earth. You know, go read Sapiens. It's, like, great book. Like, I love that book. I think the challenging thing is if we are developing an intelligence that can outcompete us, there becomes a very existential question about, like, the value of human. Like, what is the unique value that humans have that is not just intelligence? Because we've built these systems that can take our niche jobs in the intellectual world. And that's a pretty thorny existential question.

(Joel Beasley at 00:27:26) Yeah. It'll be interesting. But the humans are asking it, and then we gotta be concerned when the robots start. You know, my concern levels vary. I—for some reason, I have, maybe it's just because I am a human, but I have more of a concern when the Optimus bots start rolling in the streets than I do right now with software and stuff like that. Because as the AIs take more physical form, I think our risk factor goes up.

(Stephen Poletto at 00:27:54) Yeah. There's still a lot of physical world stuff that we can do, the machines can't do, even if they can take vast swaths of intellectual labor. There's still, like, a lot of human services that are in the physical world. Totally.

(Joel Beasley at 00:28:05) Yeah. Like, right now, if we needed to blow up all of our data centers, we could find another form of currency. It would be rough, but we could figure it out and we would still survive, and humans would be fine. We would figure it out. That being said, when we get to a point where there's two Optimuses to every one human roaming the planet—right? At that point, you can't just nuke the data centers. You have this decentralized autonomous robot thing that can kill you, that outnumbers you. That's the point to me that's scary.

(Stephen Poletto at 00:28:40) Yeah. Yeah.

(Joel Beasley at 00:28:41) Because even if there's a human bad actor controlling them—it doesn't matter if it's the—everyone's so fantastical about, like, what they always fantasize about, this idea that like the robots go—I'm not really concerned about the robots hallucinating and going crazy. I'm more concerned about the person who's evil, who's directing the swarm of robots, going crazy.

(Stephen Poletto at 00:29:01) Sure. Sure. Totally. Joel, do you normally get into these, uh, existential futuristic topics with your other CTOs, or is this—

(Joel Beasley at 00:29:09) We do.

(Stephen Poletto at 00:29:10) Okay. I love it. I love it.

(Joel Beasley at 00:29:11) It depends on the day. It depends on if you're willing.

(Stephen Poletto at 00:29:15) Like, as of late, this is what's in the ether of the world. You know? Everyone's grappling with these questions because we're seeing the real impact and the real potential. Like, I think what happened last year was everyone thought these things were toys, at least in the software development world. It was like, oh, I tried it. It kinda works. It's a toy. And then some models came out in the fall. We had, like, Opus 4.5. I think a lot of people had actual time where they weren't in the grinder and in the execution of their business over the holidays and were playing with these things. And, like, entering 2025, it feels like a new world order. It feels like everyone is like, wow. How do we become an AI-native software development loop? Like, what does that mean? How do we change roles and responsibilities? How do our processes evolve? What does it mean to leverage these tools well? And all of those questions just feel different than they did, like, middle of last year.

(Joel Beasley at 00:30:14) You're exactly right. And I think you nailed it too. When those breaks happen, that's when I get to stop running the business and play more. You know? That's when I pick up these new things. Oh, I've got, you know, three days because of the way the holidays run. I've got three days. I don't have anything I have to do. My team's kinda checked out. Like, maybe I'll experiment with this. And over two years, that happens here and there a couple times, and everyone who's currently in executive leadership or even, you know, just any position in the company—if in the past two years, you've been able to see the progress of the models in a way that—like, everybody has a different industry. So we're always watching our industries and, like, watching them progress. It's very slow. You know, if you're in happiness, it's very slow. It takes a long time to watch it. Over ten years is like—you've been in the game to watch it over ten years.

(Joel Beasley at 00:31:05) You know, but we saw a massive shift happen, and it gets significantly better and better in, like, a two-year time span from the first time GPT kind of came out to, like, right now. It's been, like, three years, I think, two or three years. We've seen it go from hallucinating and off the rails and stuff in the early versions to now being connected to search engines, to now having real-time search engine access. Like, we've been watching these things progress and get so much better, and we're all thinking the same thing.

(Joel Beasley at 00:31:35) We're like, oh, wow. This is not going to slow down. These trains don't slow down. I think they're going to keep going.

(Steven Paleto at 00:31:44) Yeah. And I mean, I think what we're seeing is a year ago, two years ago, they felt like autocomplete. They felt like a search engine at your fingertips. You've got Stack Overflow in your IDE. You know, like, that was kind of the mental model. And now it's like, well, customer reports a bug. We've got a GenAI subsystem that checks out the bug, reads our agents.md, you know, goes and drafts a fix. If it passes all of the tests and our linters and gets through the harness, the environment that we've constructed for it, it just opens a pull request.

(Steven Paleto at 00:32:25) And now there was no human reviewing that bug. It was just like a customer wrote a bug report. Here's a PR to fix it. Like, that wasn't happening two years ago. That's a totally different world.

(Joel Beasley at 00:32:39) And it's fairly reliable. In my experiences with really small things that are very specific that can be validated—like, if I had a queue that came in and I was like, yep, that's valid, and then it proposed to me the situation as a pull request, and I could look at it and say, like, all right, cool, and I just let it go into production—I think where everything gets swampy is when you do, like, larger features or things that impact the system. But on pull requests for bugs and things like that, I think it's pretty sharp.

(Steven Paleto at 00:33:09) Yeah.

(Joel Beasley at 00:33:10) Wait. What does Span do, and how can people buy it? Let's make sure we do work here too. We're having fun.

(Steven Paleto at 00:33:17) Yeah. No, do some work. Span integrates to your development tool chain. So we connect to, like, your source control, your ticket system, your GenAI tools that your team is using. So we can understand code workflow, and then we can understand AI tool engagement and the patterns around using those AI tools. And with all of that context, we're constructing a work graph of, like, what major projects, themes, and initiatives is your engineering team working on, and how are they flowing through the development process. So we can help identify bottlenecks. We can help identify investment envelopes, like roughly how much time allocation is flowing into different projects.

(Steven Paleto at 00:34:03) And so it's this software engineering intelligence layer that helps leaders understand where is human time and token spend going, and where is it encountering friction and bottlenecks. And so this has been increasingly valuable as people are embracing AI and wanting to understand, like, how is AI reshaping the development loop? Where is it creating new challenges? Where do I need to be focusing my attention? So that's what we do. That's what Span's all about.

(Joel Beasley at 00:34:32) So what is the problem people are facing when they find you? Is it out-of-control token spend? Is it not—

(Steven Paleto at 00:34:39) It's a mix. Yeah. It's a mix. You know, a big one is, like, we're spending all this money on AI tools. We're having a hard time quantifying the total impact that it's having. Can you help us understand, you know, the ROI? Can you help us understand how much more productive we are? And are we actually more productive, or do we just feel more productive? Where are there new challenges emerging, and how do we tackle those challenges? Who is proficiently using these AI tools, and who is kind of shallowly using these AI tools? How do we get everybody in the organization to become proficient?

(Steven Paleto at 00:35:18) So it's those kinds of questions. It's basically organizations who are like, we've seen the light. We think that the companies that are introspective and have an attitude of self-improvement about navigating this AI transformation are going to outperform the ones who just kind of go with the flow. And so Span can be this lens of data and consulting and services that help us figure out how to optimize our AI development.

(Joel Beasley at 00:35:46) That's awesome. So you have tooling, but you also have, like, a consultative side of things too.

(Steven Paleto at 00:35:51) We do. Yeah. We're spending a lot of time talking to customers, understanding practices that are working, doing, like, benchmark reports so we can aggregate and anonymize data across our customer base and use those to understand macro trends so that you can see, like, how you're doing relative to the industry at large. And we use all of that data and anecdata to help companies move through this transformation process.

(Joel Beasley at 00:36:17) Interesting. Now there's this study from METR. Is that how you say it? M-E-T-R? Tell me about that study.

(Steven Paleto at 00:36:24) Yeah. It was published last year, and so, obviously, things are changing quickly. But they ran an experiment on an open-source project where they gave access to AI tools to a subset of contributors. Kind of randomly—you know, basically, you would be assigned a task, and then you would be told, like, you can use AI or you can't use AI for that task. And then they tracked productivity and self-reported productivity based on that assignment, like, basically trying to do a randomly controlled AB test. And what they found was, like, engineers self-reported significant productivity gains when they were given access to AI tools.

(Steven Paleto at 00:37:11) But when they actually looked at, like, cycle times associated with the completion of the projects and other productivity stats, they saw a degradation in productivity. So there was this paradox of, like, people feel more productive, but they're not actually more productive. So, yeah, pretty interesting study. There are some more recent studies that they're working on too.

(Joel Beasley at 00:37:37) I would like to see—I like to poke at all the studies. I don't care what the information is. I always like to rip it apart because I'm a nerd. And I would be curious to figure out how they controlled for experience level. Like, because I can go in there inexperienced, do it, learn something, and then feel like, oh, wow. The next time—like, I could feel more productive and I could actually be more productive in a future task. Because I learned—like, did some of these people, were they learning how to do it for the first time with AI, or were they already, like, integrating it into their—that's the control I would—

(Steven Paleto at 00:38:13) Totally.

(Joel Beasley at 00:38:14) That's the metric I'd want to see is how many hours a week are all these participants already using AI versus this is their first interaction with it? Because that was horrible. I was so dismissive of it at first after being a software engineer for almost twenty years. And then my one of my best friends, he sent me a YouTube video of some dude, like, really doing it, like, showing another senior engineer how to use it.

(Joel Beasley at 00:38:41) And so I watched that, and I was like, oh, I was using this tool wrong. I didn't understand how to use this tool. And then it just made me crazy productive.

(Steven Paleto at 00:38:52) Yeah. That's the challenge with a lot of these studies is, like, there's multiple conflating factors, so it's hard to control for everything. You know, even—

(Joel Beasley at 00:39:00) Great for sales.

(Steven Paleto at 00:39:02) Yeah. Yeah. Well—

(Joel Beasley at 00:39:03) I mean, I do agree with the sentiment of it, though. There's definitely people out there who feel like they're faster and they're slower. I bet that is true. A hundred percent.

(Steven Paleto at 00:39:11) I mean, what we're trying to do at Span—because there's a lot of—you could do, like, a longitudinal, like, a time-based measure of productivity where you could say, look, we've rolled out a tool, or we gave, like, this cohort of engineers a tool, but not this other cohort, and then we could track the impact over time. But, you know, that also introduces a ton of conflating variables because, like, the environment, the work that you're doing three months down the road relative to that point when you started the experiment is very different. What we have been trying to do at Span is say, like, estimate the dosage of AI in any given pull request. So, like, how heavily AI-influenced was a given code change?

(Steven Paleto at 00:39:52) And then track all of the workflow signals around pull request lifecycle, number of code review comments, subsequent rework that happens after that pull request is merged. Is there, like, defect fixing and bug fixing that happens in the ninety-day window after the original merge? And so now you have something that is more controlled. It's like how much AI was actually in that given pull request, and you can cohort on a pull request basis. So that's been the foundation for a lot of our benchmarking and productivity insights because we think it does a pretty good job of controlling for, like, all these different variables.

(Steven Paleto at 00:40:31) And that's where, you know, we've been finding some really good insights around, like, actually, code review burden is way higher, and there are more rounds of feedback happening in code review as a byproduct of AI adoption. And I think a lot of attention is now being given to code review processes and workflows and how to shift left, you know, quality guardrails so that you're not catching them at the code review stage, but you're defining them in the environment earlier with linters and things like that. But, yeah, that's kind of been our methodology. It's fun to see the different study designs, and then it's fun to try to figure out, like, the best methodology that we feel can control for the most number of variables. And this is what we've landed on.

(Joel Beasley at 00:41:11) And I like—one of the things that stood out to me about your product as you're talking is the consultation-type deal because there are many SaaS-type tools that are like, all right, we'll digest your repos. We'll do this, and then we'll give you a dashboard. And it's like, well, that's great. But I think a lot of the people listening to this show or at least a lot of the high-level technology leaders, they've got a lot of other stuff going on, but they might want to figure out this token spend and human time and, like, what's going on? Is it—and so they would just hire you guys, and then you guys would tell them.

(Joel Beasley at 00:41:44) You guys would look at the data, help interpret it. Am I understanding that correctly?

(Steven Paleto at 00:41:48) Yeah. I mean, you're right. Engineering execs are busy. I mean, you've—I've been an engineering exec. You've led engineering teams in the past. You know how fragmented your attention gets. You're, like, managing people. You're doing project execution forums. You're doing exec reporting up to the CEO and the board maybe. You know, cross-functional syncs with sales and product—like, you're busy. And so, yeah, the idea is that we get a set of out-of-the-box metrics by using our software, but then the signals ultimately inform recommendations that we can make where we're like, hey. Here's how you're doing relative to the industry.

(Steven Paleto at 00:42:24) You know, it looks like you're struggling with X, Y, Z. You know, for instance, one of the things that we are seeing across a number of our customers right now is, like, elevated code review burden. So we're like, hey. You know, here's some themes of, like, repeated code review feedback that's happening in your AI pull requests. Here's some ideas for, like, things that you could do to streamline that. So just trying to give targeted recommendations because, yeah, our customers are busy, and they just want to know, like, what should I do?

(Joel Beasley at 00:42:52) Yeah. And as harness engineering kind of matures, you can watch it evolve and share knowledge across your customer base.

(Steven Paleto at 00:42:59) Totally.

(Joel Beasley at 00:43:00) Yeah. Yeah.

(Steven Paleto at 00:43:01) It's a nascent practice. I mean, it's a nascent practice in some ways, and it isn't in others. If you think about, like, hypergrowth organizations that have needed to scale up and hire lots of humans, you also need to do this practice of environmental constraints so that the junior engineer you just hired doesn't take the site down. Right? Like, people are going to make mistakes. You assume people are going to make mistakes. There is knowledge trapped in the heads of your, like, senior tenured engineers. So what do you do? You write tests.

(Steven Paleto at 00:43:30) You create, um, ways to roll back safely when failures occur. You contain the blast radius. You invest in linters. You invest in code standards. You do all of these things. It's just now every organization just hired a hundred junior engineers, and now this stuff is even more important. And so we give a fancy label to it called harness engineering, which is really just guardrail design and safe engineering environments.

(Joel Beasley at 00:43:55) Do you have kids?

(Steven Paleto at 00:43:57) I don't have kids.

(Joel Beasley at 00:43:58) Oh, man. There's so many parallels between that and having children because, like, how many mistakes you let them make and all of that. I would promise you having kids will make you a better technology leader. I've been a fan of that for a while. For about eight years.

(Steven Paleto at 00:44:14) I believe that. Yeah. Yeah. I believe that.

(Joel Beasley at 00:44:17) Okay. So you were at Lattice. They grew from 15 million to a 125 million ARR. You're director of engineering at Dropbox for eight years. You've done it all. You're Apple, Xcode stuff. You are, like, a nerd. How did you run into Span? Why did you decide to join them?

(Steven Paleto at 00:44:35) Yeah. Well, so Jazak is the CEO and cofounder, and I worked with him at Lattice. He was leading product when I was leading engineering, and I really liked working with him. Like, he's an incredible human. He's really talented, really smart. I learned and grew as a human working with him. And when he started Span, I was fortunate to be one of the early design partners. Like, I used the software when I was still leading the engineering team at Lattice and provided a lot of product input and feedback and kind of helped shape some of the early product into what it is today because I was an actual practitioner, like, using it. So, yeah, just like good old-fashioned human relationships and, you know, working with good people. That's how I ended up there.

(Joel Beasley at 00:45:22) Lead with that on sales calls? Because you should.

(Steven Paleto at 00:45:25) I do. Yeah. I mean, I tell people—yeah. They're like, I'm a former CTO. I'm a former engineering leader. Now I'm doing a customer-facing role, which is very different. How did I end up here? Like, this is a weird career path for someone to take. Right? And it's like, because I worked with the early team, and I found the product space compelling. I found the challenges intellectually stimulating. And, yeah, I just wanted to be part of it. You know? So, yeah, my role now is very different. Like, we have a cofounder and CTO, Henry, who leads the product development efforts.

(Steven Paleto at 00:46:04) Like, he oversees the engineering team. I provide ample input, and I'm, you know, an important stakeholder to that process. But, like, I'm not leading the engineering team at Span. So I'm, like, leading go-to-market and doing sales and customer support and the consulting stuff that we were talking about. And so it's a totally different role. And it's been fun. There's been a lot of learning. There's been a lot of, like, things I don't like about the role, but it's been fun.

(Joel Beasley at 00:46:29) Nice. Yeah. It keeps it interesting. Right?

(Steven Paleto at 00:46:33) Careers are meant to be interesting, I think. Like, you want intellectual stimulation. You want growth. You know?

(Steven Paleto at 00:46:40) There is a financial component to careers and needing to support and provide for yourself and your family. But beyond that, it's also just like, I want to grow as a human. You know? So like, how do I do that? How do I satisfy both of those criteria?

(Joel Beasley at 00:46:56) Yeah. How do you wake up and enjoy life? You know? Like, how do you, where do you get your variety from? And at some point, you have to have, at least for me historically, I just, I'll do something and I can stick with it for several years, but eventually I'm like, okay.

(Joel Beasley at 00:47:10) I have approached this from 99% of every possible angle. I need to solve a new problem because I found that I'm most energized when I'm doing something difficult.

(Steven Paleto at 00:47:22) Yeah. Yeah. And I will say, like, you know, careers are long and who knows what I'll do next after Span. But like, if I decide to go back into product development and leading an engineering team again, I will be a significantly better engineering leader as a byproduct of understanding how sales and marketing and customer success and customer support works way more deeply. Because like, the empathy level has just gone way, way up for the challenges on the business side of the house.

(Steven Paleto at 00:47:51) So like, to me, it's just a net good of, like, yeah, I think this will eventually make me a significantly better engineering leader if I decide to go back to that in the future.

(Joel Beasley at 00:48:01) You'll probably go COO, CEO, that's your career.

(Steven Paleto at 00:48:05) Who knows? Who knows? We'll see. We'll see. I don't have, I've never had a long-term plan with my career.

(Steven Paleto at 00:48:11) You're right that I have had a good career trajectory to date. I'm very proud of the things that I've done, but none of that was planned. Right? That just happened as a function of me chasing impact and intellectual stimulation.

(Joel Beasley at 00:48:25) And you just said the magic words. I've done almost a thousand of these interviews, and like, the top 1% of the top 1% is they, that's what they say. That's what they're doing, and they're trying different parts of the business. They're always doing interesting things. They're trying to stay curious.

(Joel Beasley at 00:48:45) They are just trying to wrap their mind around it. And a lot of them also say they didn't have like, some very specific, like, I'm gonna be this role by this time and this role. They're just like, where can I be helpful today? Totally. Yeah.

(Joel Beasley at 00:48:57) I'm curious. What's the problem happening? How can I add value right now? Like, what can I do? Like, what's going on?

(Joel Beasley at 00:49:03) You know? And then just doing it.

(Steven Paleto at 00:49:05) Yep. Yeah. That was my path. Totally. I did not have a plan. Yeah. Yeah.

(Joel Beasley at 00:49:11) I like that your LinkedIn title says CTO-ish. That's my favorite thing about you, by the way. I mean, I saw that.

(Steven Paleto at 00:49:18) How do you describe someone who was a CTO and is now doing sales and customer-facing stuff? And how do you, I don't know. I didn't know how to describe it. I was just like, I'm CTO-ish. I'm not actually CTO-ing right now, but I'm talking to CTOs every day, you know, and mapping what's happening in the software engineering landscape.

(Steven Paleto at 00:49:40) So I'm, I don't know. I'm CTO-ish.

(Joel Beasley at 00:49:42) Yeah. And who was the CEO that brought you in?

(Steven Paleto at 00:49:45) J. Zach. J. Zach Stein.

(Joel Beasley at 00:49:47) J. Zach? Okay. So one of my favorite things about J. Zach, I don't know him, but is that he was able, because he worked with you, he was able to see and understand you and how you make decisions and how you solve problems and know that like, he could put you in this role and that you would figure it out. Like, you would problem solve it. You would figure it out.

(Joel Beasley at 00:50:08) You would speak up if you were underwater because that's the big thing we all know is like, really hard to get people to do. Yeah. You would, yeah, you would self-report if there were issues. And so that says a lot about your character too, that somebody that had worked with you and you were an expert in one specific niche field would let you join the company and actually let you be in another part.

(Steven Paleto at 00:50:29) Totally. I mean, I don't think in the abstract when I was leaving Lattice that anyone would give me a head of go-to-market interview.

(Joel Beasley at 00:50:39) Right.

(Steven Paleto at 00:50:39) Right? That no one would look at my resume and be like, ah, he's the guy to hire for this. You know? So yeah, I mean, that's the beauty of deep human relationships and the trust that you build with people. And you know, the thing about the technology industry is like, it's smaller than you think it is, and just trying to do earnest good work and maintain good relationships.

(Steven Paleto at 00:51:02) They serve you in really unexpected ways down the road. So I've heard ample stories of folks who co-found or do early-stage startups on the basis of, you know, those good historical working relationships. And you know, there's, the upside to that is that a lot of this stuff is relationship-based. The downside is it can sometimes be hard to break in as an outsider. Right?

(Steven Paleto at 00:51:27) But yeah, you're spot on that I'm very fortunate to have been given the opportunity to do something totally different.

(Joel Beasley at 00:51:33) Yeah.

(Steven Paleto at 00:51:34) And I take that seriously to just try and do earnest good work and do my best, and I'm not succeeding at every aspect of the job. I'll tell you that. There's some stuff that's hard, but that's the point of, to your point, you communicate as a team. You try to figure out, okay, if I'm not good at these things, how do we cultivate an organization who can compensate for those things so that I can focus on the things that I'm best at? And it's, again, that impact orientation of aligning.

(Steven Paleto at 00:51:57) I mean, it's kind of the Japanese concept of Ikigai. I'm sure you've seen.

(Joel Beasley at 00:52:00) 100%.

(Steven Paleto at 00:52:01) Yeah. It's like trying to align those things of like, this is what I'm good at. This is what I can have impact doing. This is what the world needs from me, you know, and just try to figure out like, what that shape of role can look like over time.

(Joel Beasley at 00:52:13) Well, and the hard part, the reason why it's an art versus a science is because it changes. True. Like, it improves.

(Steven Paleto at 00:52:20) It changes. Yeah. Totally. Totally.

(Joel Beasley at 00:52:23) You can be fully aligned in one season of your life and then find yourself out of alignment in another season.

(Steven Paleto at 00:52:28) Like, five years ago, I wanted nothing more than to like, lead an engineering org, and I was given that opportunity to go do that at Lattice. When I was leaving Lattice, I was like, I don't want to do that right now. I want to do something different. You know?

(Steven Paleto at 00:52:40) And totally. So your interests, your gravitations change. So it's a constant journey of just trying to align those things.

(Joel Beasley at 00:52:46) Yeah. And I think where I got in trouble early in my life is when I ignored that and tried to force other things that I think this is the way it should be, so I'm going to force this versus listening to this is, you know, where my attention and energy is flowing.

(Steven Paleto at 00:53:02) Totally.

(Joel Beasley at 00:53:04) Dude, this is awesome. What's the next step people can take with Span? Do they go to the website? Do they?

(Steven Paleto at 00:53:09) Yeah. Span.app. Check us out. You'll see an overview of what we do. There's a lot of really exciting stuff on the roadmap. So if you want to follow along on our LinkedIn, we'll continue to be publishing some really cool updates over the next couple of months.

(Steven Paleto at 00:53:22) And the hope is that we'll be doing a combination of really cool software stuff and product stuff, but also surveys and insights from the market. So yeah, follow us along, and hopefully we can add some value and some color to this emerging conversation around AI's role in the software world.

(Joel Beasley at 00:53:41) You nailed it. Hey. Tell your design team. Send them a DM after this and be like, Joel loved the design.

(Steven Paleto at 00:53:46) Yeah.

(Joel Beasley at 00:53:47) The UI, everything looks, it looks so beautiful. It's some of my favorite typography. I really enjoyed it.

(Steven Paleto at 00:53:52) I will absolutely do that. And that's, you know, Jared Arandu and team. He's our design lead, and he does some amazing work.

(Joel Beasley at 00:54:00) Yeah. Yeah.

(Steven Paleto at 00:54:00) Yeah. Good job.

(Joel Beasley at 00:54:00) It's beautiful website. It's beautiful software, so it's worth checking out. Where are you guys located? Are you in Austin?

(Steven Paleto at 00:54:10) We're distributed. So the...

(Joel Beasley at 00:54:11) You're distributed?

(Steven Paleto at 00:54:12) Yeah. The company headquarters is in San Francisco. Our founders are in the Bay Area. We have a small office there, but the team is distributed kind of all over the place.

(Joel Beasley at 00:54:23) I don't know why I said Austin. For some reason, I had that in my head. But...

(Steven Paleto at 00:54:26) Maybe I just gave you Austin vibes. I don't know.

(Joel Beasley at 00:54:28) I don't know. Is that what it is? You've got some Austin vibes. Hey. We don't do vibes. That's the title of this episode.

(Steven Paleto at 00:54:34) Yeah. Yeah. Yeah.

(Joel Beasley at 00:54:36) Thank you so much for listening. And if you found this episode useful, please share it with a friend or colleague who you think would get value from it. And if you have topics that you'd like to hear discussed on the podcast, either add me on LinkedIn, or send me an email [email protected]. Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.