Why Software Is Never “Done” Anymore with Iccha Sethi, SVP of Engineering at Vanta

(Intro Narrator at 00:00:00) Today, we're talking to Iccha Sethi, Senior Vice President of Engineering at Vanta, about why software is never really done anymore and what engineering leadership should do about it. You're listening to Joel Beasley, Modern CTO.

(Joel Beasley at 00:00:19) We've talked about all sorts of topics over the years, but I don't think we've ever done an episode with the main theme being why software is never done anymore. So you're going to own this topic here on the podcast, and I'm super excited about it. Because for most of software history, there was a concept like, I could build a small little application. It could be done. But now nothing's ever done, and I want to know why.

(Iccha Sethi at 00:00:46) Yeah. Great question. You know, when you and me were engineers, probably, we'd be like, okay, get the specs, write the software, write tests, add some observability, and it's kind of in production.

(Iccha Sethi at 00:00:59) And now I think, you know, with people building more and more AI features or agentic features, you're kind of introducing this dependency to a vendor who, A, is always changing things under the hood, even if you're pinned to a specific version. And two, they are constantly improving and shipping new versions of models and dependencies, basically. Right? And so what that means is, you know, even if you take the first scenario, like I'm using this version of Claude or an Opus 4.6, even underneath that, there's always drift happening. And the probabilistic or the nondeterministic nature of that means that your customer experience is never consistent.

(Iccha Sethi at 00:01:49) Right? Because your customers are relying on these agentic features within your product to get work done. And so the onus really becomes on us or me as an engineering leader to make sure that I am staying on top of this to make sure I'm providing the best possible experience or high quality feature to my customer, which means that I need to stay on top of these drifts because they're never published or announced. It's all experimentation under the hood. And also evaluating every new model coming out from every new provider to make sure that I am, again, making sure we're providing that high quality customer experience. If there's a better version, it significantly improves the outcome for the customer. I want to be using or leveraging that.

(Joel Beasley at 00:02:41) When you mentioned staying on top of these drifts, what does that actually look like at an organization? Is it a team that's doing it? Is it each individual engineer? How does it actually look?

(Iccha Sethi at 00:02:51) Great question. We have a whole system kind of set up to make sure that each—many teams build AI features at Vanta. And of course, I have an AI org and an AI platform team, and they provide the frameworks around this. They have frameworks for evals, and we can talk a lot more about it. But then each team owns the quality, we call it, of their AI feature.

(Iccha Sethi at 00:03:17) And so they are constantly running experiments and evaluations to make sure that you're doing both offline evals against a consistent golden dataset, as well as online evals against incoming customer requests and stuff to make sure that you are still providing that high quality output.

(Joel Beasley at 00:03:41) Okay. So you have—and how would it bubble up to you to signal that they're not doing a good job at that?

(Iccha Sethi at 00:03:53) Great question. So we have metrics we track and dashboards we track. The obvious ones are like, we have customer thumbs up and thumbs down metrics, or we have certain metrics of like, is a customer accepting the suggestion or rejecting the suggestion? And each area has their product owner and eng owner have dashboards where they track this. And leaders like me are always being like, hey, what are your metrics? How are you tracking them? What are they looking like? How are they trending?

(Iccha Sethi at 00:04:26) If they're not trending well, what are you doing about it? So that's more like, addresses the outcome. That's what I look at. And whenever that outcome isn't there, the question becomes, what are you doing under the hood to make sure this happens? And that's where all the observability related to agentic features comes in.

(Joel Beasley at 00:04:49) Okay. So you manage the drifts through managing the quality, and you have systems in place to do that. And then you have conversations with your leaders about where their quality is at or where their scores are at. And that's how you manage the drift ultimately.

(Iccha Sethi at 00:05:06) Yeah. And that's at my level. But then at individual team level, they have a more complex setup where they're constantly running these evals, both in their CI/CD processes and just running ad hoc at a certain cron period. And they're running against consistent—like I said, offline data, golden data set that we've built with our SMEs internally and also against customer data. And some teams have this cool setup where anytime there is a violation, you get a Slack notification.

(Iccha Sethi at 00:05:47) There's an alerts channel. And I was talking to one of the engineering managers about this, and I'm like, this is kind of like, you know, principles of operational excellence as you know it from—you would have an alert in a channel when your latency, P99 latency is higher or something like that. It's very much the same concept, but the metrics you're looking at are different, and the evaluators you're setting up are different. But we're kind of doing the same thing. We're notifying the team or the engineer that, hey, we are running this continuously, and this is where we're experiencing a degradation. Go take a look at that. And that could be for many reasons. Drift is one of the reasons. Unanticipated customer use case could be another reason.

(Joel Beasley at 00:06:38) You've mentioned evals a handful of times in a way that I want you to tell me more about how you're using that word and what that means in your company.

(Iccha Sethi at 00:06:49) Yeah. Yeah. Actually, maybe let me talk a little bit holistically about this AI evaluation maturity metric—

(Joel Beasley at 00:06:58) Okay.

(Iccha Sethi at 00:06:59) Matrix. And I'll give full credit to Andy, who is the EM at the company of the AI platform team who came up with this.

(Joel Beasley at 00:07:09) Go Andy.

(Iccha Sethi at 00:07:09) Yes. Exactly. I'm just representing his framework. So basically, we are—a ton of teams are building AI features, and we want to make sure that, like I said, they are maintaining the brand of Vanta, a high quality output. And so we wanted to provide guidance to teams to, like, how to do this well. And so there's various levels to it.

(Iccha Sethi at 00:07:34) So level one is we want to make sure that all agentic operations have traces associated with them. What tools are they calling? What responses are you getting? That kind of stuff. Then we want to make sure that they all have golden datasets.

(Iccha Sethi at 00:07:56) And what I mean by that is, since we do not train on customer data—that's very important to us to maintain trust—we create kind of fake input and output data. And this is built in conjunction with our GRC SMEs, governance, risk, and compliance subject matter experts that we have hired who have done kind of the job of our customers at other companies to build this golden dataset. So to make sure that each area or AI feature has a golden dataset. So that's the second thing.

(Iccha Sethi at 00:08:30) The third thing—the way we describe an eval is, hey, if a customer tries to come in and use this AI feature with inputs X, Y, Z, we expect the output to be A, B, C, but within these parameters, because, again, it's nondeterministic. So we cannot always be like, output is equal to two. If it's not equal to two, fail this test. But instead, we should be like, it should reasonably be within these parameters. It should match this tone, and it should talk about these things. And if it's not, kind of fail this evaluation of sorts. And again, over there, we have two types of evaluators. One is we have offline evaluators, which run against these golden datasets. And then we have online evaluators, which are running against incoming customer data.

(Iccha Sethi at 00:09:32) And that's kind of like level three. And then I'm obviously happy to talk more, but level four is how do you run multiple experiments in conjunction? And level five is how do you make this a loop where you take the output of the experiments and feed it back into the cycle so you have kind of this self-improving loop of AI quality.

(Joel Beasley at 00:09:54) Okay. I'm going to ask you more detailed questions about that. But first, it sounds like I'm going to go into an ad. I'm not going into it. But first—

(Iccha Sethi at 00:10:02) Yeah. No. No.

(Joel Beasley at 00:10:02) Has Andy written about this publicly? There's going to be a lot of people that hit me up. Is there any way they can read in detail about these levels and the maturity model?

(Iccha Sethi at 00:10:13) He's working on a blog post, and I'm going to really push him to maybe publish it soon on the Vanta engineering blog.

(Joel Beasley at 00:10:20) Ask him if he knows about what Claude is and be like, write it faster.

(Iccha Sethi at 00:10:24) He has a draft that I've reviewed. We should just get on publishing it for sure.

(Joel Beasley at 00:10:30) Yeah. No. Because this is—you guys are pioneering stuff. You know? I talk to companies. I've been doing this for ten years interviewing people, and it's rare when I find a company that is not only—a lot of companies will do something great, an even smaller percentage do something great and then tell the outside world about what they're doing.

(Iccha Sethi at 00:10:53) Yes. This is a great nudge.

(Joel Beasley at 00:10:55) Yeah. This is a great nudge.

(Iccha Sethi at 00:10:57) Andy, right after this.

(Joel Beasley at 00:10:59) Okay. So I'm following you here. You've got these different levels. You've got the golden datasets. You're monitoring the evaluations and the output that it's within range. You lost me at the multiple and self-improving situation. Explain that to me.

(Iccha Sethi at 00:11:13) So, you know, once you have your evaluators running, the next step—

(Joel Beasley at 00:11:18) Do they—I'm going to interrupt you a couple of times. Do they constantly run or they run on deployment and CI/CD? Like, how are they running?

(Iccha Sethi at 00:11:26) So both. So they're running in CI/CD, and then we run them when, say, a new model comes out. Or if there is a nudge to go run them, we can go run them manually, or you can set them up as a recurring run to go happen. And each team kind of does it slightly differently depending on their use case. So now once you have an evaluator, now the next stage is kind of what we call experimentation.

(Iccha Sethi at 00:11:57) This is about running multiple evaluators in parallel to determine what gives you the best customer outcome. So for example, an experiment could be, there's a new model in Gemini, ChatGPT 5.5 came out, Opus has a 4.7. What is truly the best model for this specific use case within Vanta? And we use all three in different places because they are good at different things. So that's an example of an experiment we would go run.

(Iccha Sethi at 00:12:29) Another type of experiment could be we are tweaking something in the context that we are sending to a given LLM, or we want to play around with the temperature or the creativity or other elements of it. Those are all experiments where we can go run multiple evaluators. And then when you learn something from this, generally what happens is teams—they do this every week. They have data reviews that they do this. They'll have ideas of like, hey, I want to go add extra data to my offline golden dataset because we didn't cover X, Y, Z use case, this exposed that, or here's how we can go improve the product in other ways overall.

(Iccha Sethi at 00:13:19) And so that kind of becomes the loop of like, you feed that improvement back in to right back at the beginning. And then you make sure you're generating traces. You have the dataset capturing that, and then you're running evaluations and experiments, and that's kind of the loop.

(Joel Beasley at 00:13:36) Okay. So you build the—I'm slower than the other kids. That's why we have editors. We build the evaluator. We run the evaluator that is checking on the model. Where does the customer improvement or, like, where does the improvement happen at?

(Iccha Sethi at 00:13:53) It's kind of in the end in the compounding loop phase, where we are like, we have learned that this—we've run an experiment with multiple evaluators, and you know what? The new reasoning model is not that great or the standard model is better. So I'm—and all our teams are using, they didn't, you know, they're using, say, the reasoning model for some reason. So I'm going to go back and switch that AI feature to use the model which is better for the customer. So that's the compounding loop back in.

(Joel Beasley at 00:14:30) Got it. Okay. So where I got off the rails was the context of thinking about it in its traditional feature improvement of an application versus in the context of trying out different models and always putting the most effective model forward. Is that what we're talking about?

(Iccha Sethi at 00:14:45) Exactly. Yeah. Yeah. Yeah. Yeah. And that's one of the many ways of improving it for the customer. Yeah. Yeah.

(Joel Beasley at 00:14:53) All right.

(Iccha Sethi at 00:14:53) It's so funny, Joel. You know, a few months ago, I would just be like, hey, team. How hard can it be to just ship an AI feature? You know? Just use the SDK, write the code, and they tell me, like, Iccha, you don't get it. This is a different way of thinking. And I'd be like, how hard can it be? You know? And then you start getting into the details of it and all these elements of it, and you're like, yeah. You know? There's a lot more nondeterminism to this than it is to ship a regular feature.

(Joel Beasley at 00:15:26) How hard can it be? That's been the beginning of so many journeys.

(Iccha Sethi at 00:15:29) I know. I know. It shouldn't—

(Joel Beasley at 00:15:30) It should take an afternoon. We should get this done in an app. And then it's, like, eight months down the road, and you've got—

(Iccha Sethi at 00:15:37) I mean, imagine explaining this when you're giving estimates, and you can't give accurate estimates anymore.

(Joel Beasley at 00:15:44) Especially to nontechnical people?

(Iccha Sethi at 00:15:46) Oh my goodness. Yeah.

(Joel Beasley at 00:15:47) You just have a board asking you. It's like, uh—

(Iccha Sethi at 00:15:50) Exactly.

(Joel Beasley at 00:15:51) Oh, man. How hard can it be? That's like a trigger for me. Yeah. You know what it's caused me to do, Iccha? It's caused me to be—I'm less—I have more patience and I'm less critical now. When I see something and my brain goes, oh, that's really easy. And I'd be like, there's probably a reason why it's not. Yeah. There's more—more likely.

(Iccha Sethi at 00:16:14) That's wisdom, Joel. Is that wisdom? Yes. That's—you have lived enough scar tissue to switch your way of thinking. That's what it tells me. Yeah.

(Joel Beasley at 00:16:26) You have too. And we're going to talk about your past and your career growth and all of that. But I did want to talk about how—because we got introduced through Span. Yeah. And so tell me about what Span is to you, how you guys use Span, all of that.

(Iccha Sethi at 00:16:39) Yeah. Great question. So every company I have worked at before here, you know, me and other engineering leaders would try to hack up some version of metrics to collect to know how your engineering team is doing. It'd be somebody's side project somewhere, but Span kind of provides that and better out of the box for you. Couple of ways I personally really like to use Span—one is looking at onboarding metrics, actually.

(Iccha Sethi at 00:17:12) Especially as Vanta is growing and hiring a lot of engineers, I want to make sure that everyone coming in is set up for success. So looking at onboarding metrics is one of the things I keep up with on the regular. The other is—

(Joel Beasley at 00:17:29) Like onboarding new engineers, it's got metrics that follow them so you can see your new fish in the sea, and you can see how well they're adapting.

(Itjoseffy at 00:17:36) Exactly. For example, time to first PR and time to tenth PR are two sets of metrics that I look at in there. And that's important because it tells me how much are they fighting the system getting the access to various tools. Is there, you know, manager and onboarding buddy setting them up for success with the right first PR?

(Itjoseffy at 00:17:59) How hard is it for somebody to be successful? And then I do the tenth. We do the tenth PR too because, you know, one can just be like, get something out quickly. The tenth kind of tells you they have found their way around a little bit more.

(Joel Beasley at 00:18:15) Are these accepted PRs? I don't want to get too nerdy, but are these accepted PRs or just PRs?

(Itjoseffy at 00:18:20) Merged PRs. Yeah.

(Joel Beasley at 00:18:21) Nice. All right. Very cool.

(Itjoseffy at 00:18:24) Yeah. And I use Span to do a lot of deep dives, reflective lookbacks kind of thing. So for example, you know, we hired so many engineers last year, and I was like, I want to know what's the impact of that? Are we shipping more product? What are we doing?

(Itjoseffy at 00:18:47) And what was interesting is Span has this thing around investment mix, whether you're investing in net new things, continuous improvement, KTLO kind of stuff. And I know a lot of people use this for software capitalization, but I kind of used it to look at an interesting trend that our investment is now increasing in continuous improvement, and not surprising because the product is continuously growing deeper and deeper and not only wider. So it makes sense. That kind of analysis is super interesting to use Span for.

(Itjoseffy at 00:19:26) And then the last one is the obvious answer that a lot of people will give you is the AI adoption of, are more engineers using Copilot and Cursor and Codex and tools like that or not, and what is the impact of that to our escape rate and kind of other metrics overall.

(Joel Beasley at 00:19:47) Oh, nice. I haven't heard, I don't know escape rate. What's that?

(Itjoseffy at 00:19:52) It's basically how many bugs, you know, are in your system. Yeah.

(Joel Beasley at 00:19:57) Okay. You know, it's so funny too because there are so many different companies and so many different terminologies, and then people will use the same words but differently in different industries.

(Itjoseffy at 00:20:06) Yeah.

(Joel Beasley at 00:20:06) And so for me, I've just got, I just, when I hear something, I just don't know. I just ask all the time.

(Itjoseffy at 00:20:12) That's great. Even I used the eye icon in Span to be like, oh, what was this escape rate when I first saw it? And I'm like, oh, yeah. Yeah. So you're not alone, Joel.

(Joel Beasley at 00:20:22) You use the info bubble.

(Itjoseffy at 00:20:23) Yeah. Yeah.

(Joel Beasley at 00:20:23) I love it. So just curious, not to get too into it, but you mentioned that investment mix thing that's inside of Span where you could see continuous improvement, net new. Does that have any exposure or connection to your CFO situation, or is that just in product?

(Itjoseffy at 00:20:40) Yeah. We do, you know, connect it to software capitalization and tracking that. So there's a tie to that. But I think a lot of companies, you know, we track that internally and we track it in Span, and we kind of mix and match to see what looks more accurate. I think that definitely helps with that, but I think it also helps me quite honestly take a look at where are we investing time as an organization, and does this investment mix match what I expect it to be or not?

(Itjoseffy at 00:21:12) And if not, why? And what did we miss over there?

(Joel Beasley at 00:21:17) And then as far as, I know you had some thoughts on greenfield versus brownfield code bases and AI agents. Does any of that apply to what you do with Span?

(Itjoseffy at 00:21:27) Yeah. Good question. There's, you know, a ton of blog posts out there doing greenfield work and how they are using AI really well. And anyone I've talked to doing brownfield work with an existing code base with a ton of history, they'll talk to you about, you know, coding agents are helpful, but it's not as magical as it is when you're maybe doing something more greenfield. And Span has this really cool data about what is your AI adoption and how does that compare to the industry?

(Itjoseffy at 00:22:05) And the compare to the industry part was super interesting to me because it compared it against a lot of other companies in a similar situation as us. And I was actually quite pleasantly surprised because I'm like, oh man, are we, you know, a whole other topic about all the investments we are doing to make AI agents work within a brownfield code base. Span helps me track whether we're still trending in a good direction or not. The other thing which is super helpful is when these coding tools became more popular late last year, a lot of the engineers had concerns about AI slop, which is just really large PRs generated or one-shotted by Claude. And, you know, the burden it puts on reviewers.

(Itjoseffy at 00:23:04) And what Span's really been good at is it also tells you what your PR size is. And you can correlate, and that often correlates to higher bugs or escape rate as well. And so that kind of helps me keep pulse on things like, yes, I want people to use AI, but I don't want them to throw slop over the wall and still own the outcome. It kind of helps you track, not just usage, but how are people using it also.

(Joel Beasley at 00:23:34) Now I've been, I was hands-off programming for five to six years when the podcast, so I did it for seventeen years.

(Intro Narrator at 00:23:43) Mhmm.

(Joel Beasley at 00:23:43) And when the podcast started to take off, I kind of just took a back seat. And in the past couple months, I've built an application for myself, and it's been unbelievable. And I always like to, I use myself because I build things that I really care about. And it's funny because I can see where I would want to get lazy and then I choose not to. Like, in my mind, I'm like, okay.

(Joel Beasley at 00:24:08) Most of the people won't choose not to.

(Itjoseffy at 00:24:10) Yes.

(Joel Beasley at 00:24:11) Most of the people will get lazy. And so I was building this application to monitor our Bitcoin, we've got some Bitcoin mines. And I was building this to monitor it for our partners. And, you know, I started with some basic stuff. I had it scaffold out, you know, a basic installation of Rails. And then I did a couple things and started, and I realized, okay.

(Joel Beasley at 00:24:31) If you go this big with a full feature, it's not gonna, it's gonna create junk. There'll be small little, and so I realized, and I don't know how to put it into words because I haven't had to explain this before. But I realized the size in which I could give it a task, and then I could review them as a senior engineer that I am and be like, okay, that's 99% right. You know?

(Joel Beasley at 00:24:52) And what size of the chunks is directly related to the complexity of it. You know? And so the more complicated it is to explain or to build, the smaller chunks I break it into all the way down to a function if need be. But that is like a new art. It's like a new skill is what size chunks do I break this into to give to AI?

(Itjoseffy at 00:25:15) Yeah. 100%. I think our engineers today are learning slightly different versions of the skill sets beyond coding. Like you said, it is breaking down these tasks into right size. It is how to prompt or nudge or teach these agents.

(Itjoseffy at 00:25:34) I did a pair programming session with one of the staff engineers here at Vanta. And I was kind of watching him go between the agent, you know, give it instructions, look at the code generated, and being like, you know what? It didn't do these things right. Let me go instruct it slightly differently. And then, oh yeah.

(Itjoseffy at 00:25:56) Now I did it right. Let me just go add this now to, you know, my Claude MD file or whatever. And it almost reminded me of pair programming with a junior engineer. But instead, you're doing it with an agent. And it's a whole different paradigm.

(Joel Beasley at 00:26:16) It's pair programming with a junior engineer who has the confidence of a senior engineer.

(Itjoseffy at 00:26:22) I love that. That could be a tagline on its own. Yeah. Yeah. Yeah.

(Joel Beasley at 00:26:26) That's what it's like. Oh, that's, yeah, that's so funny. Oh man. Cool.

(Joel Beasley at 00:26:32) So overall, you're a fan. You like it. You'd recommend it to friends?

(Itjoseffy at 00:26:36) Oh, I love Span. Big fan of the team there too. We were early design partners with them. I am actually, you know, we do a lot of looking at Span to debug organizational delivery issues. Yeah.

(Itjoseffy at 00:26:51) Super helpful. Highly recommend.

(Joel Beasley at 00:26:54) Nice. Great. Well, you know what? That carries more weight than it normally would given the AI maturity model innovation that you guys have.

(Itjoseffy at 00:27:04) Yeah.

(Joel Beasley at 00:27:05) Because that shows that you're a really forward company, and forward companies are pretty good at picking tools and choosing what they need to invest their time into. So I wanted to ask you about your career before Vanta. And then also, oh, you know what we definitely need to do? I'm excited about your career. We should definitely tell people in two sentences what Vanta is and the general size of the engineering org if you can share.

(Joel Beasley at 00:27:33) I don't know if you can. But if you can, so people have context to all of this that they're hearing.

(Itjoseffy at 00:27:37) Yeah. So Vanta is an agentic trust platform. So we help you earn trust, you know, and also display that trust to your customers via things like we help you do, you know, get ready for your audit. We help you do security reviews, do third-party vendor risk management, trust centers, questionnaire automations, and much more. And the way I look at it is the agentic nature comes from, we started with, you know, hey.

(Itjoseffy at 00:28:10) Let us build the evidence to make sure that you can, you know, do these tasks to help doing these tasks for you with the various agents that we build within our platform.

(Joel Beasley at 00:28:23) That is awesome. Yeah. And then the general size?

(Itjoseffy at 00:28:26) Couple of hundred within engineering. Yeah.

(Joel Beasley at 00:28:29) That's that's plenty enough. Yeah. It'd be one thing if you guys, you were like, three. We got three. It's Andy and me.

(Joel Beasley at 00:28:36) I'm the third person. You know? Because that happens. Yeah. Every company has to start somewhere.

(Joel Beasley at 00:28:41) It all starts with one.

(Itjoseffy at 00:28:42) Yeah.

(Joel Beasley at 00:28:43) You know? Yeah. Oh, that's fun. All right. So tell me about your journey from young, interested engineer all the way up to here today.

(Itjoseffy at 00:28:55) Yeah. I feel very thankful about the experiences I've had in my career. I feel like I've worked at big companies, at startups, back to big companies, a very diverse set of experience. So I started off, you know, my career growing up the engineering track, you know, went right up to being a principal engineer at Atlassian, then was, you know, did some pre, was, to put it into, head of engineering at Envision and switched to the management track there. For those who don't know what Envision was, it was like a Figma competitor in the day, which was pretty cool at that point in time.

(Itjoseffy at 00:29:35) Then went on to being in leadership at GitHub, where I was a senior director of engineering for a multi-product portfolio, GitHub Actions, Codespaces, Packages, npm, and so on and so forth. And then landed at Vanta running its engineering org.

(Joel Beasley at 00:29:56) Okay. You're awesome, first of all. The moment you said Atlassian, I wanted to interrupt because I got excited.

(Itjoseffy at 00:30:02) Yeah. I saw your face. I was like—

(Joel Beasley at 00:30:03) I know. I was like, oh, I was like, of course. You want to know why? Archana Rao, all these great people that I have met. I think I've interviewed three or five Atlassian leaders over the course, like CTO, CPO type people over the course of these ten years.

(Joel Beasley at 00:30:20) And every time, I'm like, I would bet money. They're awesome. There's something about the culture, whoever the board, whatever is going on over at Atlassian, they're attracting the best, brightest people in the world. And every time I meet someone from there in leadership, they're just amazing. They're just phenomenal people.

(Joel Beasley at 00:30:38) And then you go to Envision, which is one of my favorite products ever back when I ran the development agency. We were a big customer of that. And let's see. And then, of course, GitHub. GitHub's great.

(Itjoseffy at 00:30:57) Yeah. Okay. Cool. Good to know that you've used all the products. Yeah.

(Joel Beasley at 00:31:00) So how have you made these decisions? Were you intentional? Like, let's say Atlassian was your first one, and you're just like, oh, I just need to make sure I work at a company as good as this. And then were you using Envision at Atlassian? How did you make these decisions and end up at a string of awesome companies, or did it just kind of happen?

(Itjoseffy at 00:31:19) Good question. I kind of always prioritize my career decisions based on what am I going to learn from this experience. And I feel like if I am going to learn something new and different, which is gonna help me grow as a leader. And often, if there is somebody I know that I respect who works there, that helps too. So at Atlassian, you know, the engineering leader running the portfolio of products that I worked into went to go be the CTO at Envision. And he was somebody I really respected, and he's like, hey.

(Itjoseffy at 00:31:58) You know, you're solid. You're really good. Come report to me. And it was a unique opportunity to report into the CTO at Envision and be a principal engineer for the entire organization when I moved there.

(Itjoseffy at 00:32:10) So I kind of look at it as, do I get to work with awesome people that I'm gonna learn something from? And, how am I gonna grow in this experience? So that's kind of how I did. I don't look at the brand of the company itself on the outside too much versus what am I gonna do there, and what am I gonna learn from that?

(Joel Beasley at 00:32:31) It's interesting. There is a trend that the high quality people create high quality products. I did a Google real quick. Yeah. That Envision is definitely the one I was thinking of.

(Joel Beasley at 00:32:42) And I saw that they shut down their services. Did they become something, did they get bought by someone else, or what happened there? Do you know?

(Itjoseffy at 00:32:49) Yeah. I know part of it got sold, and then, you know, I think Figma just took the market cap. And that's why I—

(Joel Beasley at 00:32:59) Yeah. Yeah. Yes. But at the time, when Envision came out, I was like, take my money.

(Itjoseffy at 00:33:05) Yeah.

(Joel Beasley at 00:33:05) It made my life so much easier doing mockups and building applications with clients and everything. It was just brilliant. And then it integrated into, like, Sketch.

(Itjoseffy at 00:33:15) Yeah.

(Joel Beasley at 00:33:15) Sketch. Yeah. It integrated into Sketch, and then you could drag. Yeah. And that was, I sit there with my buddy, Derek.

(Joel Beasley at 00:33:21) I'm like, this is the future.

(Itjoseffy at 00:33:23) I was there in all the good parts of Envision.

(Joel Beasley at 00:33:28) Yes. Yeah. Yes. And then how did you get over to GitHub?

(Itjoseffy at 00:33:33) So, you know, Envision was my great startup experience, influenced the whole company, and really learned a ton about end-to-end ownership there outside of just technical stuff too. And I had kind of switched into management track during my time there, and I wanted to go to a bigger company to learn about how to be a really good engineering leader. And I had a couple of options of companies to choose between, and I ended up choosing GitHub because, again, I look at what am I gonna do there? And this opportunity was related to GitHub Actions, which was in its very, very early stages of just taking off. And I kind of took a bet on, I think Actions is gonna be big.

(Itjoseffy at 00:34:24) GitHub has a big customer base. I am gonna learn a ton through this growth over here. So I kind of picked GitHub for, I think this product is really important, interesting. I'm gonna learn a ton. I didn't know anybody personally there, but I was like, I'm sure given the brand it has, it has a bunch of good people.

(Joel Beasley at 00:34:44) That is awesome. Okay. And then what was your gut right? Did it end up becoming a huge thing?

(Iccha Sethi at 00:34:51) Oh, yeah. GitHub Actions, like, I think took up a lot of the market share. It became a very popular product. I feel like anybody I talk to, they know what GitHub Actions is. And it was a good bet personally for me too, because during that time I got to not only own Actions, but also ended up owning Codespaces, Packages, NPM. And this is such a wide portfolio where Actions was like this big, scaled, popular product. NPM was open source, and Codespaces was just taking off. So it really helped me to vary how I should think about as a leader for a scaled product versus a brand new product entering the market versus an open source kind of product which exists. So it turned out to be a fantastic learning experience for me.

(Joel Beasley at 00:35:49) I'm learning a ton. Okay. GitHub and NPM, explain the relationship there.

(Iccha Sethi at 00:35:58) So NPM was, you know, it's your node package manager. It was kind of dying in the open source community. The company which was funding it, I think, was unable to support it financially anymore. And the GitHub leadership decided to, you know, being the home of open source, swoop in as benefactors to be like, let's pay for it and support the NPM team because it does good to the entire developer ecosystem. And that's how we ended up acquiring and supporting NPM.

(Joel Beasley at 00:36:35) Okay. Good. Because for a second there I was like, there's no way GitHub created it. I mean, NPM has been around for a long time. I was like, how did, how did she keep saying GitHub and NPM? And I couldn't—thank you for helping me bridge that gap. I'm learning so much today. Oh man, this is good. Okay. So then you're at GitHub. You finish your project or you reach your, like, what you believe to be as your potential there, and then that's when this Vanta opportunity came around?

(Iccha Sethi at 00:37:03) That's correct. It was, you know, I had worked with Jeremy Epling at GitHub who is the product counterpart to a lot of these products. He moved to Vanta. Vanta was Series B at that time, and this was a great opportunity for me to run all of engineering. You know, you're always like, if I ran engineering, here are ideas I would implement or experiments I would do or try. And I thought this was just a great opportunity to go do that. I was also familiar with the problem space because as an engineering leader, I've done SOC 2 multiple times and ISO multiple times. So I'm like, okay, Vanta has good leadership. Also, a big fan of Christina, the CEO. And, you know, it has solid leadership. It's solving a real problem, and it gives me also a good opportunity as a leader to grow. So it kind of fit all those boxes, multiple boxes. And then if I look back at my career, at Atlassian, I had a ton of lessons around like, hey, here's a perfect engineering system you can build. Like, I had built the most beautiful architecture for one of the chat products. And eventually, you know, we ended up killing our chat product.

(Joel Beasley at 00:38:32) That happens.

(Iccha Sethi at 00:38:32) That's a real thing. Yes. And trust me, Joel, that's the most beautiful piece of code I've ever written in my life.

(Joel Beasley at 00:38:40) I believe you.

(Iccha Sethi at 00:38:40) I could, like, recite the internals of Kafka and just became a deep expert.

(Joel Beasley at 00:38:47) Oh, and the classes read like beautiful stories. It was very clear what was happening. It was amazing. Yes.

(Iccha Sethi at 00:38:54) And but then, you know, the business impact wasn't there. And then at GitHub, I got to see that, oh my god, this is such a beloved product, such an impactful product. And, you know, the architectural investments play such an important role to be able to continue shipping value to customers and to be able to scale to customer needs. Kind of learned from like, hey, I wish GitHub had done this, you know, X years ago, hindsight kind of stuff. You learned a lot of that too. Then coming here, I was like, oh, I have all these great lessons under my belt. And it's going to be, you know, I can bring these to Vanta to thread the needle between where do you need to invest in architecture foundations versus where do you need to first seek product market fit and accept some trade-offs and then later go invest. And what's the right—it's very important to do it kind of at the right time and not over—

(Joel Beasley at 00:39:58) Timing is everything.

(Iccha Sethi at 00:39:59) Timing is everything, right? And not overload too much one way or the other at any given instance. So I, and, you know, I feel like I'm implementing them at Vanta. Like, I have an engineering strategy and initiatives and kind of guidance, but I guess only time will tell.

(Joel Beasley at 00:40:17) I would only be worried if there weren't uncertainty.

(Iccha Sethi at 00:40:20) Yeah.

(Joel Beasley at 00:40:21) Yeah. Yeah. Like, if you were like 1,000% confident, like, this is the exact—yeah. Yeah. The fact that you're always a little bit like, that has saved me quite a bit.

(Joel Beasley at 00:40:31) Yeah. That's that's the wisdom.

(Iccha Sethi at 00:40:33) Yes. It is the wisdom.

(Joel Beasley at 00:40:35) Yeah. Everything looks—I did the amount that you should do in making this decision, and we'll see how it goes.

(Iccha Sethi at 00:40:42) Yeah. Yeah. And my style is also like, I bring my team along and get their input and feedback. So like, even though I own the engineering strategy, I'm like, we are building this together. Tell me if you think I am doing something wrong. Very much that approach. Yeah.

(Joel Beasley at 00:41:00) Yeah. Which, let's talk about that a little bit for leadership stuff. So that was a hard lesson for me to learn. I kind of bounced to both sides of the spectrum. I started with dictator. Yeah. And then I went all the way to like, oh, we've got to get the whole group and make everybody happy. Neither of the extremes work. You kind of have to say, we're going on a road trip. Yep. Here's the destination. Here's how I plan on getting here. Now what do you guys see that I don't see? And then get them involved. And have you found—is that an accurate—

(Iccha Sethi at 00:41:36) Yes. Yeah. That's a great way to put it. And I have this saying, which, you know, early in the career in my first job I heard it, and then it's kind of stuck with me through my entire career. It's like, we all want to be valued members on a winning team on an inspiring mission, right? So all these components, it kind of maps to Drive and other books and their concepts out there. So, like, it's like everyone is excited about the journey we are on as a company, as a product. We get what we're trying to do. We are setting the teams up for success, which is giving them clarity, clear direction, giving them ownership and autonomy with the right amount of checkpoints. And valued is like, you're appreciating them and giving them feedback, good and bad feedback, you know? Everybody, including me, I'm like, don't always tell me I'm doing a great job, but also tell me how can I do better? Like, and, you know, being like, oh, you took it and you did an amazing job and you ran with it. I really value you over here. So I think those are all the really important components to running a successful team or an organization.

(Joel Beasley at 00:42:52) Okay. So help me understand this. I want you to share with me the time you mentioned good and bad feedback. Yeah. It's hard to give that bad feedback in a good way.

(Iccha Sethi at 00:43:05) Yes.

(Joel Beasley at 00:43:05) Yeah. It is hard. Yeah. Tell me about a time where you messed that up or figured that out.

(Iccha Sethi at 00:43:14) I think this was early in my management career-ish. It falls into the ruinous empathy trap often where, you know, I had a direct report and I knew that they were trying their hardest, and they were a good person. But their, you know, entire team was giving me feedback that this individual wasn't delivering value or providing clarity as a manager. And in my interactions with them, I would always be like, you know, you're doing all these things great, but if you could just tweak this one thing, it will be awesome. And I think it would often get lost in that. So what I've learned to be is like, really clear. Like, hey, I am hearing these themes from your team: A, B, and C. I know you're trying to do this, but this is impacting them in these ways. I am here to be your brainstorming partner on how to do this better. You know, can you come back with a plan for me? Or be like, I want you to try out A, B, and C. If you disagree, tell me why, and we can again brainstorm together. I will check back in with your team in, you know, X weeks or whatever, and then we will review this and see how it's going. And so it's very much like a nervous, you know, don't want to offend you to like, being clear is kind, kind of being very direct in listing them out. Yeah.

(Joel Beasley at 00:44:52) Yeah. That—you sound like Iccha AI. I was like, oh, there she goes.

(Iccha Sethi at 00:44:58) I mean, it's the wisdom, Joel. It's the wisdom.

(Joel Beasley at 00:45:01) It's the wisdom.

(Iccha Sethi at 00:45:02) Yeah. What is AI but the wisdom of millions, you know?

(Joel Beasley at 00:45:07) I know. I know. And we, you know, we've—so many generations have come before us, and so we're learning on top of their learnings.

(Iccha Sethi at 00:45:13) Exactly. Yeah.

(Joel Beasley at 00:45:14) Yeah. That is interesting. Yeah. You handled that well. Those were good, good little managerial clips.

(Iccha Sethi at 00:45:22) Yes. Yes. Learned it the hard way, but I think I have a system now. Yeah.

(Joel Beasley at 00:45:28) For me, it was—I tend to be a people pleaser. So, yeah. I'm on one end of the spectrum, I'm a people pleaser. The other end, I'm like, autistically direct, you know? And so I kind of bounce around. But for me, learning the feedback—to give clear, direct feedback when it's painful, when I'm shy or nervous or I feel like, oh, I don't want to do that, that's the sign that I need to do it the most. And then the way I learned—the way I got the courage to do it and do it quickly. And now, today, it's a reflex. It's just like, we can just do it. Yeah. But going through the process of learning, it took a lot of pain. And once I realized that the pain of not doing it, it's just—it's so great that you just have to do it. Yeah. Because, you know, like, you go through that cycle enough, you're just like, I have to do it. I have to give them the bad feedback, and I have to give it to them fast, and I have to learn this lesson. And then of course, you know, a year or two after I learned that, somebody writes a book on it, and I'm like, I wish this book had existed before. Yeah. Yeah.

(Iccha Sethi at 00:46:34) Yeah. Yeah. And I think like, even if AI is there today as a brainstorming partner for managers, which I think is a great resource, right? Like, I think a ton of people you hear use AI to help them brainstorm for performance reviews and stuff like that. I think there's still the human element of learning how to synthesize it and deliver it, which kind of comes only with experience.

(Joel Beasley at 00:47:03) You know, you just triggered a thought of—I'll hold my comment until after I get your answer. How much time do you spend with the customer? Like, how often do you get pulled into large organizational sales type stuff?

(Iccha Sethi at 00:47:21) It's very much company dependent. At various companies, it's been different for me. Like, GitHub, I was on customer calls multiple times a week versus Vanta. It's a couple of times a month. And it kind of depends on what the customer wants to talk about, where it makes sense to have a pure product roadmap conversation versus an engineering plus product conversation.

(Joel Beasley at 00:47:52) Yeah. I've got this working theory of mine that one of the—because young engineers always ask for advice. They write in and all this stuff. And one of the things I've been playing with lately is telling them that the top 1% of the guests that I have on—like, I'm always trying to figure out magic. I call them magic people. Yeah. Yeah. Yeah. Like, why is—like, what's the top 1% of the top 1%? Like, what are those people doing different? And I have noticed a trend. They have exposure to the sales side of the organization. Yes. They've spent some time with customers. Yes. And you would be surprised that that's not a lot. It's not as much as I would hope it would be, but it is a trend in the top 1%. So—

(Iccha Sethi at 00:48:33) Oh, yeah. I mean, I have regular one-on-ones with post-sales leaders, with the leader of the solutions architects. I have these conversations with them exchanging feedback back and forth. And I think customer calls are—if you're an engineer and you're not included in one, I would say you can always go watch a recording of one, which, you know, I try to do all the time. I ask a query for like, can you give me snippets related to X? And go listen to segments of customer conversation on a certain topic. I think that's a huge time saver too, as an engineering leader.

(Joel Beasley at 00:49:16) Oh, that's awesome. Are you—so you're hooked into, like, the—like, we use Fireflies here, and that records all of our customer calls. And then I can query Fireflies and ask it for specific things. Is that what you're kind of referring to?

(Iccha Sethi at 00:49:28) Some version of that now. Yeah. Uh-huh.

(Joel Beasley at 00:49:30) Oh, yeah. Gong. That's a very popular one as well.

(Iccha Sethi at 00:49:32) Yeah. Yeah.

(Joel Beasley at 00:49:32) I think they were actually like the pioneers of the space.

(Iccha Sethi at 00:49:35) Yeah. I'm not quite sure, but I feel like it's super easy for me to go look up customer clips on a certain topic. And then, you know, obviously I love going into specific customer calls because especially with all the AI product features we're building, asking them questions about like, especially the users who are in the forefront of this usage, how are they thinking about it, definitely informs my perspective.

(Joel Beasley at 00:50:01) Interesting. Does Gong advertise it for that use case, or did you just—where you're like, all the information is there, I'm going to go see?

(Iccha Sethi at 00:50:08) I think a lot of us at Vanta very much use Gong this way apart from—I don't know if they advertise it or not, but we're very much like, okay, we have dedicated Slack channels where we get alerts on certain topics of clips and stuff. Oh, come on. Yeah.

(Joel Beasley at 00:50:23) That is—you guys have got it going on.

(Iccha Sethi at 00:50:25) Yeah.

(Joel Beasley at 00:50:27) Yeah. This is good. This is good. Yeah. Christina, your CEO, or someone else is like, you are sharing too many of our amazing secrets. We might continue to attract amazing talent and more customers. That's great. Oh, this has been fantastic. And so last question I have for you. A piece of leadership—I'm going to give you constraints. It's a piece of leadership advice that somebody gave you early on that you implemented, and you're like, wow, this works really well, and you've kept it with you for the most of your career.

(Iccha Sethi at 00:51:00) I look at it as like, I've learned certain things that certain leaders have done that I have adopted into my own working model of how I operate as a leader. Like, you know, I had a leader who's really good at the operational side of things when it comes to delivery, when it comes to the operational health of your services. He had a very solid framework instituted that I adopted, and I've carried with me every job I've worked at. So, you know, we have weekly operational reviews, monthly business reviews. It's very much a model I have learned from him.

(Itjoseffy at 00:51:43) And then the other skill I learned, and this was from actually somebody who was a direct report of mine back at one of my jobs. She was very good at enabling the EM layers under her. And every change we wanted to roll out, she would pair it with enablement and AMAs and carry the human elements of getting something done. And that's something I tried to take with me too.

(Joel Beasley at 00:52:19) That's brilliant. Yeah. Okay. So you're out there. You're observing.

(Joel Beasley at 00:52:22) You find behaviors and traits, and you're like, oh, that's good. And then you grab that, and you just keep doing that. Alright.

(Itjoseffy at 00:52:29) Keep adding it to my tool belt. Yeah.

(Joel Beasley at 00:52:31) Yeah. Yeah. Like, what, are you trying to be a successful person over here? That's good. That's good.

(Joel Beasley at 00:52:39) This is great. Wow. Iccha, you crushed it, man. This is a good—you know, it's days like today that I'm like, I love doing this show. Thank you so much for listening.

(Joel Beasley at 00:52:52) And if you found this episode useful, please share it with a friend or colleague who you think would get value from it. And if you have topics that you'd like to hear discussed on the podcast, either add me on LinkedIn or send me an email [email protected]. Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.