6 Critical Inputs on AIOps with Tony Davis, Chief Strategic Advisor of AIOps at Broadcom

(Intro Narrator at 00:00:03) Hello, my friends. Today, Joel is talking to Tony, the Chief Strategic Advisor of AIOps at Broadcom, and they discuss how the real value of AIOps is observations on how IT is impacting your customers, the six critical inputs to analyze what part of the stack is most impacting the customer experience, and why technical leaders have to be motivated by growing the people that they manage. All of this right here, right now, on the Modern CTO podcast.

(Joel Beasley at 00:00:37) Here we go. This is the Modern CTO podcast.

(Tony at 00:00:49) So I started in IT in 1991. I've been there a long time doing different things, just like everybody else, right? My first major gig was for FedEx in their IT division. I worked in the FedEx.com division.

(Tony at 00:01:07) And the exciting thing about that — I started working in that area around 1998, and the thing that was so funny to me, and I still look back and laugh, is going to work for FedEx.com was like you were going to work on some crazy project. No one will ever ship packages on the internet. Who would ever ship packages on the internet, you know?

(Tony at 00:01:30) And that's exactly the way it felt back then. And then I ended up spending about ten years working just in FedEx.com, from software development to operations side for FedEx.com. And then I eventually ended up — by far the best gig that I had in my FedEx world was over what they called their reliability engineering team. And so for about three years, for FedEx.com, I worked on the team that was there to protect the reliability of that revenue.

(Tony at 00:02:04) So whatever amount of revenue we were making, and it was in the billions, the job of this team was to protect it and make the systems reliable. And so that's actually where I worked on a new methodology all the way back then, about 2009 and 2010, is when I worked on it. And it was sort of creating a methodology for people who are in that world. But instead of just creating great operational dashboards and operational metrics, we decided to use the same data but bubble it up into customer experience type data. And so it's a methodology.

(Tony at 00:02:46) And then after that, for the next few years, I spent some time with the original CA Technologies and did some evangelizing with our customers of how to use that methodology. And then what it's morphed into now — and now I'm the strategic advisor for the company — but it's a very customer-focused position. So I get, it's like the best job in the world because I literally get to go to our customers and look at their entire landscape using Broadcom products, but more importantly, using any product. So maybe I have a lot of history with Splunk and AppDynamics, and so I'm able to go in and look at what they have from Broadcom, but then also match it up with their Splunk or whatever they have.

(Tony at 00:03:31) And then I create sort of their whole AIOps architecture. It's easily the best job I've had in my life. It's not even close. So yeah.

(Joel Beasley at 00:03:40) So is it site reliability engineering that you're still doing, or is it AIOps, or are they the same thing?

(Tony at 00:03:46) Yeah, that's a great — so if you were to go back and ask me that back in 2010 when I had just come from FedEx.com, I would say that my thinking and what I do is really focused on site reliability engineering because we did have some of the classic roles at FedEx for site reliability. So we had, you know, a single team, for instance, with hardcore Java developers and hardcore back-end developers as well as network engineers on the same team. And so basically, you couldn't get anything past the reliability engineering team because they knew everything. So if you asked me back then, I would say that's what I was more focused on.

(Tony at 00:04:28) What I'm more focused on, and I sort of morphed into, I guess you'd say evolved with what we've been through over the years, is how do you take that same data and make it more customer and business relevant? And so now, as you've heard people talk about AIOps and observability — so we have that same thing, of course, at Broadcom, AIOps and trying to get you to a level of observability. But what, in my role as sort of the chief advisor for our customers, what I keep trying to keep at the front of their face is don't get caught up on the technologies necessarily that drive AIOps and observability. Make sure that you're obsessed with your business observability.

(Tony at 00:05:13) So it uses the same tools. It's funny you'd ask that question. I do use the same technologies and tools that I did ten years ago. But what I'm asking our clients to do is look at what they get from those tools very differently than what we did ten years ago, if that answers your question.

(Joel Beasley at 00:05:30) Yeah. Actually, when I saw AIOps, I was really excited to talk with you because I don't really understand — I mean, I talk to people all day, and when I saw AIOps, it was one of the first times I'd ever seen it, for your prep for your interview. So I was like, you know, I can always Google it, but you don't know the people behind what you're getting there. So I want to know, like, how do you explain what AIOps is?

(Tony at 00:05:51) Yeah, and I think you'll find, you know, as you talk to people, you'll get sort of different versions of what they think AIOps is. So when I try to talk to our customers, I try —

(Joel Beasley at 00:06:02) I try to put it in 100% real world because I don't want it to come across as academic. Here's what AIOps is. I saw one thing one time that sort of —

(Tony at 00:06:12) It didn't crack me up, it's okay. But I saw one thing where somebody said, well, the true meaning of observability is based on control theory, and, you know, here's where it falls in the chain and all that. But from the real world perspective, that's not what observability is. Observability is the end result. You've got all these monitoring tools, whether they are intelligent monitoring or what I call legacy type monitoring. But you have all these tools and they're all bubbling up this amazing amount of information. Well, once you apply AI engines, some form of artificial intelligence or some form of machine learning, once you apply that to that data, then you start to get things that are very valuable as long as you're looking at them from a business perspective. So for me, when I talk about AIOps and observability, it's three layers. The first layer is all of that — blah — data. It's all the data that's coming from these endless monitoring tools: application, infrastructure, network. The next layer is what Broadcom specializes in, which is that AI layer. So as we bring in or ingest all of this disparate data, we have a platform where we then take it and apply AI principles. So machine learning, we look at the data and recognize patterns. That's probably the most important.

(Tony at 00:07:32) And then we do some AI as far as automation and, you know, what happens with that data. But by far and large, it's that top level that matters. So after I've applied AI to all that data, what comes out of that is — I'll use the term observability — but more importantly, I would say the word observations. So if you have the right people looking at that outcome, they will then have what I call truly valuable observations about how IT is impacting their customer. And so for me, it gets me really excited because I do see a lot, and again, I'm not knocking any academic views of these terms that we're talking about.

(Tony at 00:08:15) But I can tell you that in the real world, these terms have two very specific meanings. AIOps applies that machine learning and that artificial intelligence to all this data, and then observability is where we really take what we've learned and apply it to make our business better. I hope that wasn't too much there, what you asked for, you know?

(Joel Beasley at 00:08:36) No, I agree with you on the academic thing. I've never once gone through academic experience and then when I go into the real world to do the actual job, it's virtually like introduction to acronyms.

(Intro Narrator at 00:08:50) Right.

(Joel Beasley at 00:08:51) It's like, here's some concepts you'll be learning about when you do the work. And until I get into some of the private education stuff, like you buy a course from a practitioner or someone — especially when engineering, like I'd learn from Martin Fowler, his books were very, very specific, right, and good about different programming concepts. But yeah, so what — so right now, that's what you're doing at Broadcom. Broadcom's a massive organization. How big is it? Where do you sit in the organization?

(Tony at 00:09:19) So I work for what you would call the AIOps division. So that part is in what would be enterprise software. And so for myself, my specialty and why I'm in the role that I'm in is sort of the things that people sometimes call NetOps or AIOps tools. So those, a lot of times, that boils down into actual tools. So for instance, at Broadcom, you know, in the division that I work in and that I'm sort of a customer advisor to, we have — on the application side of AIOps, we have things like Application Performance Manager.

(Tony at 00:09:55) So, you know, we have the ability to go deep inside of, for instance, Java or .NET code and pull out the performance characteristics of the code. So we're on — that's on one side, we're in the application. So we have a dedicated team of system engineers and they know that stuff cold, right? So they not only know our tool, but if you have them on site, then they'll literally dig in, help you dig into your code, right? So that's on one side. On the other side, we have network and infrastructure capabilities that are just that deep.

(Tony at 00:10:25) So, you know, we have the ability to look at network flow, analyze network traffic. We have the ability to look at different infrastructure components in a customer's infrastructure and then bubble all that up into our AI engine as well. So I think the one thing that really excited me about coming to Broadcom for this specific role that I'm in now is that they truly have the full capability of something that I would have been dreaming about ten years ago. Literally, it would have been a nirvana, and I would have questioned if you could have done it. But Broadcom has the capabilities now to actually — when I talk about bringing up all that data through AI and into a business layer, the weakness in that methodology back when I first started on it — so if I go back to about 2009, 2010 — it had one glaring weakness.

(Tony at 00:11:18) The weakness was it was really dependent upon application data. So I had to have data from the application layer, exceptions, errors in the code. And if I didn't have that, I really couldn't produce a customer view. But now it's awesome because we have, at Broadcom, we have the ability — this sounds crazy, but I'm telling you, we do this — we have the ability to tie application data, infrastructure data, and network data, not just to the top layer of what are we doing for the company, but down to the individual steps that our customers take.

(Tony at 00:11:55) I can't explain how much of a nirvana that is. I'll give you an example. I'll just go back to my time at FedEx. One of the main business services that we had, obviously, back especially in the beginning, was just shipping a package, all right? So that really boils down when you look at it to just these things. A customer logs in to FedEx.com and they tell us what their package size is and where they want it to go, from where to where. And then we give them back how much it'll cost, and they say, okay, print a label. So I just named off like six steps, right?

(Tony at 00:12:29) Where we're at now, we can literally look at each of the six steps in real time, and I can tell you down to a perfect score how perfect that step is in real time. So if we have a million customers today, I can literally watch and monitor and see how perfect each step of the customer's experience is, not from what we think from IT. So my role, I would say, if I were to sum it up for Broadcom as the chief advisor for this space, is that I need to help our customers see that they can actually produce that outcome. That's a business outcome. That's not an IT outcome. But my role, I have to show customers they can do that. So it's perfect for me, like I said, because I get really excited about it. Maybe I'm just geeking out or whatever, but I get so excited about it, and I need to make sure that our customers see it like I do.

(Joel Beasley at 00:13:29) So help me understand a little bit deeper. Let's take — let's continue with your FedEx concept, the different steps of getting the package shipped. Pick one of the steps and tell me the actual usefulness of understanding that step.

(Tony at 00:13:47) Sure. Okay. Let's just pick the one where they're shipping their package and they say how big it is and where they want it to go. And then let's pick the step we literally had that said customer rate — get the customer a rate. How much is it going to cost? All right. So before you use this type of methodology that I'm talking about, the best you could probably hope for is to look at an APM product and find that method, you know, or those methods in the code that actually go and fetch rates. So if I find those methods, then I could sit there and I could analyze all day if those methods are slow or fast. That's about the —

(Joel Beasley at 00:14:24) Like New Relic. For me, the tool is New Relic. How do I do that?

(Tony at 00:14:29) Yeah, I've used New Relic. I'm a big user in the past of AppDynamics, and of course I use Wily, which back in the day, you know — so now it's the Broadcom APM module. But using them all, you nailed it. We're looking for the performance characteristics of that method, okay? That is valuable. It still is valuable to this day. But when you have — like a recent client I worked with, this was a pharma, big pharma, and they had something like, I don't know, 5 million executions —

(Intro Narrator at 00:15:05) Of —

(Tony at 00:15:05) Of a certain set of business code every day. And the idea that someone is going to sit there and analyze all the method performance for 5 million transactions a day — is it realistic, right? So you have to have something that bubbles up real data. So here's how we would do it with what you just said. So I'm going to use the rating. I know, for instance, that — and FedEx, I'm not talking about something we did. I'm talking in general here. But I'll give you an example. I just want to be clear about that.

(Tony at 00:15:33) So for that rating, I'm going to bring in what I call the six critical inputs. So for rating for a package, I'm going to bring in six things. The first is application error rate. So this is a ratio of how many times in any given time period, like a time picker, right? So in any given time period, what is the rate of exceptions being thrown in the code? That's number one. Number two is application performance. What is the rate in that same time period? What is the rate of methods for getting a rating for a package being outside the third deviation of performance?

(Tony at 00:16:10) So they're really outliers, you know, stuff like that. Next, I'm going to bring in the third, which is synthetic — what I would call a robot. What is the failure rate for a robot that is constantly pulling that same service? Okay, so that's three. Number four, I'm going to bring in specific syslog characteristics. So I'm going to go ahead and ingest logs from syslog related to the infrastructure that I'm running this out of, so whether that's cloud or local on-premise. And I'm going to pull in, for instance, everything above a two as far as error. So that's number four.

(Tony at 00:16:47) Number five, I don't know if you've seen — I'm sure you have, of course — but you've seen how we can now analyze traffic between hop, between hop, no matter whether it's internal network or to the cloud. And so I can analyze packet loss, and I can also analyze latency for each of the hops. So number five is packet loss. So I'm going to look — is there any loss on the route, the path visualization that I'm taking to provide the service of rating?

(Tony at 00:17:15) Because many times you go to a third party and you have internal components both. I want that number. And then the final, the number six, is latency. So what kind of latency am I experiencing? And what you do is you simply take those six negative — those are all negative.

(Tony at 00:17:33) You take those six negative experiences and you subtract them from a 100% perfect customer experience. Now it's juvenile, right? It's elementary math. There's nothing complex about that.

(Tony at 00:17:46) But guess what? You can definitely look at the output of that. So let's say that I look at the last 24 hours. And in the last 24 hours, when I do the math that I just told you, I can tell you that the customer experience ratio for rating a package is 88%. Well, that's not going to be satisfactory.

(Tony at 00:18:05) But if I look now at the six critical inputs and I see that 8% of that number was strictly application performance, now as a big organization that's looking at millions and millions of transactions, now I can literally tell, you know, one or two people on the team, we've got to focus on this. So what it does is it really drives them into what's impacting the customer. So in many cases, I've personally found that this mathematics that I'm using or that we're using at Broadcom to implement AIOps, I can tell you it is by far, in my 27 years of doing stuff like this, the closest measurement of the customer experience I've ever seen. It is by far the closest.

(Joel Beasley at 00:18:47) And I'm assuming, so I'm going to make a statement and correct me if I'm wrong. Your product would have some sort of interesting interface that would understand these business outcomes and then report. Because I was thinking about New Relic. I can monitor methods and I can check a lot of these individual things, but it's definitely a developer tool. It's not presented in a way that here's the—I just see it in my head. It feels like I should just see the chain of experience, see the steps, and then see the rates for them. Is that how it is?

(Tony at 00:19:18) Yeah, it's nice. You nailed the user experience, you nailed the use case right there. So we have a product that I would in my own vernacular, being a practitioner in the real world, I would call it the aggregation level. So for us at Broadcom, it's our AI level. So this is a product called Operational Intelligence or DX OI. So this layer can ingest from nearly anything out there as far as data sources. So you mentioned New Relic, right? So in my past, I had a lot of APD. So if I were producing those two metrics that I just told you at the first number one and number two, which is error rate percentage and the standard deviation of performance, those are both negative ratios. Using Broadcom's OI layer or our AIOps dashboards, we would just simply ingest those two numbers into our overall business dashboard and then it would become one of those six critical inputs that we're talking about here. So it's important, in my opinion, you know, you just said how you could sit down and look at New Relic, same for myself. We could sit down and look at New Relic, and we could see the methods, and we could make judgments of that looks slow to me and stuff like this. What we do is we take it that one step further, and we bring it up into an AI level, and we apply this methodology I'm talking about for business.

(Tony at 00:20:36) And it is just as simple as you said. We don't take this methodology and try to make it real complex. If I was working with FedEx and I have those six steps, and they wanted to get intelligence about those six customer steps, that's all it would be. We would literally outline, like you said, the six steps, and we would ingest and pipe into each of the steps what the score is. And as you know, I mean, once you start producing an accurate score, your whole executive team can key in on that. I mean, you literally can have a conversation with your board of directors. That's why I think it's so important.

(Joel Beasley at 00:21:11) Who's helping me? Like, how does it work? Is it like a consultative sale where you guys actually help understand, like, identify those metrics and find the data and help connect it all?

(Tony at 00:21:24) Yeah. And trust me, that's the most exciting part of this job. So let's say the customer wants to go on that journey with us, and they're like, okay, yeah, we will try this AIOps thing and see how it looks. The first thing we do, and this is a pivot, okay, from if when you've worked maybe with vendors before in this space, you'll know that one of the first things that happens is you get a technical architect, so somebody who's really good at the code level, and they come in and sort of assess your abilities and look at the code level of what you're doing. We certainly will bring in that resource, but the first resource we bring in is what we call an AIOps designer. So it's even a higher level view. So we bring in an AIOps designer, and it's somebody like me who has, like, you know, 20 years of experience with this. So when they come in, they're not just talking about Broadcom products.

(Tony at 00:22:16) Right? We come in and look at your Splunk installation. We look at your Logic Monitor. We look at everything you have. And then we say, okay. How do we bubble all of this, you know, Broadcom tools or not? How do we bubble it up and design your new AIOps from a 30,000-foot level? Because I can tell you from my personal experience, when we skip the designer level, when we skip that, we never end up with anything real from AIOps. You end up with some good bells and whistles. You end up with some nice gadgets.

(Tony at 00:22:48) But when the CEO or CIO of your company come to you and say, what are we getting out of this? It's hard to show them anything other than these little features and whistles and everything. What we do is provide that designer who comes in and says, no, no, no, no. Here's where we're going to get you to from a business perspective. And I think if you want to know the truth, that's one of the most valuable partnership things that we provide at Broadcom. Because, you know, everybody wants to be a partner, and you want to be a partner to your clients. Well, one of the major ways that we do that is we just don't throw product at them and say, hey, good luck on your AIOps journey. We literally bring in somebody who designs your journey for you, and then you can go back and forth with them on things that you want to tweak. But we do bring in a senior designer, and it's not a technical position. They're a business designer.

(Intro Narrator at 00:23:38) Ryan Vaughn (35:33):

(Joel Beasley at 00:23:38) So I've gotten to go around and, you know, meet a lot of the guests that I've had on the show. I've gotten to tour their offices and inevitably at all of them, there's a dark room with

(Intro Narrator at 00:23:49) with

(Joel Beasley at 00:23:49) a bunch of IT people in it with giant screens on the wall, like collections. And it's things like, you know, Twitter service monitoring for different companies that tools, APIs that they're relying on, all these types of things, server uptime and current CPU usage, all these. So with this concept, like, let's say if we're at FedEx, with the quality of those six steps, would that be something that's, like, up on the screen?

(Tony at 00:24:18) Yeah. And the beauty of it is it's real time. So even though you can use, like, a time picker functionality and say, well, just show me what the last 24 hours look like, you can also, you know, run real time, and let's just watch what the scores are for each of the six steps. And then, you know, it's funny you mentioned that because one of the things that's so important about this methodology is that it is not in any way meant to replace what a company is doing with their individual monitoring tools. If anything, this methodology, when it identifies a problem, simply focuses the resources on, let's say, the resource for New Relic.

(Tony at 00:24:56) When this methodology identifies a problem with the application portion of the customer step, it really does crystallize what your New Relic developer or New Relic engineer needs to go look at. And then the exciting thing is when they make changes, it will show up mathematically on the score. I can't tell you how exciting it is when, in my own past working at a company recently, an IoT manufacturer, when we had a perfection score for a customer workflow step that was in the eighties. And we simply used this methodology to identify what was the most important thing to do to improve the customer experience. And I'll just make one up, but, you know, because it happens a lot.

(Tony at 00:25:38) Let's just say that there's a piece of code that's running that is doing, you know, a thousand calls. You know, all this chatter, right, when it could just do one call, you know, that's just efficiency stuff, defensive coding type things. When you find that with this methodology, and you literally go and you roll out your fix, you release your bug fix. When you make that improvement and your score goes from 88 to 98% because of what you did, now you can quantify why you're important so much on the monitoring side. I struggled many years proving business value from monitoring data, but now you can actually tie it to your customer experience.

(Tony at 00:26:18) It's like the best thing.

(Joel Beasley at 00:26:20) Robert Leonard (39:33): Yeah. As a CTO, I agree with you. That is awesome. But I think the best thing is that if I have those customer journey steps up on my giant screens, and it'll be a totally different conversation when the CEO walks in to talk, because usually they're staring at hieroglyphics on the screen, but then they can actually see the steps that the customer is going through. And so that would make me look really good.

(Joel Beasley at 00:26:44) Eddie Reynolds (00:03:40

(Tony at 00:26:45) seven): That happens all the time. And I'll tell you, like you said, CEO comes, you're the CTO. He comes in here. She comes in your office and goes, oh, so what's going on there with, you know, shipping rating? What's going on there?

(Tony at 00:26:59) And you come back and go, well, we've analyzed it. That's AI, basically, right there telling us that our weakness is in this one specific area, but we've already started fixing it, and it'll be released next week. I mean, that's a totally different conversation. And especially when you can marry what you produce from this methodology, you can actually marry it with, let's say you have a call center that handles customer complaints. You can actually marry this data to your call center data.

(Tony at 00:27:26) It's awesome because then you know that your customer experience, that's why I say it's the most accurate customer experience measurement I've ever seen because I have married this data to call center data. It's right on the money. It's on the money.

(Joel Beasley at 00:27:40) Is anyone taking this data? Obviously, you get the score, the percentage, you talk about the ratios and all of that. And you can see like 88% and you can see the changes, get it up to 98%. Is anyone connecting a dollar value to that?

(Tony at 00:27:52) I mean, yeah. And it's different for every company. So one place might say that, I'll give you an example. The pharma company that I was working with had a specific, I guess you'd call it like a distribution system. And they had complaints from one of their clients that they were distributing the pharmaceuticals to.

(Tony at 00:28:13) They had complaints about usability of the application that they were required to use. So, you know, we made improvements based on this methodology to that specific area that was giving them trouble in the distribution system. It has a financial impact, but you would have to literally sit down and see what is the value of that distribution center system, let's say, by minute or by hour. What is lost productivity? But I think that math is, that dollar value is pretty easy to calculate, when you take it to that level.

(Intro Narrator at 00:28:47) Robert Leonard (35:30:

(Joel Beasley at 00:28:48) So some of your customers are doing that?

(Intro Narrator at 00:28:50) Robert Leonard (35:30:

(Tony at 00:28:50) I've had a couple that I've worked with over the past three years that took the output from this methodology and moved it into a cash value. So, yeah, like a showback.

(Joel Beasley at 00:29:01) Robert Leonard (35:30): Yeah. That'd be, because for some reason I'm in that zone of the CEO walking in the room and I was like, how cool would it be? I'm being like a customer right now. Do you do this? Do you do this? No, but how cool would it be to see a list of opportunities and be like, oh, here's—we prioritize, we're working on the million dollar opportunity, that'll be out next week, then we'll work on the $700,000 opportunity.

(Joel Beasley at 00:29:22) It would just seem so cool to—I don't know, it just, I could see myself implementing this feature incorrectly at first by putting errors in giant red negative dollars and then realizing that that would not be the right way. Because you wouldn't want the CEO to see like, that you're losing money. You would rather see them that, like, you have opportunities.

(Tony at 00:29:44) Yeah. Exactly. And it's sort of cool because, like, a couple of customers I've worked with have taken—like, I like to look at this as, don't come in and try to boil the ocean. So when I talk with our customers, I don't ever want to present that they should take all their business services and immediately try to get this out there because sometimes that makes things go so slow that you never really see value because it just takes forever. So what I like to do is I like to go into—you mentioned earlier is this consultative, and yes, indeed, that designer role, when we go in with a customer, we like to look at what their big hitters are.

(Tony at 00:30:19) So, um, example, a bank. You know, if you go work with one of the large banks in this country, they're going to have tons of lines of business. But no doubt, they will probably have a lot of interest in their online banking. So there that will be a workflow that customers do with them every day that is seriously important to them. So from our vantage point, we would probably just say, why don't we just do a little AIOps journey on online banking, the mortgage part.

(Tony at 00:30:48) So, you know, how get customers coming in to do their online banking and convert to a mortgage application. So we literally just pick something that simple, and then we'll do this methodology on it, right? But the exciting thing is when a customer sees so much value from that one chunk that we did, that they decide to just simply replicate it all over their company on little chunks of things. So what you end up with, which is just such eye candy, is you get these dashboards where you have this all up view of your company with the business services that you're trying to track.

(Tony at 00:31:24) And so to watch it grow is what's really exciting at a company. Not just one work stream, but like 10. And then you have an all up view of your company.

(Intro Narrator at 00:31:34) Jeremy Schneider (00:23:40):

(Joel Beasley at 00:31:34) No. You got—it's a little bit contagious. I'm all excited now. How did we—we got connected through ANI Solutions, right?

(Intro Narrator at 00:31:43) Jeremy Schneider (00:23:43): Yeah.

(Joel Beasley at 00:31:44) What's the relationship there?

(Tony at 00:31:46) Obviously, they're a great partner and integrator for us. And as a matter of fact, one of the most exciting things was with ANI. Bill Lavoy was his name. And I met him back in maybe 2012, I think it was. And one of the first implementations that we ever did of this methodology, now mind you, it was very dependent back then on application. Now we can go way further than that. But one of the very first implementations ever done was done by ANI Solutions and Bill. And so I have a soft spot because that was like my baby. You know?

(Tony at 00:32:21) It was like, oh, actually watching it be implemented at a client site. So but yeah. No. Great partner. Extreme talent. I can certainly speak for my, I guess you'd say work with them is that, you know, we're talking about the best of the best there.

(Joel Beasley at 00:32:36) Well, so I'm, you know, I'm like entrepreneurial. I like to understand business relationships. Are they like, like a sales arm? Are they sales and consulting? Like how do they actually—I mean, you say partner or maybe I'm completely wrong. Like how do they actually interface? How do your businesses interface?

(Tony at 00:32:52) Yeah. So, obviously, Broadcom being such a big company and everything, we have different tiers in the way that we approach customers and work with them. And so in the case of ANI, for instance, we do both a sales and a service arm with them. So they represent us when they have customers that are looking for—I'll just pick my space, the AIOps and NetOps solutions. But they're also, for me, very critical for the integration arm.

(Tony at 00:33:19) So when we have that type of relationship, and let's say a customer decides to go with our full AIOps stack, so they may have other products and everything, but they decided to go with our full AIOps stack, then AI Solutions will be the type of company where we would work with them, not just to get the deal done, but to actually do the post-sales implementation. And I would say, I'm not saying this is unique, but it is very special that we have AIOps designers that work through and with ANI. So in other words, this isn't a knock or anything like that on anything, but I've made purchases in my role on the customer side, so when I'm practitioning, I've made purchases before of software, and because a company is so big, they send us to like a partner and integrator, and that's who you end up dealing with.

(Tony at 00:34:11) But what happens is I've had where I've lost all contact with, you know, maybe the architects of the original software that I was hoping to have. And we don't have that happen, especially on this methodology. We literally provide designers, AIOps designers, through our partners.

(Joel Beasley at 00:34:29) That's pretty neat.

(Tony at 00:34:30) Yeah, it's a value add. So, like, we don't hit our customers up for charges for this service. This is how we partner with our customer. We want to provide them experts in every field, so this is just one of those experts that we provide.

(Joel Beasley at 00:34:48) I like it. Yeah. Now, I have on the notes to ask you about mainframe software, and I was like, did my team write that correctly?

(Tony at 00:34:59) On the mainframe side, you know, the way that mainframe touches me and what I do and what we do in AIOps is we have mainframe operational intelligence, so we have the ability to pull insights from mainframe. But for what I'm talking about with AIOps, the most important part of that is that it becomes part of those six critical inputs. So back in, if I go back in the day when I was working at FedEx with a mainframe department, in our, from FedEx.com, we would have been more client servers, so, you know, Java or .NET-based applications. And then from there, we'd hit our databases, but also we'd hit IMS.

(Tony at 00:35:39) So we would go back to the mainframes to pull certain things. Back then, a lot of the AI type intelligence that we have now was not really available for mainframe. So it really was a black box. Now, we can actually, especially with our APM products, we are able to actually go all the way back in through the mainframe and come back out with performance characteristics. So there's no possible way that, well, I shouldn't say that.

(Tony at 00:36:07) I think it's extremely unlikely now that any problems happening on the mainframe would not show up in the six critical inputs and would not show up in the calculation of perfection. So I haven't seen anything yet where I don't see that show up in the measurement.

(Joel Beasley at 00:36:30) I'm going to ask you a silly question. I mean, there's no silly questions, but it'll definitely show you my age here. But okay, so mainframes, I'll tell you what my understanding, they're like really old computers and they're still very operational, because I talked to somebody who actually listened to the show and they reached out to me and they were taking over a role at a company that looks like a mainframe farm.

(Joel Beasley at 00:36:52) And I think a lot of financial software was still using that, even credit card processing stuff. I was blown away. When you were using it just now, it almost sounded like you were using it as a brand name. I really don't know. What's the difference between a mainframe and a computer? Is it a brand?

(Joel Beasley at 00:37:10) Can you help me understand this better?

(Tony at 00:37:12) Yeah, and I'm not a mainframe specialist, so I won't be the best person to answer it. But just in general, if you were to go back to back when I started in IT, a mainframe would be the way, let's say it was provided by IBM. So they bring in these huge mountainous mainframe computers, and the reason that you would want to go with them was because of their bulletproof nature: extremely reliable. Run over and over transactions, unbelievable amounts of transactions, and never really run into trouble.

(Tony at 00:37:42) And so there was a time where you would say you couldn't get into a production data center if you weren't a mainframe, right? So you have all these characteristics that have been around forever. I think what maybe, mainframes are still out there, and I'm sure they always will be. And that's why you mentioned, like, bank transactions and how they still go to the mainframe.

(Tony at 00:38:04) I can certainly appreciate that you have maybe less worries at night over systems going down when they go back to a mainframe because they're so bulletproof. I think the thing that maybe has changed in the past 15 years or so has been when you come down to a server-class machine, especially now with virtualization and the cloud. It doesn't do away with the need for mainframes, but it does complement the fact that now all of us can do computing of any nature without having to make that kind of investment. So think about what we can all do right now, for instance, in AWS or GCP. We can all go out there and create these really powerful compute cycles that we can run our applications on.

(Tony at 00:38:50) So I think when you have a mainframe specialist on, they're going to really give you the rundown of why it's so important, but I never worked in mainframe myself, so I can't tell you that.

(Joel Beasley at 00:39:00) All right, great. So I'm not too far off then. I think one of the things when I was talking with this fellow was, what I gathered from it, like intangibly, was that it's like a, there's like a suite of tools and a whole ecosystem that all kind of runs together with these giant physical, like, IBM machines. And that's, you don't necessarily find all of those same things in computers. But we'll get a mainframe specialist on, and we'll get to the bottom of this, Tony, and we're going to figure it out.

(Tony at 00:39:29) Absolutely. But hey, Broadcom's one of the biggest providers of mainframe software. I've just not made it over to that side yet.

(Joel Beasley at 00:39:36) So are you, like, currently, professionally, are you, are you individual contributor helping with this type of stuff, or are you running a team?

(Tony at 00:39:44) For right now, my role is Chief Strategic Advisor, and so it's sort of a role where, working, I guess you'd say, across, at Broadcom, we're very flat, which is a wonderful thing. So we work together regardless of titles or anything like that. But what I am doing is I'm sharing and have been sharing the methodology that we provide in our AIOps layer, sort of sharing how I go with customers and work with them to make that as beneficial as possible. So at Broadcom, we're growing this org, and so over the next, actually, six months, I think we're probably going to increase the number of designers, you know, probably by threefold and really maybe attack the market a little bit more, not from a sales perspective, but from an education and here's the art of the possible.

(Tony at 00:40:34) You know, a lot of times, to be honest, when I go and may meet with a client and maybe they're just starting to think about AIOps and what it can do, when you present sort of those six critical inputs I mentioned and when you present what a dashboard and a strategy would look like, it's a little bit foreign because, you know, a lot of times a customer might think that if they have APM, you know, any form of APM, they may think that they already have customer experience. And certainly, you do have some customer experience from an APM or a RUM tool, any of those. You certainly have some. The difference is that this methodology doesn't stop with application. It brings in both network and infrastructure metrics, and that's what makes it so different.

(Tony at 00:41:24) The granularity of what I can discover, or observe, just to stick with the term, the granularity of what I can observe now using this methodology is unreal compared to where it used to be. So that's why I think, you know, like I said, in the next few months, we're really going to maybe hit the market harder to share. This is, you don't have to do it, right? We just want to share it with you.

(Tony at 00:41:50) So that's why I was so excited about today, right? Because the number of followers that you have is a chance for me to at least say, this is why we want to come show you. And then, you know, it's sometimes, even if you're really good at what you're doing, like, I can think back to my days at FedEx. That team, that reliability engineering team that I had, was the most brilliant.

(Tony at 00:42:11) They did everything that seemed right. But you know what? They also invited in, like, I don't know, like Amazon's engineers, and they just benchmark. Nobody's right or wrong. We're just benchmarking.

(Tony at 00:42:25) That's what we want to do for our customers. Let us share this, let us share this AI vision that we have, and if it doesn't work for you, no big deal. But at least we share it.

(Joel Beasley at 00:42:36) Yeah. This type of media is incredibly useful for getting information out. Do you guys have, do you have a YouTube channel where you're sharing this type of stuff?

(Tony at 00:42:43) We do. We actually have our CMO for the enterprise software division is actually putting out, I think they've already taped it, like, this week, so I'm excited to see the final. But we'll have out on YouTube, like, a three-minute, I guess you would call it, like, a one-pager video of what BSR, this methodology and our AIOps, what this can do for our customers. So anyway, when that comes out, I'll make sure I get you the link, because I'm hoping people have a chance to spend just that three minutes to hear sort of the high level. I think we've covered a lot like what we've talked today, but our CMO really put a lot into making that film.

(Tony at 00:43:26) So I'm excited about it.

(Joel Beasley at 00:43:28) Yeah. Let me know. Send it over to us and Adam can make a note of that and we can make sure that we post it on the show page when it comes out.

(Tony at 00:43:36) That's awesome.

(Joel Beasley at 00:43:37) Cool. I want to talk a little bit about leadership. Well, our whole mission here, educate, entertain, and elevate the next generation of technology leaders. So you've gotten very far in your career. You're a wealth of knowledge for an engineer looking to take on some more management responsibilities, looking to move up in their career. What comes to mind as far as how you would give them advice or insight?

(Tony at 00:44:05) When I think back, so in the 27 years, probably, you know, 19, 18, 19 of them were with staff, so people manager, right? So what you're sort of talking about on the management side. So if I were to give advice on moving toward that goal, especially in a technical field, I would go back to my time at FedEx where a lot of that was spent leading teams. And probably the biggest thing is, I had a mentor of mine there tell me, if you decide to go into management, technical management, just make sure that you're doing it because you want to help grow the people that will be reporting to you.

(Tony at 00:44:44) And don't think that it's anything about you or like that. I thought that was really astute because, you know, I mean, it's sort of hard to be ambitious and not think some about yourself. But I thought it was astute that he told me, look, you need to remember, these people are going to be looking to you for leadership, they're going to probably come to you for advice, and they're going to want you to have their backs, protect them, you know, just from the corporate politics or anything like that. And so when I went into that, with that mindset, I just found myself really focusing, like, on that reliability engineering team that I said, all those men and ladies were so brilliant. Well, you know, there's a responsibility that comes with that.

(Tony at 00:45:26) You know, my focus was on, you know, what is bothering them every day? What do they need to make them feel like they're advancing? And you make mistakes, to be honest. You know? You think that you're doing it right and you're going to make mistakes, but if you can keep focused on that, that your people are what matters, then you'll do fine.

(Tony at 00:45:47) Because without a doubt, I think all of us know this, you're only as good as your people. If you think you're great and if you think you've got it all figured out, and then you don't take care of your 10 direct reports, and they don't do what they're supposed to do, and they're not able to, or they're demoralized, you, no matter how great you think you are, you failed, and it'll show.

(Joel Beasley at 00:46:11) I like how direct you are. It's good. I enjoy direct people. I was talking with Aaron Suzuki. What you said made me think of this.

(Joel Beasley at 00:46:21) He's a founder of SmartDeploy. They do, how do I describe it? They solve the problem of deploying endpoints at scale, like automation, like provisioning machines, imaging machines, things like that. When we were talking, we were discussing open communication between IT and other departments. At first I wasn't going to ask you about it because you were in your current, like, evangelist type role or consulting type role.

(Joel Beasley at 00:46:47) But then you said you had like 17 years of managing people and working with them. Have you ever had a situation where there's not great communication between IT and other departments? And if so, how did that get resolved?

(Tony at 00:46:58) Yeah. I mean, that's a common theme for sure. And so there have been plenty of times where I've had that happen. Direct communication is probably the biggest thing.

(Tony at 00:47:11) So I'll think of an example here. We had a user group at one company I worked with. So these were the people who did the requirements for a particular system. This was a customer service call center, so call center software. And so the user group would provide the requirements to a development team.

(Tony at 00:47:31) The development team would go build them and bring it back for user acceptance testing, and then they'd release it. When I had a leadership role over part of the development team, what I would hear pretty consistently is, I just don't understand that user group. They certainly don't, they're not in touch with our customer. They don't get it. You know?

(Tony at 00:47:52) So a lot of negativity, right? And then when I went and had a chance to talk with the manager of the user group, what she told me at the time was, you know, we love your team and everything, but we feel like there's a communication gap. You know? Sometimes we don't really get what we ask for, you know, that type of thing.

(Tony at 00:48:12) So, of course, direct communication, I'm a big believer in that. And so at the time, you know, that user group represented what you would call the call center population. So, I mean, this could be a little bit wacky, I don't know, but one of the things that I chose to do then is of all of our development team, we took a couple of days out of a week, and we all went and put on our headset, and we sat with customer service.

(Tony at 00:48:39) And we watched them use the product we were building, not just like cursory, but we literally sat there and watched them, and then we listened to the calls. And when there was a problem, like, with the software not responding quickly enough, we got to see the agent sweat because they've got the customer on the phone, and they're going, well, I need a minute. My computer's, and, you know, I think when we did that, we had a lot more sympathy for what our user group was going through. And then instead of this, I mean, as you know, you're doing your best to never have this, you should never have where you're on the same team, but you're sort of hitting heads all the time.

(Tony at 00:49:16) You should be working in some form of a tandem. And so I think after we did that and we walked in their shoes, we were a little bit more understanding, and then that helped us to create a better product, if you want to know the truth. We did better with our product after we saw people using it on that intimate level.

(Joel Beasley at 00:49:34) Yeah, I actually, when I was talking about that real estate product I made, because I was in the office and building this and seeing that, that was one of my first experiences. And so when I went to do it next, after I had had some success and made money, I did it in a chamber and then it fell flat. And I realized the reason why it didn't work this time is I wasn't sitting next to someone who had a problem and I wasn't getting that emotional satisfaction of resolving their attention or helping them do what they need to do to bring them value. And so that's how I learned that lesson and I've never repeated that mistake again.

(Tony at 00:50:09) Getting closer and closer to the customer as always. I mean, that's not to go back to the AI ops thing, but when we use this methodology, our whole goal here is to get closer to the customer. What are they really experiencing? Because I can't tell you how many times I've been to visit a customer and they show me, like you said, those dashboards in that dark room where everybody, we all love it. But I can't tell you how many times I've gone to those rooms and looked up and all I see is green and 99.99.

(Tony at 00:50:36) And then my next meeting, my very next meeting is with an executive saying, "So how do we improve customer experience?" And I just chuckle inside because all it was 99.9 perfect on everything. So why do we even have a customer experience problem? It's because we're not close to what the customer is doing sometimes.

(Joel Beasley at 00:50:54) Oh, that's good. That hits home to a lot of people listening right now.

(Tony at 00:50:58) Yeah, it happens. And I mean, I've probably been guilty of it myself, especially going back before I really embarked on this journey. You know, you want your numbers to be good. You want to show goodness, but it doesn't matter that much if you show goodness and your customer is still complaining about you.

(Joel Beasley at 00:51:14) Yeah. You guys work with companies of all sizes?

(Tony at 00:51:18) We do. Absolutely. Yeah. We tier that up and a lot of times that's where our partners come into play more. So, you know, if we get down to a smaller company, you know, maybe we engage a partner to work with us.

(Tony at 00:51:31) But even with the smaller companies, we share and use the same methodology. And I don't care how big or small. Even in some ways, even a small company might need this more, you know, because they're trying to grow their business, and they need to be really close to what their customers are experiencing with them. So, but no, the answer to your question is absolutely. And that's why I mentioned before that even, you know, if we work with one of our partners, an integration partner, we still provide this AIOps designer. So we want whoever they are, we want them to try to look at the AIOps journey from the Broadcom standpoint, which is definitely more customer and business focused.

(Joel Beasley at 00:52:17) Thank you so much for listening. And if you found this episode useful, please share it with a friend or colleague who you think would get value from it. And if you have topics that you would like to hear discussed on the podcast, either add me on LinkedIn, or send me an email [email protected]. Every time I get an email or LinkedIn message, it absolutely makes my day and inspires me to keep going.